问如何配置Spark 的log4j的日志？

提问于 2020-05-28 10:34:15

回答 1关注 0查看 1.8K

我使用的是CDH5.16安装的集群。跑的是spark程序，想要自定义log4j输出日志到客户端目录

配置的log4j如下

log4j.rootCategory=INFO, console,FILE
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=WARN
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=WARN
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

log4j.appender.FILE = org.apache.log4j.RollingFileAppender
log4j.appender.FILE.Threshold=DEBUG
#log4j.appender.FILE.File=./log/rzp.log
log4j.appender.FILE.File=/home/rzpt/logs/spark.log
log4j.appender.logFile.Encoding = UTF-8
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=[%-5p] [%d{yyyy-MM-dd HH:mm:ss}] [%C{1}:%M:%L] %m%n
log4j.appender.FILE.MaxFileSize=10mb

这个log4j配置文件我在项目中放在了resource文件下，服务器上也放了一份（因为网上找了答案两种都有），然后这些命令我都试过了，但是全部都没有日志输出

spark2-submit \
    --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
    --master yarn \
    --deploy-mode cluster \
    --files "./log4j.properties" \
    ./sgp-1.0.jar
    
    spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain --master local[*]  \
    --deploy-mode client sgp-1.0.jar

spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
--files "./log4j.properties" \
--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--master yarn --deploy-mode cluster  sgp-1.0.jar

spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
--files "./log4j.properties" \
--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--conf spark.driver.extraJavaOptions='-Dlog4j.configuration=file:/opt/centralLogs/conf/log4j.properties' \
--master yarn --deploy-mode cluster  \
sgp-1.0.jar

专用宿主机