如何扩展python的logging组件支持json日志输出

我是攻城师

发布于 2019-03-06 07:13:30

3.1K00

代码可运行

运行总次数：0

代码可运行

这两天在优化公司一个python的项目，顺便研究了一下如何将python日志转成json格式，原来在Java的项目中搞过类似的事情，知道日志转成json之后有很多便利的之处，最常见的就是可以直接对接各种日志分析系统，如开源的ELK，将数据导入之后就能快速的进行查询和分析，方便做各种统计，监控或报警等。

python里面的logging组件，其实已经是组件化了，有Logger组件，Handler组件，Fomatter组件，如下图所示：

logger=>handler=>formatter分别是一对多的关系，日志的格式其实是由formatter决定的，所以想要扩展成你想要的各种格式，就重写定制formatter组件就可以了，它实际上和Java里面Log4j的LayOut组件类似。

核心代码如下：

REMOVE_ATTR = ["filename", "module", "exc_text", "stack_info", "created", "msecs", "relativeCreated", "exc_info", "msg"]


class JSONFormatter(logging.Formatter):
    host_name, host_ip = HostIp.get_host_ip()

    def format(self, record):
        extra = self.build_record(record)
        self.set_format_time(extra)  # set time
        self.set_host_ip(extra)  # set host name and host ip
        extra['message'] = record.msg  # set message
        if record.exc_info:
            extra['exc_info'] = self.formatException(record.exc_info)
        if self._fmt == 'pretty':
            return json.dumps(extra, indent=1, ensure_ascii=False)
        else:
            return json.dumps(extra, ensure_ascii=False)

    @classmethod
    def build_record(cls, record):
        return {
            attr_name: record.__dict__[attr_name]
            for attr_name in record.__dict__
            if attr_name not in REMOVE_ATTR
        }

    @classmethod
    def set_format_time(cls, extra):
        now = datetime.datetime.utcnow()
        format_time = now.strftime("%Y-%m-%dT%H:%M:%S" + ".%03d" % (now.microsecond / 1000) + "Z")
        extra['@timestamp'] = format_time
        return format_time

    @classmethod
    def set_host_ip(cls, extra):
        extra['host_name'] = JSONFormatter.host_name
        extra['host_ip'] = JSONFormatter.host_ip

使用的时候，可以通过简单配置即可：

[loggers]
keys=root

[handlers]
keys=console,file

[formatters]
keys=json,json_pretty

[logger_root]
level=DEBUG
;handlers=console,file,rotate
handlers=console,file

[handler_console]
class=StreamHandler
level=INFO
formatter=json_pretty
args=(sys.stderr,)


[handler_file]
class=FileHandler
level=INFO
formatter=json
args=('log/base_conf.log','a','utf-8')

[handler_rotate]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=json
args=('log/rotate.log', 'D',1,0,'utf-8')

[formatter_json]
class=format.json_formatter.JSONFormatter

[formatter_json_pretty]
format=pretty
class=format.json_formatter.JSONFormatter

如下的一段异常代码：

fileConfig('log_conf.ini')
    log = logging.getLogger(__name__)
    try:
        a = 1 / 0
    except Exception:
        log.exception(" occurred exception ")

输出结果如下：

{
 "name": "__main__",
 "args": [],
 "levelname": "ERROR",
 "levelno": 40,
 "pathname": "C:/texx.py",
 "lineno": 17,
 "funcName": "base_configuration",
 "thread": 10608,
 "threadName": "MainThread",
 "processName": "MainProcess",
 "process": 11916,
 "@timestamp": "2019-01-10T12:50:20.392Z",
 "host_name": "your-PC",
 "host_ip": "192.168.10.11",
 "message": " occurred exception ",
 "exc_info": "Traceback (most recent call last):\n  File \"C:/txxx.py\", line 14, in base_configuration\n    a = 1 / 0\nZeroDivisionError: division by zero"
}