基於可視化配置的日誌結構化轉換實現

• 2019 年 10 月 3 日
• 筆記

???????DBus????????????????????????????????????????????????????????Web?????????????????????DBus??????????RDBMS?????????????????????????????Logstash?flume?filebeat???????????????????????????????????????DBus??????????????????????

??????????

1.1 ??????

DBus??????log???????Logstash?Flume?Filebeat??????????????????????????????????????????????????????????????????????????????????raw data log???????????Kafka????DBus?????

1.2 ??????????????

???????????????????????????????????????“???-???”?????????????????????????????????????????????schema?????????????

1.3 ????

DBus??????????????????????????????????????????????????????????????????????????????????????????

1.4 ????

???????????????????????????????????????????Kafka??????????????????????

??DBus log?????????????????????????????????????????schema???

• ??????????????????????1??????????
• ??????????????????????????????

?????????????raw data log????????????

??????????????T1,T2…??????????????????????????????????

• ?????T1?????????????
• ??????????????????????????????
• ????????????????????
• ??T1?????????????????????????T2??????????
• ???????????????????????_unknown_table_?

??????????????????????????Table????????????Table??????????????????????????????????????????????????Table?

?????????????????????????????????????

??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??DBus??????

?DBus???????DBus??????????master-slave?????????????????????????????????????????????????????????????????????????????????????DBus log??????????

DBus??????????Logstash?Flume?Filebeat??????Logstash?????????DBus???????????

???dbus-n2?dbus-n3????????????????????????????????Logstash????????Logstash????????????????DBus?????????????????????????Logstash????????Flume?Filebeat????????????????????????????????Logstash?????Kafka????????????????????????

????????2????Logstash??????DBus?Logstash????????????????????????????????????Logstash???????????????Kafka Topic??DBus????????????????????????????

2.1 ??Logstash

???Logstash??????????topic : heartbeat_log_logstash?????????????

1)????

2)??????

2.2 ????

??????????DBus Web????????????????????

?????????sink_info_table???????sink????????????????????????????????????????????schema????heartbeat_log_logstash??????topic?????????????????????????????????

1)????????

?????Logstash?????????log4j????????path?@timestamp?level????????????????log?????????????????????????log????????

2)???????

?????timestamp?log ?????????????????toIndex???????????

?????????????????????????? - ???????????????flume??????????split??????????? - ??????????????????python??? ???1:3??1?2??? ???????????????????

????????????????????

3)???????

?????????????“Sink to influxdb OK!”?????????????filter??????7????”Sink to influxdb OK!”???????

?????????????????????

4)????????

????select???????1?3???????????????????

??select????????????1?3???

5)?????????????

?????1?????????????????????regexExtract??????????????????http_code=(d*).*type=(.*),ds=(.*),schema=(.*),table=(.*)s.*errorCount=(d*)????????????????

????????????????????

6)?????

?????????????????saveAs??? ??????????????????????

??saveAs????????????????????

2.3 ?????????

???????????????????DBus???????????????????????????????DBus??????UMS?????????UMS?????????????????

??UMS?DBus?????????????????????JSON????????schema????????UMS?????DBus???????????????https://github.com/bridata/dbus

??????????????UMS??????

2.4 ????

??????????????????????????????????????????????????????????????

• ??????
• ????????????????????????????????????????????????DBus???????????????????????
• ??????
• ?????????

??????????????????????????IP??????????????????????

??????????_unkown_table_ ???????????????????Logstash???????5???????????????????3???????????????????_unkown_table_????

DBus??????Flume?Filebeat?UMS??????????????????????Logstash???????????????DBus?log??????????

• https://bridata.github.io/DBus/install-logstash-source.html

• https://bridata.github.io/DBus/install-flume-source.html

• https://bridata.github.io/DBus/install-filebeat-source.html

??????DBus????????????????????????Kafka????????????????????Wormhole???????????????DBus?Wormhole???????????????????????????

??????