{"id":1379,"date":"2023-03-19T10:54:47","date_gmt":"2023-03-19T02:54:47","guid":{"rendered":"https:\/\/www.appblog.cn\/?p=1379"},"modified":"2023-04-28T21:12:41","modified_gmt":"2023-04-28T13:12:41","slug":"logstash-grok-configuration-debugging","status":"publish","type":"post","link":"https:\/\/www.appblog.cn\/index.php\/2023\/03\/19\/logstash-grok-configuration-debugging\/","title":{"rendered":"Logstash grok\u914d\u7f6e\u8c03\u8bd5"},"content":{"rendered":"

grok\u662f\u4e00\u79cd\u91c7\u7528\u7ec4\u5408\u591a\u4e2a\u9884\u5b9a\u4e49\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\uff0c\u7528\u6765\u5339\u914d\u5206\u5272\u6587\u672c\u5e76\u6620\u5c04\u5230\u5173\u952e\u5b57\u7684\u5de5\u5177\u3002\u901a\u5e38\u7528\u6765\u5bf9\u65e5\u5fd7\u6570\u636e\u8fdb\u884c\u9884\u5904\u7406\u3002logstash\u7684filter\u6a21\u5757\u4e2dgrok\u63d2\u4ef6\u662f\u5176\u5b9e\u73b0\u4e4b\u4e00\u3002<\/p>\n

logstash\u5185\u7f6e\u7684grok\u5339\u914d\u89c4\u5219\u53ef\u53c2\u8003\uff1ahttps:\/\/github.com\/logstash-plugins\/logstash-patterns-core\/blob\/master\/patterns\/grok-patterns<\/a> \u3002grok\u8fd8\u652f\u6301\u81ea\u5b9a\u4e49\u5339\u914d\u5b57\u6bb5\u89c4\u5219\uff0c\u53ef\u4ee5\u7075\u6d3b\u6ee1\u8db3\u6269\u5c55\u7684\u9700\u6c42\u3002<\/p>\n

<\/p>\n

\u65e5\u5fd7\u65b9\u5f0f<\/h2>\n
input {\n    ...\n}\n\nfilter {\n    ...\n}\n\noutput {\n    if "_grokparsefailure" in [tags] {\n        file { path => "\/data\/logs\/logstash\/grok_failures.txt" }  #\u89e3\u6790\u5931\u8d25\u65e5\u5fd7\n    } else {\n        elasticsearch {\n            hosts => ["192.168.165.239:9200"]\n            index => "%{type}"\n        }\n        stdout {\n           codec => rubydebug  #\u63a7\u5236\u53f0\u8f93\u51fa\u65e5\u5fd7\n        }\n    }\n}<\/code><\/pre>\n
\u6b64\u65f6\u53ef\u4ee5\u901a\u8fc7\u524d\u53f0\u542f\u52a8\u67e5\u770b\u63a7\u5236\u53f0\u8f93\u51fa\u65e5\u5fd7\uff1a<\/p>\n
# bin\/logstash -f config_file\/log.conf<\/code><\/pre>\n
Kibana Dev Tools<\/h2>\n
Console<\/h3>\n
\u67e5\u8be2ES\u91c7\u96c6\u7684\u6570\u636e\u683c\u5f0f\u662f\u5426\u7b26\u5408grok\u5207\u5272\u9884\u671f<\/p>\n
GET \/appblog\/_search\n{\n  "query": {\n    "match_all": {\n\n    }\n  },\n  "sort": [{ "@timestamp": { "order" : "desc"} }]\n}<\/code><\/pre>\n
Grok Debugger<\/h3>\n
Sample Data \u8f93\u5165\u65e5\u5fd7\u6837\u4f8b\uff0c\u5982<\/p>\n
2019-05-25 15:23:32.009 [cn-appblog-provider-channel-gateway-alipay][ INFO ] [65117] [nio-8851-exec-8] [47a999cec484e6b5] [0ea76f03cdf92c57] [true] --- [cn.appblog.provider.channel.gateway.alipay.helper.XStreamHelper] [parseAlipayCreateReturn] [39] : This is log content<\/code><\/pre>\n
Grok Pattern \u8f93\u5165\u5339\u914d\u89c4\u5219\uff0c\u5982<\/p>\n
%{TIME_STAMP_A:logtime}\\s+\\[%{APP_NAME:appname}\\]\\[\\s+%{LOG_LVL:loglvl}\\s+\\]\\s+\\[%{PROCESS_ID:pid}\\]\\s+\\[%{PROCESS_NAME:pname}\\]\\s+\\[%{TRACE_ID:traceid}\\]\\s+\\[%{SPAN_ID:spanid}\\]\\s+\\[%{SPAN_EXPORTABLE}\\]\\s+---\\s+\\[%{CLASS_PATH:classpath}\\]\\s+\\[%{METHOD_NAME:methodname}\\]\\s+\\[%{CODE_LINE:codeline}\\]\\s+:\\s+%{CONTENT:content}<\/code><\/pre>\n
Custom Patterns \u8f93\u5165\u81ea\u5b9a\u4e49\u89c4\u5219\uff0c\u5982<\/p>\n
TIME_STAMP_A \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}.\\d{3}\nTIME_STAMP_T \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}.\\d{3}Z\nTIME_STAMP_P \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\nTIME_STAMP_S \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3}\nHOST_NAME_PATTERN [a-zA-Z0-9._-]+\nAPP_NAME [a-zA-Z0-9._-]+\nLOG_LVL [a-zA-Z0-9._-]+\nCORRELATION_ID [0-9a-f-]{36}\nCIP ((?:(?:25[0-5]|2[0-4]\\d|((1\\d{2})|([1-9]?\\d)))\\.){3}(?:25[0-5]|2[0-4]\\d|((1\\d{2})|([1-9]?\\d))))\nID_PATTERN [0-9a-f\\-]{36}\nRPC_ID_PATTERN [0-9\\.]+\nAPP_OR_METHOD [\/a-zA-Z0-9._-]+\nTRACE_ID [0-9a-f]*\nSPAN_ID [0-9a-f]*\nPROCESS_ID \\d{0,5}\nPROCESS_NAME [a-zA-Z0-9._-]+\nSPAN_EXPORTABLE [a-z]{0,5}\nCLASS_PATH [a-zA-Z0-9._]+\nMETHOD_NAME [a-zA-Z0-9_]+\nCODE_LINE \\d{1,5}\nCONTENT [\\s\\S]*$<\/code><\/pre>\n
\u70b9\u51fbSimulate<\/code>\uff0c\u5f97\u5230Structured Data<\/code><\/p>\n
{\n  "traceid": "47a999cec484e6b5",\n  "classpath": "cn.appblog.provider.channel.gateway.alipay.helper.XStreamHelper",\n  "loglvl": "INFO",\n  "pname": "nio-8851-exec-8",\n  "pid": "65117",\n  "content": "This is log content",\n  "codeline": "39",\n  "spanid": "0ea76f03cdf92c57",\n  "appname": "cn-appblog-provider-channel-gateway-alipay",\n  "logtime": "2019-05-25 15:23:32.009",\n  "methodname": "parseAlipayCreateReturn"\n}<\/code><\/pre>\n
\u914d\u7f6e\u793a\u4f8b<\/h2>\n
2019-05-23 11:50:36.022 [cn-appblog-provider-channel-core][ INFO ] [21992] [nio-8888-exec-1] [143da285c068e5e1] [cb964a4c7b09ee0e] [true] --- [cn.appblog.provider.channel.core.helper.ChannelInfoHelper] [checkChannelInfo] [35] : ChannelPayRequest.checkChannelInfo [MerchantId: 142019050800009001, TransSerialNo: 122019052300016001, ChnlCode: alipay_offline_payment]<\/code><\/pre>\n
input {\n    kafka {\n        bootstrap_servers => "192.168.1.10:9092"\n        topics => "logstash"\n        group_id => "logstash"\n        consumer_threads => 5\n        decorate_events => true\n        codec => json\n        type => "thaipay"\n        #auto_offset_reset => "smallest"\n        #reset_beginning => true\n   }\n}\n\nfilter {\n    if [type] == "thaipay" {\n        if [message] =~ "^\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}.\\d{3}\\s+\\[[a-zA-Z0-9._-]+\\]\\s*\\[\\s*[a-zA-Z0-9._-]+\\s*\\][\\s\\S]*$" {\n            grok {\n                patterns_dir => "\/data\/server\/logstash\/config_file\/patterns"\n                #add_field => {"logmatch" => "100001"}\n                #match => { "message" => "%{TIME_STAMP_A:logtime}" }\n                #match => { "message" => "%{TIME_STAMP_A:logtime}\\s+\\[%{APP_NAME:appname}\\]\\s+\\[%{LOG_LVL:loglvl}\\]" }\n                #match => { "message" => "%{TIME_STAMP_A:logtime}\\s+\\[%{APP_NAME:appname}\\]\\[\\s+%{LOG_LVL:loglvl}\\s+\\]\\s+\\[%{PROCESS_ID:pid}\\]\\s+\\[%{PROCESS_NAME:pname}\\]\\s+\\[%{TRACE_ID:traceid}\\]\\s+\\[%{SPAN_ID:spanid}\\]\\s+\\[%{SPAN_EXPORTABLE}\\]\\s+---\\s+\\[%{CLASS_PATH:classpath}\\]\\s+\\[%{METHOD_NAME:methodname}\\]\\s+\\[%{CODE_LINE:codeline}\\]" }\n                match => { "message" => "%{TIME_STAMP_A:logtime}\\s+\\[\\s*%{APP_NAME:appname}\\s*\\]\\[\\s*%{LOG_LVL:loglvl}\\s*\\]\\s+\\[\\s*%{PROCESS_ID:pid}\\s*\\]\\s+\\[\\s*%{PROCESS_NAME:pname}\\s*\\]\\s+\\[\\s*%{TRACE_ID:traceid}\\s*\\]\\s+\\[\\s*%{SPAN_ID:spanid}\\s*\\]\\s+\\[\\s*%{SPAN_EXPORTABLE}\\s*\\]\\s+---\\s+\\[\\s*%{CLASS_PATH:classpath}\\s*\\]\\s+\\[\\s*%{METHOD_NAME:methodname}\\s*\\]\\s+\\[\\s*%{CODE_LINE:codeline}\\s*\\]\\s+:\\s+%{CONTENT:content}" }\n            }\n            #date {\n            #    match => ["logtime", "yyyy-MM-dd HH:mm:ss.SSS"]\n            #    target => "messagetime"\n                #locale => "en"\n                #timezone => "+00:00"\n                #remove_field => ["logtime"]\n            #}\n        }\n    }\n}\n\noutput {\n    if "_grokparsefailure" in [tags] {\n        file { path => "\/data\/logs\/logstash\/grok_failures.txt" }\n    } else {\n        elasticsearch {\n            hosts => ["192.168.1.10:9200"]\n            index => "%{type}"\n        }\n        stdout {\n           codec => rubydebug\n        }\n    }\n}<\/code><\/pre>\n
TIME_STAMP_A \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}.\\d{3}\nTIME_STAMP_T \\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}.\\d{3}Z\nTIME_STAMP_P \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\nTIME_STAMP_S \\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3}\nHOST_NAME_PATTERN [a-zA-Z0-9._-]+\nAPP_NAME [a-zA-Z0-9._-]+\nLOG_LVL [a-zA-Z0-9._-]+\nCORRELATION_ID [0-9a-f-]{36}\nCIP ((?:(?:25[0-5]|2[0-4]\\d|((1\\d{2})|([1-9]?\\d)))\\.){3}(?:25[0-5]|2[0-4]\\d|((1\\d{2})|([1-9]?\\d))))\nID_PATTERN [0-9a-f\\-]{36}\nRPC_ID_PATTERN [0-9\\.]+\nAPP_OR_METHOD [\/a-zA-Z0-9._-]+\nTRACE_ID [0-9a-f]*\nSPAN_ID [0-9a-f]*\nPROCESS_ID \\d{3,5}\nPROCESS_NAME [a-zA-Z0-9._-]+\nSPAN_EXPORTABLE [a-z]{0,5}\nCLASS_PATH [a-zA-Z0-9._]+\nMETHOD_NAME [a-zA-Z0-9_]+\nCODE_LINE \\d{1,5}\nCONTENT [\\s\\S]*$<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
grok\u662f\u4e00\u79cd\u91c7\u7528\u7ec4\u5408\u591a\u4e2a\u9884\u5b9a\u4e49\u7684\u6b63\u5219\u8868\u8fbe\u5f0f\uff0c\u7528\u6765\u5339\u914d\u5206\u5272\u6587\u672c\u5e76\u6620\u5c04\u5230\u5173\u952e\u5b57\u7684\u5de5\u5177\u3002\u901a\u5e38\u7528\u6765\u5bf9\u65e5\u5fd7\u6570\u636e\u8fdb\u884c\u9884\u5904 […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[345],"tags":[347,343],"class_list":["post-1379","post","type-post","status-publish","format-standard","hentry","category-elk","tag-grok","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts\/1379","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/comments?post=1379"}],"version-history":[{"count":0,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts\/1379\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/media?parent=1379"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/categories?post=1379"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/tags?post=1379"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}