使用ilogtail k8s 部署方式收集日志一段时间后夯住不再收集日志,然后使用nohup 启动ilogtail 处理相同的日志,检测正则等语法是没有任何问题的,也是可以收集的 #923
-
#nohup 启动的ilogtail 配置文件
k8s 启动的ilogtail 的配置文件apiVersion: v1 #日志内容 |
Beta Was this translation helpful? Give feedback.
Replies: 11 comments 1 reply
-
ilogtail.LOG, logtail_plugin.LOG麻烦提供一下 |
Beta Was this translation helpful? Give feedback.
-
好的,昨天忙没顾上,立马上传 |
Beta Was this translation helpful? Give feedback.
-
ilogtail.LOG[2023-06-12 10:44:55.021622] [info] [000010] /src/core/app_config/AppConfigBase.cpp:156 AppConfigBase AppConfigBase:success [2023-06-12 10:44:55.021795] [info] [000010] /src/core/app_config/AppConfigBase.cpp:601 logtail checkpoint path:/usr/local/ilogtail/checkpoint/logtail_check_point [2023-06-12 10:44:55.033009] [info] [000010] /src/core/config_manager/ConfigManagerBase.cpp:1099 load user_yaml_config.d config:true file:/usr/local/ilogtail/./user_yaml_config.d/saas_stdout.yaml now config count:1 [2023-06-12 10:44:55.102128] [info] [000010] /src/core/controller/EventDispatcherBase.cpp:607 start to verify existed checkpoints, total checkpoint count:0 |
Beta Was this translation helpful? Give feedback.
-
logtail_plugin.LOG2023-06-12 10:44:55 [INF] [plugin_export.go:250] [setGCPercentForSlowStart] set startup GC percent from 30 to 20 2023-06-12 10:44:55 [INF] [plugin_manager.go:120] [Init] init plugin, local env tags:[node_name mediaprocess30241node node_ip 172.17.230.241] |
Beta Was this translation helpful? Give feedback.
-
采集的业务日志有什么特殊的地方吗?之前好像提过有单行很大的日志? |
Beta Was this translation helpful? Give feedback.
-
ilogtail_plugin.LOG
|
Beta Was this translation helpful? Give feedback.
-
上边的插件日志清晰的可以看到因为日志很大,被截断了,被截断后的日志,因为不符合下一行的开头,显示not match |
Beta Was this translation helpful? Give feedback.
-
unmatch this log |
Beta Was this translation helpful? Give feedback.
-
配置文件如下
|
Beta Was this translation helpful? Give feedback.
-
本来想放一份原始日志,无奈超过了最大字数限制 |
Beta Was this translation helpful? Give feedback.
-
自行解决了一下几个问题 |
Beta Was this translation helpful? Give feedback.
自行解决了一下几个问题
1、我们的单行日志很大,平均是一条20K,比较大的100K的也有,然后ilogtail 的默认批处理行数应该是BulkMaxSize=2048(default),但是我们的kafka 接受生产者的消息一次是最大1MB,很容易导致我们的日志大量丢失,我们修改了kafka 接受的最大大小为5MB,但是会因为我们的日志太大了,初步按照20KB来算,我们修改BulkMaxSize=200,同时设置MaxMessageBytes 为5Mb,这样就可以适当的保证数据不会被kafka drop 掉。
2、后来发现日志完全没了,是因为一个同事失误操作把我在deploy yaml 李的lable ilogtail:enable 的标签给删除了,自然也就不收集了,这个dicussion 就是这个问题,后来我又加回来了。
3、我设置了单行大小不要超过512K,这个从某种意义上是必要的取舍,日志太大日志收集就没法玩了。