Skip to content

使用ilogtail k8s 部署方式收集日志一段时间后夯住不再收集日志,然后使用nohup 启动ilogtail 处理相同的日志,检测正则等语法是没有任何问题的,也是可以收集的 #923

Closed Answered by charles-liming
charles-liming asked this question in Help
Discussion options

You must be logged in to vote

自行解决了一下几个问题
1、我们的单行日志很大,平均是一条20K,比较大的100K的也有,然后ilogtail 的默认批处理行数应该是BulkMaxSize=2048(default),但是我们的kafka 接受生产者的消息一次是最大1MB,很容易导致我们的日志大量丢失,我们修改了kafka 接受的最大大小为5MB,但是会因为我们的日志太大了,初步按照20KB来算,我们修改BulkMaxSize=200,同时设置MaxMessageBytes 为5Mb,这样就可以适当的保证数据不会被kafka drop 掉。
2、后来发现日志完全没了,是因为一个同事失误操作把我在deploy yaml 李的lable ilogtail:enable 的标签给删除了,自然也就不收集了,这个dicussion 就是这个问题,后来我又加回来了。
3、我设置了单行大小不要超过512K,这个从某种意义上是必要的取舍,日志太大日志收集就没法玩了。

Replies: 11 comments 1 reply

Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
1 reply
@linrunqi08
Comment options

Comment options

You must be logged in to vote
0 replies
Answer selected by messixukejia
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Help
Labels
None yet
2 participants