Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Omni stops returning machine logs after running for some time #656

Open
Unix4ever opened this issue Sep 30, 2024 · 4 comments
Open
Labels
bug Something isn't working

Comments

@Unix4ever
Copy link
Member

Hard to say what might cause this, instance was running for 11 days.
Logs appeared right after the instance was restarted.
It seems like the whole log history was obtained, so the logs are not dropped, it's just the gRPC API doesn't return anything.

Might worth checking the library for any errors related to log reading.

@Unix4ever Unix4ever added the bug Something isn't working label Sep 30, 2024
@smira
Copy link
Member

smira commented Oct 1, 2024

Look for Talos logs at the moment the logs stopped flowing - if it failed to deliver, it should log that I think

@Unix4ever
Copy link
Member Author

It was sending logs all that time as I could see them after restart, so that's why I think the problem is with reading them

@smira
Copy link
Member

smira commented Oct 2, 2024

It was sending logs all that time as I could see them after restart, so that's why I think the problem is with reading them

ah... so you think it's the problem with the circular buffer, so it had all of the logs, but never returned them?

@smira
Copy link
Member

smira commented Oct 21, 2024

Bumping go-circular might fix it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants