Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set up alerting for ERROR logs from dialogue origins #815

Open
iamdanfox opened this issue Jun 4, 2020 · 1 comment
Open

Set up alerting for ERROR logs from dialogue origins #815

iamdanfox opened this issue Jun 4, 2020 · 1 comment

Comments

@iamdanfox
Copy link
Contributor

What happened?

My BALANCED_RTT PR contained a bug which caused pds-121981 requiring lots of manual bouncing of services to remediate. We should have been able to spot this very early on, because we were logging ERROR messages from the com.palantir.dialogue.core.RttSampler class, which have never been seen before.

Unfortunately, we didn't have any alerting on error logs from our libraries, so we didn't catch this until users reported an outage.

What did you want to happen?

Dialogue maintainers should get alerted when log.error messages come from our origins, even if they come from different products.
@ferozco
Copy link
Contributor

ferozco commented Jun 4, 2020

We definitely want to alert on anything from com.palantir.dialogue.core.NeverThrowLimitedChannel origin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@iamdanfox @ferozco