Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix server.address field in HTTP logs #4142

Merged
merged 3 commits into from
Nov 22, 2024

Conversation

michel-laterman
Copy link
Contributor

What is the problem this PR solves?

server.address is empty in HTTP request logs.

How does this PR solve the problem?

Populate server.address using the interface associated with http.LocalAddrContextKey.

How to test this PR locally

start a fleet-server and make an HTTP request.

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool

@michel-laterman michel-laterman added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Nov 22, 2024
@michel-laterman michel-laterman requested a review from a team as a code owner November 22, 2024 13:33
Copy link
Contributor

mergify bot commented Nov 22, 2024

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Nov 22, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Nov 22, 2024
Comment on lines 257 to 262
if zlog.Info().Enabled() {
e.Uint64(ECSHTTPRequestBodyBytes, rdCounter.Count())
e.Uint64(ECSHTTPResponseBodyBytes, wrCounter.Count())
e.Int(ECSHTTPResponseCode, wrCounter.statusCode)
e.Int64(ECSEventDuration, time.Since(start).Nanoseconds())

e.Msgf("%d HTTP Request", wrCounter.statusCode)
e.Msg("HTTP Request")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed the middleware layer to log an info level message for each request if info is enabled.

The previous behaviour was to log an (info level) message if Debug is enabled, or the server responded in a non 2xx status

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure this is a good idea? How frequent would these logs be in the median case?

Copy link
Contributor Author

@michel-laterman michel-laterman Nov 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It depends on the number of connected agents.
IIRC agents will long poll 10m by default, so we would expect a log every 10m per agent for checkins + additional entries for each action's acks (when an action or policy change is sent)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds like potentially quite a bit. Could we rate limit or aggregate these based on return code?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I'm fine with the rest of this PR, it's just this change that seems potentially dangerous.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually the previous condition is broken; it would only output the (info level) message if debug is enabled.

Copy link
Contributor Author

@michel-laterman michel-laterman Nov 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll revert & fix the condition; then make an issue for sampling logs

EDIT:
issue: #4145

@michel-laterman michel-laterman merged commit 4132a50 into elastic:main Nov 22, 2024
8 checks passed
@michel-laterman michel-laterman added backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify labels Nov 22, 2024
@michel-laterman michel-laterman deleted the fix-http-addr-field branch November 22, 2024 16:49
mergify bot pushed a commit that referenced this pull request Nov 22, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)
mergify bot pushed a commit that referenced this pull request Nov 22, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)
mergify bot pushed a commit that referenced this pull request Nov 22, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)
michel-laterman added a commit that referenced this pull request Nov 25, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)

Co-authored-by: Michel Laterman <[email protected]>
michel-laterman added a commit that referenced this pull request Nov 25, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)

Co-authored-by: Michel Laterman <[email protected]>
michel-laterman added a commit that referenced this pull request Nov 25, 2024
* Fix server.address field in HTTP logs, fix request log condition

(cherry picked from commit 4132a50)

Co-authored-by: Michel Laterman <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants