Ensure "received invalid w3c traceparent" has resolved

[robrap]

Our hypothesis is that logs with "received invalid w3c traceparent" will resolve once New Relic is no longer enabled. We think New Relic may be setting a trace header that gets misinterpreted by Datadog.

We need to do the following in order:

• Wait until New Relic has been disabled.
• Ensure "received invalid w3c traceparent" is not appearing in the Live Tail view of logs.
• If still appearing, work with DD support to resolve.
• Remove the temporary log filter from edx-logs once fixed.

[robrap]

Update: I updated the exclusion rule from 99% to 99.99%, because it was still allowing too many logs.

[robrap]

Update: I moved the retention rule up so it happens before the EKS rules.

Additionally, we may ultimately need a DD support ticket for this, if the DD trace support ticket doesn't resolve this.

[jristau1984]

Currently, I do not see any instances of this log message in the last one month, however, the exclusion rule is currently set to 100%, so there would be no records kept. I have not seen it appearing in Live Tail after 15 minutes.

I think we can update the retention rule to something like 0% and then check back in 24 hours. If no logs, we can remove the retention rule and call this ticket complete.

[jristau1984]

I have updated the retention rule to exclude 0%. Now we wait...

[jristau1984]

We have logged almost 36k of these since the exclusion was turned off yesterday

So, it appears this is not yet fully resolved. Over 95% of the messages are in the edx-analytics-api repo, and a handful % is from LMS.
https://app.datadoghq.com/logs?query=%22received%20invalid%20w3c%20traceparent%22&agg_m=count&agg_m_source=base&agg_q=service&agg_q_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=desc&top_n=10&top_o=top&viz=toplist&x_missing=true&from_ts=1729431977933&to_ts=1729604777933&live=true

I will move the exclusion rule back to 99% so that it is still apparent, but not costing us much $.