Enable log correlation for edxapp

[robrap]

AC:

• Enable log correlation for edxapp.
• Update docs and communicate.

Notes:

• See https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/python/.
  • The GSRE ticket: https://2u-internal.atlassian.net/browse/GSRE-1687 was completed to enable the appropriate setting.
• We should continue to rollout to stage, then edge, then prod.
• See Slack thread from another team wanting this for their service.
• We may want to communicate and document what we learn.

[robrap]

[question] If we decide edxapp observability is the next in line for priority, we may want to move to the existing Datadog Migration epic, or create a new one.

[robrap]

The following screenshot is from the Logs tab of a Trace. It makes it seem like we just need to inject trace_id and span_id, event though the docs include much more.

[robrap]

I wrote my findings of the challenges in https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1277394947/Challenges+with+edx-platform+log+formatting.

• This private page includes some log examples, which I wanted to keep private.

Here is a copy of the most relevant information from that page:

The log formatting in edx-platform is not easily configurable. You can find some of the configuration in code the edx-platform logsettings.py.

The following are past discussions about making it more easily configurable, but this work was not completed at the time:

• https://discuss.openedx.org/t/improving-the-flexibility-of-the-edx-platform-log-configuration/5379/4
• A draft for making a decision on managing logging configurations  openedx/edx-platform#28603

@timmc-edx: I'm going to punt this to your containerization epic as a P5. Once that initial work is complete, which should affect some of the log file usage, we can determine whether to P4 or close this issue.