Add OTLP logs to calendar-rest-go

[liustanley]

What does this PR do?

Adds logs over OTLP gRPC to the Go Calendar example. Also updates the timeout in the server to avoid errors.

I override the existing logger when adding the loggerProvider because we use the initial logger in the process of creating the loggerProvider. I'm not sure if there's a better way to set this up.

Motivation

I plan to use this example for end to end pipeline tests in datadog-agent for traces, metrics, and logs.

[jackgopack4]

will overriding the logger cause us to lose the logs that are sent if there are errors initializing components, like Tracer provider?
Are we allowed to have a temporary logger to use before setup instead of overwriting the value of logger?

[jade-guiton-dd]

@jackgopack4 It looks to me like any errors during initialization of the tracer/metrics provider should be sent through the OTel-enabled logger. I think this 2 step approach where an initial stderr-only logger is replaced by an OTel-enabled one makes sense.

However, I wonder if it may be useful to still log things to stderr as well? I also think keeping the original logger config would be good. Maybe something like:

otelCore := otelzap.NewCore(service, otelzap.WithLoggerProvider(loggerProvider))
logger = logger.WithOptions(
  zap.WrapCore(func(core zapcore.Core) zapcore.Core {
    return zapcore.NewTee(core, otelCore)
  }),
  zap.Fields(zap.String("service", service)),
)

I don't have any experience with zap, so no idea if this is idiomatic or not.

[liustanley]

@jade-guiton-dd Good callout, I updated my PR to include the stdout exporter in the loggerprovider which should be the idiomatic way to do this.

[jade-guiton-dd]

That is definitely simpler! Note that this still erases the logger-level configuration of the original logger (ie. the options set by NewDevelopment). It doesn't seem to change anything in non-error cases however, so I won't block you on that.