[otlp] Add mTLS Support for OTLP Exporter

[sandy2008]

Fixes #2009
Based on PR

• Otlp grpc tls certificates #4731
• Improve Integration Test by Generating TLS/mTLS Certificates via MSBuild #5818

Changes

This pull request introduces support for mutual TLS (mTLS) in the OpenTelemetry Protocol (OTLP) Exporter, allowing secure gRPC connections over HTTPS and mTLS supports for HTTP Protocol.

Key Changes:

• Certificate file. OTEL_EXPORTER_OTLP_CERTIFICATE
• Client key file. OTEL_EXPORTER_OTLP_CLIENT_KEY
• Client certificate file. OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE

Merge requirement checklist

• CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
• Unit tests added/updated
• Appropriate CHANGELOG.md files updated for non-trivial changes
• Changes in public API reviewed (if applicable)

[CodeBlanch]

We can't just throw away the HttpClient and make a new one. Users are able to configure SSL/TLS today using factory:

opentelemetry-dotnet/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpExporterOptions.cs

Line 134 in 5dff99f

public Func<HttpClient> HttpClientFactory

This change will break any user doing that or doing anything else to the HttpClient they are intending to use here.

[sandy2008]

I modified a little bit, does it look fine? @CodeBlanch

[sandy2008]

If the design looks ok, I will start to write tests :)

[codecov]

Codecov Report

Attention: Patch coverage is 73.07692% with 35 lines in your changes missing coverage. Please review.

Project coverage is 84.92%. Comparing base (6250307) to head (9df6f06).
Report is 391 commits behind head on main.

Files with missing lines	Patch %	Lines
...orter.OpenTelemetryProtocol/OtlpExporterOptions.cs	68.65%	21 Missing ⚠️
...TelemetryProtocol/OtlpExporterOptionsExtensions.cs	77.77%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5918      +/-   ##
==========================================
+ Coverage   83.38%   84.92%   +1.53%     
==========================================
  Files         297      269      -28     
  Lines       12531    12397     -134     
==========================================
+ Hits        10449    10528      +79     
+ Misses       2082     1869     -213     

Flag	Coverage Δ
unittests	?
unittests-Project-Experimental	84.71% <73.07%> (?)
unittests-Project-Stable	84.88% <73.07%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...TelemetryProtocol/OtlpExporterOptionsExtensions.cs	80.78% <77.77%> (-12.84%)	⬇️
...orter.OpenTelemetryProtocol/OtlpExporterOptions.cs	87.50% <68.65%> (-8.87%)	⬇️

... and 240 files with indirect coverage changes

[sandy2008]

Just for your info, AddCertificatesToHttpClient is not needed, so will delete it in the next commit.
@alanwest @CodeBlanch @sokoide Does other changes look ok?

[rajkumar-rangaraj]

Just for your info, AddCertificatesToHttpClient is not needed, so will delete it in the next commit. @alanwest @CodeBlanch @sokoide Does other changes look ok?

@sandy2008 I recommend putting this on hold for a month. We are working on a custom serialization (#5730) and also we plan to handle GRPC without relying on Grpc.Net packages and removing dependencies on Google Protobuf and GRPC libraries. This will impact the part where you are adding support for mTLS.

[sandy2008]

Just for your info, AddCertificatesToHttpClient is not needed, so will delete it in the next commit. @alanwest @CodeBlanch @sokoide Does other changes look ok?

@sandy2008 I recommend putting this on hold for a month. We are working on a custom serialization (#5730) and also we plan to handle GRPC without relying on Grpc.Net packages and removing dependencies on Google Protobuf and GRPC libraries. This will impact the part where you are adding support for mTLS.

Hi :) @rajkumar-rangaraj
Since there is also http part as well, shall I change this PR to support HTTP first, and add GRPC support afterwards?

[rajkumar-rangaraj]

Since there is also http part as well, shall I change this PR to support HTTP first, and add GRPC support afterwards?

Today, we had a discussion on this topic in the SIG meeting. We all feel it is better to hold off on this work as this space is expected to change drastically in the coming weeks. We thought it would be beneficial if you could join the next SIG meeting to discuss it further.

[sandy2008]

Since there is also http part as well, shall I change this PR to support HTTP first, and add GRPC support afterwards?

Today, we had a discussion on this topic in the SIG meeting. We all feel it is better to hold off on this work as this space is expected to change drastically in the coming weeks. We thought it would be beneficial if you could join the next SIG meeting to discuss it further.

Thank you for the update. @rajkumar-rangaraj

Could you please let me know the time and participation details for the next SIG meeting? I think it’s important for @sokoide , as a co-author, to attend as well, since I might not be able to cover everything alone.

[rajkumar-rangaraj]

Could you please let me know the time and participation details for the next SIG meeting? I think it’s important for @sokoide , as a co-author, to attend as well, since I might not be able to cover everything alone.

Please check the details here - https://github.com/open-telemetry/opentelemetry-dotnet?tab=readme-ov-file#contributing

[sandy2008]

Could you please let me know the time and participation details for the next SIG meeting? I think it’s important for @sokoide , as a co-author, to attend as well, since I might not be able to cover everything alone.

Please check the details here - https://github.com/open-telemetry/opentelemetry-dotnet?tab=readme-ov-file#contributing

Thank you for the details! We’ll join the SIG meeting scheduled for November 5 at 16:00 PT (November 6, 11:00 JST). Looking forward to it!

[psmulovics]

Do you want users of this API being able to add their custom validation callbacks next to this one?

[sandy2008]

@rajkumar-rangaraj @alanwest

    public static void TryEnableIHttpClientFactoryIntegration(this OtlpExporterOptions options, IServiceProvider serviceProvider, string httpClientName)
    {
        if (serviceProvider != null
            && options.Protocol == OtlpExportProtocol.HttpProtobuf
            && options.HttpClientFactory == options.DefaultHttpClientFactory)
        {
            options.HttpClientFactory = () =>
            {
                Type? httpClientFactoryType = Type.GetType("System.Net.Http.IHttpClientFactory, Microsoft.Extensions.Http", throwOnError: false);
                if (httpClientFactoryType != null)
                {
                    object? httpClientFactory = serviceProvider.GetService(httpClientFactoryType);
                    if (httpClientFactory != null)
                    {
                        MethodInfo? createClientMethod = httpClientFactoryType.GetMethod(
                            "CreateClient",
                            BindingFlags.Public | BindingFlags.Instance,
                            binder: null,
                            new Type[] { typeof(string) },
                            modifiers: null);

                        if (createClientMethod != null)
                        {
                            HttpClient? client = (HttpClient?)createClientMethod.Invoke(httpClientFactory, new object[] { httpClientName });

                            if (client != null)
                            {
                                client.Timeout = TimeSpan.FromMilliseconds(options.TimeoutMilliseconds);

                                // Set up a new HttpClientHandler to configure certificates and callbacks
                                var handler = new HttpClientHandler();

    #if NET6_0_OR_GREATER
                                // Add server certificate validation
                                if (!string.IsNullOrEmpty(options.CertificateFile))
                                {
                                    var trustedCertificate = X509Certificate2.CreateFromPemFile(options.CertificateFile);
                                    handler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) =>
                                    {
                                        if (cert != null && chain != null)
                                        {
                                            chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
                                            chain.ChainPolicy.CustomTrustStore.Add(trustedCertificate);
                                            return chain.Build(cert);
                                        }
                                        return false;
                                    };
                                }

                                // Add client certificate
                                if (!string.IsNullOrEmpty(options.ClientCertificateFile) && !string.IsNullOrEmpty(options.ClientKeyFile))
                                {
                                    var clientCertificate = X509Certificate2.CreateFromPemFile(options.ClientCertificateFile, options.ClientKeyFile);
                                    handler.ClientCertificates.Add(clientCertificate);
                                }
    #else
                                throw new PlatformNotSupportedException("mTLS support requires .NET 6.0 or later.");
    #endif

                                // Re-create HttpClient using the custom handler
                                return new HttpClient(handler) { Timeout = client.Timeout };
                            }
                        }
                    }
                }

                return options.();
            };
        }
    }

Something like this would work?

[rajkumar-rangaraj]

@sandy2008 Sorry, I don't have the bandwidth to review this now. As we discussed in the SIG, I will get to this part once the dependencies on Google.Protobuf and Grpc.Net are removed. Additionally, until we have a solution in OpenTelemetry, you could implement a custom solution for HTTP/protobuf.