OTEL plugin traceparent header points to inexistent entity instead of Kong when aws-lambda plugin is used

[AlexTheAndroid]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.7

Current Behavior

We are having a setup where a request comes in to a Kong route, which in turn executes four plugins in this exact order (screenshot from NewRelic also attached)

• openid-connect
• rate-limiting-advanced
• opentelemetry
• aws-lambda
  After the last plugin is executed, an AWS Golang Lambda is invoked using the Event type (instead of RequestResponse) and the aws_gateway_compatible property set to true. Everything works as expected except for OTEL. We are trying to create a trace inside the invoked lambda which will have as parent our Kong Gateway. However, as you can see in the attached NewRelic screenshot, although they appear in the same trace, they are parallel to each other and the lambda has a missing parent (which should be the Kong span from above).

Initially, the OTEL plugin was being executed after the aws-lambda plugin, which caused the w3c headers (traceparent and tracestate) to not be injected in the request headers. We changed the execution order so that the OTEL plugin is executed first, which seems to work to some extent, meaning that now we get a traceparent w3c header injected in the headers (but no tracestate header). However, it seems that it is generated incorrectly, as it points to nothing when trying to search by it in NewRelic.

Our assumption is that maybe there is something going on in the header_filter lifecycle step, where the traceparent is being replaced? But we are not sure. Therefore we wanted to check with you whether this behavior is normal or whether you have experienced it before.
It is also important to note that we are manually instrumenting our Golang Lambda. But it should not be that relevant to this use case as the traceparent is generated by the OTEL plugin, and it seems to be incorrectly generated.

I am also attaching our configuration for the OTEL and aws-lambda plugins.

• otel plugin

---
_format_version: "3.0"

add-plugins:
  - selectors:
      - $.services[?(@.name == "custom-service")]

    plugins:
      - name: opentelemetry
        enabled: true
        instance_name: otel-custom-service
        config:
          traces_endpoint: http://<custom-domain>/v1/traces
          http_response_header_for_traceid: trace-id
          propagation:
            default_format: w3c
            inject: 
              - w3c
            extract:
              - w3c
              - aws
            clear:
              - aws
          queue:
            initial_retry_delay: 0.01
            max_batch_size: 200
            max_bytes:
            max_coalescing_delay: 5
            max_entries: 10000
            max_retry_delay: 60
            max_retry_time: 60
            concurrency_limit: 1
          connect_timeout : 1000
          send_timeout: 5000
          read_timeout: 5000
        
        ordering:
          before:
            access:
              - aws-lambda

• aws-lambda plugin

---
_format_version: "3.0"

add-plugins:
  - selectors:
      - $.services[?(@.name == "custom-service")]

    plugins:
      - name: aws-lambda
        instance_name: aws-lambda-custom-service
        enabled: true
        protocols:
          - http
          - https
        config:
          aws_imds_protocol_version: v1
          aws_region: <some-region>
          aws_role_session_name: kong
          awsgateway_compatible: true
          base64_encode_body: false
          disable_https: false
          empty_arrays_mode: legacy
          forward_request_body: true
          forward_request_headers: true
          forward_request_method: true
          forward_request_uri: true
          function_name: custom-service-lambda
          invocation_type: Event
          is_proxy_integration: false
          keepalive: 60000
          log_type: Tail
          port: 443
          skip_large_bodies: false
          timeout: 60000

• Order of execution of the plugins:

Expected Behavior

The OTEL plugin should generate a correct traceparent that points to Kong, so that the span of the invoked AWS Lambda would not have a missing parent anymore.

Steps To Reproduce

1. Configure an OTEL plugin that injects w3c headers.
2. Configure an aws-lambda plugin using the Event type.
3. Make sure that in the access lifecycle step the aws-lambda is executed AFTER the OTEL plugin.
4. Set up OTEL tracing in the invoked lambda.
5. Extract the w3c headers in the lambda.
6. Create a span in the lambda that needs to have as parent Kong (meaning that it needs to point or be linked to the traceparent from the w3c headers)

Anything else?

No response