feat(lambda-event-sources): starting position timestamp for kafka

[nikovirtala]

Issue # (if applicable)

Closes #31808

Reason for this change

It was impossible to start consuming a Kafka topic from a specific point in time.

Description of changes

The user may now set startingPositionTimestamp to ManagedKafkaEventSource and SelfManagedKafkaEventSource to start consuming a Kafka topic from a specific point in time.

Lambda and CloudFormation have supported the functionality for a while, and other stream event sources like KinesisEventSource already had a CDK implementation for it. So, technically, this change is doing nothing new; it is only repeating the pattern that has already been proven to work on other sources.

Description of how you validated changes

The change is tested by adding similar unit tests to other event sources supporting this functionality.

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[moelasmar]

could you please add a new GH issue as a Feature request, and link it to this issue.

[moelasmar]

IMO, this is a feature and not a chore, and it needs update in Readme, and new integration test case

[sumupitchayan]

Agreed with @moelasmar, this seems like a feat to me - @nikovirtala can you please add an integ test and README documentation on this?

[aws-cdk-automation]

This PR has been in the CHANGES REQUESTED state for 3 weeks, and looks abandoned. To keep this PR from being closed, please continue work on it. If not, it will automatically be closed in a week.

[aws-cdk-automation]

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

[nikovirtala]

Agreed with @moelasmar, this seems like a feat to me - @nikovirtala can you please add an integ test and README documentation on this?

README is clear, but creating the integration test for this is a lot of work:

1) there are no existing integration tests for any other Event Source Mappings
2) testing asserting Kafka Event Source Mapping requires creating an MSK cluster or such, which is not as trivial as creating a simple resource like DynamoDB Table (I hope there is something I can borrow from the MSK module, if one exist)

[nikovirtala]

Exemption Request

2. testing Kafka Event Source Mapping requires creating an MSK cluster or such, which is not as trivial as creating a simple resource like DynamoDB Table (I hope there is something I can borrow from the MSK module, if one exist)

Ok, the MSK module is in alpha state, and even that doesn't have an integration test that would create a topic and messages to the topic — which would be required to test this feature 😞

I doubt that creating all that is too much for me.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.52%. Comparing base (e4703c1) to head (b287c83).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #31439   +/-   ##
=======================================
  Coverage   81.52%   81.52%           
=======================================
  Files         224      224           
  Lines       13762    13762           
  Branches     2414     2414           
=======================================
  Hits        11220    11220           
  Misses       2270     2270           
  Partials      272      272           

Flag	Coverage Δ
suite.unit	81.52% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	80.93% <ø> (ø)
packages/aws-cdk-lib/core	82.15% <ø> (ø)

[nikovirtala]

It seems there are zero integration tests in this module. So, can anyone change this module before there is one? 🤔

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

Pull request has been modified.

[moelasmar]

a minor comment, I know that we already use number as a type for this property in Kinesis, but I was thinking that it is better for customers to use a real time stamp here, and we do calculate the epoch number on behalf of the customers.

I can accept to have this property to keep it aligned with Kinesis, but I prefer to add another property that use Time stamp instead of number .. what do you think?

[nikovirtala]

a minor comment, I know that we already use number as a type for this property in Kinesis, but I was thinking that it is better for customers to use a real time stamp here, and we do calculate the epoch number on behalf of the customers.

I can accept to have this property to keep it aligned with Kinesis, but I prefer to add another property that use Time stamp instead of number .. what do you think?

Well, our current internal implementation of this very same functionality is using Date but I chose to use the number here to keep the consistency. I will take a look if I can find a nice way to get them both and we can then choose which looks/feels the best.

If we end up to improve the API here, I can then implement those changes also to the Kinesis source (naturally on another pull request).

[moelasmar]

thanks @nikovirtala

[moelasmar]

sorry @nikovirtala for not coming back to this PR. I see that you already handled the integ testing point. I left one minor comment.

[nikovirtala]

sorry @nikovirtala for not coming back to this PR. I see that you already handled the integ testing point. I left one minor comment.

No worries! I was stuck in thinking that the integration test requirement meant something else than it actually does — I had set the bar "a little" too high in my mind 😅

[nikovirtala]

@moelasmar, here is one way to achieve it but is it very idiomatic to do it that way in CDK?

[moelasmar]

in CDK we do not accept the union types in L2 any more. Personally I was thinking on replacing the type to be TimeStamp instead of number, but to be align with kinesis, I think we can add another property, and let customers choose which one to use, and for sure they can only use one of them.

[nikovirtala]

After digging into how this kind of property type change should happen in CDK, I found out that there is no clearly defined convention for it. There is a long discussion, e.g., on Enhanced L1s RFC about using the _v2 pattern, but no consensus on going with it.

So, I suggest that we merge this like this:

I can accept to have this property to keep it aligned with Kinesis

and worry about the API later once there is a clear convention on how to proceed.

The changes I made to test the union type are now reverted.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: b287c83
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[samson-keung]

Left a small comment, then I think it will be good to merge!

[samson-keung]

I'd suggest giving warnings instead of throw errors on these validations so that the following use cases become possible:

• when customers use Tokens for props.startingPosition and/or props.startingPosition (although probably a rare case)
• when customers use escape hatches (e.g. using addPropertyOverride method) to set either of the props after initializing the construct

Same comment on the other validations that throws error (line 260 - 268).