Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: awslabs/amazon-kinesis-client-nodejs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v2.1.0
Choose a base ref
...
head repository: awslabs/amazon-kinesis-client-nodejs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Loading
5 changes: 5 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
*Issue #, if available:*

*Description of changes:*

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
11 changes: 11 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
version: 2
updates:
- package-ecosystem: "maven"
directory: "/"
schedule:
interval: "weekly"
# Add assignees, reviewers
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
31 changes: 31 additions & 0 deletions .github/workflows/node.js.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs

name: Node.js CI

on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]

jobs:
build:

runs-on: ubuntu-latest

strategy:
matrix:
node-version: ['18.x', '20.x', '21.x']
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/

steps:
- uses: actions/checkout@v3
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- run: npm ci
- run: npm run build --if-present
- run: npm test
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -4,5 +4,4 @@
/node_modules
*.log
.idea
package-lock.json

81 changes: 79 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -158,7 +158,7 @@ Amazon KCL for Node.js uses [MultiLangDaemon][multi-lang-daemon] provided by [Am
### Setting Up the Environment
Before running the samples, make sure that your environment is configured to allow the samples to use your [AWS Security Credentials](http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html), which are used by [MultiLangDaemon][multi-lang-daemon] to interact with AWS services.

By default, the [MultiLangDaemon][multi-lang-daemon] uses the [DefaultAWSCredentialsProviderChain][DefaultAWSCredentialsProviderChain], so make your credentials available to one of the credentials providers in that provider chain. There are several ways to do this. You can provide credentials through a `~/.aws/credentials` file or through environment variables (**AWS\_ACCESS\_KEY\_ID** and **AWS\_SECRET\_ACCESS\_KEY**). If you're running on Amazon EC2, you can associate an IAM role with your instance with appropriate access.
By default, the [MultiLangDaemon][multi-lang-daemon] uses the [DefaultCredentialsProvider][DefaultCredentialsProvider], so make your credentials available to one of the credentials providers in that provider chain. There are several ways to do this. You can provide credentials through a `~/.aws/credentials` file or through environment variables (**AWS\_ACCESS\_KEY\_ID** and **AWS\_SECRET\_ACCESS\_KEY**). If you're running on Amazon EC2, you can associate an IAM role with your instance with appropriate access.

For more information about [Amazon Kinesis][amazon-kinesis] and the client libraries, see the
[Amazon Kinesis documentation][amazon-kinesis-docs] as well as the [Amazon Kinesis forums][kinesis-forum].
@@ -273,6 +273,83 @@ In this release, we have abstracted these implementation details away and expose


## Release Notes
### Release 3.0.0 (November 6, 2024)
* New lease assignment / load balancing algorithm
* KCL 3.x introduces a new lease assignment and load balancing algorithm. It assigns leases among workers based on worker utilization metrics and throughput on each lease, replacing the previous lease count-based lease assignment algorithm.
* When KCL detects higher variance in CPU utilization among workers, it proactively reassigns leases from over-utilized workers to under-utilized workers for even load balancing. This ensures even CPU utilization across workers and removes the need to over-provision the stream processing compute hosts.
* Optimized DynamoDB RCU usage
* KCL 3.x optimizes DynamoDB read capacity unit (RCU) usage on the lease table by implementing a global secondary index with leaseOwner as the partition key. This index mirrors the leaseKey attribute from the base lease table, allowing workers to efficiently discover their assigned leases by querying the index instead of scanning the entire table.
* This approach significantly reduces read operations compared to earlier KCL versions, where workers performed full table scans, resulting in higher RCU consumption.
* Graceful lease handoff
* KCL 3.x introduces a feature called "graceful lease handoff" to minimize data reprocessing during lease reassignments. Graceful lease handoff allows the current worker to complete checkpointing of processed records before transferring the lease to another worker. For graceful lease handoff, you should implement checkpointing logic within the existing `shutdownRequested()` method.
* This feature is enabled by default in KCL 3.x, but you can turn off this feature by adjusting the configuration property `isGracefulLeaseHandoffEnabled`.
* While this approach significantly reduces the probability of data reprocessing during lease transfers, it doesn't completely eliminate the possibility. To maintain data integrity and consistency, it's crucial to design your downstream consumer applications to be idempotent. This ensures that the application can handle potential duplicate record processing without adverse effects.
* New DynamoDB metadata management artifacts
* KCL 3.x introduces two new DynamoDB tables for improved lease management:
* Worker metrics table: Records CPU utilization metrics from each worker. KCL uses these metrics for optimal lease assignments, balancing resource utilization across workers. If CPU utilization metric is not available, KCL assigns leases to balance the total sum of shard throughput per worker instead.
* Coordinator state table: Stores internal state information for workers. Used to coordinate in-place migration from KCL 2.x to KCL 3.x and leader election among workers.
* Follow this [documentation](https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html#kcl-migration-from-2-3-IAM-permissions) to add required IAM permissions for your KCL application.
* Other improvements and changes
* Dependency on the AWS SDK for Java 1.x has been fully removed.
* The Glue Schema Registry integration functionality no longer depends on AWS SDK for Java 1.x. Previously, it required this as a transient dependency.
* Multilangdaemon has been upgraded to use AWS SDK for Java 2.x. It no longer depends on AWS SDK for Java 1.x.
* `idleTimeBetweenReadsInMillis` (PollingConfig) now has a minimum default value of 200.
* This polling configuration property determines the [publishers](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/polling/PrefetchRecordsPublisher.java) wait time between GetRecords calls in both success and failure cases. Previously, setting this value below 200 caused unnecessary throttling. This is because Amazon Kinesis Data Streams supports up to five read transactions per second per shard for shared-throughput consumers.
* Shard lifecycle management is improved to deal with edge cases around shard splits and merges to ensure records continue being processed as expected.
* Migration
* The programming interfaces of KCL 3.x remain identical with KCL 2.x for an easier migration. For detailed migration instructions, please refer to the [Migrate consumers from KCL 2.x to KCL 3.x](https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html) page in the Amazon Kinesis Data Streams developer guide.
* Configuration properties
* New configuration properties introduced in KCL 3.x are listed in this [doc](https://github.com/awslabs/amazon-kinesis-client/blob/master/docs/kcl-configurations.md#new-configurations-in-kcl-3x).
* Deprecated configuration properties in KCL 3.x are listed in this [doc](https://github.com/awslabs/amazon-kinesis-client/blob/master/docs/kcl-configurations.md#discontinued-configuration-properties-in-kcl-3x). You need to keep the deprecated configuration properties during the migration from any previous KCL version to KCL 3.x.
* Metrics
* New CloudWatch metrics introduced in KCL 3.x are explained in the [Monitor the Kinesis Client Library with Amazon CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html) in the Amazon Kinesis Data Streams developer guide. The following operations are newly added in KCL 3.x:
* `LeaseAssignmentManager`
* `WorkerMetricStatsReporter`
* `LeaseDiscovery`

### Release 2.2.6 (April 25, 2024)
* [PR #327](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/327) Upgraded amazon-kinesis-client from 2.5.5 to 2.5.8
* [PR #329](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/329) Upgraded aws-sdk from 2.1562.0 to 2.1603.0
* [PR #95](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/95) Upgraded jcommander from 1.72 to 1.82
* [PR #271](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/271) Upgraded org.codehaus.mojo:animal-sniffer-annotations from 1.20 to 1.23
* [PR #266](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/266) Upgraded com.amazonaws:aws-java-sdk-core from 1.12.370 to 1.12.512
* [PR #313](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/313) Upgraded logback.version from 1.3.12 to 1.5.3
* [PR #305](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/305) Upgraded org.slf4j:slf4j-api from 2.0.5 to 2.0.12
* [PR #325](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/325) Upgraded mocha from 9.2.2 to 10.4.0
* [PR #307](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/307) Upgraded com.google.protobuf:protobuf-java from 3.21.7 to 3.25.3
* [PR #262](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/262) Upgraded checker-qual from 2.5.2 to 3.36.0
* [PR #303](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/303) Upgraded commander from 8.3.0 to 12.0.0
* [PR #287](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/287) Upgraded sinon from 14.0.2 to 17.0.1
* [PR #318](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/318) Upgraded awssdk.version from 2.20.43 to 2.25.11
* [PR #319](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/319) Upgraded org.reactivestreams:reactive-streams from 1.0.3 to 1.0.4
* [PR #320](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/320) Upgraded netty-reactive.version from 2.0.6 to 2.0.12
* [PR #330](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/330) Upgraded io.netty:netty-codec-http from 4.1.100.Final to 4.1.108.Final
* [PR #331](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/331) Upgraded ion-java from 1.5.1 to 1.11.4
* [PR #211](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/211) Upgraded fasterxml-jackson.version from 2.13.4 to 2.14.1

### Release 2.2.5 (February 29, 2024)
* [PR #309](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/309) Updated amazon-kinesis-client and amazon-kinesis-client multilang from 2.5.4 to 2.5.5 and updated awssdk.version to match amazon-kinesis-client from 2.19.2 to 2.20.43

### Release 2.2.4 (January 16, 2024)
* [PR #298](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/298) Added dependency on aws-sdk arns
* [PR #293](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/293) Updated logback-classic to 1.3.12

### Release 2.2.3 (December 18, 2023)
* [PR #291](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/291) Updated KCL and KCL multilang to the latest version 2.5.4
* [PR #284](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/284) Updated netty to 4.1.100.Final, fasterxml-jackson to 2.13.5, and guava to 32.1.1-jre
* [PR #277](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/277) Updated com.google.protobuf:protobuf-java from 3.21.5 to 3.21.7

### Release 2.2.2 (January 4, 2023)
* [PR #207](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/207) Add endpoints-spi dependency

### Release 2.2.1 (January 3, 2023)
* [PR #202](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/202) Keep Java dependencies in sync with the KCL V2.4.4
* Updated dependencies to match the v2.4.4 KCL Java release
* Updated slfj to resolve the logger's incompatibility problem

### Release 2.2.0 (September 15, 2022)
* [PR #165](https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/165) Update Java dependencies
* KCL and KCL-multilang are updated to the latest version 2.4.3

### Release 2.1.0 (January 31, 2020)
[Milestone #4](https://github.com/awslabs/amazon-kinesis-client-nodejs/milestone/4)
@@ -346,7 +423,7 @@ __Updating minimum requirement for the JDK version to 8__
[amazon-kinesis-python-github]: https://github.com/awslabs/amazon-kinesis-client-python
[amazon-kinesis-ruby-github]: https://github.com/awslabs/amazon-kinesis-client-ruby
[multi-lang-daemon]: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java
[DefaultAWSCredentialsProviderChain]: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
[DefaultCredentialsProvider]: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html
[kinesis-forum]: http://developer.amazonwebservices.com/connect/forum.jspa?forumID=169
[aws-console]: http://aws.amazon.com/console/
[jvm]: http://java.com/en/download/
Loading