v2.8.0

PIPs

• [PIP 45] Pluggable metadata interface #9148 #9221 #9240 #9273 #9274 #9338 #9346 #9351 #9412 #9485 #9586 #9967 #9973 #10391 #10457 #10532 #10545 #10647
• [PIP 50] Package management service #8637 #8680 #8744 #8764 #8816 #8817 #8858 #8861 #8893 #8907
• [PIP 68] Exclusive producer #8685 #8992 #9554 #9600
• [PIP 70] Lightweight broker entry metadata #8618 #9067 #9088 #9091
• [PIP 71] Pulsar SQL migrate SchemaHandle to Presto decoder #8422
• [PIP 74] Client memory limits #8965 #9676
• [PIP 75] Perform serialization/deserialization with LightProto #9046
• [PIP 76] Streaming offloader #9096
• [PIP 82] Tenant and namespace level rate limiting #10008 #10201 #10204 #10218
• [PIP 83] Message consumption with pooled buffer #10184
• [PIP 85] Support get reader schema for a message in Java Client #10476

Transactions

• Support pending ack state persistence #8881
• Stable position and low watermark for the transaction buffer #9195
• Transaction timeout support #9229
• Transaction coordinator retry to complete the transaction #9236
• Fix race condition when appending transaction log #9238
• Transaction buffer snapshot #9490
• Add metrics for transaction coordinator #9706
• Clean the useless transaction individual acknowledgements based on low watermark #9722
• Fix memory leak when deleting transaction marker #9751
• Check the transaction state at the client side #9776
• Clean aborted transactions for the transaction buffer #9974
• Fix transaction coordinator retry to end transaction #10131
• Fix NPE when opening a new transaction #10139
• Fix transaction log failed to recover #10146
• Fix transaction coordinator recover timeout #10162
• Handling committing and aborting state when recovering transaction coordinator #10179
• Fix NPE in case of enableTransactionCoordinator=false #10182
• Fix transaction buffer client timeout #10206
• Fix recover max local id issue for the transaction coordinator #10224
• Support using transactions on standalone #10238
• Fix transaction buffer lookup issue #10257
• Fix transaction timeout issue at the client side #10323
• Fix transaction client reconnect issue after transaction coordinator unloaded #10327
• Fix transaction timeout not canceled after expired #10366
• Transaction log low watermark optimization #10422
• Fix the issue of transaction buffer client channel is inactive #10407
• Add Admin API for getting transaction coordinator stats #10639
• Add Admin API for getting transaction in transaction buffer stats #10642
• Add Admin API for getting transaction in pending ack stats #10648
• Add Admin API for getting transaction buffer stats and pending ack stats #10650
• Add Admin API for getting transaction coordinator internal stats #10653
• Setup transaction metadata with metadata store #10677
• Fix issue with acknowledge messages from multiple subscriptions of a topic #10689
• Admin API for getting transaction metadata #10690
• Admin API for getting slow transactions #10701
• Fix transaction log handle managed ledger WriteFail state #10711
• Admin API for getting pending ack internal stats #10725
• Fix transaction ack delete marker position when don't have transaction ack #10741
• Fix transaction pending ack generate managedLedgerStats fail exception #10797
• Use Zookeeper Prometheus metric provider to export Zookeeper metrics #10803
• Always allow system topic for transaction buffer snapshot auto create #10876

Security

• Optional auth method name header in HTTP authentication #6799
• Topics level authorization support #7523
• Authorization for setting topic/subscription auto-creation and subscription expire time on namespace #7900
• Allow serializable stream-provider field into AuthenticationTls #10020
• Add configuration for running OWASP Dependency Check for all modules #10288
• Update default TLS protocols to TLSv1.3 and TLSv1.2 for the broker and proxy #10598

Broker

• Share EventLoopGroup between broker and BookKeeper client #2603
• Dispatch batch messages according to consumer permits 7266
• Improve the max pending publish buffer mechanism 7406
• Allow disabling HTTP TRACE/TRACK verbs #7907
• Fix DispatchRateLimiter does not take effect #8611
• Validate namespace isolation policy regex before updating #8804
• Perform automatically cert refresh for pulsar-admin #8831
• Fix updating rack info dynamically #8844
• Fix deadlock when checking message expiration #8877
• Support limit max topics per namespace #8942
• Make ledger rollover check task internally #8946
• Clean up topic that failed to unload from the cache #8968
• Support get broker response for the message acknowledgement #8996
• Support message index for a topic #9039
• Supporting limit max topics per namespace by namespace policy #9042
• Streaming dipsatcher support #9056
• Make Netty acceptor threadPool size con...

v2.6.4

Official release notes:
http://pulsar.apache.org/release-notes/#2.6.4

Download area:
http://pulsar.apache.org/en/download/

Broker

• Disallow parsing of token with none signature in authenticateToken #9172
• Fix marking individual deletes as dirty #9732
• Issue 9082: Broker expires messages one at a time after topic unload #9083
• [logging] Upgrade Log4j2 version to 2.14.0, replace legacy log4j dependency with log4j-1.2-api #8880
• Upgrade Bouncy castle to newest version #8047
• Fixed logic for forceful topic deletion #7356
• Perform periodic flush of ManagedCursor mark-delete posistions #8634
• Fix the batch index ack persistent issue. #9504
• Fix the partition number not equals expected error #9446
• fix the closed ledger did not delete after expired #9136
• Fix testBrokerSelectionForAntiAffinityGroup by increasing OverloadedThreshold #9393

Tiered storage

• [tiered-storage] Allow AWS credentials to be refreshed #9387

Java client

• Compression must be applied during deferred schema preparation and enableBatching is enabled #9396

C++ client

• [C++] Remove namespace check for MultiTopicsConsumerImpl #9520

v2.7.2

Official release notes:
http://pulsar.apache.org/release-notes/#2.7.2

Download area:
http://pulsar.apache.org/en/download/

Broker

• Fix the useless retry when the maximum number of subscriptions is reached #9991
• wrong timeunit in updating lastLedgerCreationInitiationTimestamp #10049
• Avoid spammy logs in case of BK problems #10088
• Fix NonDurableCursorImpl initialPosition by startCursorPosition greater than lastConfirmedEntry problem. #10095
• fix 8115 Some partitions get stuck after adding additional consumers to the KEY_SHARED subscriptions #10096
• Add underReplicate state in the topic internal stats #10013
• Continue graceful shutdown even if web service closing fails #9835
• Issue 9804: Allow to enable or disable the cursor metrics #9814
• Allow to configure BookKeeper all BK client features using bookkeeper_ prefix #9232
• Fix NPEs and thread safety issue in PersistentReplicator #9763
• Non Persistent Topics: Auto-create partitions even when the auto-creation is disabled #9786
• Issue 9602: Add schema type validation #9797
• Fix message not dispatch for key_shared sub type in non-persistent subscription #9826
• zkBookieRackAffinityMapping bug to support for bookkeeper dnsResolver #9894
• Messaging Fix delay message block #10078
• Make PersistentDispatcherMultipleConsumers.readMoreEntries synchronized #10435
• Fix issue in reusing EntryBatchIndexesAcks instances #10400
• Fix schema not added when subscribing an empty topic without schema #9853
• Support advertisedListeners for standalone #10297
• Fix schema ledger deletion when deleting topic with delete schema. #10383
• Fix primitive schema upload for ALWAYS_COMPATIBLE strategy. #10386
• Fix schema type check issue when use always compatible strategy #10367
• Fix CPU 100% when deleting namespace #10337
• add return statement to exit asyncMarkDelete early on failure #10272
• Adding more permits debug statements to better diagnose permit issues #10217

Bookie

• Fallback to PULSAR_GC if BOOKIE_GC is not defined #9621
• Fallback to PULSAR_EXTRA_OPTS if BOOKIE_EXTRA_OPTS isn't defined #10397

Dependency upgrade

• Upgrade Bouncy Castle to 1.68 #9199
• Upgrade athenz version and remove yahoo.bintray.com repository #10471
• Upgrade Netty version to 4.1.60.final #10073
• Upgrade commons-io to address CVE-2021-29425 #10287
• Upgrade Jetty libraries to 9.4.39.v20210325 #10177

Proxy

• Issue 10221: Fix authorization error while using proxy and Prefix subscription authentication mode #10226

Pulsar Admin

• Add get version command for pulsar rest api, pulsar-admin, pulsar-client #9975

Pulsar SQL

• Using pulsar SQL query messages will appear NoSuchLedger… #9910

Docker

• Allow DockerImage to be built from source tarball #9846
• Fix docker standalone image error #10359
• Suppress printing of "skip Processing" lines in startup scripts #10275
• Issue 10058:apply-config-from-env.py to commented default values #10060

Client

• [Java] Fix: seemingly equal ClientConfigurationData's objects end up not being equal #10091
• [Java] Fix AutoConsumeSchema KeyValue encoding #10089
• [Java] Fix error OutOfMemoryError while using KeyValue<GenericRecord, GenericRecord> #9981
• [Java] Fix concurrency issue in incrementing epoch (#10278) #10436
• [Java] Allow pulsar client receive external timer #9802
• [Java] Handle NPE while receiving ack for closed producer #8979
• [Java] Fix batch size not set when deserializing from byte array #9855
• [Java] Fix ensure single-topic consumer can be closed #9849
• [Java] Issue 9585: delete disconnected consumers to allow auto-discovery #9660
• [Python] Support Python Avro schema set default value. #10265
• [Python] Fix nested Map or Array in schema doesn't work #9548
• [C++,Python] [PIP-60] Add TLS SNI support for cpp and python clients #8957
• [C++] Fix C++ client cannot be built on Windows #10363
• [C++] Fix paused zero queue consumer still pre-fetches messages #10036
• [C++] Fix segfault when get topic name from received message id #10006
• [C++] SinglePartition message router is always picking the same partition #9702
• [C++] Reduce log level for ack-grouping tracker #10094
• [WebSocket Client] WebSocket url token param value optimization #10187
• [WebSocket Client] Make the browser client support the token authentication #9886

Functions and Pulsar IO

• Allow customizable function logging #10389
• Pass through record properties from Pulsar Sources #9943
• ISSUE 10153: Pulsar Functions Go fix time unit ns -> ms #10160
• Kinesis Connector: Fix kinesis sink can not retry to send messages #10420
• Kinesis Connector: Fix null error messages in onFailure exception in KinesisSink. #10416

Tiered Storage

• Prevent Class Loader Leak; Restore Offloader Directory Override #9878
• Add logs for cleanup offloaded data operation #9852

v2.7.1

Broker

• Fix topic ownership is not checked when getting topic policy #9781
• Fix the issue of consumers cannot be created for older subscriptions if the limit of maxSubscriptionsPerTopic is reached #9758
• Fix marking individual deletes as dirty #9732
• Fix broker-address header added when response has already been committed #9744
• Fix ByteBuffer allocate error in the AirliftUtils #9667
• Use Atomic Field Updater to increment volatile messagesConsumedCounter #9656
• Schema comparison logic change #9612
• Add metrics for the cursor ack state #9618
• Fix race condition in BrokerService topic cache #9565
• Avoid introducing bookkeeper-common into the pulsar-common #9551
• Async read entries with max size bytes #9532
• Fix the metric data of msgDelayed for partitioned topics is not aggregated #9529
• Fix the batch index ack persistent issue #9504
• Fix logic in ManagedLedgerWriter when config threadNum >= ledgerNum #9479
• Do not use a static map of listeners in TopicPoliciesService #9486
• Makes subscription start from MessageId.latest as default #9444
• Fix setting backlogQuota will always succeed #9382
• Skip clear delayed messages while dispatch does not init #9378
• Expose offloaded storage size to the topic stats #9335
• Expose more info with unknown exception #9323
• Add alerts for expired/expiring soon tokens #9321
• Fix fake complete issue in offloading #9306
• Fix system topic can not auto created #9272
• Fix BookkeeperSchemaStorage NPE #9264
• Fix race condition on producer/consumer maps in ServerCnx #9256
• Fix interceptor disabled in ResponseHandlerFilter.java #9252
• Fix the interceptor that not handle boundary for multipart/form-data #9247
• Add authentication metrics #9244
• Handle web application exception to redirect request #9228
• Skip the interceptor for MediaType.MULTIPART_FORM_DATA #9217
• Keep topic-level policies commands consistent with that for namespace level #9215
• Fix handle topic loading failure due to broken schema ledger #9212
• Fix issue with topic compaction when compaction ledger is empty #9206
• Fix incoming message size issue that introduced in #9113 #9182
• Disallow parsing of token with none signature in authenticateToken #9172
• Fix locking for ConsumerImpl when creating deadLetterProducer #9166
• Fix maxProducersPerTopic cannot be disabled at the namespace level #9157
• Fix wrong default value #9149
• Fix the closed ledger did not delete after expired #9136
• Additional error checks in TwoPhasesCompactor #9133
• Fix master broker while subscribing to non-persistent partitioned topics without topic auto-creation #9107
• Support chained authentication with same auth method name #9094
• Broker expires messages one at a time after topic unload #9083
• Add refresh authentication command in broker #9064
• Add updateRates method for kop to collect publish rate #9094
• Fix DelayedDelivery at the broker level has a default value #9030
• Getting the stats of a non-persistent topic that has been cleaned causes it to re-appear #9029
• Add raw Prometheus metrics provider #9021
• Improve error handling when broker doesn't trust client certificates #8998
• Remove duplicated broker Prometheus metrics type 8995
• Peeking at compressed messages throws an exception (Readonly buffers not supported by Airlift) #8990
• Make namespaces isolation policy updates take effect on time #8976
• Fix NPE in PersistentStickyKeyDispatcherMultipleConsumers #8969
• Fix the recovery not respect to the isolation group settings #8961
• Add properties default value for SchemaInfoBuilder #8952
• Consumer support update stats with specified stats #8951
• Support configure max subscriptions per topic on the topic level policy #8948
• Fix subscription dispatch rate does not work after the topic unload without dispatch rate limit #8947
• Avro custom schema not working in consumer #8939
• Expose non-contiguous deleted messages ranges stats #8936
• Intercept beforeSendMessage calls #8932
• Monitor if a cursor moves its mark-delete position #8930
• Capture stats with precise backlog #8928
• Support configure max subscriptions per topic on the namespace level policy #8924
• Export Prometheus metric for messageTTL #8871
• Add pulsar-perf new feature: one subscription has more than one consumer #8837
• Execute removing non-persistent subscription of a topic from a different thread to avoid deadlock when removing inactive subscriptions #8820
• Fix get partition metadata problem for a non-existed topic #8818
• Fix the problem that batchMessageId is converted to messageIdImpl #8779
• Clear delayed messages when clear backlog #8691
• Fixes first automatic compaction issue #8209

Proxy

• Fix Proxy Config bindAddress does not working for servicePort #9068
• Return correct authz and auth errors from proxy to client #9055
• Fix the metadata setup compatibility issue #8959
• Support HAProxy proxy protocol for broker and proxy #8686

Pulsar Perf

• Dump JVM information #9769
• pulsar-perf uses DefaultCryptoKeyReader for E2E encryption #9668
• Add --batch-index-ack for the pulsar-perf #9521

Transaction

• Fix deleteTransactionMarker memory leak #9752
• Fix transaction messages order error and deduplication error #9024
• Fix transaction log replay not handle right #8723

Pulsar Admin

• Validate offload param #9737
• Inform user when expiring message request is not executed. #9561
• Fix get-message-by-id throwing NPE when message is null #9537
• Expire message by position #9519
• Add subscription backlog size info for topicstats #9302
• Expose schema ledger in topic stats-internal #9284
• Fix potential HTTP get hangs in the Pulsar Admin #9203
• Fix admin-api-brokers list failed #9191
• Fix force delete namespace did not delete all topics of the namespace [#8806](https://github.co...

v2.7.0

The following lists fixes and enhancements in the 2.7.0 release.

Transactions

• Implement the Transaction Buffer Client #6544
• Support produce messages with transaction and commit transaction #7552
• Support consume transaction messages #7781 #7833
• Message acknowledgment with transaction #7856 #8007
• Support transaction abort on partition #7953
• Support transaction abort on subscription #7979
• Handle pending ack at the client side #8037
• Pending ack state implementation #8426
• Support get reponse for message acknowledge #8161
• Refactor the transaction buffer implementation #8291 #8347
• Transaction marker deletion #8318
• Support produce messages with transaction in batch #8415
• Register transaction metadata before send or ack messages #8493
• Expose transaction interface #8505
• Guarantee transaction metadata handlers connected #8563
• Add the batch size in transaction ack command #8659
• Implement the Transaction Log #8658

Topic policy

• Support setting message TTL on topic level #7738
• Support setting retention on topic level #7747
• Support setting delayed delivery policy on topic level #7784
• Support setting max unacked message per subscription on topic level #7802
• Support setting persistence policie on topic level #7817
• Support setting max unacked messages per consumer on topic level #7818
• Support setting deduplication policy on topic level #7821
• Support setting message dispatch rate on topic level #7863)
• Support setting compaction threshold on topic level #7881
• Support setting offload policy on topic level #7883
• Support setting max producers for a topic #7914
• Support setting max consumers for a topic #7968
• Support setting publish rate limitation for a topic #7948
• Support setting inactive topic policy on topic level #7986
• Support setting subscribe rate for a topic #7991
• Support setting max consumers per subscription on topic level #8003
• Support setting subscription dispatch rate on topic level #8087
• Support setting deduplication snapshot interval on topic level #8552

Broker

• Upgrade BookKeeper version to 4.12.0 #8447
• Capture the add entry latency of managed-ledger #4419
• Keep max-concurrent http web-request configurable #7250
• Perform the unload in background after bundle split #7387
• Cleanup already deleted namespace topics when remove cluster #7473
• Support partitioned topics in the Reader #7518
• Support partitioned topic lookup #7605
• Make OrderedExecutor threads number configurable #7765
• Add config to lazily recover cursors when recovering a managed ledger #7858
• Make BookKeeper throttle configurable #7901
• Report compacted topic ledger info when calling get internal stats #7988
• Add broker config to enforce producer to publish encrypted message #8055
• Expose ensemble placement policy in bookkeeper.conf #8210
• Support limit topic publish rate at the broker level #8235
• Support limit the max tenants of the Pulsar cluster #8261
• Support limit the max namespaces per tenant #8267
• Support limit max subscriptions per topic #8289
• Added metrics for topic lookups operations #8272
• Added REST handler for broker ready probe #8303
• Configure namespace anti-affinity in local policies #8349
• Handle hash collision in KeyShared subscription mode #8396
• Configure maxMsgReplDelayInSeconds for each repl-cluster #8409
• Support taking de-duplication snapshots based on time #8474
• Support namespace-level duplication snapshot #8506
• Expose consumer names after the mark delete position for the Key_Shared subscription #8545
• Close topics that remain fenced forcefully #8561

Functions

• Separate out FunctionMetadata related helper functions #7146
• Attach names for all producers/readers in worker service #7165
• Add support to read compacted topic #7193
• Re-work Function MetaDataManager to make all metadata writes only by the leader #7255
• Fix leader/scheduler assignment processing lag problem #7237
• Set source spec's negativeacktimeout as well as timeout #7337
• Add an endpoint to check whether function worker service is initialized #7350
• Functions metadata compaction #7377
• Implement rebalance mechanism #7388
• Improve security setting #7424
• Allow function rebalance to be run periodically #7449
• Log scheduler stats for Pulsar Functions #7474
• Add BatchPushSource interface #7493
• Rejigger contract between LeaderService and rest of components #7520
• Allow null consume in BatchPushSource #7573
• Add readiness api for the worker leader #7601
• Reduce in the leader init time in Pulsar Functions #7611
• Export Function worker internal stats via Prometheus #7641
• Allow ability to specify retain key ordering in functions #7647
• Added ability to specify runtime for localrunner #7681
• Add additional metrics for Pulsar Function Worker #7685
• Use available cores for io thread processing #7689
• Added ability to specify producer config for functions and sources #7721
• Allow the option to make producers thread local #7764
• Add ability for BatchPushSource to notify errors asynchronously #7865
• Allow ability to specify sub position in functions #7891
• Add hostname to consumer/producer properties in Pulsar Functions #7897
• Allow specifying state storage url for Source/Sink localrun #7930
• Enable function worker JVM metrics to be reported via Prometheus #8097
• Add ability to specify EnvironmentBasedSecretsProvider in LocalRunner #8098
• Added ability to specify secrets class in localrunner builder #8127
• Add access to the current m...

v2.6.2

The following lists fixes and enhancements in the 2.6.2 release.

Broker

[Broker] Catch throwable when start pulsar 7221
[Broker] Protobuf-shaded package can not update version 7228
[Broker] Check for null arguments in Namespaces Rest API 7247
[Broker] Handle SubscriptionBusyException in resetCursor api 7335
[Broker] Converted the namespace bundle unload into async operation 7364
[Broker] Update Jersey to 2.31 7515
[Broker] Stop to dispatch when skip message temporally since Key_Shared consumer stuck on delivery 7553
[Broker] Fix bug where producer for geo-replication is not closed when topic is unloaded 7735
[Broker] Make resetting cursor in REST API asynchronous 7744
[Broker] Reestablish namespace bundle ownership from false negative releasing and false positive acquiring 7773
[Broker] make pulsar executor pool size configurable 7782
[Broker] Redirect Get message by id request when broker not serve for the topic 7786
[Broker] Make zk cache executor thread pool size configurable 7794
[Broker] Implement toString() method for TopicMessageIdImpl class 7807
[Broker] Fix pending batchIndexAcks bitSet batchSize in PersistentAcknowledgmentsGroupingTracker 7828
[Broker] Fix deadlock when adding consumer 7841
[Broker] Split message ranges by ledger ID and store them in individualDeletedMessages 7861
[Broker] Fix pulsar metrics providing wrong information 7905
[Broker] Don't fail the health check request when trying to delete the previous subscription 7906
[Broker] Add configuration to set number of channels per bookie 7910
[Broker] Fix publish buffer of one ServerCnx calculated multi-times when check broker's publish buffer 7926
[Broker] Add some logging to improve Authentication debugging and Fix typos in code "occured" -> "occurred" 7934
[Broker] Fix NPE when acknowledge messages at the broker side 7937
[Broker] Fix the wrong issuer url concatenate 7980
[Broker] Upgrade the snakeyaml version to 1.26 7994
[Broker] Exclude vertx from bookkeeper-http package 7997
[Broker] Check null point before setting auto read 7999
[Broker] Fix IndexOutOfBoundsException in the KeyShared subscription when dispatching messages to consumers 8024
[Broker] Upgrade jetty-util version to 9.4.31 8035
[Broker] Add replicated check to checkInactiveSubscriptions 8066
[Broker] Add get-last-message-id admin for v1 api 8081
[Broker] Fix client lookup hangs when broker restarts 8101
[Broker] Should not cache the owner that does not belong to current server 8111
[Broker] Support to specify multi ipv6 hosts in brokerServiceUrl 8120
[Broker] Intercept messages to consumers and add intercept exception 8129
[Broker] Add ChannelFutures utility class to pulsar-common 8137
[Broker] Support Disable Replicated Subscriptions 8144
[Broker] Fix error code returned to client when service unit is not ready 8147
[Broker] Skip intercepting multipart requests 8156
[Broker] Enable intercept filters only when interceptors are configured 8157
[Broker] Clean inactive non-persistent subscriptions 8166
[Broker] Add a new state for namespace-level TTL 8178
[Broker] Fix peek messages failed with subscriptionName not exist 8182
[Broker] Fix pulsar service close exception 8197
[Broker] Use ThreadPoolExecutor instead of EventLoop 8208
[Broker] Close ZK connections at end of metadata setup 8228
[Broker] Delete associated ledgers before deleting cluster metadata 8244
[Broker] Fix stuck lookup operations when the broker is starting up 8273
[Broker] Fix Broker enters an infinite loop in ManagedLedgerImpl.asyncReadEntries 8284
[Broker] Fix message TTL on Key_Shared subscription and Fix ordering issue when replay messages 8292
[Broker] Fix race condition in updating readPosition in ManagedCursorImpl 8299
[Broker] Refresh ZooKeeper-data cache in background to avoid deadlock and blocking IO on ZK thread 8304
[Broker] Upgrade hdfs2 version to 2.8.5 8319
[Broker] Upgrade solr version to 8.6.3 8328
[Broker] Fix deadlock that occurred during topic ownership check 8406

Proxy

[Proxy] Add advertisedAddress config field to ProxyConfiguration 7542
[Proxy] Fix deadlock in pulsar proxy 7690
[Proxy] Handle NPE while updating proxy stats 7766
[Proxy] Fix the null exception when starting the proxy service 8019
[Proxy] Add proxy plugin interface to support user defined additional servlet 8067

Pulsar SQL

[Pulsar SQL] Upgrade Presto version to 332 7194
[Pulsar SQL] Replace com.ning.asynchttpclient with org.asynchttpclient 8099

Java Client

[Java Client] Support input-stream for trustStore cert 7442
[Java Client] Avoid subscribing the same topic again 7823
[java Client] Add autoPartitionsUpdateInterval for producer and consumer 7840
[Java Client] Avoid resolving address for sni-host + thread-safe connection creation 8062
[Java Client] Websocket interface decode URL encoding 8072
[Java Client] Always use SNI for TLS enabled Pulsar Java broker client 8117
[Java Client] Improve timeout handling in ClientCnx to cover all remaining request types (GetLastMessageId, GetTopics, GetSchema, GetOrCreateSchema) 8149
[Java Client] Fix ConsumerImpl memory leaks 8160
[Java Client] Fix issue where paused consumer receives new message when reconnecting 8165
[Java Client] Improve refactored client connection code 8177
[Java Client] Add log level configuration in pulsar-client 8195
[Java Client] Remove unnecessary locks 8207
[Java Client] Fix AutoUpdatePartitionsInterval setting problem 8227
[Java Client] Add read position when joining in the consumer stats 8274
[Java Client] Support reset cursor to a batch index of the batching message 8285
[Java Client] Support exclude the message when reset cursor by message ID 8306
[Java Client] Increasing timeout for pulsar client io threads to shutdown 8316
[Java Client] Support cancelling message & batch futures returned from Reader & Consumer 8326
[Java Client] Disable batch receive timer for Readers 8381
[Java Client] Fix pause does not work for new created consumer 8387

CPP Client

[CPP Client] Wait for all seek operations completed 7216
[CPP Client] Ensure parallel invocations of MultiTopicsCo...

v2.6.1

2.6.1 — 2020-08-21

The following lists fixes and enhancements in 2.6.1 release.

Broker

• [Broker] Limit batch size to the minimum of the maxNumberOfMessages and maxSizeOfMessages #6865
• [Broker] Fix hash range conflict issue in Key_Shared with sticky hash range #7231
• [Broker] Fix the issue that get lookup permission error #7234
• [Broker] Update Jetty to version 9.4.29 #7235
• [Broker] Fix readers backlog stats after data is skipped #7236
• [Broker] Fix the regression in isSupperUser #7241
• [Broker] Avoid introducing null read position for the managed cursor #7264
• [Broker] Fix permission operation check on setRetention admin operation #7281
• [Broker] Fix error in creation of non-durable cursor #7355
• [Broker] Fix bug related to managedLedger properties #7357
• [Broker] Add tenant name check in list namespaces function #7369
• [Broker] Avoid the NPE occurs in method ManagedLedgerImpl.isOffloadedNeedsDelete #7389
• [Broker] Fix producer stuck issue due to NPE thrown when creating a new ledger #7401
• [Broker] Avoid NPEs at ledger creation when DNS failures happen #7403
• [Broker] Support decompression payload if needed in KeyShared subscription #7416
• [Broker] Fix update-cluster cli updates proxy-url #7422
• [Broker] Handle NotAllowed Exception at the client side #7430
• [Broker] Shade jclouds to avoid Gson conflict #7435
• [Broker] Consumer is registered on dispatcher even if hash range conflicts on Key_Shared subscription #7444
• [Broker] Add pulsar-client-messagecrypto-bc into pulsar-client dependency to avoid method not found #7447
• [Broker] Fix update partitions error for non-persistent topic #7459
• [Broker] Use CGroup CPU usage when present #7475
• [Broker] Fix ArrayIndexOutOfBoundsException when dispatch messages to consumer #7483
• [Broker] Get last entry is trying to read entry -1 #7495
• [Broker] Fix timeout opening managed ledger operation #7506
• [Broker] Fixes the exception that occurred when the geo-replication policy is updated #7514
• [Broker] Update Jackson to version 2.11.1 and ensure all dependencies are pinned #7519
• [Broker] Fix protobuf generation on handling repeated long number #7540
• [Broker] Add more logging to the auth operations on failure #7567
• [Broker] Use Consume/Produce/Lookup interfaces for specific operations in allowTopicOperation #7587
• [Broker] Support configuring DeleteInactiveTopic setting in namespace policy #7598
• [Broker] Fix NPE when using advertisedListeners #7620
• [Broker] Fix the issue that deduplication cursor can not be deleted after disabling message deduplication #7656
• [Broker] Add missing AuthenticationDataSource to canConsumeAsync method call #7694
• [Broker] Close the previous reader of the health check topic #7724
• [Broker] Change some WebApplicationException log level to debug #7725
• [Broker] Replay delayed messages in order #7731
• [Broker] Fix the wrong returned URL for lookup when specify advertised listener #7737
• [Broker] Fix topic getting recreated immediately after deletion #7524
• [Broker] Set default root log level to debug #7789
• [Broker] Fix producer stucks on creating ledger timeout #7319
• [Broker] AllowTopicOperationAsync should check the original role is super user #7788

Zookeeper

• [Zookeeper] Use hostname for bookie rackawareness mapping #7361

Pulsar SQL

• [Pulsar SQL] Make Pulsar SQL get correct offload configurations #7701

Pulsar Schema

• [Schema] Fix the error that occurs when getting schemaName by partitioned topic name #7708

Java Client

• [Java Client] Fix the issue that the HTTP header used in Athenz authentication can not be renamed #7311
• [Java Client] Add more detail information of retry errors #7341
• [Java Client] Check NPE when a tombstone (null value) is produced. #7408
• [Java Client] Fix batch ackset recycled multiple times. #7409
• [Java Client] Support Oauth2 authentication #7420
• [Java Client] Ensure the create subscription can be completed when the operation timeout happens #7522
• [Java Client] Fix race condition on the close consumer while reconnecting to the broker. #7589
• [Java Client] Fix validation never return false #7593
• [Java Client] Make OAuth2 auth plugin to use AsyncHttpClient #7615
• [Java Client] Support to set listener name for client CLI #7621
• [Java Client] Fix batch index filter issue in Consumer #7654
• [Java Client] Fix the backward compatibility issues with batch index acknowledgment. #7655
• [Java Client] Fix the issue that batchReceiveAsync is not completed exceptionally when closing consumer #7661
• [Java Client] Fix producer stats recorder time unit error #7670
• [Java Client] Fix shutdown AsyncHttpConnector.delayer #7687

CPP Client

• [CPP Client] Fix partition index error in closing callback #7282
• [CPP Client] Reduce log level for ack-grouping tracker #7373
• [CPP Client] Support Oauth2 authentication #7467
• [CPP Client] Fix segment crashes that caused by race condition of timer in cpp client #7572
• [CPP Client] Fix multitopic consumer segfault on connect error #7588
• [CPP Client] Add support to read credentials from file #7606
• [CPP Client] Fix the issuer_url parsing failure in oauth2 #7791
• [CPP Client] Fix reference leak when reader create #7793

Pulsar Functions

• [Pulsar Function] Use fully qualified hostname as default to advertise worker #7360
• [Pulsar Function] Fix the function BC issue introduced in release 2.6.0 #7528
• [Pulsar Function] Improve security setting of Pulsar Functions #7578
• [Pulsar Function] Differentiate authorization between source/sink/function operations #7466

Go Function

• [Go Function] Fix Go instance config port #7322
• [Go Function] Remove timestamp from metrics #7539

Pulsar Perf

• [Pulsar Perf] Supports tlsAllowInsecureConnection in pulsar-perf produce/consume/read #7300

v2.6.0

2.6.0 — 2020-06-17

Features

PIPs

• [PIP-37] Large message size support #4400
• [PIP-39] Namespace change events (System Topic) #4955
• [PIP-45] Switch ManagedLedger to use MetadataStore interface #5358
• [PIP 53] Contribute DotPulsar to Apache Pulsar
• [PIP-54] Support acknowledgment at batch index level #6052
• [PIP-58] Support consumers set custom message retry delay #6449
• [PIP-60] Support SNI routing to support various proxy-server #6566
• [PIP-61] Advertise multiple addresses #6903
• [PIP-65] Adapting Pulsar IO Sources to support Batch Sources #7090

Broker

• [Broker] Add threshold shedder strategy and metrics exporter for loadbalancer #6772
• [Broker] Add consistent hashing in the Key_Shared distribution #6791
• [Broker] Fixed ordering issue in KeyShared subscription dispatcher when adding consumer #7106 #7108 #7188
• [Broker] Add support for key hash range reading in Key_Shared subscription #5928
• [Broker] Allow for schema reader and writer registration on SchemaDefinition #6905
• [Broker] Support use null key and null value in KeyValue Schema #7139
• [Broker] Support multiple pulsar clusters to use the same bk cluster #5985
• [Broker] Add a flag to skip broker shutdown on transient OOM #6634
• [Broker] Make zookeeper cache expiry time configurable #6668
• [Broker] Check replicator periodically to avoid issue due to zookeeper missing watch #6674
• [Broker] Expose managedLedgerCache, managedLedger, loadBalance metrics to Prometheus #6705
• [Broker] Optimize consumer fetch messages in case of batch message #6719
• [Broker] Add configuration to limit max partitions for a partitioned topic #6794
• [Broker] Change default FlushEntryLogBytes to 256MB to improve bookie io throughput #6915
• [Broker] Introduce precise topic publish rate limiting #7078
• [Broker] Expose new entries check delay in the broker.conf 7154
• [Broker] Add broker interceptor for intercepting all Pulsar command and REST API requests #7143
• [Broker] Only close active consumer for Failover subscription when seek() #7141
• [Broker] Allow to delete topics that are failing to recover #7131
• [Broker] Support set netty max frame size in bookkeeper.conf #7116
• [Broker] Trigger rollover when meeting maxLedgerRolloverTimeMinutes #7111
• [Broker] Cap the dispatcher batch size in bytes to fixed max #7097
• [Broker] Support specify managedLedgerMaxSizePerLedgerMbytes in broker.conf #7085
• [Broker] Allow to grant permissions when the authorization is disabled #7074
• [Broker] Add messages and bytes counter stats to broker-stats/topics #7045
• [Broker] Expose new entries check delay in the broker.conf #7154

Function

• [Function] Built-in functions support #6895
• [Function] Add Go Function heartbeat (and gRPC service) for production usage #6031
• [Function] Add custom property option to functions #6348
• [Function] Separate TLS configuration of function worker and broker #6602
• [Function] Added ability to build consumers in functions and sources #6954
• [Function] Support DLQ on sources and sinks #7032

Pulsar SQL

• [SQL] KeyValue schema support #6325
• [SQL] Multiple version schema support #4847
• [SQL] Fix presto SQL does not start metrics service before queue execute #7030

Pulsar IO

• Added ability for sources to publish messages on their own #6941
• [RabbitMQ] Allow routing key per message to RabbitMQ sink connector #5890
• [RabbitMQ] Add passive config options #6679
• [debezium] Upgrade from v0.10.0-Final to v1.0.0-Final #5972
• [debezium] Support avro schema for debezium connector #6034
• [influxdb2] Add support for influxdb2 in pulsar-influxdb-sink #6601
• [jdbc] Add jdbc sinks: postgres, mariadb, clickhouse #6835

Pulsar Proxy

• [Proxy] Add REST API to get connection and topic stats #6473
• [Proxy] Add advertised address option #6942
• [Proxy] Add proxyLogLevel into config #6948

Clients

• [Java Client] Use pure-java Air-Compressor instead of JNI based libraries #5390
• [Java Client] Change the time unit of patternAutoDiscoveryPeriod to seconds #5950
• [Java Client] Support waiting for inflight messages while closing producer #6648
• [Java Client] Add support to load TLS certs/key dynamically from input stream #6760
• [Java Client] Support return sequence ID when throwing Exception for async send message #6825
• [Java Client] Change the default value of maxLookupRedirects of Java client to 20 #7126
• [Java Client] Limit the number of times lookup requests are redirected #7096
• [CPP Client] Support seek by time on partitioned topic #7198
• [CPP Client] Refresh authentication credentials #7070
• [CPP Client] Fix Result can't be serialized to string inside the library #7034
• [CPP Client] Support zstd and Snappy compression to C API #7014
• [Python Client] Add deliver_at and deliver_after for the producer #6737

Admin

• [Admin] Support delete inactive topic when subscriptions caught up #6077
• [Admin] Add configuration to disable auto-creation of subscriptions #6456
• [Admin] Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies #5936
• [Admin] Support get a message by message ID in pulsar-admin #6331
• [Admin] Support delete subscription forcefully #6383
• [Admin] Add subscribe initial position for consumer CLI #6442
• [Admin] Support to get managed ledger info of a partitioned topic #6532
• [Admin] Support compact all partitions of a partitioned topic #6537
• [Admin] Support multi-hosts in PulsarAdmin #6547
• [Admin] Support to get internal stats for a partitioned topic #6624
• [Admin] Support enable or disable subscription auto-creation at namespace level #6637
• [Admin] Enable to set the subscription expiration time for each namespace #6851

Fixes

• [Broker] Fixed increasing number of partitions with attached readers #7077
• [Broker] Make ZkBookieRackAffinityMapping work as expected #6917
• [Broker] Fix backlog and backlog size stats keeps growing #7082
• [Java Client] Fix connection leak [#6524](#6524...

v2.5.2

2.5.2 — 2020-05-19

Fixes and Enhancements

Broker

• [Broker] Implement AutoTopicCreation by namespace level override #6471
• [Broker] Add custom deletionLag and threshold for offload policies per namespace #6422
• [Broker] Invalidate managed ledgers zookeeper cache instead of reloading on watcher triggered #6659
• [Broker] Retention policy should be respected when there is no traffic #6676
• [Broker] Fixed double delete on a namespace #6713
• [Broker] fix get batch message from http response, only get the first message#6715
• [Broker] Fix Deadlock by Consumer and Reader#6728
• [Broker] avoid backpressure by skipping dispatching if consumer channel is not writable #6740
• [Broker] fix when producing encrypted messages, MessageMetadata objects are not released after they are created. #6745
• [Broker] Bump netty version to 4.1.48.Final #6746
• [Broker] Increase timeout for loading topics #6750
• [Broker] Fix wrong cursor state for cursor without consumer #6766
• [Broker] change nondurable cursor to active to improve performance #6769
• [Broker] register loadbalance znode should attempt to wait until session expired #6788
• [Broker] Fix some empty message related problems in the compacted topic. #6795
• [Broker] Avoid creating partitioned topic for partition name #6846
• [Broker] Add Tls with keystore type config support #6853
• [Broker] fix consumer stuck when batchReceivePolicy maxNumMessages > maxReceiverQueueSize #6862
• [Broker] use originalAuthMethod on originalAuthChecker in Proxy Authentication #6870
• [Broker] Close producer when the topic does not exists. #6879
• [Broker] Handle all exceptions from topic.addProducer #6881
• [Broker] fix topicPublishRateLimiter not effective after restart broker #6893
• [Broker] Expose pulsar_out_bytes_total and pulsar_out_messages_total for namespace/subscription/consumer. #6918
• [Broker] Policy ttlDurationDefaultInSeconds not applying #6920
• [Broker] Fix pulsar admin thread number explode bug. #6940

Pulsar Schema

• [Schema] Fix long field parse in GenricJsonRecord #6622
• [Schema] Fix the leak of cursor reset if message encode fails in Avro schema. #6695
• [Schema] fix Get schema by version can get the deleted schema info #6754 #6754
• [Schema] Fix serialization of enums with json/avro schemas in python #6808
• [Schema] Pulsar SQL Support Avro Schema ByteBuffer Type #6925

CPP Client

• [CPP Client] Auto update topic partitions #6732
• [CPP Client] Subscription InitialPosition is not correctly set on regex consumers. #6810
• [CPP Client] Fix message id is always the default value in send callback #6812
• [CPP Client] Fix message id error if messages were sent to a partitioned topic #6938

Python Client

• [Python Client] Fix Python function protobuf missing field#6641

Pulsar Functions

• [Functions] Support function with format: Function<I, CompletableFuture>#6684
• [Functions] Function endpoint admin/v3/functions/{tenant}/{namespace} always returns 404 #6767
• [Functions] Ensure that all dangling consumers are cleaned up during failures #6778
• [Functions] Fix localrunner netty dependency issue #6779
• [Functions] Fix SerDe validation of function's update #6888

Tiered Storage

• [Tiered Storage] Extract common SerDe method in tiered storage to managed-ledger module #6533
• [Tiered Storage] Make SchemaStorage accessible in Offloader #6567
• [Tiered Storage] Avoid prefetch too much data causing OutOfMemory, when offloading data to HDFS #6717

Pulsar IO

• [IO] JDBC sink does not handle null in schema #6848

v2.5.1

2.5.1 — 2020-04-20

Features

• PIP-55: Refresh Authentication Credentials #6074
• Namespace level support offloader #6183
• Upgrade Avro to 1.9.1 #5938
  • Avro 1.9.1 enables the JSR310 datetimes by default, which might introduce some regression problems if users use generated source code by Avro compiler 1.8.x and contains datetimes fields. It's better to use Avro 1.9.x compiler to recompile.
• Support unload all partitions of a partitioned topic #6187
• Supports evenly distribute topics count when splits bundle #6241
• KeyValue schema support for pulsar sql #6325
• Bump netty version to 4.1.45.Final #6424
• Support BouncyCastle FIPS provider #6588
• Improve Key_Shared subscription message dispatching performance. #6647
• Add JSR310 logical type conversion. #6704
• Introduce maxMessagePublishBufferSizeInMB configuration to avoid broker OOM #6178

Fixes

Broker

• [Broker] Fixed NPE occurs when getting partitioned topic stats #6060
• [Broker] Fixed zero queue consumer message redelivery #6076
• [Broker] Fixed message redelivery for zero queue consumer while using async api to receive messages #6090
• [broker] Fixed bug that backlog message that has not yet expired could be deleted due to TTL #6211
• [Broker] Remove problematic semicolon from conf #6303
• [Broker] Fixed broker to specify a list of bookie groups #6349
• [Broker] Fixed create consumer on partitioned topic while disable topic auto creation #5572
• [Broker] Fix maven broken link #6068
• [Broker] Fixed java code errors reported by lgtm. #6398
• [Broker] Fixed memory leak when running topic compaction. #6485
• [Broker] Fixed admin getLastMessageId return batchIndex #6511
• [Broker] Fixed topic with one partition cannot be updated #6560
• [Broker] Fixed negative un-ack messages in consumer stats #5929
• [broker] Fixed bug that tenants whose allowed clusters include global cannot be created/updated #6275
• [Broker] Fixed log compaction for flow control/empty topic/last deletion #6237
• [Broker] Fixed restore clusterDispatchRate policy for compatibility #6176
• [Broker] Fix some async method problems at PersistentTopicsBase. #6483
• [Broker] This "earlier" message should be avoided to emit when reset cursor.#6393

Managed Ledger

• [Managed Ledger] Fixed consumer received duplicated delayed messages upon restart #6404

Pulsar Proxy

• [Proxy] Fixed correct name for proxy thread executor name #6460
• [Proxy] Fixed logging for published messages #6474
• [Proxy] Fixed proxy routing to functions worker #6486

Zookeeper

• [Zookeeper] Fixed casting in ZooKeeperCache.getDataIfPresent() #6313

Pulsar Functions

• [Function] remove future.join() from PulsarSinkEffectivelyOnceProcessor #6361

Pulsar SQL

• [SQL] Fixed integration Pulsar SQL test failed #6279

Security

• Fixed publish buffer limit does not take effect #6431
• Fixed the bug of authenticationData is't initialized. #6440

Pulsar Schema

• [Schema] Fixed get schema version in HttpLookupService. #6193
• [Schema] Fixed avro schema decode error ClassCastException in Pulsar Function #6662
• [Schema] Fixed channel write error handling for send get raw schema request #6650

Java client

• [Client] Fixed available permits may be greater than 1 even though queue size is 0. #6106
• [Client] Fixed broker client tls settings error #6128
• [Client]Fixed hasMessageAvailable() #6362
• [Client] Fixed duplicate key to send propertys #6390
• [Client] fixed deadlock on send failure #6488
• [Client] Fixed NPE while call getLastMessageId #6562
• [Client] Fixed the max backoff configuration for lookups #6444

C++ client

• [C++] Fixed static linking on C++ lib on MacOS #5581
• [C++] Fixed memory corruption on ExecutorService destructor #6270
• [C++] Fixed handling of canceled timer events on NegativeAcksTracker #6272
• [C++] Fixed for possible deadlock when closing Pulsar client #6277
• [C++] Fixed Unacked Message Tracker by Using Time Partition on C++ #6391
• [C++] Fixed Redelivery of Messages on UnackedMessageTracker When Ack Messages . #6498

Python Client

• [Python Client]Fixed the enum34 package not found #6401

Pulsar Websocket

• [Websocket] Fixed Websocket doesn't set the correct cluster data #6102

Deployments

• [Helm] Autorecovery - Fixed could not find or load main class #6373
• [Helm]: Start proxy pods when at least one broker pod is running #6158

Enhancements

Pulsar Broker

• [Broker] close managed-ledgers before giving up bundle ownership to avoid bad zk-version #5599
• [Broker] Add timeout to search for web service URLs to avoid web threads getting stuck #6124
• [Broker] Flush the potential duplicated message when add messages to a batch. #6326
• [Broker] Avoid getting partition metadata while the topic name is a partition name. #6339
• [Broker] Fixed create partitioned topic with a substring of an existing topic name. #6478
• [Broker] Do not retry on authorization failure #6577
• [Broker]Handle BadVersionException thrown by updateSchemaLocator() #6683
• [Broker] Expose bookkeeper expose explicit lac configuration in broker.conf #5822
• [Broker] Allow to enable/disable delayed delivery for messages on namespace #5915
• [Broker] Prevent creation of regular topic with the same name as existing partitioned topic #5943
• [Broker] Reset cursor with a non-exists position #6120
• [Broker] Use fully qualified hostname as default to advertise brokers #6235
• [broker] Timeout API calls in BrokerService #6489
• [Broker] Start namespace service and schema registry service before start broker. #6499
• [Broker] Disable channel auto read when publish rate or publish buffer exceeded #6550
• [Broker] Resume some servercnx method to public #6581
• [Broker] Enable get precise backlog and backlog without delayed messages. #6310
• [Broker] Avoid using same OpAddEntry between different ledger handles #5942
• [Broker] Clean up closed producer to avoid publish-time for producer #5988
• [Broker] Support delete inactive topic when subscriptions caught up #6077
• [Broker] Add a message on how to make log refresh immediately when starting a component #6078
• [Pulsar Admin] allow tenant ...