0.13.0

What's Changed

• Support snappyer 2 by @Argonus in #437
• Misc doc changes by @kianmeng in #432
• Fix typos by @kianmeng in #444
• Kafka 11 - record headers support by @habutre in #414
• CI retry failed tests only by @TC-aLi in #440
• Use offset_fetch api_version when fetching messages by @GDegrove in #450
• Doc: Update GenConsumer example by @EduardoGHdez in #443
• Doc: Auto offset reset config by @EduardoGHdez in #442
• Update metadata before topic creation by @jbruggem in #446
• Apply mix format by @Argonus in #464
• GitHub actions checks cdn by @Argonus in #460
• Trying out github actions by @dantswain in #422
• Migrate GitHub Actions to erlef/setup-beam by @kianmeng in #467
• Use full versions for OTP & Elixir in static checks by @Argonus in #469
• Added examples for topic create/delete functions. by @alfetahe in #470
• Fix compiler warnings by @b1az in #473
• Release 0.13.0 by @jbruggem in #468

New Contributors

• @Argonus made their first contribution in #437
• @kianmeng made their first contribution in #432
• @TC-aLi made their first contribution in #440
• @GDegrove made their first contribution in #450
• @EduardoGHdez made their first contribution in #443
• @alfetahe made their first contribution in #470
• @b1az made their first contribution in #473

Full Changelog: 0.12.1...0.13.0

0.12.1

0.12.1

Increases the version of ex_doc to allow publishing

Includes all the 0.12.0 changes.

0.12.0

NOTE: not released due to issues with ex_doc.

Breaking Changes

• Drop support for Elixir 1.5

Features

• Allow passthrough of ssl_options when starting a consumer or consumer group (#413)
• Allow GenConsumer callbacks to return :stop - allows graceful shutdown of consumers (#424)

Bugfixes

Misc

• Use Dynamic Supervisor rather than simple_one_for_one - fixes warnings on newer elixir versions (#418)
• Update to Kayrock 0.1.12 - fix compile warnings (#419)
• Tests pass the first time more often now (#420)
• Fix deprecation warning about Supervisor.terminate_child (#430)
• Remove Coveralls - was not being used, caused test failures (#423)

PRs Included

• #413
• #418
• #419
• #420
• #430
• #423
• #424

0.11.0

KafkaEx 0.11.0 is a large improvement to KafkaEx that sees the introduction of the Kayrock client, numerous stability fixes, and a critical fix that eliminates double-consuming messages when experiencing network failures under load.

Breaking changes

• Drop support for Elixir < 1.5
• Partitioner has been fixed to match the behavior of the Java client. This will cause key assignment to differ. To keep the old behavior, use the KafkaEx.LegacyPartitioner module. (#399)

Fixes

• Numerous fixes to error handling especially for networking connections (#347, #351 )
• Metadata is refreshed after topic create/delete(#349)
• Compression actually works for producing now 😬 (#362)
• Don't crash when throttled by quotas (#402)
• Default partitioner works the same way as the Java client now (#399)
• When fetching metadata for a subset of topics, don't remove the metadata for the unfetched topics. (#409)

Improvements

• Any GenServer can be used as a KafkaEx.GenConsumer (#339)
• Can specify wait time before attempting reconnect (#347)
• KafkaEx.stream/3 now uses the KafkaEx.Server interface, which inherits the sync_timeout setting. This avoids timeouts when sync_timeout needs to be greater than default. (#354)
• KafkaEx can now use Kayrock as the client implementation. This is a large change, and is pushing toward the improvements we want in 1.0 to allow the library to easily support new versions of the Kafka protocol. (#356, #359, #364, #366, #367, #369, #370, #374, #375, #377, #379, #387, #406, #408)
• KafkaEx.Protocol.Fetch.Message includes the topic and partition, allowing consumers to know which topic and partition they consumed from.
• Add KafkaEx.start_link_worker/1-2 to start a working and link it to the current process.
• Allow setting the client_id for the application - supports better monitoring, debugging, and quotas. (#388)
• Send metadata requests to a random broker rather than the same one each time (#395)
• Retry joining a consumer group 6 times rather than failing (#375, #403)

Kayrock client

See Kayrock and the Future of KafkaEx

Note that the Kayrock implementation doesn't support Kafka < 0.11

Improvements over default client:

• Can specify message API versions for the KafkaEx.stream/3 API
• Can specify message API versions for the consumer group API - NOTE that if you specify OffsetCommit API version >= 2, it will attempt to store the offsets in kafka, which will have no data about the topic unless you migrate the data. This could result in losing or reprocessing data. Migration can be done out of band, or can be done via the appropriate API calls within KafkaEx.
• Allow specifying OffsetCommit API version in KafkaEx.fetch

Misc

• Test suite uses KafkaEx 0.11 now in preparation for fully supporting Kafka API versions.
• KafkaEx.Protocol.Produce cleaned up (#380)
• Documentation improvements (#383, #384)
• Test against Elixir 1.9 (#394)
• Use OTP 22.3.3+ for OTP 22.2 testing to avoid SSL bug. (#405)

Thanks

Thanks to @gerbal, @ukrbublik, @jbruggem, @thulio, @dantswain, @thiamsantos, @kennethito, @habutre, @davidrusu, @getong, @shamilish, @adrienmo, @mjparrott, @richardlarocque for their contributions!

0.10.0

Features

• Allow passing in state to a KafkaEx.GenConsumer by defining a KafkaEx.GenConsumer.init/3 callback. Adds a default implementation of init/3 to ensure backward compatibility. -- @mtrudel
• Support DeleteTopics API for Kafka 0.10+ -- @jbruggem
• Add a default partitioner using murmur2 hashing when key is provided, or random partitioning otherwise. Use the KafkaEx.Partitioner behaviour to define a different partitioner, and enable it by adding it to the partitioner configuration setting.

Misc

• Thanks to @mjparrott , @clambodile, @soteras for documentation fixes
• Thanks to @gorenje for adding Elixir 1.8 testing and fixing 1.8 compile warnings

PRs Included:


#343
#338
#337
#335
#329
#333
#331

0.9.0

Features:

• CreateTopics API for Kafka 0.10.1+ https://issues.apache.org/jira/browse/KAFKA-2945 (@jbruggem)
• ApiVersions API for Kafka 0.10.0+ https://kafka.apache.org/protocol.html#api_versions (@jbruggem)
• Support callback function to set the broker configuration, allowing easier runtime broker discovery (@thecodeboss)
• Use the ApiVersions API to choose which API message version to send (@jbruggem)

Bug fixes:

• Send better logging messages on failure (@cdegroot)

Misc:

• Dialyzer & Credo cleanup/fixes
• Codebase formatted with mix format
• Documentation fixes (@IanVaughan , @jasongoodwin , @jhchabran , @amorphid , @skateinmars )
• CI tests now run Credo

PRs included:

#296
#297
#302
#303
#305
#312
#313
#316
#317
#319
#320
#323
#325
#327

Correctly handle sync_timeout

The sync_timeout setting should now be honored through GenServer calls.

Includes PR #196