errors.tolerance support?

[lavalex]

Hello,

Does the connector support ignoring errors or putting errors and responses to another topic?

[ivanyu]

Hi,
No, not at the moment

[CodeLikeAGecko]

My company is considering implementing support for errors.tolerance and ErrantRecordReporter. Is there a recommendation with respect to the http-connector batch mode? Using batch mode will be a lot of extra work to figure out which records worked and which had an issue and then likely need to report multiple Errant records. It might be simpler to remove batch support to support reporting Errant records. Any other suggestions on direction to implement would be helpful so changes can be posted back here as a PR. Thanks!

[CodeLikeAGecko]

@ivanyu Hi, I am wondering what your opinion is on the above? Is it okay to remove the batch mode logic when adding errors.tolerance and ErrantRecordReporter functionality?

[Oduig]

This would be useful functionality. We use this connector to replicate traffic from a production environment to a staging environment, forwarding the raw payload. Currently, any bad request to the main API causes the connector to stop.

[ivanyu]

Sorry for the very long delay!

Do I understand correctly that you think of integration like in this test, i.e. the connector will itself send messages to the DLQ?

[CodeLikeAGecko]

No worries @ivanyu.

Yes, the Sink will report the message with the ErrantRecordReporter if configured with errors.tolerance and DLQ handling, and the Connector will send the message to the DLQ. It would involve increasing the minimum supported Kafka version to 2.6.3, as ErrantRecordReporter showed up in Kafka 2.6.
I have already implemented this for my company's internal use and I can submit it as a PR if you would like. I did it in a way that preserved the current "batch send" concept. I felt that for errant record reporting we should only be sending one record at a time, so if batch send is being used then it won't use the errant record reporting. This should maintain backwards compatibility.

[ivanyu]

Sounds good to me, @CodeLikeAGecko, please submit the PR!

I think a good strategy would be to fail somewhere in the configuration checking with the error message if both DLQ and batching are configured.

[dbayonacode]

@lavalex
We could ignore errors and putting them and in a Dead Letter Queue Topic adding the following configuration properties to the sink connector configuration:

errors.tolerance = all
errors.deadletterqueue.topic.name =

Kafka Connect framework does the work.

[miguelaeh]

Hi!
Is there any update on this? #57 was merged but still errors.tolerance does not seem to work

[adaliszk]

Sorry for the ever-so-slight necro, but it seems that at least the error.tolerance = all does work. However, batched mode does not support it, which would be really handy to have.