Scaling Streams Application

[bugrakosen]

Hello @LGouellec ,

I have a stream application that reads messages from the "orders" topic with 10 partitions, processes it with "FlatMapValues", then writes the outputs to another topic. When I open 1 more instance of the streams application without changing anything, the re-balancing process takes place without any problems and the tasks are shared. However, when I produce new messages to the topic, the first streams application I run gives the following error, sometimes processing some records and sometimes not processing records at all (which probably depends on the offset interval). How can I resolve this error. Partition-based parallel processing is very important to me, so I have to solve this problem.

 fail: Streamiz.Kafka.Net.Processors.StreamThread[0]
  stream-thread[stream_app-5761c47c-9b02-4c34-abfd-c8e3982b62f8-stream-thread-0] Failed to close stream thread due to the following error:
  Confluent.Kafka.KafkaTxnRequiresAbortException: Failed to commit offsets to transaction on broker localhost:9092/1: orders[5](Broker: Specified group generation id is not valid) (after 0ms)
       at Confluent.Kafka.Impl.SafeKafkaHandle.SendOffsetsToTransaction(IEnumerable1 offsets, IConsumerGroupMetadata groupMetadata, Int32 millisecondsTimeout)
       at Confluent.Kafka.Producer2.SendOffsetsToTransaction(IEnumerable1 offsets, IConsumerGroupMetadata groupMetadata, TimeSpan timeout)
       at Streamiz.Kafka.Net.Processors.StreamTask.Commit(Boolean startNewTransaction)
       at Streamiz.Kafka.Net.Processors.StreamTask.Suspend()
       at Streamiz.Kafka.Net.Processors.StreamTask.Close()
       at Streamiz.Kafka.Net.Processors.Internal.TaskManager.Close()
       at Streamiz.Kafka.Net.Processors.StreamThread.CompleteShutdown()

[LGouellec]

Hi @bugrakosen,

Thanks for your question, could you forward the complete debug logs when the issue appears ?

Thanks,

[bugrakosen]

I solved the problem, I don't need help anymore, thanks. @LGouellec