Question about Avro Schema

[JonasMueller1991]

Hey,
first of all: thanks for the great work!
We have some troubles with one of our Avro schemas. A minimal example of the Avro schema is attached. The import of the schema works fine, however, we get the following error message while running the test:

ERROR o.a.j.p.j.s.AbstractJavaSamplerClient: Failed to send message
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Record cannot be cast to class java.util.Collection (org.apache.avro.generic.GenericData$Record is in unnamed module of loader org.apache.jmeter.DynamicClassLoader @6833ce2c; java.util.Collection is in module java.base of loader 'bootstrap') in field dataItems in field data
at org.apache.avro.generic.GenericDatumWriter.addClassCastMsg(GenericDatumWriter.java:191) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:229) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:103) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:925) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at net.coru.kloadgen.sampler.KafkaSchemaSampler.runTest(KafkaSchemaSampler.java:153) [kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:638) [ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) [ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) [ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) [ApacheJMeter_core.jar:5.4.1]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Record cannot be cast to class java.util.Collection (org.apache.avro.generic.GenericData$Record is in unnamed module of loader org.apache.jmeter.DynamicClassLoader @6833ce2c; java.util.Collection is in module java.base of loader 'bootstrap')
at org.apache.avro.generic.GenericDatumWriter.getArraySize(GenericDatumWriter.java:281) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:251) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:137) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:103) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:925) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at net.coru.kloadgen.sampler.KafkaSchemaSampler.runTest(KafkaSchemaSampler.java:153) ~[kloadgen-3.6.0-kloadgen-plugin-3.6.0.jar:?]
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) ~[ApacheJMeter_java.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:638) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4.1]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4.1]
at java.lang.Thread.run(Thread.java:829) ~[?:?]

Would it possible to give us some guidance if the Avro schema is compatible with the current kloadgen plugin.

Thank you in advance!
Cheers,
Jonas

Avro Schema:

{ "name" : "TopicName", "type" : "record", "fields" : [{ "name" : "data" , "type" : { "name" : "Data", "type" : "record", "fields" : [{ "name" : "stringData", "type" : "string" },{ "name" :"dataItems", "type" : { "type" : "array", "items" : { "name" : "DataItems", "type" : "record", "fields" : [{ "name" : "dataItemName", "type" : "string" }] } } }] } }] }

[t-h-e]

Hi, this seems to be a bug. I have created a test case as well as a fix in the following commit: t-h-e@84f41c7
If one of the KLoadGen maintainers can confirm this is a bug, I will create a PR for this.

[kszosze]

Hi @t-h-e,
honestly yes, this is a bug. we detected it in the current development for Protobuf support and we were preparing a fix. But your solution looks good. If you open a PR we will happy to approve it. May be @JonasMueller1991 can test it as well and give a double verification?

Thanks for the amazing job.
Cheers!

[t-h-e]

I have created a PR: #149
There was a second bug regarding simple arrays in sub entities. See the second commit in the PR.

[Rivarsal]

Hello! @t-h-e Thanks for the solution, we actually developed a similar one while developing the protobuf support, so it´s cool to know that we shared the same logic to fix this problem!

Also, we have added better conditions in the SchemaProcessorLib in CheckArray(), CheckMap(), CheckArrayRecord(), in order to get the right fields at the right time. We will approve your changes for now since the logic is great and it works! But we will change it again when the protubuf support is added to master, since we need it to behave that way due to the nature of the protobuf´s schemas.

Thank you so much!!! It is really appreciated the support you gave us!

Cheers!