Docker Compose #2
Comments
ptuckey
pushed a commit
to ptuckey/kbrowse
that referenced
this issue
Jul 25, 2018
Avro deserialization was resulting in an exception:
```
:cause Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:via
[{:type com.fasterxml.jackson.core.JsonGenerationException
:message Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:at [cheshire.generate$generate invokeStatic generate.clj 152]}]
```
This is handled in to-result-string, so we now use that instead of
invoking cheshire/generate-string directly.
----
Test Plan
As williamla points out in PandoraMedia#18, we don't currently have schema/avro testing.
That is in part because the fetch-kafka-tgz approach is retrieving
binaries that do not include Schema Registry support.
One option is to update to use a Docker approach with Schema support,
as outlined in PandoraMedia#2. Another would be to figure out how to use the
Confluent downloads in a scripted way (atm it requires an email,
so I have only tested manually).
In the absense of proper integration tests, I have tested this change
using the Confluent binaries:
```
./bin/kafka-topics --zookeeper localhost:2181 --create --topic
kbrowse-avro --partitions 10 --replication-factor 1
echo '"k"','{"label": "v"}' | ./bin/kafka-avro-console-producer
--broker-list localhost:9092 --topic kbrowse-avro --property
parse.key=true --property key.separator=, --property
key.schema='{"type": "string", "name": "label"}' --property
value.schema='{"type": "record", "name": "value", "fields": [{"type":
"string", "name": "label"}]}'
CONFIG=config/default-with-avro.yml ./lein run server
http://localhost:4000/?{%22key%22:%22%22,%22valRegex%22:%22%22,%22bootstrapServers%22:%22localhost:9092%22,%22topic%22:%22kbrowse-avro-1%22,%22relativeOffset%22:%22%22,%22follow%22:false,%22defaultPartition%22:false,%22valueDeserializer%22:%22io.confluent.kafka.serializers.KafkaAvroDeserializer%22,%22schemaRegistryURL%22:%22http://localhost:8081%22,%22partitions%22:%22%22}
```
ptuckey
pushed a commit
to ptuckey/kbrowse
that referenced
this issue
Jul 25, 2018
Avro deserialization was resulting in an exception:
```
:cause Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:via
[{:type com.fasterxml.jackson.core.JsonGenerationException
:message Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:at [cheshire.generate$generate invokeStatic generate.clj 152]}]
```
This is handled in to-result-string, so we now use that instead of
invoking cheshire/generate-string directly.
----
Test Plan
As williamla points out in PandoraMedia#18, we don't currently have schema/avro testing.
That is in part because the fetch-kafka-tgz approach is retrieving
binaries that do not include Schema Registry support.
One option is to update to use a Docker approach with Schema support,
as outlined in PandoraMedia#2. Another would be to figure out if the Confluent
downloads are intended to be used in a scripted way (atm it requires
an email, so I have only tested manually).
In the absense of proper integration tests, I have tested this change
using the Confluent binaries:
```
./bin/kafka-topics --zookeeper localhost:2181 --create --topic
kbrowse-avro --partitions 10 --replication-factor 1
echo '"k"','{"label": "v"}' | ./bin/kafka-avro-console-producer
--broker-list localhost:9092 --topic kbrowse-avro --property
parse.key=true --property key.separator=, --property
key.schema='{"type": "string", "name": "label"}' --property
value.schema='{"type": "record", "name": "value", "fields": [{"type":
"string", "name": "label"}]}'
CONFIG=config/default-with-avro.yml ./lein run server
http://localhost:4000/?{%22key%22:%22%22,%22valRegex%22:%22%22,%22bootstrapServers%22:%22localhost:9092%22,%22topic%22:%22kbrowse-avro-1%22,%22relativeOffset%22:%22%22,%22follow%22:false,%22defaultPartition%22:false,%22valueDeserializer%22:%22io.confluent.kafka.serializers.KafkaAvroDeserializer%22,%22schemaRegistryURL%22:%22http://localhost:8081%22,%22partitions%22:%22%22}
```
ptuckey
pushed a commit
that referenced
this issue
Jul 27, 2018
Avro deserialization was resulting in an exception:
```
:cause Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:via
[{:type com.fasterxml.jackson.core.JsonGenerationException
:message Cannot JSON encode object of class: class org.apache.avro.generic.GenericData$Record: {"label": "v"}
:at [cheshire.generate$generate invokeStatic generate.clj 152]}]
```
This is handled in to-result-string, so we now use that instead of
invoking cheshire/generate-string directly.
----
Test Plan
As williamla points out in #18, we don't currently have schema/avro testing.
That is in part because the fetch-kafka-tgz approach is retrieving
binaries that do not include Schema Registry support.
One option is to update to use a Docker approach with Schema support,
as outlined in #2. Another would be to figure out if the Confluent
downloads are intended to be used in a scripted way (atm it requires
an email, so I have only tested manually).
In the absense of proper integration tests, I have tested this change
using the Confluent binaries:
```
./bin/kafka-topics --zookeeper localhost:2181 --create --topic
kbrowse-avro --partitions 10 --replication-factor 1
echo '"k"','{"label": "v"}' | ./bin/kafka-avro-console-producer
--broker-list localhost:9092 --topic kbrowse-avro --property
parse.key=true --property key.separator=, --property
key.schema='{"type": "string", "name": "label"}' --property
value.schema='{"type": "record", "name": "value", "fields": [{"type":
"string", "name": "label"}]}'
CONFIG=config/default-with-avro.yml ./lein run server
http://localhost:4000/?{%22key%22:%22%22,%22valRegex%22:%22%22,%22bootstrapServers%22:%22localhost:9092%22,%22topic%22:%22kbrowse-avro-1%22,%22relativeOffset%22:%22%22,%22follow%22:false,%22defaultPartition%22:false,%22valueDeserializer%22:%22io.confluent.kafka.serializers.KafkaAvroDeserializer%22,%22schemaRegistryURL%22:%22http://localhost:8081%22,%22partitions%22:%22%22}
```
|
Got docker-compose working on my branch below, but not configured to run tests given it's using a dockerhub zk and kafka image. Remaining work would be to customize the docker image directly and rewrite the tests to suit this model. Leaving it as a branch for now, but sharing in case this might help if you or anyone else who might want to reference or complete it. https://github.com/williamla/kbrowse/tree/docker-compose |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The "run-zookeeper-and-kafka" script fetches binaries and runs them locally as child processes.
A solution using Docker Compose would be more robust.
The text was updated successfully, but these errors were encountered: