How do I go about adding provider/producer side tests for asyncapi spec where events are sent to Kafka?

[tomknee-accelins]

I am a bit confused here, I see docs for Kafka stub server but that seems like it should be used more on consumer side? Any leads to look at here would be appreciated. Thanks.

My setup is we have APIs defined in OpenAPI spec and one of those is a KafkaService which then produces events, with another repo consuming those events/messages. I have made an AsyncAPI spec which is in a central contract repo.

Many thanks.

[harikrishnan83]

Hello @tomknee-accelins,

Thanks for reaching out and sharing the details.

Specmatic Kafka support can be used for both mocking as well as testing.

1. Sample project using Specmatic Kafka Mock Server
2. Sample project using Specmatic Kafka Contract Test to verify a pattern on the lines request-reply / single request multiply reply etc.

The system under test you have described (a service exposing HTTP API to its consumers which produces events as a result of HTTP interactions) is very similar to the sample project number 1 above.

The BFF service exposes an OpenAPI HTTP API to its consumers and upon receiving an HTTP request it produces a Kafka message (details of which are in an AsyncAPI spec similar to your setup). The Kafka mock server in this case is used to ascertain that the BFF produced the right message on the right topic and more. Hope this clarifies.

The next step here will be to test the application that is consuming the messages in Kafka that the BFF layer produced. However, how we test it depends on the architecture of that application. For example: Does it store the details from the message into a DB, write it to a file, send any further messages etc. Event Driven Architecture has several interaction patterns (fire and forget, request reply, fan out and more) in comparison to sync (which is largely request response). It will be great if you can elaborate on what aspect you are trying to test and then I suggest the approach accordingly.

About the Docker image documentation, we are in the process of adding details about running contract tests (which is already available in programmatic setup) from CLI. Apologies for any confusion caused by the Docker image documentation. It will be fixed shortly.

Please let me know if you have any questions on this.

[tomknee-accelins]

The BFF service exposes an OpenAPI HTTP API to its consumers and upon receiving an HTTP request it produces a Kafka message (details of which are in an AsyncAPI spec similar to your setup). The Kafka mock server in this case is used to ascertain that the BFF produced the right message on the right topic and more. Hope this clarifies.

Thank you, this makes sense, I will focus on this part and then come back to discuss the other side, cheers.

[tomknee-accelins]

Hi, so our BFF is in Node.JS, so cannot use JVM here, any docs on running the tests from CLI ? Thanks

EDIT: I also emailed you last week with what I originally tried, but now realise that is probably wrong.

[tomknee-accelins]

Also do you support swagger.json (v2), as I am using that and getting some errors like

Load Error has FAILED
Reason: >> region
           
              In POST /api/organizations request
              Enum values must contain null since the schema  is nullable


Could not load report configuration, coverage will be calculated but no coverage threshold will be enforced
The Open API coverage report generated is blank.
This can happen if you have included all the endpoints in the 'excludedEndpoints' array in the report section in specmatic.json, or if your open api specification does not have any paths documented.

[harikrishnan83]

Hi, so our BFF is in Node.JS, so cannot use JVM here, any docs on running the tests from CLI ? Thanks

EDIT: I also emailed you last week with what I originally tried, but now realise that is probably wrong.

Hello @tomknee-accelins, Specmatic has a npm wrapper for JS. This article goes over the same BFF application built with node and express. Hope this helps. Thanks.

[tomknee-accelins]

Hi Hari, I have made decent progress now, but when running java -jar specmatic.jar test --config=./specmatic.yaml --testBaseURL=http://localhost:3001 I now get 429 Too Many Requests. Any ideas how to resolve please? I assume we have some sort of rate limiting in place.

Maybe I NEED to write tests programatiically, and/or use specific json test fixtures?

[harikrishnan83]

hello @tomknee-accelins, good to know you are able to make progress.

As per my understanding you are running the OpenAPI spec as a test on your service. And looks like your service is returning 429. My guess is your OpenAPI spec does not have any inline co-ordinated examples.

And without examples, Specmatic will try to generate all possible test cases based on the request schema. If your API spec has a lot of optional and nullable fields, this can lead to several tests getting generated to verify various valid combinations. Looks like your application is not able to handle this many requests on your machine.

My suggestion will be to add a few co-ordinated examples and try it out. If you have any difficulty adding examples to the OpenAPI spec itself, you can keep the examples externalised. We even have an utility command called java -jar specmatic.jar examples <spec file> which will generate template files where you can fill in values as per your convenience.

Hope this helps.

Thanks.

UPDATE: Fixing typos

[tomknee-accelins]

Thanks this helps, we do have examples but it is a swagger.json file using "openapi": "3.0.0",

I will look at using externalised examples. Cheers.

[tomknee-accelins]

If running all tests via CLI then I believe there will be LOADS of examples I will need to externalise.

Is my understanding correct that I can instead (for this proof-of-concept), programatically add a smaller handful of tests with matching externalised examples?

Thanks.

[harikrishnan83]

No you will not need "LOADS of examples I will need to externalise". Specmatic "Generative Tests / API Resiliency Tests" will create several requests based on a handful of examples that you have added.

I did not understand what you mean by "programatically add a smaller handful of tests", can you please elaborate?

[tomknee-accelins]

Hi @harikrishnan83, I am getting closer, but still have a couple questions please.

I have a RESTful service where I make a submission via a POST request to a specific endpoint (e.g. /api/submit).

In order to succeed in the above call, I first need to get a bearer token by sending a POST request to the endpoint '/api/auth/login'.

Once I succeed in making a submission to '/api/submit', another service (/api/scan-results) then needs to get called downstream (manually though, unless i code something in for this contract test specifically).

When this downstream service is called, an event listener picks this up and sends a message to a kafka topic.

Because of these downstream services, how do I handle this with Specmatic, as the examples provided only show the one provider service which instantly sends a message to kafka, rather than another downstream service? Also what about the initial auth call?

My current assumption would be that I need to get the API app set up first so that I then just need to do a contract test calling api/scan-results which then produces the message on the kafka topic? I feel like this is the last piece of the puzzle for me, appreciate any help here.

Thanks 🙏

[harikrishnan83]

hello @tomknee-accelins, Contract Tests require isolation and thereby authentication and authorisation are usually considered orthogonal concerns which we do not need to exercise during this type of testing. We certainly need to validate auth, however that comes later, in workflow tests, etc.

Here is detailed documentation on this and also a sample project that demonstrates how to bypass auth during Contract Testing. While this sample project is written in Java and Sprintboot similar concepts are applicable across stacks such as Node also.

Secondly with the auth related workflow and how the downstream system is called seems to again represent several components at play here. The ideal way to deal with this is to break it down into component level contract tests and isolate dependencies like we did with the BFF layer.

Thanks again for all the great questions. It will be great if you could raise each of these questions as a separate discussion so that it is helpful for other people also.