Specmatic Expects 4xx Response for Valid Request

[premsvmm]

openapi: 3.0.1
info:
  title: Order Fetch API
  description: API for retrieving orders
  version: 1.0.0
servers:
  - url: https://localhost:90
paths:
  /v1/orders:
    get:
      summary: Get all orders
      operationId: getOrders
      security:
        - basicAuth: []
      parameters:
        - name: count
          in: query
          required: false
          schema:
            type: integer
            maximum: 100
            minimum: 1
            example: 2
          description: Number of orders to retrieve
        - name: skip
          in: query
          required: false
          schema:
            type: integer
            maximum: 100
            minimum: 1
            example: 1
          description: Number of orders to skip
      responses:
        '200':
          description: A list of orders
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/OrdersResponse'
              examples:
                SUCCESS:
                  value:
                    entity: collection
                    count: 10
                    items:
                      - id: order_OLiLVkokheKr4c
                        entity: order
                        amount: 646
                        amount_paid: null
                        amount_due: 646
                        currency: USD
                        receipt: ""
                        offer_id: null
                        status: created
                        attempts: 0
                        notes: []
                        created_at: 1718165792
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                COUNT_EXCEEDED:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The count may not be greater than 100.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: count
                INVALID_SKIP:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The skip must be an integer.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: skip
                INVALID_COUNT:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The count must be an integer.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: count
      examples:
        example-1:
          summary: Example request
          description: This is an example of a request to fetch orders with specific count and skip parameters.
          value:
            count: 2
            skip: 1
components:
  schemas:
    OrdersResponse:
      type: object
      properties:
        entity:
          type: string
          example: collection
        count:
          type: integer
          example: 10
          maximum: 100
          minimum: 1
        items:
          type: array
          items:
            type: object
            properties:
              id:
                type: string
                example: order_OLiLVkokheKr4c
                pattern: '^order_[A-Za-z0-9]{14}$'
              entity:
                type: string
                example: order
              amount:
                type: integer
                example: 646
              amount_paid:
                type: integer
                nullable: true
                example: null
              amount_due:
                type: integer
                example: 646
              currency:
                type: string
                example: USD
              receipt:
                type: string
                nullable: true
                example: ""
              offer_id:
                type: string
                nullable: true
                example: null
              status:
                type: string
                example: created
              attempts:
                type: integer
                example: 0
              notes:
                oneOf:
                  - type: object
                    additionalProperties:
                      type: string
                  - type: array
                    items:
                      type: object
              created_at:
                type: integer
                example: 1718165792
    ErrorResponse:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              example: BAD_REQUEST_ERROR
            description:
              type: string
              example: The count may not be greater than 100.
            source:
              type: string
              example: business
            step:
              type: string
              example: payment_initiation
            reason:
              type: string
              example: input_validation_failed
            metadata:
              type: object
              additionalProperties:
                type: string
            field:
              type: string
              example: count
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
security:
  - basicAuth: []

--------------------
  Request to https://localhost at 2024-6-24 11:54:34.90
    GET /v1/orders
    Authorization: <Auth>
    Host: localhost
    Accept-Charset: UTF-8
    Accept: */*
    Content-Type: NOT SENT
    
    

  Response at 2024-6-24 11:54:34.93
    200 OK
    Date: Mon, 24 Jun 2024 06:24:34 GMT
    Content-Type: application/json
    Transfer-Encoding: chunked
    Connection: keep-alive
    cache-control: max-age=0, must-revalidate, no-store, nocache, private
    expires: Fri, 01 Jan 1990 00:00:00 GMT
    pragma: no-cache
    strict-transport-security: max-age=315360000; includeSubDomains
    via: rws
    x-frame-options: SAMEORIGIN
    x-pam: 0
    x-xss-protection: 0
    
    {
        // Response
    }


java.lang.AssertionError: Testing scenario "Get all orders. Response: Bad Request"
API: GET /v1/orders -> 400

  >> RESPONSE.STATUS
  
     Expected status 400, actual was status 200

  at in.specmatic.test.SpecmaticJUnitSupport$dynamicTestStream$1.invoke$lambda$1(SpecmaticJUnitSupport.kt:310)
  at java.base/java.util.Optional.ifPresent(Optional.java:178)
  at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

The test generation by Specmatic sent a request expecting a 4XX response, but the request payload appears to be valid. How can we control the test data in Specmatic?

Additionally, Specmatic does not indicate what changes were made to the payload or why it expects a 4XX response for this request.

[yogeshnikam671]

Hi @premsvmm, thanks for reporting this issue.
We will get back to you soon on this.

[harikrishnan83]

@premsvmm Thanks again for the details.

In your spec the examples COUNT_EXCEEDED, INVALID_SKIP and INVALID_COUNT are only in the responses section. However I do not see the corresponding named examples under the request section. Did you intend to setup the corresponding named example under the request section also as per your test data so that your application responds with appropriate error status?

[premsvmm]

Thanks @harikrishnan83. After making the named example changes, it works as expected.