Specmatic Generated Value Exceeds Defined Maximum for Query Parameter in OpenAPI Specification #1146

premsvmm · 2024-06-17T04:58:06Z

premsvmm
Jun 17, 2024

Open API Spec

info:
  title: Order Fetch API
  description: API for retrieving  orders
  version: 1.0.0
servers:
  - url: https://localhost:90
paths:
  /v1/orders:
    get:
      summary: Get all orders
      operationId: getOrders
      security:
        - basicAuth: []
      parameters:
        - name: count
          in: query
          required: false
          schema:
            type: integer
            maximum: 100
            minimum: 1
            example: 2
          description: Number of orders to retrieve
        - name: skip
          in: query
          required: false
          schema:
            type: integer
            maximum: 100
            minimum: 1
            example: 1
          description: Number of orders to skip
      responses:
        '200':
          description: A list of orders
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/OrdersResponse'
              examples:
                SUCCESS:
                  value:
                    entity: collection
                    count: 10
                    items:
                      - id: order_OLiLVkokheKr4c
                        entity: order
                        amount: 646
                        amount_paid: null
                        amount_due: 646
                        currency: USD
                        receipt: ""
                        offer_id: null
                        status: created
                        attempts: 0
                        notes: []
                        created_at: 1718165792
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
              examples:
                COUNT_EXCEEDED:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The count may not be greater than 100.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: count
                INVALID_SKIP:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The skip must be an integer.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: skip
                INVALID_COUNT:
                  value:
                    error:
                      code: BAD_REQUEST_ERROR
                      description: The count must be an integer.
                      source: business
                      step: payment_initiation
                      reason: input_validation_failed
                      metadata: {}
                      field: count
components:
  schemas:
    OrdersResponse:
      type: object
      properties:
        entity:
          type: string
          example: collection
        count:
          type: integer
          example: 10
          maximum: 100
          minimum: 1
        items:
          type: array
          items:
            type: object
            properties:
              id:
                type: string
                example: order_OLiLVkokheKr4c
                pattern: '^order_[A-Za-z0-9]{14}$'
              entity:
                type: string
                example: order
              amount:
                type: integer
                example: 646
              amount_paid:
                type: integer
                nullable: true
                example: null
              amount_due:
                type: integer
                example: 646
              currency:
                type: string
                example: USD
              receipt:
                type: string
                nullable: true
                example: ""
              offer_id:
                type: string
                nullable: true
                example: null
              status:
                type: string
                example: created
              attempts:
                type: integer
                example: 0
              notes:
                oneOf:
                  - type: object
                    additionalProperties:
                      type: string
                  - type: array
                    items:
                      type: object
              created_at:
                type: integer
                example: 1718165792
    ErrorResponse:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              example: BAD_REQUEST_ERROR
            description:
              type: string
              example: The count may not be greater than 100.
            source:
              type: string
              example: business
            step:
              type: string
              example: payment_initiation
            reason:
              type: string
              example: input_validation_failed
            metadata:
              type: object
              additionalProperties:
                type: string
            field:
              type: string
              example: count
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
security:
  - basicAuth: []

Although the count query parameter has a defined minimum of 1 and a maximum of 100 in the OpenAPI specification, Specmatic generated a value greater than 100 and expected a 200 status code response for this invalid input.

    Request to https://localhost at 2024-6-17 10:1:55.399
    GET /v1/orders?count=347
    Authorization: Basic <AUTH>
    Host: <HOST>
    Accept-Charset: UTF-8
    Accept: */*
    Content-Type: NOT SENT

How can this be resolved or what are the possible workarounds?

Answered by harikrishnan83

Jun 25, 2024

@premsvmm I tested with your API specification above with Specmatic 1.3.29 and it is generating values within the set minimum and maximum values for your amount field. Can you please check again with the latest version of Specmatic?

Also, I see that your name examples are not co-ordinated between request and response. In request section you have two examples example and [WIP] SUCCESS, however in responses you have examples_1 and examples_2. So your examples were not getting picked up as test data. Since OpenAPI inherently does not have a way to associate request and response examples, in Specmatic we leverage example names as a way to associate request and responses.

Once I fixed this (re…

View full answer

yogeshnikam671 · 2024-06-17T06:43:23Z

yogeshnikam671
Jun 17, 2024
Maintainer

Hi @premsvmm, thanks for reporting this issue.
We will have a look at it and get back.

0 replies

yogeshnikam671 · 2024-06-17T06:54:25Z

yogeshnikam671
Jun 17, 2024
Maintainer

Hey @premsvmm, this has been taken care of in the latest version of Specmatic [1.3.26].
Let us know if you face this issue even with this version.
Thanks!

0 replies

premsvmm · 2024-06-17T10:51:58Z

premsvmm
Jun 17, 2024
Author

Hi @yogeshnikam671 , Thank you for informing me that the reported issue has been resolved in the latest version. However, I'm encountering a problem with version 1.3.26 when running the tests. It fails to load the specmatic.json config file, which wasn't an issue in version 1.3.24.

Here are the logs to help diagnose the issue:

specmatic.json

  "sources": [
    {
      "provider": "filesystem",
      "test": [
        "./api-spec/order_get/order_get.yaml"
      ],
      "stub": [
        "./api-spec/upi-switch/verification_token.yaml",
        "./api-spec/order_get/order_get.yaml",
        "./api-spec/order/order.yaml"
      ]
    }
  ],
  "report": {
    "formatters": [
      {
        "type": "text",
        "layout": "table"
      }
    ],
    "types": {
      "APICoverage": {
        "OpenAPI": {
          "successCriteria": {
            "minThresholdPercentage": 20,
            "maxMissedEndpointsInSpec": 10,
            "enforce": false
          },
          "excludedEndpoints": []
        }
      }
    }
  }
}

While running the contract test:

'in.specmatic.core.SpecmaticConfigJson in.specmatic.core.SpecmaticConfigKt.loadSpecmaticJsonConfig(java.lang.String)'

2 replies

yogeshnikam671 Jun 17, 2024
Maintainer

Hey @premsvmm, can you try refreshing your dependencies and running these tests?
If you encounter the same problem, please provide a more detailed log.
You can replace any sensitive info within the logs with some dummy values, if any.

Thanks again.

premsvmm Jun 17, 2024
Author

@yogeshnikam671, the issue has been resolved. The problem was due to not updating the provider dependency to the latest version. Updating junit5-support to the latest version fixed it.

yogeshnikam671 · 2024-06-17T11:11:21Z

yogeshnikam671
Jun 17, 2024
Maintainer

Hi @premsvmm, thanks for reporting this.
We will have a look at this and get back

0 replies

premsvmm · 2024-06-24T06:23:48Z

premsvmm
Jun 24, 2024
Author

@yogeshnikam671

openapi: 3.0.1
info:
  title: Order API
  description: API for creating orders
  version: 1.0.0
servers:
  - url: https://localhost
components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
  schemas:
    OrderRequest:
      type: object
      required:
        - amount
        - currency
      properties:
        amount:
          type: integer
          minimum: 100
          maximum: 100000
          example: 100
          description: Amount in the smallest currency unit (e.g., paise for INR)
        currency:
          type: string
          description: Currency code
          enum: [INR, USD]
        receipt:
          type: string
          nullable: true
          description: Receipt identifier
        notes:
          type: object
          additionalProperties:
            type: string
          description: Additional notes
    OrderResponse:
      type: object
      properties:
        id:
          type: string
        amount:
          type: integer
          minimum: 100
          maximum: 100000
        currency:
          type: string
        receipt:
          type: string
          nullable: true
        status:
          type: string
        notes:
          oneOf:
            - type: object
              additionalProperties:
                type: string
            - type: array
              items:
                type: object
        amount_due:
          type: integer
        amount_paid:
          type: integer
        attempts:
          type: integer
        created_at:
          type: integer
        entity:
          type: string
        offer_id:
          type: string
          nullable: true
    ErrorResponse:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
            description:
              type: string
            field:
              type: string
              nullable: true
            source:
              type: string
            step:
              type: string
            reason:
              type: string
            metadata:
              type: object
              additionalProperties: true
security:
  - basicAuth: []

paths:
  /v1/orders:
    post:
      summary: Create an Order
      operationId: createOrder
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/OrderRequest'
            examples:
              example:
                summary: Success Request
                value:
                  amount: 1000000
                  currency: "INR"
                  receipt: "Receipt no. 1"
                  notes:
                    notes_key_1: "Tea, Earl Grey, Hot"
                    notes_key_2: "Tea, Earl Grey… decaf."
              "[WIP] SUCCESS":
                summary: Success Request
                value:
                  amount: 10000000000
                  currency: "INR"
                  receipt: "Receipt no. 1"
                  notes:
                    notes_key_1: "Tea, Earl Grey, Hot"
                    notes_key_2: "Tea, Earl Grey… decaf."
      responses:
        '200':
          description: Order created successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/OrderResponse'
              examples:
                example_1:
                  summary: Successful response with notes as object
                  value:
                    amount: 1000000
                    amount_due: 1000000
                    amount_paid: 0
                    attempts: 0
                    created_at: 1717991270
                    currency: "INR"
                    entity: "order"
                    id: "order_OKumxaVkPEnex7"
                    notes:
                      notes_key_1: "Tea, Earl Grey, Hot"
                      notes_key_2: "Tea, Earl Grey… decaf."
                    offer_id: null
                    receipt: "Receipt no. 1"
                    status: "created"
                example_2:
                  summary: Successful response with notes as array
                  value:
                    amount: 1000000
                    amount_due: 1000000
                    amount_paid: 0
                    attempts: 0
                    created_at: 1717991270
                    currency: "INR"
                    entity: "order"
                    id: "order_OKumxaVkPEnex7"
                    notes: []
                    offer_id: null
                    receipt: "Receipt no. 1"
                    status: "created"
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

--------------------
  Request to https://localgost at 2024-6-24 11:48:53.486
    POST /v1/orders
    Authorization: <Auth>
    Host: sample.com
    Accept-Charset: UTF-8
    Accept: */*
    Content-Type: application/json
    
    {
        "amount": 2,
        "currency": "INR"
    }

With the latest version 1.3.26 , the minimum and maximum constraints are not being enforced in the properties field:

properties:
  amount:
    type: integer
    minimum: 100
    maximum: 100000
    example: 100
    description: Amount in the smallest currency unit (e.g., paise for INR)

Even though the min and max values are set, the test payload is generating values less than the specified minimum.

0 replies

harikrishnan83 · 2024-06-25T13:27:38Z

harikrishnan83
Jun 25, 2024
Maintainer

@premsvmm I tested with your API specification above with Specmatic 1.3.29 and it is generating values within the set minimum and maximum values for your amount field. Can you please check again with the latest version of Specmatic?

Also, I see that your name examples are not co-ordinated between request and response. In request section you have two examples example and [WIP] SUCCESS, however in responses you have examples_1 and examples_2. So your examples were not getting picked up as test data. Since OpenAPI inherently does not have a way to associate request and response examples, in Specmatic we leverage example names as a way to associate request and responses.

Once I fixed this (renamed example_1 to example and example_2 to [WIP] SUCCESS), Specmatic also highlighted that this inline example is not as per your contract.

Unsuccessful Scenarios:
  "Load Error FAILED"
        Reason: Error loading test data for Scenario: POST /v1/orders -> 200
           
           >> REQUEST.BODY.amount
           
              Expected number <= 100000 as per the specification, but the example example_1 had 1000000 (number)

Then once I fixed this, the tests ran with the values in the examples. Here is the full updated spec.

openapi: 3.0.1
info:
  title: Order API
  description: API for creating orders
  version: 1.0.0
servers:
  - url: https://localhost
components:
  securitySchemes:
    basicAuth:
      type: http
      scheme: basic
  schemas:
    OrderRequest:
      type: object
      required:
        - amount
        - currency
      properties:
        amount:
          type: integer
          minimum: 100
          maximum: 100000
          example: 100
          description: Amount in the smallest currency unit (e.g., paise for INR)
        currency:
          type: string
          description: Currency code
          enum: [INR, USD]
        receipt:
          type: string
          nullable: true
          description: Receipt identifier
        notes:
          type: object
          additionalProperties:
            type: string
          description: Additional notes
    OrderResponse:
      type: object
      properties:
        id:
          type: string
        amount:
          type: integer
          minimum: 100
          maximum: 100000
        currency:
          type: string
        receipt:
          type: string
          nullable: true
        status:
          type: string
        notes:
          oneOf:
            - type: object
              additionalProperties:
                type: string
            - type: array
              items:
                type: object
        amount_due:
          type: integer
        amount_paid:
          type: integer
        attempts:
          type: integer
        created_at:
          type: integer
        entity:
          type: string
        offer_id:
          type: string
          nullable: true
    ErrorResponse:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
            description:
              type: string
            field:
              type: string
              nullable: true
            source:
              type: string
            step:
              type: string
            reason:
              type: string
            metadata:
              type: object
              additionalProperties: true
security:
  - basicAuth: []

paths:
  /v1/orders:
    post:
      summary: Create an Order
      operationId: createOrder
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/OrderRequest'
            examples:
              example:
                summary: Success Request
                value:
                  amount: 10000
                  currency: "INR"
                  receipt: "Receipt no. 1"
                  notes:
                    notes_key_1: "Tea, Earl Grey, Hot"
                    notes_key_2: "Tea, Earl Grey… decaf."
              "[WIP] SUCCESS":
                summary: Success Request
                value:
                  amount: 100000
                  currency: "INR"
                  receipt: "Receipt no. 1"
                  notes:
                    notes_key_1: "Tea, Earl Grey, Hot"
                    notes_key_2: "Tea, Earl Grey… decaf."
      responses:
        '200':
          description: Order created successfully
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/OrderResponse'
              examples:
                example:
                  summary: Successful response with notes as object
                  value:
                    amount: 10000
                    amount_due: 1000000
                    amount_paid: 0
                    attempts: 0
                    created_at: 1717991270
                    currency: "INR"
                    entity: "order"
                    id: "order_OKumxaVkPEnex7"
                    notes:
                      notes_key_1: "Tea, Earl Grey, Hot"
                      notes_key_2: "Tea, Earl Grey… decaf."
                    offer_id: null
                    receipt: "Receipt no. 1"
                    status: "created"
                "[WIP] SUCCESS":
                  summary: Successful response with notes as array
                  value:
                    amount: 10000
                    amount_due: 1000000
                    amount_paid: 0
                    attempts: 0
                    created_at: 1717991270
                    currency: "INR"
                    entity: "order"
                    id: "order_OKumxaVkPEnex7"
                    notes: []
                    offer_id: null
                    receipt: "Receipt no. 1"
                    status: "created"
        '400':
          description: Bad Request
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'

Here are the logs from my testing.

% specmatic test test.yaml
API Specification Summary: test.yaml
  OpenAPI Version: 3.0.1
  API Paths: 1, API Operations: 1
  Schema components: 3, Security Schemes: [http]

Endpoints API not found, cannot calculate actual coverage


--------------------
  Request to http://localhost:9000 at 2024-6-25 7:7:50.841
    POST /v1/orders
    Authorization: Basic UEpPUVI6V0NYQ1Q=
    Accept-Charset: UTF-8
    Accept: */*
    Content-Type: application/json
    
    {
        "amount": 10000,
        "currency": "INR",
        "receipt": "Receipt no. 1",
        "notes": {
            "notes_key_1": "Tea, Earl Grey, Hot",
            "notes_key_2": "Tea, Earl Grey… decaf."
        }
    }

  Response at 2024-6-25 7:7:50.842
    200 OK
    Vary: Origin
    X-Specmatic-Result: success
    Content-Length: 398
    Content-Type: application/json
    Connection: keep-alive
    
    {
        "amount": 10000,
        "amount_due": 1000000,
        "amount_paid": 0,
        "attempts": 0,
        "created_at": 1717991270,
        "currency": "INR",
        "entity": "order",
        "id": "order_OKumxaVkPEnex7",
        "notes": {
            "notes_key_1": "Tea, Earl Grey, Hot",
            "notes_key_2": "Tea, Earl Grey… decaf."
        },
        "offer_id": null,
        "receipt": "Receipt no. 1",
        "status": "created"
    }

 Scenario: POST /v1/orders -> 200 | EX:example has SUCCEEDED


--------------------
  Request to http://localhost:9000 at 2024-6-25 7:7:50.864
    POST /v1/orders
    Authorization: Basic RkVEWEU6QUlHSEU=
    Accept-Charset: UTF-8
    Accept: */*
    Content-Type: application/json
    
    {
        "amount": 100000,
        "currency": "INR",
        "receipt": "Receipt no. 1",
        "notes": {
            "notes_key_1": "Tea, Earl Grey, Hot",
            "notes_key_2": "Tea, Earl Grey… decaf."
        }
    }

  Response at 2024-6-25 7:7:50.864
    200 OK
    Vary: Origin
    X-Specmatic-Result: success
    Content-Length: 297
    Content-Type: application/json
    Connection: keep-alive
    
    {
        "amount": 10000,
        "amount_due": 1000000,
        "amount_paid": 0,
        "attempts": 0,
        "created_at": 1717991270,
        "currency": "INR",
        "entity": "order",
        "id": "order_OKumxaVkPEnex7",
        "notes": [],
        "offer_id": null,
        "receipt": "Receipt no. 1",
        "status": "created"
    }

 Scenario: POST /v1/orders -> 200 | EX:[WIP] SUCCESS has SUCCEEDED

Could not load report configuration, coverage will be calculated but no coverage threshold will be enforced
|-----------------------------------------------------------------------|
| API COVERAGE SUMMARY                                                  |
|-----------------------------------------------------------------------|
| coverage | path       | method | response | # exercised | remarks     |
|----------|------------|--------|----------|-------------|-------------|
|      33% | /v1/orders | POST   |      200 |           2 | covered     |
|          |            |        |      400 |           0 | did not run |
|          |            |        |      401 |           0 | did not run |
|-----------------------------------------------------------------------|
| 33% API Coverage reported from 1 path                                 |
|-----------------------------------------------------------------------|


Saving Open API Coverage Report json to ./build/reports/specmatic ...


Tests run: 2, Successes: 2, Failures: 0, Errors: 0

Please feel free to reach out to us if you have any questions on this.

0 replies

premsvmm · 2024-06-26T09:00:01Z

premsvmm
Jun 26, 2024
Author

Thanks @harikrishnan83 for the detailed explanation. The named example changes made it work perfectly.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Specmatic Generated Value Exceeds Defined Maximum for Query Parameter in OpenAPI Specification #1146

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 7 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Specmatic Generated Value Exceeds Defined Maximum for Query Parameter in OpenAPI Specification #1146

premsvmm Jun 17, 2024

Replies: 7 comments · 2 replies

yogeshnikam671 Jun 17, 2024 Maintainer

yogeshnikam671 Jun 17, 2024 Maintainer

premsvmm Jun 17, 2024 Author

yogeshnikam671 Jun 17, 2024 Maintainer

premsvmm Jun 17, 2024 Author

yogeshnikam671 Jun 17, 2024 Maintainer

premsvmm Jun 24, 2024 Author

harikrishnan83 Jun 25, 2024 Maintainer

premsvmm Jun 26, 2024 Author

premsvmm
Jun 17, 2024

Replies: 7 comments 2 replies

yogeshnikam671
Jun 17, 2024
Maintainer

yogeshnikam671
Jun 17, 2024
Maintainer

premsvmm
Jun 17, 2024
Author

yogeshnikam671 Jun 17, 2024
Maintainer

premsvmm Jun 17, 2024
Author

yogeshnikam671
Jun 17, 2024
Maintainer

premsvmm
Jun 24, 2024
Author

harikrishnan83
Jun 25, 2024
Maintainer

premsvmm
Jun 26, 2024
Author