MP70: Implement MicroProfile OpenAPI 4.0

[Azquelt]

Description

Add support for MicroProfile OpenAPI 4.0.

This is likely to include:

• Support for the OpenAPI 3.1.0 standard
  • The largest change is to use JSON schema directly for the schemas
• Miscellaneous small improvements

See the specification milestone for the full list of features included in this release of the specification.

In addition, we plan to add the following features to OpenLiberty beyond what is required by the spec:

• Continue to support OpenAPI 3.0 output format
• Continue to do validation checks on the output document
• Produced merged OpenAPI documentation for all deployed applications by default
• Server.xml config to control which applications are included in the OpenAPI documentation
  • backported to mpOpenAPI-2.0 and later
• Server.xml config to set the info section of the output OpenAPI document
  • backported to mpOpenAPI-2.0 and later

---

Documents

When available, add links to required feature documents. Use "N/A" to mark particular documents which are not required by the feature.

• UFO: https://ibm.box.com/s/f5qzwtyvqiz7hrlrsahyt31keyf4matk
• FTS: Feature Test Summary mpOpenAPI-4.0 #29962
• Beta Blog: BETA BLOG - MicroProfile OpenAPI 4.0 #29097
• GA Blog: GA BLOG - MicroProfile OpenAPI 4.0 #30139
• Docs issue: Documentation: MicroProfile OpenAPI 4.0 docs#7654

---

Process Overview

• Prioritization
• Design
• Implementation
• Legal and Translation
• Beta
• GA
  • Focal Point Approvals
• Other Deliverables

General Instructions

The process steps occur roughly in the order as presented. Process steps occasionally overlap.

Each process step has a number of tasks which must be completed or must be marked as not applicable ("N/A").

Unless otherwise indicated, the tasks are the responsibility of the Feature Owner or a Delegate of the Feature Owner.

If you need assistance, reach out to the OpenLiberty/release-architect.

Important: Labels are used to trigger particular steps and must be added as indicated.

---

Prioritization (Complete Before Development Starts)

The (OpenLiberty/chief-architect) and area leads are responsible for prioritizing the features and determining which features are being actively worked on.

Prioritization

• Feature added to the "New" column of the Open Liberty project board
  • Epics can be added to the board in one of two ways:
    • From this issue, use the "Projects" section to select the appropriate project board.
    • From the appropriate project board click "Add card" and select your Feature Epic issue
• Priority assigned
  • Attend the Liberty Backlog Prioritization meeting

---

Design (Complete Before Development Starts)

Design preliminaries determine whether a formal design, which will be provided by an Upcoming Feature Overview (UFO) document, must be created and reviewed. A formal design is required if the feature requires any of the following: UI, Serviceability, SVT, Performance testing, or non-trivial documentation/ID. Furthermore, each identified item places a blocking requirement on another team so it must be identified early in the process. The feature owner may check-off the item if they know it doesn't apply, but otherwise they should work with the focal point to determine what work, if any, will be necessary and make them aware of it.

Design Preliminaries

• UI requirements identified, or N/A. (Feature owner and UI focal point)
• Accessibility requirements identified, or N/A. (Feature owner and Accessibility focal point)
• ID requirements identified, or N/A. (Feature owner and ID focal point)
  • Refer to Documenting Open Liberty.
  • Feature Owner adds label ID Required, if non-trivial documentation needs to be created by the ID team.
  • ID adds label ID Required - Trivial, if no design will be performed and only trivial ID updates are needed.
• Serviceability requirements identified, or N/A. (Feature owner and Serviceability focal point)
• SVT requirements identified, or N/A. (Feature owner and SVT focal point)
• Performance testing requirements identified, or N/A. (Feature owner and Performance focal point)

Design

• POC Design / UFO review requested.
  • Feature owner adds label Design Review Request
• POC Design / UFO review scheduled.
  • Follow the instructions in POC-Forum repo
• POC Design / UFO review completed.
• POC / UFO Review follow-ons completed.
• POC Design / UFO approval requested.
  • Feature owner adds label Design Approval Request
• Design / UFO approved. (OpenLiberty/chief-architect) or N/A
  • (OpenLiberty/chief-architect) adds label Design Approved
  • Add the public link to the UFO in Box to the Documents section.
  • The UFO must always accurately reflect the final implementation of the feature. Any changes must be first approved. Afterwards, update the UFO by creating a copy of the original approved slide(s) at the end of the deck and prepend "OLD" to the title(s). A single updated copy of the slide(s) should take the original's place, and have its title(s) prepended with "UPDATED".

No Design

• No Design requested.
  • Feature owner adds label No Design Approval Request
• No Design / No UFO approved. (OpenLiberty/chief-architect) or N/A
  • Approver adds label No Design Approved
• Feature / Capability stabilization or discontinuation or N/A
  • Feature owner adds label Product Management Approval Request and notifies OpenLiberty/product-management
  • Approver adds label Product Management Approved (OpenLiberty/product-management)
  • Note: For stabilized, superseded, and discontinued feature/capability, skip the Beta section of the template (you may delete it). Otherwise, proceed as normal.

FAT Documentation

• "Feature Test Summary" child task created
  • Use the Feature Test Summary Template
  • Add FTS issue link to the Documents section.

---

Implementation

A feature must be prioritized before any implementation work may begin to be delivered (inaccessible/no-ship). However, a design focused approach should still be applied to features, and developers should think about the feature design prior to writing and delivering any code.
Besides being prioritized, a feature must also be socialized (or No Design Approved) before any beta code may be delivered. All new Liberty content must be inaccessible in our GA releases until it is Feature Complete by either marking it kind=noship or beta fencing it.
Code may not GA until this feature has obtained the Design Approved or No Design Approved label, along with all other tasks outlined in the GA section.

Feature Development Begins

• Add the In Progress label
• Complete the specification and TCK
• Implement the spec required features in SmallRye OpenAPI
• Consume SmallRye OpenAPI implementation in liberty and pass the TCK
• Allow 4.0 to generate OpenAPI v3.0 as well as OpenAPI v3.1 smallrye/smallrye-open-api#1891
• Allow OpenAPI document version to be set with server.xml for MP OpenAPI 4.0 #28859
• Adapt existing MP OpenAPI tests for 4.0 #28856
• Add MP OpenAPI server.xml config to include or exclude apps (2.0+) #28857
• Allow MP OpenAPI server.xml info section to be set with server.xml (2.0+) #28858
• Implement OpenAPI document validation on MP OpenAPI 4.0 #28860
• Implement document merging on MP OpenAPI 4.0 #28861
• Document all modules by default on MP OpenAPI 4.0 #29793
• Apply version config before performing OpenAPI validation #29975

Legal and Translation

In order to avoid last minute blockers and significant disruptions to the feature, the legal items need to be done as early in the feature process as possible, either in design or as early into the development as possible. Similarly, translation is to be done concurrently with development. Both MUST be completed before Beta or GA is requested.

Legal (Complete before Feature Complete Date)

• Changed or new open source libraries are cleared and approved, or N/A. (Legal Release Services/Cass Tucker/Release PM).

Innovation (Complete 1 week before Feature Complete Date)

• Consider whether any aspects of the feature may be patentable. If any identified, disclosures have been submitted.

Translation (Complete by Feature Complete Date)

• PII (Program Integrated Information) updates are merged (i.e. all English strings due for translation have been delivered), or N/A.

---

Beta

In order to facilitate early feedback from users, all new features and functionality should first be released as part of a beta release.

Beta Code

• Beta fence the functionality
  • E.g. kind=beta, ibm:beta, ProductInfo.getBetaEdition()
• Beta development complete and feature ready for inclusion in a beta release
  • Add label target:beta and the appropriate target:YY00X-beta (where YY00X is the targeted beta version).
• Feature delivered into beta
  • (OpenLiberty/release-manager) adds label release:YY00X-beta (where YY00X is the first beta version that included the functionality).

Beta Blog (Complete by beta eGA)

• Beta blog issue created and populated using the Open Liberty BETA blog post template.
  • Add a link to the beta blog issue in the Documents section.
  • Note: This is for inclusion into the overall beta release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

---

GA

A feature is ready to GA after it is Feature Complete and has obtained all necessary Focal Point Approvals.

Feature Complete

• Feature implementation and tests completed.
  • All PRs are merged.
  • All related/child issues are closed.
  • All stop ship issues are completed.
• Legal: all necessary approvals granted.
• Translation: Feature may only proceed to GA if it has either Translation - Complete or Translation - Missing label
  • If all translation has been delivered to release branch, feature owner adds label Translation - Complete.
  • If missing translation does not cause a break in functionality, nor a security or production outage risk, feature owner adds label Translation - Missing.
    • Once all missing translations are delivered, the Translation - Missing label is replaced with Translation - Complete.
  • If missing translation could cause a break in functionality or a security or production outage risk, feature owner adds the Translation - Blocked label.
    • Featues with Translation - Blocked may NOT proceed to GA until the label has been replaced with either Translation - Missing or Translation - Complete.
  • For further guidance, contact Globalization focal point or the Release Architect.
• GA development complete and feature ready for inclusion in a GA release
  • Add label target:ga and the appropriate target:YY00X (where YY00X is the targeted GA version).
  • Inclusion in a release requires the completion of all Focal Point Approvals.

Focal Point Approvals (Complete by Feature Complete Date)

These occur only after GA of this feature is requested (by adding a target:ga label). GA of this feature may not occur until all approvals are obtained.

All Features

• APIs/Externals - Externals have been reviewed or N/A. (OpenLiberty/externals-approvers)
  • Approver adds label focalApproved:externals
• Demo - Demo is scheduled for an upcoming EOI or N/A. (OpenLiberty/demo-approvers)
  • Add comment @OpenLiberty/demo-approvers Demo scheduled for EOI [Iteration Number] to this issue.
  • Approver adds label focalApproved:demo.
• FAT - All Tests complete and running successfully in SOE or N/A. (OpenLiberty/fat-approvers)
  • Approver adds label focalApproved:fat.

Design Approved Features

• ID - Documentation is complete or N/A. (OpenLiberty/id-approvers)
  • Approver adds label focalApproved:id.

  • NOTE: If only trivial documentation changes are required, you may reach out to the ID Feature Focal to request a ID Required - Trivial label. Unlike features with regular ID requirement, those with ID Required - Trivial label do not have a hard requirement for a Design/UFO.

• InstantOn - InstantOn capable or N/A. (OpenLiberty/instantOn-approvers)
  • Approver adds label focalApproved:instantOn.
• Performance - Performance testing is complete or N/A. (OpenLiberty/performance-approvers)
  • Approver adds label focalApproved:performance.
• Serviceability - Serviceability has been addressed or N/A. (OpenLiberty/serviceability-approvers)
  • Approver adds label focalApproved:sve.
• STE - Skills Transfer Education chart deck is complete or N/A. (OpenLiberty/ste-approvers)
  • Approver adds label focalApproved:ste.
• SVT - System Verification Test is complete or N/A. (OpenLiberty/svt-approvers)
  • Approver adds label focalApproved:svt.

Remove Beta Fencing (Complete by Feature Complete Date)

• Beta guards are removed, or N/A
  • Only after all necessary Focal Point Approvals have been granted.

GA Blog (Complete by Friday after GM)

• GA Blog issue created and populated using the Open Liberty GA release blog post template.
  • Add a link to the GA Blog issue in the Documents section.
  • Note: This is for inclusion into the overall release blog post. If, in addition, you'd also like to create a dedicated blog post about your feature, then follow the "Standalone Feature Blog Post" instructions under the Other Deliverables section.

Post GM (Complete before GA)

• After confirming this feature has been included in the GM driver, feature owner closes this issue.

Post GA

• Remove the target:ga and target:YY00X labels, and add the appropriate release:YY00X. (OpenLiberty/release-manager)

---

Other Deliverables

• Standalone Feature Blog Post - A blog post specifically about your feature or N/A. (Feature owner and OpenLiberty/release-architect)
  • This should be strongly considered for larger or more prominent features.
  • Follow instructions in the blogs repo.
• OL Guides - OL Guides assessment is complete or N/A. (OpenLiberty/guide-assessment)
• Dev Experience - Developer Experience & Tools work is complete or N/A. (OpenLiberty/dev-experience-assessment)

---

[jhanders34]

UFO Socialization notes from today:

Slide 11 To-Be

• Update version from 3.1 to 4.0
• Add an explanation about ignore vs merge / override and conflict resolution when both are configured
• Add a statement about adding a message if mp config is ignored due to new server configuration.
• Add an example that shows using the module in an include / exclude

Slide 13 To-Be

• Specify that by takes precedence that the mp config is ignored
• Add a statement about adding a message if mp config is ignored due to new server configuration.

Slide 15 Feature Design

• Include information that SmallRye is the one that is providing the conversion to v3.0 and not something that liberty is doing.

Slide 17 End User Overview

• Add more details to this slide. Need to talk about what is new in OpenAPI 3.1 Maybe some server.xml snippets as well.

Slide 20 Java APIs / SPIs

• Need to provide details about what changes were made to the API

Slide 23, 24 and 25 Admin / Config / Command Line

• Need to determine what example differences you wanted to have in 23 vs 24 since they are the same
• Need to add details to earlier As-Is / To-Be section to state that we are adding support for the include / exclude in older versions of mpOpenAPI (2.0 or later). May also talk about that in the End User Overview section as well
• Include an example of include all and exclude a specific one for previous mpOpenAPI versions
• Have examples include / exclude modules as well
• Include a statement that if you include all others are excluded

Slide 28 Deprecation / Stabilization

• Check to see if any previous deprecated function can be deleted in 4.0

Slide 33 Open Source Software

• Don't say unchanged since we will probably update to the latest version of the UI

Slide 34 Beta

• Add statement about new mpOpenAPI attributes being marked beta / internal so it doesn't leak out

Slide 35 Automated Testing

• State that New tests added for new server.xml configuration options will include mpOpenAPI 2.0 and later, not just for the 4.0 feature.

Slide 38 Platform / Cloud Considerations

• Should be Java 11, not Java 17

Slide 40 Serviceability

• Additional messages as mentioned earlier for config
• Add a slide earlier that explains what the new features are in OpenAPI 3.1 since it was stated earlier in the UFO. Maybe somewhere in the Technical Overview.

[NottyCode]

@Azquelt

• I do not like the idea of an include value of first because I have no idea how to know what is first or to know if first will always be the same application. It seems non-deterministic behaviour.
• It isn't clear to me what the values for include or exclude should be.

[Azquelt]

• I agree, the value of first is not very useful, it's only included because it's the default behaviour for older versions of mpOpenAPI, and it matches the MP Config values those features currently accept.
  • We could exclude it, though that would mean there's no way for users to retain the old behaviour when upgrading from older versions, and no way for users of older versions to explicitly set the default value
• I agree that the valid values for include and exclude are not clear, I will add a new slide to clarify
  • The permitted values are
    • <application name> (for any application)
    • <application name>/<web module name> (for a .ear application)
  • The application name is as reported in the CWWKZ0001I: Application started message
    Edit: Found a mistake in my test here, we're matching against the Jakarta application name, whereas the name listed in the server log is the deployment name. These can be different if the Jakarta application name is set in the .ear deployment descriptor.
    This is the current behaviour for mpOpenAPI-2.0+ when reading the value from MP Config, but it doesn't seem ideal so I will look at it again tomorrow.
  • The module name is as reported in the SRVE0169I: Loading Web Module message

[Azquelt]

@NottyCode Slides 30 and 31 have been updated with the information above.

[Azquelt]

The conclusions from #29541 have been added to the UFO:

• <include> and <exclude> elements split into <includeApplication>, <includeModule>, <excludeApplication>, <excludeModule> to make it easier to understand a config file without already being familiar with these options
• when referencing an application by name with one of the server.xml elements, the same name used in the <application> element in server.xml must be used (or the name of the archive file with the extension removed if the name attribute is unset)
  • if the user tries to use the name set in the application deployment descriptor, a more specific error message will be logged explaining how to reference an application by name and which name they might need to use.
• when referencing an application by name using MP Config properties, we will continue to match against the deployment descriptor name to maintain zero migration
  • docs will be updated to give prominence to the server.xml config elements over the MP Config properties, and to note the difference in matching behaviour of the MP Config properties.

Slides updated 30-34, 49-50

@NottyCode the UFO is ready for re-review.

[Azquelt]

@OpenLiberty/id-approvers the docs issue for this feature has been raised here: OpenLiberty/docs#7654

[donbourne]

Serviceability Approval Comment - Please answer the following questions for serviceability approval:

1. UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. IBM Support, test team, or another development team).
   a) What problem paths were tested and demonstrated?
   b) Who did you demo to?
   c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
   a) Who conducted SVT tests for this feature?
   b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

4. Which IBM Support / SME queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective IBM Support/SME teams know they are supporting it. Ask Don Bourne if you need links or more info.

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

[Azquelt]

@donbourne

1. UFO -- does the UFO identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation?

The UFO does identify likely problems on slides 49 and 50. We have implemented informative warning messages for most of these scenarios but two are outstanding: #30036 and #30046. We plan to resolve those issues after release.

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. IBM Support, test team, or another development team).
   a) What problem paths were tested and demonstrated?
   b) Who did you demo to?
   c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

I demoed the following problem paths to @benjamin-confino who does most of our L3 support:

• User configures an OpenAPI spec version in the wrong format
• User configures an unsupported OpenAPI spec version
• User configures <info> with specifying required fields
• User attempts to include an application and module in the OpenAPI documentation which are not deployed
• User attempts to reference an application by the name from its deployment descriptor, rather than the name in server.xml

All were found to be satisfactory.

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered.
   a) Who conducted SVT tests for this feature?
   b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that IBM Support should be able to quickly address those problems without need to engage SMEs?

I believe SVT is being done by @hanczaryk, but I don't know if it has yet been completed. I'll need to wait for him to comment on the serviceability.

4. Which IBM Support / SME queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective IBM Support/SME teams know they are supporting it. Ask Don Bourne if you need links or more info.

WAS L3: CDI

5. Does this feature add any new metrics or emit any new JSON events? If yes, have you updated the JMX metrics reference list / Metrics reference list / JSON log events reference list in the Open Liberty docs?

No.

[jdmcclur]

I have approved performance, but opened #30058 to investigate more when we have time.

[Azquelt]

@OpenLiberty/demo-approvers Demo scheduled for EOI 24.24

[Azquelt]

@OpenLiberty/ste-approvers STE is available on the L3 site (internal link)

[Azquelt]

@OpenLiberty/id-approvers the docs issue for this feature has been raised here: OpenLiberty/docs#7654

Unless there is more information that's needed, please add the ID approval tag.

[chirp1]

Ram has the info that he needs to document OpenApi 4.0 in doc issue at OpenLiberty/docs#7654. Approving the epic.

[Azquelt]

@donbourne

In the end it was @tam512 that did the SVT testing. She gave me this report on the serviceability:

issue I saw was specifying wrong module name and it can't find it but I think the message is ok
CWWKO1684W: The includeModule configuration element includes "SocialSphere-mp/SocialSphereApp-AllergyForecastMPOpenAPI" but that does not match any deployed application or web module.

Between this information and the comment above, is that enough information for the serviceability approval?