Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request for Comment (RFC) Process #222

Merged
merged 25 commits into from
Feb 21, 2024
Merged

Request for Comment (RFC) Process #222

merged 25 commits into from
Feb 21, 2024

Conversation

joshmoore
Copy link
Member

@joshmoore joshmoore commented Dec 22, 2023

Fix #132 (image.sc announcement)

This is a proposal for the use of the RFC model,
established by the Internet Engineering Task Force (IETF), for decision making
within the NGFF community. As a proof-of-concept, the PR uses the RFC process
that it itself is proposing in order to demonstrate that process. The description
here on GitHub will be kept to a minimum in favor of reading the RFC text.

One possible order of review would include:

  • template.md: Read the template document which describes each of the sections
    in an RFC in approachable language (largely based on the Hashicorp model from
    [https://works.hashicorp.com/articles/rfc-template]). A review of the
    sections from templates in several RFC-using communities can be found in
    a google spreadsheet.

  • diagram.png: Take a quick look at the process diagram linked below to get an
    overview of the number of steps and phases, but don’t be too concerned about
    the details since they are described in the RFC.

  • RFC-x.md: Read the RFC itself which has all of the sections described in
    the template.md and walks through each of the phases and states, describing
    all terms.

  • Read some of the other proposed RFCs (once they are ready)

Expectations

As mentioned in the RFC text, not all PRs are expected to go through the RFC
process. In fact, the number that are in the process at any given time is
likely limited to a handful at most. (These numbers have been omitted from the
RFC text since they are estimates and likely to go stale.)

Open questions

As a WIP, the RFC text currently has a number of TODOs listed throughout that
need resolution before this PR will be ready for review. Additional open questions
that could use consideration before a closer review is possible include:

  • The times suggested in the drawing are initial suggestions and up for
    discussion (as is the general flow).

  • It is not yet defined which sign-offs are required for this RFC! Work on
    that will begin in the new year. There will likely also be a separation
    between the reviewers of technical PRs and of process PRs.

  • In IETF RFC, texts are not edited once they are adopted (see rfc2026).
    If that model is followed, RFCs opened to modify existing processes will need
    to copy the entire text of the original RFC. This is likely more complicated
    than we would like. What modifications, however, are permissible on published
    RFCs? Do they need versioning?

  • How do we gauge the impact on implementors? This could likely use
    clarification in the SPEC phase.

  • Do versions of the full spec get their own RFCs? How do alpha/beta releases
    work?

In some cases, these questions may be moved to the "Future possibilities"
section of the RFC to be answered by future RFCs.

Open actions

  • Add a PR which summarizes the RFC to the main documentation with a
    reference to this page. Then when other RFCs are added, that central location
    can be updated (and we can decide whether or not we update the RFCs
    retroactively, or perhaps only in the metadata)
  • Write up the the consensus model from issue Process definition #132 as RFC-0.
  • Finalize the a metadata model under "Technical considerations", or
    move it to "Future possibilities"

Copy link
Contributor

github-actions bot commented Dec 22, 2023

Automated Review URLs

@imagesc-bot
Copy link

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ngff-rfc-process-proposal-draft-now-available/90181/1

@normanrz
Copy link
Contributor

normanrz commented Jan 2, 2024

Happy new year, everyone! Fantastic write up @joshmoore! I think this RFC process will help to drive innovation in the NGFF space.

While reading through the proposal, 2 things came to my mind:

  • Guidance on experimental version numbers. I think it would be useful to recommend a version schema for experimental features so that implementations can work with features pre-approval. I think that is important because it is desirable that new features are validated in a number of implementations and we should keep the feature set of main versions stable. For example, an experimental version number could be 0.4-dev-pr222 where 222 is the RFC PR that describes the changes. To be a bit defensive about the spread of experimental versions, the spec could include a note like Implementations SHOULD only read/write experimental versions when a feature flag is explicitly activated by a user.
  • Changes to the spec documents. Currently, changes to the spec are formulated as PRs against the spec document. I think that leads to a suboptimal review experience, because there are many small changes and it is easy to lose the big picture. Also, all incremental changes need to be maintained and reviewed throughout the process. With the RFC documents, we have an opportunity to motivate and describe the high-level ideas of the change separately. I wonder at what point in the process the spec document (and all the json files) should be updated. Doing that in an early draft stage might be wasteful and contra-productive. Should these changes go in the same RFC PR?

Copy link
Contributor

@thewtex thewtex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, spectacular @joshmoore !

I love 😍 :

  • The well-defined process!
  • Building on the process of arguably the most successful and impactful efforts for interoperability, the internet.
  • Multiple stages for incremental evolution and improvements.
  • A process that promotes a "defaults to yes" as opposed to "defaults to no" so we can progress and effort is not wasted.
  • Artifacts that result in a clear record of what has been proposed and discussed.
  • The diagram that provides a succinct high-level overview.
  • The time limits at stages.
  • A recognition in the process that full consensus is not what will allow the community to move forward, which is what we all really want.
  • The process essentially utilizes the same GitHub pull request-based system, but formalizes the phases and content, and ensures drafts and reviews and responses are more easily navigated.

Regarding possible areas of improvement, there are a few tweaks we could make to have the intended effect. In particular, I am thinking of dynamics common in specification development where:

  • Changes are prematurely or unjustifiably suppressed because reviewers do not personally have interest in or and understanding of the change.
  • Strong-willed, arm-chair speculation on how a change should work without experience in implementation or usage.
  • Overly complex proposals that are difficult to implement.
  • Changes are rejected too early in an incubation stage.

Another successful community process to cite that I think has lessons to learn from, also a W3C project, is the WebAssembly process.

In particular, I think it is helpful to specify expectations on different degrees of implementations throughout the process. As we know, speculation on what should work well or how it should work is often clarified or modified during implementation and usage. There is also an iterative, evolutionary process to a final specification and implementation for good specifications. It is also helpful to clarify the specifics of a proposal to reviewers with a concrete example. And concerns about the impact of the complexity of a change on practical deployment are often resolved with reference implementation(s). By evolving an implementation with the spec, even if the implementation lags a bit, this helps avoid inconsistencies, which can be problematic even if it is just schema / documentation inconsistencies.

The following changes could help address these issues with this in mind:

  1. Reserve input from Reviewers and Commenters until a spec has gone into the RFC stage (I am afraid we may not see a practical difference with the status quo otherwise).
  2. Before exiting the DRAFT phase, request some partial implementation reference to link to.
  3. Before exiting the RFC phase, a full implementation with a test suite is required.
  4. Before entering the SPEC adopted status, there must be at least two implementations for the spec in the community.

rfc/x/index.md Outdated Show resolved Hide resolved
rfc/x/index.md Outdated Show resolved Hide resolved
rfc/x/index.md Outdated
The DRAFT phase begins when **Authors** propose (D1) a new idea and
subsequently gather support (or "socialize") the idea (D2) before opening a PR
(D3). Open PRs can be discussed on GitHub. Any clarifications that are needed
can be requested by **Editors**, **Reviewers** or **Commenters** (D4) which may
Copy link
Contributor

@thewtex thewtex Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In keeping with the last paragraph, i.e. the DRAFT stage promotes ideation without premature criticism and procedural indecision, it would be helpful to remove **Reviewers** and **Commenters** here and invite **Commenters** participation in the RFC stage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've combined the previous text with your suggestions in 225c762, @thewtex, if you'd like to take a look.

rfc/x/index.md Outdated Show resolved Hide resolved
rfc/x/index.md Outdated

This brings a critical, and iterative, decision point (R6). If a "Reject"
recommendation remains, then the RFC is closed. The text remains on the
specification pages for posterity. If sufficient (TODO: define minimal, etc.)
Copy link
Contributor

@thewtex thewtex Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sufficient: two released implementations?

@joshmoore
Copy link
Member Author

@normanrz commented last week

  • Guidance on experimental version numbers. I think it would be useful to recommend a version schema for experimental features ...

I definitely agree that something like this is needed, but I also know that Thar Be Dragons. I'm inclined to say that should follow as a fully fledged RFC itself, but it's worth considering if we can make an initial stab at it.

  • Changes to the spec documents. Currently, changes to the spec are formulated as PRs against the spec document. I think that leads to a suboptimal review experience ... Should these changes go in the same RFC PR?

My feeling is that it shouldn't be part of the initial RFC PR because that makes merging that PR much more difficult. Currently the change happens at S1 which may then be too late.


@thewtex left a comment

Another successful community process to cite that I think has lessons to learn from, also a W3C project, is the WebAssembly process.

Thanks for the link. I'll read and respond.

  1. Reserve input from Reviewers and Commenters until a spec has gone into the RFC stage (I am afraid we may not see a practical difference with the status quo otherwise).

It's a good point and I think reflects how GitHub has not been ideal for what we need. Are you proposing explicitly removing D4? Two options I considered to that end were: using GH locking mechanism to prevent D4 and submit the RFC outside of GH. In the end, I thought it would be simplest if we tried to develop a culture of only making the minimal clarifications during D4 (with the burden being on the Editors of shaping those discussions). But I'd love to hear what everyone thinks.

  1. Before exiting the DRAFT phase, request some partial implementation reference to link
    to.

Interesting. That's quite the bar but I can definitely see the benefit.

  1. Before exiting the RFC phase, a full implementation with a test suite is required.

Evaluating the "full"ness and the "suiteness" of the tests may be an issue, but I concur. I could also see adding "...a released implementation..." to the requirement.

  1. Before entering the SPEC adopted status, there must be at least two implementations for the spec in the community.

I've left your comment on line 215 open for further discussion.

Thanks, both!

@thewtex
Copy link
Contributor

thewtex commented Jan 16, 2024

In the end, I thought it would be simplest if we tried to develop a culture of only making the minimal clarifications during D4 (with the burden being on the Editors of shaping those discussions).

@joshmoore yes, I agree with this approach. That is, keep D4 and limit the GitHub pull request review discussion to minimal clarification on what is being proposed. This could come from the Editor. It would be helpful to also set the expectation that PR comments on how a specification is implemented or what could be proposed instead from Reviewers or Commenters should be come in the RFC stage, and the Editor will direct these comments to the RFC stage when appropriate. I am not sure if we want to invite limited clarification questions on what is being proposed from Reviewers and Commenters because it will be tempting to derail into more comprehensive discussions. These suggestions and discussion on the content of RFC 0 should probably be made once RFC 0 reaches the RFC Stage, though! 🤷

@joshmoore
Copy link
Member Author

joshmoore commented Jan 17, 2024

These suggestions and discussion on the content of RFC 0 should probably be made once RFC 0 reaches the RFC Stage, though! 🤷:

😄 Exactly. So we clarify the few remaining TODOs, then move this forward for fuller reviews ❤️


At the **Editors** discretion (D5), the PR can be merged at which point it
becomes an RFC or closed if there is no interest in progressing with the
proposal. In the latter case, **Authors** can use any feedback to open a new PR
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diagram.png doesn't mention that the DRAFT PR gets merged or closed at D5. Only approve Yes/No. The PR persists state doesn't indicate that the PR is closed.
Is there a point at which the DRAFT PR loses the "DRAFT" status and becomes a "ready to review" PR, or maybe DRAFT PR is a PR to DRAFT and RFC, rather than a PR with "DRAFT" status on github?

It wasn't clear to me from the diagram whether the RFC section was reviewing an open PR, but from reading the text here is appears not.

Who is responsible for D4? It's the only block that isn't assigned to a stakeholder role. I assume it is REVIEWERS but are they invited or notified about the PR by EDITORS?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR persists state doesn't indicate that the PR is closed.

👍 I'll change "PR persists" to "PR closed"

Is there a point at which the DRAFT PR loses the "DRAFT" status and becomes a "ready to review" PR

The GitHub "draft" state doesn't currently play a role in the RFC, but I can see that that could cause confusion.

It wasn't clear to me from the diagram whether the RFC section was reviewing an open PR, but from reading the text here is appears not.

Correct. I'll include "EDITORS ... merge PR" in R1.

Who is responsible for D4?

Anyone in the community, but it won't be requested of anyone.

rfc/x/index.md Outdated Show resolved Hide resolved
@joshmoore
Copy link
Member Author

Another successful community process to cite that I think has lessons to learn from, also a W3C project, is the WebAssembly process.

Thanks for the read, @thewtex! I could definitely see lifting some of the formatting (e.g., phases) or phrasing (e.g., champion) for this proposal.

The barrier I see to making more use of the process in general is the amount of dedicated capacity that I imagine it takes. We're not yet in a position to get that number of decision makers to synchronously come to consensus, not to mention to get them colocated physically in the Bay Area! 😄 It's a given, though, that this RFC process will need to evolve as the NGFF community changes. Assuming the number of individuals dedicated to driving the specification forward starts to increase, let's come back to this model and review.

I've now taken this PR out of GitHub draft mode and expect to merge it soon and solicit reviews. I'm currently trying to finalize RFC-0 with a summary of the original consensus model, but don't want to block too long on that more aesthetic addition.

@joshmoore joshmoore marked this pull request as ready for review February 14, 2024 09:22
@joshmoore joshmoore changed the title Request for Comment (RFC) Process Proposal (WIP) Request for Comment (RFC) Process Feb 14, 2024
@thewtex
Copy link
Contributor

thewtex commented Feb 15, 2024

colocated physically in the Bay Area! 😄

No offense to the Bay Area, but I would prefer Napari. Or Kaibu 😄

#mergeit

@normanrz normanrz mentioned this pull request Feb 15, 2024
@kevinyamauchi
Copy link
Contributor

Hey @joshmoore . I think this is great. I agree with @thewtex 's list of strengths of this RFC above. I think there are some details to be worked out, but I think this is clear enough to move to the next stage. The fact that there is already an RFC drafted (#227) is a fairly strong endorsement in my eyes. I agree with your plan to merge and solicit reviews! 🚀

@joshmoore
Copy link
Member Author

No offense to the Bay Area, but I would prefer Napari. Or Kaibu 😄

😄

#mergeit
I agree with your plan to merge and solicit reviews! 🚀

Thanks all. Merging for the rebasing of #227, and I'll push RFC-0 separately.

@joshmoore joshmoore merged commit 24499ae into ome:main Feb 21, 2024
8 checks passed
@joshmoore joshmoore deleted the rfc branch February 21, 2024 16:29
github-actions bot added a commit that referenced this pull request Feb 21, 2024
Request for Comment (RFC) Process

SHA: 24499ae
Reason: push, by @joshmoore

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@joshmoore joshmoore mentioned this pull request Feb 21, 2024
@kevinyamauchi
Copy link
Contributor

Nice! Thanks @joshmoore !

joshmoore added a commit to joshmoore/ngff that referenced this pull request Apr 24, 2024
@joshmoore joshmoore mentioned this pull request Apr 26, 2024
@joshmoore joshmoore mentioned this pull request Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Process definition
6 participants