Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: Support for limited tests run from PR description #34736

Merged
merged 20 commits into from
Jul 8, 2024
Merged

ci: Support for limited tests run from PR description #34736

merged 20 commits into from
Jul 8, 2024
+49 −20

Conversation

sharat87
Copy link
Member

@sharat87 sharat87 commented Jul 4, 2024 •
edited by coderabbitai bot
Loading

Specify the specs we want to run, in a plain code fence, where the first line has to exactly be /test. The remaining lines have to be one spec file per line.

Like this:

/test
cypress/e2e/Regression/ClientSide/Fork/ForkApplication_spec.ts
cypress/e2e/Regression/ClientSide/JSLibrary/Library_spec.ts
  1. It's not possible to run with tags and specs and the same time. This will need a slightly larger effort. But we'll get there.
  2. The status message shows spec files like they are space-separated, and isn't very easy to read. Don't take it too seriously yet, we'll improve.
  3. Globs should also work fine. The spec content, almost directly lands into the --spec argument of Cypress. See docs at https://docs.cypress.io/guides/guides/command-line#cypress-run-spec-lt-spec-gt.
  4. Multiple runners is disabled for this, have to solve it separately. The ci-test-limited.yml also doesn't do multiple runners either.

Warning

Workflow run: https://github.com/appsmithorg/appsmith/actions/runs/9803353429
Commit: e99ae38
Cypress dashboard.
Tags:
Spec: cypress/e2e/Regression/ClientSide/Fork/ForkApplication_spec.ts
cypress/e2e/Regression/ClientSide/JSLibrary/Library_spec.ts

It seems like no tests ran 😔. We are not able to recognize it, please check workflow here.

Fri, 05 Jul 2024 05:20:23 UTC

Summary by CodeRabbit

  • Chores
    • Enhanced GitHub Actions workflows to improve test specification handling and reporting.
    • Refactored tag parsing logic in scripts for more accurate and informative outputs.

@coderabbitai coderabbitai
Copy link
Contributor

coderabbitai bot commented Jul 4, 2024 •
edited
Loading

Walkthrough

The recent changes across the GitHub workflows and scripts focus on refining the handling of tag parsing, adding output specifications, and enhancing the cypress test configuration. These updates streamline the tag parsing process, spec inclusion, and test specification handling for more efficient and insightful CI/CD operations.

Changes

File Change Summary
.github/workflows/pr-automation.yml Added spec output, modified logic to check @tag.All, and updated job details to include spec.
.github/workflows/pr-cypress.yml Added ${{ inputs.spec }} to job configurations for specifying a test spec file.
.github/workflows/scripts/test-tag-parser.js Refactored tag parsing logic, improved error handling, updated tag parsing logic, and added a function to extract specifications.

Poem

In the code's grand parade, a rabbit did weave,
Outputs and specs it sought to achieve.
Cypress tests enhanced, tags parsed with care,
Efficiency and insight filling the air.
In this garden of code, improvements bloom,
Thanks to the rabbit’s diligent loom. 🌸🐇

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions github-actions bot added skip-changelog Adding this label to a PR prevents it from being listed in the changelog CI labels Jul 4, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 removed the ok-to-test Required label for CI label Jul 5, 2024
@sharat87 sharat87 added the ok-to-test Required label for CI label Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 added ok-to-test Required label for CI and removed ok-to-test Required label for CI labels Jul 5, 2024
@sharat87 sharat87 marked this pull request as ready for review July 5, 2024 05:24
coderabbitai[bot]
coderabbitai bot reviewed Jul 5, 2024
View reviewed changes
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 4382484 and 03e97e8.

Files selected for processing (3)
  • .github/workflows/pr-automation.yml (4 hunks)
  • .github/workflows/pr-cypress.yml (3 hunks)
  • .github/workflows/scripts/test-tag-parser.js (2 hunks)
Additional comments not posted (12)
.github/workflows/pr-automation.yml (3)

22-22: LGTM! Adding spec to outputs.

The addition of spec to the outputs section is correct.

46-54: LGTM! Updating the checkAll step to handle spec.

The modifications correctly handle the presence of spec and adjust the matrix accordingly.

68-68: LGTM! Adding Spec to the output details.

The addition of Spec to the output details is correct.

.github/workflows/scripts/test-tag-parser.js (6)

2-3: LGTM! Handling predictable newlines.

Replacing carriage return newlines with predictable newlines improves consistency.

4-7: LGTM! Handling empty payload body.

The code correctly handles the case where the body is empty by setting the action to failed.

23-24: LGTM! Setting outputs for tags and specifications.

The code correctly sets the outputs for tags and specifications based on the parsing results.

9-20: LGTM! Updating error handling.

The updated error handling improves the clarity of error messages and provides useful links for documentation.

31-43: LGTM! Refactoring the parseTags function.

The refactored parseTags function improves the parsing logic and adds support for code fences.

95-100: LGTM! Adding matchCodeFence function.

The matchCodeFence function correctly matches code fences in the body content to extract specifications.

.github/workflows/pr-cypress.yml (3)

Line range hint 6-10:
LGTM! Adding spec to inputs.

The addition of spec to the inputs section is correct.

Line range hint 81-81:
LGTM! Adding spec to the perform-test job.

The addition of spec to the perform-test job is correct.

197-197: LGTM! Adding spec to the output details.

The addition of spec to the output details is correct.

@riodeuno
Copy link
Contributor

riodeuno commented Jul 5, 2024

@sharat87 I am not sure if I got the message in the PR description correctly. The question is -- we're getting the message that "no tests ran", is that expected because we're only running individual specs right now?

I don't see a problem if this is actually the case, as only power users would use the spec list way of testing and would very likely know that the "no tests ran" message is a consequence of not running tags.

@ApekshaBhosale
Copy link
Contributor

@sharat87 one small thing, can we use less machines based on number of specs to run? we have reduced to only one machine for ci-test-limited. Before it used to be three and majorly only one was required, wasting 2 resources.

@sharat87
Copy link
Member Author

sharat87 commented Jul 5, 2024

Thanks for checking this out folks! ❤️

@riodeuno, I should've mentioned this, please ignore the Cypress report part of the PR description. It's not behaving correctly because there's changes in pr-automation.yml that have to be in release branch before the rest of the functionality in this PR can take any affect. I've verified that part before opening this up for review. I can force-merge the changes in just the pr-automation.yml file, and then test this PR here. Then the Cypress report should be more accurate.

@ApekshaBhosale, yes, indeed. This is how ci-test-limited is also operating today. It has no matrix in it, and it only runs on one runner, irrespective of how many spec files we're running. So I kept it to that in this PR. I want to add support for multiple runners for this, in a future PR. But since this aligns with what we have today, I preferred to keep the PR focused just to that.

Let me know what you guys think! 🙏

@sharat87 sharat87 merged commit 3264ac3 into release Jul 8, 2024
18 checks passed
@sharat87 sharat87 deleted the ci/test-list branch July 8, 2024 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coderabbitai coderabbitai[bot] coderabbitai left review comments

@yatinappsmith yatinappsmith yatinappsmith approved these changes

@riodeuno riodeuno Awaiting requested review from riodeuno

@nidhi-nair nidhi-nair Awaiting requested review from nidhi-nair

@hetunandu hetunandu Awaiting requested review from hetunandu

@ApekshaBhosale ApekshaBhosale Awaiting requested review from ApekshaBhosale

Assignees
No one assigned
Labels
CI ok-to-test Required label for CI skip-changelog Adding this label to a PR prevents it from being listed in the changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@sharat87 @riodeuno @ApekshaBhosale @yatinappsmith