Migrate to mono-repository

[davimacedo]

New Feature / Enhancement Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.

Current Limitation

We currently have several JS repositories to manage (Parse Server, Parse Dashboard, Parse JS SDK, Docs, Push Adapter, Node APN, etc), which make it harder to create/maintain features requiring changes in multiple packages. It also requires extra effort to create new releases when we need to update all the packages bottom up.

Feature / Enhancement Description

Combine all JS repositories into a single monorepository (perhaps using lerna).

Example Use Case

• Release all JS packages at once
• Single PR involving multiple packages
• Requiring docs and dashboard changes in the same PR of a new feature to the Parse Server
• Track the docs we had in place at the moment we released each Parse Server version

Alternatives / Workarounds

• We will have to decide if we will create a new repo and move all current repos to the new one or if we will choose one current repo to move the others in
• We can first try with two or three repos and then move the others
• We can choose to continue as is now but at least try to automate the release of multilpe repos at once

3rd Party References

Next.js
Apollo GraphQL Server
Parse React

[mtrezza]

With regards to the new versioning we want to introduce, would that mean that there is only one version number for the mono repo, instead of individual version numbers for each repo?

If so one would always have to read the changelog to find out what exactly changed and whether to upgrade or how would that work?

[davimacedo]

With regards to the new versioning and release cycles we want to introduce, that would mean that there is only one version number for the mono repo, instead of individual version numbers for each repo, correct? So mono repo v5.0.0 would mean Parse Server 5.0.0, Parse Dashboard 5.0.0, etc?

The packages are independent (besides they reside in the same GitHub repository) so they can be either released separately with their own versions or together with the same version. I'd probably try to release major versions all together but let minor and patch versions to be released separately.

I think another advantage would be easier integration testing. There are some modules that are correctly not fully integration-tested, such as S3 adapter.

Yes. We can continue having some tests running inside each of the packages but we can also have a test package to do integration tests across all packages.

[mtrezza]

I wonder how would that work branch-wise? Would there be a single master branch for all and a release for an individual package would always be made from that common master?

[davimacedo]

Yes. Since they will be all in the same repository, we will have only one master branch for all packages. If we use lerna, when we use lerna deploy command, it checks the packages the suffered changes and release all of them at once.

[mtrezza]

That sounds pretty neat. Do you think we should set up a sample repo to try this out with dummy sub-repos? Just to try this out hands on.

I am doing the same currently for the new branch system and discovered some challenges that are just built into GitHub because of how it works.

[davimacedo]

Yes. I can create a test repo and even copy all others inside so we can check how it would work. I've also worked at this repo (https://github.com/parse-community/parse-react) which uses lerna and is a monorepo with multilpe npm packages.

[mtrezza]

The question is whether we run into other trouble when we go about release automation. I assume there would be less tools available for mono-repos, but that's just a guess.

[mtrezza]

I think it makes sense to combine repos like S3 adapter into monorepo together with Parse Server, because these would not be used on their own and they are very dependent on Parse Server, an adapter change mostly leads to a Parse Server change as well. But I am not sure about Parse Dashboard for example, which seems more de-coupled from Parse Server.

[davimacedo]

You may be right. We can start with the most obvious ones and then analyze the others.

[parse-github-assistant]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[mtrezza]

From the recent attempt to migrate we have learned that:

• Monorepo must use npm packages
• We stick with npm, no switching to yarn or any other package manager
• An experimental monorepo must be created first with dummy repos to test the concept (auto-release, etc); well keep this repo to test future changes before applying them to the "live" monorepo

Aimed timeframe for this: during 2023, to release Parse Server 7 in 2024 from a monorepo and incorporate any needed breaking changes.

[KodeyThomas]

@mtrezza NX (Commercial version of Lerna but still OSS) should allow you to use Yarn wanted at a later date.

Think it's also important we consider tests across the entire codebase, we should be able to leverage the NX command affected --target=PROJECT_CHANGED.

Have we scoped out the packages to incorporate into the monorepo?

[mtrezza]

We'd start with Parse Server and the JS SDK. These are two most entangled repos where we already see difficulties when a change in one repo requires a change in the other repo.

I'm not convinced using a commercial product if the native npm packages cover all we need.

[mtrezza]

Try auto instead of semantic-release due to issues with mono-repo compatibility, see parse-community/Parse-SDK-Flutter#668 (comment).

[Moumouls]

Here some feedback:

In my company we run successfully ( like 50 packages)

• Mono repo based on Yarn V3 with workspace-tools
• Complementary use of Turbo repo to speed bootstrap of all packages ( we are TS only), and prune correctly the mono repo for Docker images
• Shared TS/Pretter/Eslint config across the monorepo
• Semantic release with monorepo semantic release add on
• Scoped and optimized CI/CD

I know that many OSS teams use also https://github.com/changesets/changesets for better changelog

[mtrezza]

Thanks, if you recall, as we've discussed in the past, your setup is unlikely to work for Parse Platform.

We are looking for an auto-release solution that:

• is based on npm (not yarn)
• works with mono-repos
• is reliable and robust (semantic-release doesn't support mono-repo and extensions are fragile, the maintainers of all these extensions themselves explicitly mention that the underlying framework doesn't natively support it so they had to hack something together that is likely to break with future releases)
• is maintainable and simple (as few tools as possible)
• supports pre-releases
• works across ecosystems (single solution across all Parse Platform repos, customizable to bump versions in various files)
• supports various registries (npm, packagist, maven, etc)
• creates changelog based on commit message

A lot of factors, so it will take a while to find something that actually works. In theory it often seems easy, but when putting the things together they turn out to not work so fluently. So if you like to put together a very simple test mono-repo (with empty dummy repos, no Parse repos) that demonstrates that it works, it would be a good start.