v7.0 rewrite checklist

[yamalight]

This is an issue to track overall progress on v7.0 rewrite.
Plan is to rewrite / update code to use latest libs (not just versions) and ES modules (ES modules are not ready yet, dropping them?).
Additionally, I'm planning to implement exoframe-client library that provides simple API for interacting with server (would allow to create new client, e.g. dashboard as some people wanted).

All work is done on v7-rewrite branch.

Packages:

• exoframe-client
• exoframe-server
• exoframe-cli
• exoframe-recipe-ghost
• exoframe-template-java
• exoframe-website (incl. docs)

Planned major changes:

• Drop yarn support
  Reasoning: yarn v1 is deprecated and v2 is too different from v1. npm v7 now does most of things yarn v1 did.
• Drop plugins support
  Reasoning: swarm is deprecated. There are no other plugins and doesn't seem like they'd be useful 🤔
• Drop docker swarm support.
  Reasoning: support for it adds complexity, almost nobody uses swarm (k8s is a better solution for that problem), exoframe doesn't work well in those scenarios as a whole.
• Drop docker-compose support.
  Reasoning: compose-based deployments don't work well with exoframe (users has to manually add traefik labels, etc), which causes a lot of issues. there's also an issue with tearing down and rebuilding / restarting all services when you only want to update one. IMO the best way would be to deprecate it and describe how to deploy multi-service projects using plain exoframe configs.
• Drop exoframe-faas support.
  Reasoning: it's half-baked, rarely used but adds quite a bit of overhead to the server. Would be better to provide a guide to extra-slim docker containers (not quite FaaS, but should close enough for this project scope)
• Drop old / unused recipes and templates (include one of each as an example)
  Reasoning: it's hard to maintain / fix all of them with time (I kinda ended up abandoning most of them), so it's better to just have a few simple examples.

Things to address in code / deps:

• Fix update tests for exoframe-server (failing in CI)
• Fix deprecation warnings in exoframe-server
• Replace @zeit/ncc with @vercel/ncc. New ncc version doesn't work well, removed, will publish exoframe-server as-is.
• Update to eslint 8.x and eslint-plugin-import 3.x (?) to enable top-level await support
  • Re-enable eslint in CI for exoframe-server once done
• Refactor CLI config loading to be executed on every program creation, not just on load (helpful for tests)
• Try using new @vercel/ncc -> doesn't work since some deps are still CJS and exoframe-server is ESM. Mixing those two doesn't work well when bundling code 😑

Planned issues:

• use new standard crypto.randomUUID API for UUID generation
• Use XDG_CONFIG_HOME for config location (Move config file #309)
  • Server
  • Client
• More private key types (Allow using any type of private keys for auth #191)
  • Server
  • Client
• Removal of multiple deployments (Allow multiple IDS to be removed #291) (after some more thinking - bad idea, disregarding)
• Docker images hosted on github (Also provide docker images through github registry #302)
• Allow other types of mounts (Support other types of volumes #319)
• Allow passing build args from config to docker during build (Allow passing build args from config to docker during build #323)
• Deterministic deployment IDs (Keep the same UUID on Update Deploy, when using baseDomain on the server #275)
• New website (New website #259)
• Setup E2E tests in CI for that run basic tests for all packages

ToDos:

• Go through Dockerfile best practices repo (and this) and apply all that make sense to default dockerfiles used by exoframe
• Combine all repos to monorepo
• Archive all old repos

Things to document:

• Migration guide
• Docker-compose (why it's no longer supported, how to replace it with individual deployments)
• Docs for typical deployment cases (Document typical deployment cases #235)
• Docs for usage in CI / github action (Github Action for Exoframe #260)
• Docs for different types of mounts (Support other types of volumes #319)
• Docs for new baseDomain based deployment URLs (Keep the same UUID on Update Deploy, when using baseDomain on the server #275)
• Docs for more private key types (Allow using any type of private keys for auth #191)
• Docs for install script (release 6.2.1 #307)
• Docs for new traefik config (Supply Traefik config file #139, Document traefik config via config file #303)
• Docs for enabling traefik dashboard (Add way to enable traefik dashboard #147)
• Docs for build args config (Allow passing build args from config to docker during build #323)

Breaking changes (to keep track of them somehow):

• Node 18+ as minimum requirement (due to ESM)
• Removal of yarn
• Removal of plugins
• Removal of swarm
• Removal of compose
• Removal of exoframe-faas
• Config folder change (using XDG_CONFIG_HOME now)
• [todo]

Any feedback welcome.

[niieani]

Hi @yamalight! I'm really sad to see "Drop docker-compose support." on the list. This is basically how 90% of my exoframe deployments are done.

I really love this feature and it would be really sad to see it go, as I'll have to either keep using the old version or create a fork. At this point I'd be willing to even contribute a small amount of cash to support for docker-compose, or help by contributing to maintaining that feature myself.

I've previously contributed a PR related to this feature, and as you can see I've been using exoframe for over 5 years now.

I really don't mind the fact that I need to specify labels manually, in fact, I prefer it, because it gives me more flexibility as to how the deployments should be routed via traefik.

I really hope you reconsider. Let me know if you'd like to know more about how my use cases. Using exoframe over the years has been fantastic!

[yamalight]

@niieani I still think it's one of the worst ways to deploy things via exoframe - you make your server tear-down, re-build and re-start all containers defined in the compose file on every deployment. it's incredibly wasteful in all sorts of ways. vast majority of the time you want to update just one (app) container, and keep all the others unchanged (database, message queue, etc).

this is the main reason I think "exoframe way" (tm) is just a better approach.
and it's not like doing it "exoframe way" is a lot of work - it takes a few minutes more than deploying compose, but only once.

here's an example. Imagine you have following compose file:

version: '3.6'
services:
  postgres:
    image: postgres:15
    ports:
      - 5432:5432
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword
  redis:
    image: redis:7
    ports:
      - '6379:6379'
  app:
    build: .
    ports:
      - 8080:8080
    depends_on:
      - postgres
      - redis
    environment:
      DATABASE_URL: postgres://postgres:postgrespassword@postgres:5432/postgres
      REDIS_HOST: redis

volumes:
  db_data:

This work perfectly fine for local testing / dev, but deploying it via exoframe 6 will tear-down and re-start postgres and redis every time you deploy. Why waste that time?

Here's how you would deploy it using exoframe configs:
First, you define config for postgres:

{
  "name": "my-postgres",
  "domain": false,
  "image": "postgres:15",
  "volumes": ["db_data:/var/lib/postgresql/data"],
  "hostname": "mypostgres",
  "env": {
    "POSTGRES_PASSWORD": "postgrespassword"
  }
}

Next, you define exoframe config for redis:

{
  "name": "my-redis",
  "domain": false,
  "image": "redis:7",
  "hostname": "myredis"
}

And finally, you define config for your app:

{
  "name": "my-app",
  "domain": "my-app.com",
  "env": {
    "DATABASE_URL": "postgres://postgres:postgrespassword@postgres:5432/postgres",
    "REDIS_HOST": "askaredis"
  }
}

That's it, that's all the work it takes and you do it once.
After doing that - you can individually (re-)deploy any of the containers without touching the other ones.

I think it's a fairly simple procedure that does make deployments a lot faster / simpler.
It obviously has to be documented a lot better than it is right now - but that should come along with a new website.

I was also thinking about a possibility of writing a CLI script that would auto-convert compose files into sub-deployments like this.

Would love to know your thoughts on this - is that easy enough? Would you still want compose?