Implementing compose spec directly

[Aposhian]

As mentioned in #10976, it is an ongoing effort to maintain compatibility with an evolving Docker Compose. However, now that there is the generic compose-spec, we could potentially help users to use existing docker-compose.yaml files without worrying about providing compatibility with the Docker API.

related: #11780

[Aposhian]

My background is that I have an application that is defined with docker-compose.yaml files. It is desirable to leave it in that format since many developers have Docker, and it is a familiar development tool. However, for certain environments, it would be nice to run the same application, but with Podman. Eventually, I would love to be able to manage those individual services with systemd, but using the forking model of Podman vs the client-server architecture of Docker is a first step.

Right now, users have two main options for this in the Podman ecosystem:

1. podman-compose, which parses yaml files, and then runs podman CLI commands.
   • Pros:
     • No podman socket is required
   • Cons:
     • It doesn't use the compose-go library to parse compose YAML files, so that makes it more difficult to maintain as the spec evolves
     • Being a python script doesn't make it as pluggable into other parts of the podman libraries, such as systemd unit generation (which is what I am most interested in the long run)
     • The code is harder to read since it is executing a CLI from Python, versus calling library functions
2. Using docker-compose, but communicating with a podman socket over the Docker API.
   • Pros:
     • Compose YAML files should be parsed correctly
   • Cons:
     • Requires a podman socket to be running.
     • Compatibility is dependent on podman keeping up with the Docker API

What I am proposing is that we add a library to Podman that does the following:

• Uses compose-go to load yaml files, and convert them into a collection of libpod.Container and/or libpod.Pod
• Add a compose verb to the Podman CLI that loads YAML files and exposes a various functionalities to create, start, run, and stop containers, etc. Keep in mind that while it could be a good idea for podman compose to have similar options to docker-compose and docker compose, it needn't necessarily have the same options. The Compose spec defines the format of the files, not the format of the CLI: it doesn't require the implementing program to have "up" and "down" verbs, for example.

[baude]

Thanks @Aposhian ... before you start work on this, I would suggest we get more of the team to comment; moreover, I think a community cabal would be appropriate for just this thing. @TomSweeneyRedHat could you add this to the next meeting and share here when that is as well as get it on the agenda?

[afbjorklund]

@Aposhian including support for compose V2 (Go version) was the path taken by nerdctl compose.
It didn't have a docker socket, so going with compose V1 (Py version) was never an option, but anyway...

However, it is a long journey to make it compatible:
https://github.com/containerd/nerdctl#compose

Writing directly to the spec (https://compose-spec.io/) seems like the best, long term. Chosen for containerd.
But I think that if you ask the users they would ask What Would Docker Do. So you end up with daemons etc.

So it seems like options 2. followed by 1. are most "likely".
Maybe it would be possible to share code, for option 3. ?

[Aposhian]

I don't have code yet that accomplishes option 3, but that is what I am proposing to develop. Yes, I agree that full compose-spec compatibility is a lot, but I think that at least in my experience, a minority of features make up a majority of use-cases. The compose spec also notes:

We acknowledge that no Compose implementation is expected to support all attributes, and that support for some properties is Platform dependent and can only be confirmed at runtime.

[afbjorklund]

Looks like your best option is to rewrite podman-compose in Go ?

[Luap99]

There is already confusion if users should use podman-compose or docker-compose. Adding a new implementation directly into podman will make it even worse. All of the tools will end up having differences so migrating between them may not be possible.

I also do not think we have the manpower to maintain this. Even if someone would do the initial work we would still need to maintain this in the long run (fixing bugs and keep it up with the compose spec). There is a significant amount of work here just to reimplement already working functionality via podman/docker-compose, so why not just improve these tools?

[afbjorklund]

I didn't know that Podman supported CLI plugins.

I don't think that it has plugins - yet...

But it could be an alternative, if not accepting more code into the project but yet wanting to offer a podman compose user experience ?

[mheon]

We'd like to implement them, but Docker upstream deprecated them and pulled the specs. They still use them, though there seems to be discussions about changing them.

We'll probably still end up implementing them, assuming they aren't completely removed before we get around to it...

[mheon]

Regarding implementing Compose directly in Podman, I share the concerns of the other maintainers, but it really depends on how much code it ends up being. If it ends up being quite easy to implement 80% of the functionality (that will probably end up being ~95% of usecases) then I wouldn't mind merging that. Meanwhile, if this ends up being thousands of lines to implement more like 50% of the functionality, I get a lot more hesitant.

[Aposhian]

Well, I could try an initial Go implementation, maybe trying to enlist some help from contributors at podman-compose, and then it would be easier to evaluate whether or not it should belong in the main Podman codebase?

[muayyad-alsadi]

@Aposhian My personal preference is to have podman-compose registered as a CLI plugin outside podman code base. That CLI plugin can be written in any language (go or python). Currently I'm going for python because of how fast I can iterate and because I want to keep it process model (not remote/socket API). I'll keep an eye on any GO implementation and if you need any help there just ping me and I'll help you port it.