GPU (and PCIe) Support in Firecracker

[xmarcalx]

Dear All,

Thanks to everyone who attended the first community meeting to discuss GPU support in Firecracker on 2024-10-09.
This meeting was scheduled to kick the project as part of the engagement we had in #1179.

Moving forward, we will track the notes of our community meeting around the topic in this discussion instead of the issue.

Starting from 2024-10-09, we will meet every 4 weeks, Wednesday at 18:00 GMT for up to one hour at:

https://chime.aws/4350163809
Meeting ID: 4350 16 3809
United States Toll-Free (1): +1 855-552-4463
United Kingdom Toll-Free (1): +44 800 085 5175
International: https://chime.aws/dialinnumbers/

The meeting is open to everyone who is interested in this initiative.

On 2024-10-09, we discussed the following points:

1. We are aligned to collaborate together with the community to support PCIe devices and consequentially PCIe-enabled GPUs into Firecracker microVMs.
2. Supporting PCIe passthrough may impact the capability to overcommit guest memory, and may slow down the boot of microVMs. However, those constrains are acceptable for the community especially for a first iteration. We will evaluate SLO and define what looks like acceptable from a performance point of view at a later stage of the project.
3. In the initial iteration, we will not aim to be able to snapshot GPU devices.
4. Firecracker Team will help the community to build PCIe and GPU support sheparding the project. The team will be responsible to project manage this initiative, set up roadmap, milestones and clear KPI in order to evaluate/gate progresses. The team will also drive security reviews, provide technical guidance/support and code reviews, and last but not the least help to build the infrastructure required to enable the community to run tests during the development of this feature. @Manciukic is going to be the tech leader of the Firecracker team who will lead this initiative.
5. The community will be responsible for the actual co-design and implementation of the feature.
6. Considering the implication of supporting PCIe passthrough, we will perform our day by day development task in a feature branch to do not impact the normal Firecracker release lifecycle. We will aim to merge the change back in main at each milestone if they do meet KPI, functional and security reviews.
7. We will held regular Community Meeting, every 4 Wednesday at same time 18:00 GMT for up to 1 hour to sync and share progresses. Once the project will be rolling, we will evaluate if there is there is the need of a bi-weekly sync too.

Next steps :

1. Firecracker team will provide to the community a roadmap to support PCIe devices and GPU to be reviewed. This will milestones, and a proposed set of tasks for the community by 2024-11-06.
2. Firecracker Team will setup the initial feature branch to kick off the deployment activities.

Kind Regards,
Marco

[Manciukic]

Hey folks,

ahead of today meeting, I have resurrected and extended the old PCIe PoC (sandreim/pci_experiment) into a dedicated branch in the main repository (poc/pcie). The implementation is scrappy, ugly, likely not free of bugs, and doesn't pass style and unit tests, but:

• it's possible to enable PCI for all virtio devices with a new vm config json parameter (I haven't had to add the API endpoint yet): "pci": { "enabled": true }
• it's possible to attach multiple PCIe devices through vfio (I got up to 8 GPUs working on a single FC VM): "pci": { "enabled": true, "vfio_devices": [ { "path": "/sys/bus/pci/devices/0000:18:00.0/" } ] }
• VM memory is pre-allocated if there are VFIO devices, so oversubscription with device pass-through is not possible
• snapshot/resume of PCI devices is not supported (not even for virtio devices yet, as I've taken some shortcuts).

Initial performance results on the PoC look promising, but I need more time to prepare the testing artifacts and infrastructure, refine the PoC, and gather more data. You're welcome to play with it as well.

In today's meeting we'll be reviewing the draft of the roadmap, which I have published as a draft PR to make it easy to leave comments on it, discuss it, and iterate on it: #4894 . Feel free to take a read, leave comments, and share your opinion!

Finally, I have also created a new (but empty) feature branch for PCIe development: feature/pcie. I'm looking forwards to the contributions!

Thanks,
Riccardo

[Manciukic]

Thank you all for joining the meeting on Wednesday, November 6!

We're sorry for the confusion around the timezone caused by daylight saving changes. For the future, any meeting will always use UK timezone, which is currently the same as UTC/GMT, in case there's any doubt.

The next meeting is scheduled for Wednesday, December 4 at 18:00 GMT for up to one hour in the same Chime room (https://chime.aws/4350163809).
We would be able to accommodate an extra occurrence next week or the following, in case there's enough interest. If you're interested, let us know here, or in the public slack workspace (thread, join link. I will update this discussion if a new meeting is scheduled.

Meeting notes:

1. The plan needs more clarity on the objectives and features supported for the MVP, refining the acceptance criteria to narrow down the targeted use-cases.
   • are we going to support one single or multiple GPUs? If multiple, what about P2P? We are aiming for simple support of a single GPU.
   • are we going to support just PF or also VF? In the initial iteration, we're focusing on PF, but VF is something we want and we will call it out explicitly
   • are we going straight to hotplugging or do we want to focus on cold-plugging first? In the MVP, we want to focus on simple cold plugging with the intention to support hotplugging in the future.
     • note that hotplugging is a requirement for Kata-like workloads due to their API. Also, it introduces issues around detecting PCI root port topology as the required aperture size might not be known in advance as it depends on GPU.
     • note that PCIe native hotplugging is only supported with PCIe root ports
   • what about other features like GPU-direct, NVME support? Will not be supported in the first iterations.
2. We discussed about new features introduced in VFIO core from kernel 6.1, supporting iommufd as backend. We will look into these.
3. The kata-containers initiative for confidential compute is interested in including Firecracker GPU support. Details on how they interact with hardware devices can be found here (thanks @zvonkok):
   • Virtualization Reference Architecture: https://github.com/kata-containers/kata-containers/blob/main/docs/design/kata-vra.md
   • What happens if you type kubectl apply -f kata-gpu-pod.yaml https://docs.google.com/presentation/d/13TDKyASpMfDrVBSRj4JiU6gFeChx0ws4DTenBN1qUnA/edit?usp=sharing
   • The Kubernetes KEP: Pass down resources to CRI kubernetes/enhancements#4113
   • Issues tracking the crio and containerd changes: Pass-through resource allocations from runtime-config (CRI) to oci-spec cri-o/cri-o#8321, Pass-through resource allocations from runtime-config (CRI) to oci-spec containerd/containerd#10282

Next steps:

1. Firecracker team will review the draft roadmap to address the comments identified in the meeting [PCIe] Community Guidelines and Roadmap #4894
2. Firecracker team will setup testing artifacts with PCIe support for the first milestone (just virtio-pci device support, no GPU or device passthrough yet).

For anything, feel free to drop us a message here or in the slack workspace.

Thanks,
Riccardo

[Manciukic]

Thank you for joining the meeting on Wednesday, December 4!

The next meeting is scheduled for Wednesday, January 8 2025 at 18:00 GMT for up to one hour in the same Chime room (chime.aws/4350163809).

Meeting notes:

• Firecracker team explained the work they carried on since last meeting to refine the POC branch and setup CI artifacts which are compatible with PCIe. We also managed to gather some performance results:
  • block: virtio pci is 50% faster than mmio on 1vcpu sync reads and 46% in writes; and 20% faster in 2vcpu sync reads and 7% faster in writes
  • net throughput: virtio pci is between 2% and 11% faster in tx and 9%-17% faster in rx compared to mmio
  • net latency: virtio pci has 27% lower latency than mmio
  • boot time: less than 4GB uVM with virtio-pci devices are 8% slower to boot, but 4GB VMs are 20% faster (we think it's due to a bug with the current PoC causing BAR relocations).
• Firecracker team has completed all the prerequisite work for kicking off the project and is now waiting for external contributors and volunteers to help move to the implementation stage. In case there's anybody interested, the team would be available to help contributors with high level designs, guidance, and reviews.

For anything, feel free to drop us a message here or in the slack workspace.

Thanks,
Riccardo