Skip to content

lightonai/composer

 
 

Repository files navigation

Composer fork for Mamba models

This repository is a fork of the Composer library to train Mamba models with the following features:

  • Custom Block-wise activation checkpointing
  • Custom FSDP layer wrapping for Mamba
  • The WSD scheduler
  • FLOPs computation for Mamba
  • Custom and efficient dataloading
  • Improved logging

More details and instructions can be found in the dedicated mamba directory on how to use and train Mamba models with the provided codebase.

Releases

No releases published

Packages

No packages published

Languages

  • Python 99.5%
  • Other 0.5%