Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[T170073014] Rewrite distributed examples for Tensor Parallel, Sequence Parallel, 2D (FSDP + TP) #1201

Merged
merged 20 commits into from
Nov 22, 2023
Merged

[T170073014] Rewrite distributed examples for Tensor Parallel, Sequence Parallel, 2D (FSDP + TP) #1201

merged 20 commits into from
Nov 22, 2023

Commits on Nov 15, 2023

  1. update requirements.txt

    @lessw2020
    lessw2020 committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    21a5fcf View commit details
    Browse the repository at this point in the history

  2. add torchrun support, move to init_device_mesh

    @lessw2020
    lessw2020 committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    f962b60 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. update twod fully working

    @lessw2020
    lessw2020 committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    bc3c1dd View commit details
    Browse the repository at this point in the history

  2. ensure proper dp group seeding for synth data

    @lessw2020
    lessw2020 committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    11a3bb2 View commit details
    Browse the repository at this point in the history

  3. swiglu model added

    @lessw2020
    lessw2020 committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    9cebdf0 View commit details
    Browse the repository at this point in the history

  4. sequential running of custom, auto, seq parallel models

    @lessw2020
    lessw2020 committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    2447883 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. streamline to 2D TP only for two_d_parallel example

    @lessw2020
    lessw2020 committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    a388c20 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2023

  1. sequence parallel working...needs init_device_mesh update

    @lessw2020
    lessw2020 committed Nov 18, 2023
    Configuration menu
    Copy the full SHA
    842c3f0 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. seq parallel now using init_device_mesh

    @lessw2020
    lessw2020 committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    3aa1c53 View commit details
    Browse the repository at this point in the history

  2. tp and sp examples all working and updated

    @lessw2020
    lessw2020 committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    b54e2ec View commit details
    Browse the repository at this point in the history

  3. updates from code review

    @lessw2020
    lessw2020 committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    4889e3b View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2023

  1. remove utils.py. Sample models created in example files

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    b215178 View commit details
    Browse the repository at this point in the history

  2. remove originals.py, leftover imports, various updates from code revi… 

    …ew feedback.
    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    242c328 View commit details
    Browse the repository at this point in the history

  3. code linting via ruff

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    2f4a083 View commit details
    Browse the repository at this point in the history

  4. code formatting via ruff

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    742966b View commit details
    Browse the repository at this point in the history

  5. move rank_log to utils.py, update example files

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    7da71bc View commit details
    Browse the repository at this point in the history

  6. move logging imports and config to log_utils, update examples with ne… 

    …w import
    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    836f798 View commit details
    Browse the repository at this point in the history

  7. add gpu verification, update run_python_examples.sh

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    2de0144 View commit details
    Browse the repository at this point in the history

  8. update min gpu = 4 for fsdp+tp

    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    77fe3d8 View commit details
    Browse the repository at this point in the history

  9. move gpu check to top of examples, but before import init_device_mesh… 

    … to clear CI
    @lessw2020
    lessw2020 committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    5f4a5d3 View commit details
    Browse the repository at this point in the history