V0.10.0 Release Plan

[cp5555]

Release Manager

@cp5555

Endgame

• Code freeze: Dec. 8th, 2023
• Bug Bash date: Dec. 11th, 2023
• Release date: Dec. 25th, 2023

Main Features

SuperBench Improvement

1. • Support Monitoring for AMD GPUs (Run benchmark failed (superbenchmark-0.8.0) #518 and Monitor - Upgrade pyrsmi to amdsmi python library. #601)
2. • Support ROCm5.7 and ROCm6.0 dockerfile (Dockerfile - Upgrade to rocm5.7 dockerfile #587, Dockerfile - Bug fix for rocm docker build and deploy #598, and Dockerfile - Add rocm6.0 dockerfile #602)
3. • Add MSCCL Support for Nvidia GPU (Benchmarks: Add MSCCL Support for Nvidia GPU #584)
4. • Fix NUMA Domains Swap Issue in NDv4 Topology File (Bug Fix - Fix NUMA Domains Swap Issue in NDv4 Topology File #592)
5. • Add ndv5 topo file (CI/CD - Add ndv5 topo file #597)
6. • Fix NCCL and NCCL-test to 2.18.3 for hang issue in CUDA12.2 (Benchmarks: Build Pipeline - fix nccl and nccl test version to 2.18.3 to resolve hang issue in cuda12.2 docker #599)

Micro-benchmark Improvement

1. • Add HPL random generator to gemm-flops with ROCm (Benchmarks: Micro benchmark - add initialization options for rocm gemm flops #578)
2. • Add DirectXGPURenderFPS Benchmark to measure the FPS of rendering simple frames (Benchmarks: micro benchmarks - add source code for DirectXRenderPerf #549)
3. • Add HWDecoderFPS Benchmark to measure the FPS of hardware decoder performance (Benchmarks: micro benchmark - source code for evaluating NVDEC decoding performance #560)
4. • Update Docker image for H100 support (Dockerfile - Upgrade Docker image to CUDA 12.2 #577)
5. • Update MLC version into 3.10 for CUDA/ROCm dockerfile (Dockerfile - update mlc version into 3.10 for cuda and rocm dockerfiles #562)
6. • Bug fix for GPU Burn test (Bug Fix - remove cp ptx file command in gpu burn test  #567)
7. • Support INT8 in cublaslt function (Benchmarks: micro benchmarks - add int8 support for cublaslt function #574)
8. • Add hipBLASLt function benchmark (Benchmarks: Micro benchmark - Add hipBLASLt function benchmark #576)
9. • Support cpu-gpu and gpu-cpu in ib-validation (Benchmarks: micro benchmark - Support cpu-gpu and gpu-cpu in ib-validation #581)
10. • Support graph mode in NCCL/RCCL benchmarks for latency metrics (Benchmarks: Micro benchmark - Add graph mode in NCCL/RCCL benchmarks for latency metrics #583)
11. • Support cpp implementation in distributed inference benchmark (Benchmarks: Microbenchmark - Add distributed inference benchmark cpp implementation #586 and Benchmarks: Bug Fix - Make metrics of dist-inference-cpp aligned with PyTorch version #596)
12. • Add O2 option for gpu_copy ROCm build (Benchmark: Revision - Fix -O2 option passing in gpu_copy ROCm build #589)
13. • Support different hipblasLt data types in dist_inference (Benchmarks: Microbenchmark - Support different hipblasLt data types in dist_inference #590 and Benchmarks: Microbenchmark - Adapt to hipblasLt data type changes #603)
14. • Support in-place in NCCL/RCCL benchmark (Benchmarks: Microbenchmark - Support in-place for NCCL/RCCL benchmark #591)
15. • Support data type option in NCCL/RCCL benchmark (Benchmarks: Microbenchmark - Add data type option for NCCL and RCCL tests #595)
16. • Improve P2P performance with fine-grained GPU memory in GPU-copy test for AMD GPUs (Benchmarks: Microbenchmark - Improve AMD GPU P2P performance with fine-grained GPU memory #593)
17. • Update hipblaslt GEMM metric unit to tflops (Benchmarks: Micro benchmarks - Update hipblaslt metric unit to tflops #604)
18. • Support FP8 for hipblaslt benchmark (Benchmarks: Micro benchmarks - add fp8 and initialization for hipblaslt benchmark #605)

Model Benchmark Improvement

1. • Change torch.distributed.launch to torchrun (Benchmarks: model benchmarks - change torch.distributed.launch to torchrun #556)
2. • Support Megatron-LM/Megatron-Deepspeed GPT pretrain benchmark (Benchmarks: Add benchmark: Megatron-LM/Megatron-Deepspeed GPT pretrain benchmark #582 and Bug Fix - Bug fix for latest megatron-lm benchmark #600)

Result Analysis

1. • Support baseline generation from multiple nodes (Analyzer - Generate baseline given results from multiple nodes.  #575)

Backlog

Micro-benchmark Improvement

1. Support cuDNN Backend API in cudnn-function.

Model Benchmark Improvement

1. Support VGG, LSTM, and GPT-2 small in TensorRT Inference Backend
2. Support VGG, LSTM, and GPT-2 small in ORT Inference Backend
3. Support more TensorRT parameters (Related to TensorRT parameter passing can be enhanced #366)