Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ResNet50/TensorFlow] Inconsistent benchmark result on two citations? #434

Open
372046933 opened this issue Mar 27, 2020 · 0 comments
Open
Assignees
Labels
bug Something isn't working

Comments

@372046933
Copy link
Contributor

Related to Model/Framework(s)
The training script is from https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/Classification/RN50v1.5/main.py

Describe the bug
on https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#stories_tensorflow Mixed precision of different models are given as follows.

Model Speedup
BERT Q&A 3.3X speedup
GNMT 1.7X speedup
NCF 2.6X speedup
ResNet-50-v1.5 3.3X speedup
SSD-RN50-FPN-640 2.5X speedup

While training performance results on this repo is different:
https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/RN50v1.5#training-performance-benchmark

number of GPUs mixed precision img/s FP32 img/s mixed precision speedup mixed precision weak scaling FP32 weak scaling
1 825.2 364.9 2.20 1.00 1.00
4 3197.4 1419.4 2.25 3.96 3.89
8 6209.9 2778.5 2.24 7.74 7.61

We can see that the mixed precision is 2.2 on this repository, but 3.3 on Nvidia's another site.

To Reproduce
Just read the documentation is enough

Expected behavior
Give consistent benchmark result.

Environment
Please provide at least:

  • Container version (e.g. nvcr.io/nvidia/tensorflow:19.12-tf1-py3):
  • GPUs in the system: (e.g. 8x Tesla V100-PCIE-16GB):
  • CUDA driver version (e.g. 418.87):
@372046933 372046933 added the bug Something isn't working label Mar 27, 2020
@hXl3s hXl3s self-assigned this Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants