Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix problem with gradients accumulating #3

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix problem with gradients accumulating #3

wants to merge 1 commit into from

Conversation

AlekseySh
Copy link

The problem arises because a gradient accumulates in the layers with each new transformation.
Let's imagine, that original model works well with batch_size 128 (for example) and GPU was fully loaded, then wrapped model with several transform will crash for similar batch_size = 128, it will work only for batch size ~= 40.
To fix this problem I added torch._no_grad().

PS. If you now about this problem and assume that this code must be added in outer function, then you have to change your example snippet from readme because it doesn't include no_grad(). But in my opinion, it will be better left this code here, inside the forward.

fix problem with gradients accumulating
@qubvel
Copy link
Owner

qubvel commented Nov 14, 2019

Hi, thanks for proposal.
In my opinion, despite it is called "test" time augmentation wrapper, it is still possible to use it for training. So, it is better to add a note about no_grad().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants