You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
hi,
thanks for this repo,
here are a couple efficiency questions: a) In REU function Why is there a cast to double ? Why only in the backward pass ? I think this is unefficient, maybe useless and could be left in FP32, what do you think ?
@staticmethod
def backward(ctx, grad_output: torch.Tensor) -> torch.Tensor:
"""Performs a backpropagation."""
(data, ) = ctx.saved_tensors
data = data.double()
grad = torch.where(data >= 0.0, 1.0, torch.exp(data) * (data + 1))
return grad_output * grad
def forward(ctx, data: torch.Tensor) -> torch.Tensor:
"""Performs a forward pass."""
ctx.save_for_backward(data)
return torch.where(data <= 0.0, data * torch.exp(data), data)
@staticmethod
def backward(ctx, grad_output: torch.Tensor) -> torch.Tensor:
"""Performs a backpropagation."""
(data, ) = ctx.saved_tensors
data = data.double()
grad = torch.where(data >= 0.0, 1.0, torch.exp(data) * (data + 1))
return grad_output * grad
The exponential is computed 2 times (forward, backward) over the whole range (positive and negative).
Is think is could be computed only 1 time, and only for the negative value, for efficiency
The text was updated successfully, but these errors were encountered:
hi,
thanks for this repo,
here are a couple efficiency questions:
a)
In REU function Why is there a cast to double ? Why only in the backward pass ? I think this is unefficient, maybe useless and could be left in FP32, what do you think ?
b)
why no usage of FP16 mixed prediction which could speed up training and inference ?
c) in REU function
The exponential is computed 2 times (forward, backward) over the whole range (positive and negative).
Is think is could be computed only 1 time, and only for the negative value, for efficiency
The text was updated successfully, but these errors were encountered: