-
Notifications
You must be signed in to change notification settings - Fork 228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tests for ck mha fp16 solver #3304
Add tests for ck mha fp16 solver #3304
Conversation
…into bharriso/add-tests-ck-mha-fp16
if(m_bernulliProbability > 0.0f) | ||
{ | ||
// Due to GPU version using a different dropout generator we will compare to CPU without | ||
// dropout and verify that dropout causes a large difference when comparing results. | ||
EXPECT_GT(oError, errorThreshold); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's only check which does make some sense with the current approach, but the ideal case is specifically prepared data which will allow us to check statistical properties of the dropout - tensor V must have identity matrixes, so it won't affect the data after dropout and we can say that approximately half of the elements are zero for m_bernulliProbability = 0.5
.
It's just a side note of how it can be actually checked.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh interesting idea.
CK also allows us to capture the dropout data.
I was thinking it would make sense to capture, and use that for verification, but I figured we could add that as part of enabling the additional optional params.
Purpose of review:
Note: