Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Another unmatch between code and paper #153

Open
lizhe1531 opened this issue Nov 7, 2024 · 0 comments
Open

Another unmatch between code and paper #153

lizhe1531 opened this issue Nov 7, 2024 · 0 comments

Comments

@lizhe1531
Copy link

image

Congrats for the great work!
I find another unmatch between code and paper. In picture of the paper, different scales of qkv will be sent to LiteMLA first, and then concatenated. But in code, I find they are concatenated first, and then sent to LiteMLA.
I wonder the reason for this design and how the results would be different if I sent them to LiteMLA first and then concatenated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant