Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LoRA support #621

Open
4 tasks done
MithrilMan opened this issue Dec 12, 2024 · 3 comments
Open
4 tasks done

LoRA support #621

MithrilMan opened this issue Dec 12, 2024 · 3 comments
Labels
question Further information is requested

Comments

@MithrilMan
Copy link
Contributor

Checks

  • This template is only for question, not feature requests or bug reports.
  • I have thoroughly reviewed the project documentation and read the related paper(s).
  • I have searched for existing issues, including closed ones, no similar questions.
  • I confirm that I am using English to submit this report in order to facilitate communication.

Question details

Has been LoRA subject to the study?
Is it currently "pluggable" in current architecture? (e.g. tie it to the attention layer?)

Is this something we may expect in a future version?

Has someone tested this?

@MithrilMan MithrilMan added the question Further information is requested label Dec 12, 2024
@SWivid
Copy link
Owner

SWivid commented Dec 12, 2024

yes, it's just pluggable

@jpgallegoar
Copy link
Collaborator

Hello, in Oct 12, @SWivid you said SWivid commented on Oct 12
"Full finetune is currently supported, lora or adapter not yet." and now you say it is possible, did something change? Will you release a guide on how to do this? Have you tested it? I'm interested for accent LORAs in the same language

@SWivid
Copy link
Owner

SWivid commented Dec 16, 2024

Hi @jpgallegoar , lora is pluggable but not added yet
pr is always welcome

thought the dit blocks are wrapped there, add some lines code for common lora practice is fine
@Jerrister has got something with lora, so i think lora just fits in here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants