Skip to content

Commit

Permalink
Add MoE FAQ about --clone-tensors option. (#296)
Browse files Browse the repository at this point in the history
Add this answer from this issue to a FAQ about MoE: #279
  • Loading branch information
PhilipMay authored May 4, 2024
1 parent 8d689e4 commit 46b432f
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions docs/moe.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,12 @@ experts:
- "<|im_start|>user\nWie geht es dir?<|im_end|>"
- "Das ist ein Satz auf Deutsch."
```

## FAQ

### What does the "Your model has duplicated tensors but the --clone-tensors flag is not set" warning mean?

Answer from [Charles O. Goddard (cg123)](https://github.com/cg123)
(also see [this GitHub issue](https://github.com/arcee-ai/mergekit/issues/279#issuecomment-2081818104)):

> This is completely benign. This happens when a single tensor from a model is used in multiple places, like when doing sparse upcycling with the moe script or doing passthrough merges that repeat layers. Having `--clone-tensors` set can use slightly more memory, but having it unset will slow down saving and introduce small memory usage spikes in cases where this warning occurs. It's honestly a small enough difference that the warning could be removed entirely.

0 comments on commit 46b432f

Please sign in to comment.