Resolves huggingface#2206

NOT READY TO MERGE YET.

Tentative solution to that issue.

The problem is that we keep a "global" modules_to_save on the model
which contains all possible modules_to_save for each adapter. When the
first adapter targets layer "foo" with modules_to_save and the second
adapter targets "bar", then "foo" will create a copy of the original
module for the second adapter, even though it's not needed.

This does not change the result but is unnecessary and takes up memory.
Thus it should be avoided.

TODO: Tests.