CI Fix Windows permission error on merge test (#1952)

For some reason, Windows CI suddenly started throwing permission
errors on test_merge_layers. These errors occur when using the
TempDirectory() context manager, which raises a PermissionError on
Windows when it tries to clean up after itself. Therefore, this context
manager is now avoided in favor of manual clean up.

More context:

I investigated this issue first in #1947. My suspicion that this could
be caused by a new pytest version was not confirmed. Maybe the reason is
that GH rolled out a new Windows worker, not sure.

Also note that this is not the first time that this workaround is
required, e.g. also here:

https://github.com/huggingface/peft/blob/e6cd24c907565040ee1766a5735afe3d13a71164/tests/test_custom_models.py#L1465

tests/testing_common.py

@@ -16,6 +16,7 @@
 import os
 import pickle
 import re
+import shutil
 import tempfile
 from collections import OrderedDict
 from dataclasses import replace
@@ -621,9 +622,17 @@ def _test_merge_layers(self, model_id, config_cls, config_kwargs):
         # test that the logits are identical after a save-load-roundtrip
         if hasattr(model, "save_pretrained"):
             # model is a transformers model
-            with tempfile.TemporaryDirectory() as tmp_dirname:
+            tmp_dirname = tempfile.mkdtemp()
+            # note: not using the context manager here because it fails on Windows CI for some reason
+            try:
                 model.save_pretrained(tmp_dirname)
                 model_from_pretrained = self.transformers_class.from_pretrained(tmp_dirname).to(self.torch_device)
+            finally:
+                try:
+                    shutil.rmtree(tmp_dirname)
+                except PermissionError:
+                    # windows error
+                    pass
         else:
             # model is not a transformers model
             model_from_pretrained = pickle.loads(pickle.dumps(model))