Poor result of MEMIT over llama3.2 3B

[rabi-fei]

Thank you for the wonderful jobs and provide the hyper-meters of MEMIT over llama3.2 3B.

However, my edit accuracy under 200 facts is low that 50%, while the MEMIT over gpt2-xl have 99% accuracy under 2000 facts. Do you have any tips, thank you very much?

[XeeKee]

It's strange, I've been working on fixing MEMIT multi-gpu errors recently, and I'll debug this issue as well.

[rabi-fei]

Thank you for your response. To reduce GPU memory usage, I have adjusted the maximum token sequence length in the layer stat file to 512. I'm not sure if this change has negatively affected the results. Looking forward your response after debug.

[XeeKee]

Hi, my download speed for wikipedia/20200501.en is very slow. Do you have a precomputed cache of Wikipedia (with the file extension .npz) so that I can debug and help you identify the problem faster?

[rabi-fei]

Sure. Upload the file will cost a few hours.

[rabi-fei]

I upload my npc file in the following link:
https://drive.google.com/file/d/1-7BiEG3hZKFBd_Kz3YTgEttGjAHLJydn/view?usp=sharing

By the way, I guess you can use hf-mirror to load the wikipedia/20200501.en.
link: https://hf-mirror.com

[XeeKee]

I'm glad to have received your npz file. We are currently debugging the issue, and we will inform you promptly once we have results.