Regarding the Reference Attention Issue

[ShowLo]

Why comment out the self.bank.clear() inside the if MODE == "read": block in mutual_self_attention.py, and then perform the clear operation within the if accelerator.sync_gradients: block in train_stage1.py?

Won’t this lead to the bank accumulating multiple states in the interim? For instance, when the gradient accumulation steps are set to 3:
During the second forward pass, the bank would use the first hidden state.
During the third forward pass, the bank would use the first two hidden states.

[xumingw]

We only test on gradient accumulation = 1

[ShowLo]

We only test on gradient accumulation = 1

I have tested it, and when gradient accumulation > 1, it causes errors. It is necessary to perform the clear operation before each iteration begins.