High memory usage

[francoishernandez]

Hi there,
Thanks for the nice papers and implementation!
As part of a few set of experiments I'm trying to run this on a custom dataset with around 1500 recordings, accounting for a total of 100 hours of speech.
Though, it seems to use way too much RAM. It filled 64G of RAM, and almost 64G of swap, and I had to stop to prevent the system to freeze.
Is this a known behavior?
Are there some workarounds to limit memory usage?
Thanks!

Note: the mini_librispeech recipe seems to run properly though, using around 2G of memory. There might be some issues with my setup/data.