write files as they are done, rather than "don't write until everything is done"?

[Pomax]

It looks like right now every single page is kept in memory until the entire site has been mirrored, after which it writes everything to file. This means you can easily need 30GB of ram to fit everything in memory, and completely locks up a computer once things are done and filewriting starts happening.

Can this be changed to simply writing files to disk as they finish, before resolving all links?

[s0ph1e]

Hey @Pomax 👋

You are definitely right, now all content is stored in memory and this can be improved.
It's related to website-scraper module, not to website-scraper-phantom, so I've created an issue in main repo website-scraper/node-website-scraper#386 and closing this one

[Pomax]

Ah, I see. Thanks you for filing that issue, I will subscribe that it!