Can redis/this infrastructure be used for stateful crawl?

[birdsarah]

We would, I think, be saving a version of the profile between each site.

Perhaps overwriting after, say 10.

[motin]

As long as the profiles are stored outside the container (in S3, redis or similar) this architecture does not limit stateful crawls. This should be considered when general support for stateful crawls are restored: openwpm/OpenWPM#423
@englehardt What do you think?