Should we add tooling for a search + load mementos pipeline?

[Mr0grog]

We have a lot of code we left behind in web-monitoring-processing around knitting together the search and get_memento methods in a high-performance way across many threads. I think it’s good that we didn’t include any of that here to start with, but would it make sense to add some of that back in eventually?

i.e. Some tooling that supports the workflow:

┌──────────────────┐   ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐   ┌─────────────────┐
│   Sequence of    │     ┌─────────────┐   ┌─────────────┐     │   Sequence of   │
│     URLs/URL     │──▶│ │   search    │──▶│ get_memento │ │──▶│    Mementos     │
│     Patterns     │     └─────────────┘   └─────────────┘     │                 │
└──────────────────┘   └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘   └─────────────────┘

It should be significantly more abstract than the way it is currently implemented in https://github.com/edgi-govdata-archiving/web-monitoring-processing/blob/master/web_monitoring/cli.py, of course. 😉