Domain/pattern-based user agent overrides

[JustAnotherArchivist]

• Facebook doesn't like our UA.
• Twitter requires the AB UA for useful archival.
• Tumblr requires a browser UA.
• Flickr blocks AB UA requests with a 503.
• GitHub serves an ugly 'unsupported browser' banner if requested with the default AB UA.
• and so on...

While annoying, this can be handled for jobs explicitly targeting those sites using --useragent. However, outlinks from other sites might be screwed. I propose adding a mechanism that overrides the user agent on a per-request basis. If the request matches a regular expression, the UA is overridden for that request. As an optimisation, hostname matching would also be possible to skip expensive regex matching, possibly also with wildcards for any subdomain. This may require changes in wpull.