Rust based HTTP client as an alternative for got-scraping

[B4nan]

Feature

The got-scraping client is getting obsolete, and because of it being written in Node, supporting HTTP3 is not likely a possibility in the near future. We want to build a new client in rust to allow better low level control over the request as its fingerprint.

Once implemented for node, we also want to reuse the same client for the python version.

[barjin]

Not sure where to put the issues now (likely to https://github.com/retch-http/retch later), but I wanted to have it in one place first. Here's a list of the biggest points I can think of (Important are important, Tip are IMO optional).

Stealth

Important

• Patch h2 (and likely reqwest) to modify the h2 pseudoheaders order
• Impersonate the HTTP2 markers in h2 (see e.g. the Akamai paper).

Tip

• Automate fingerprint collecting
• Automatic CI tests, comparing our fingerprints to e.g. https://ja4db.com/

Technical

Important

• Add testing
• Investigate HTTP3 capabilities
• Remove bogus cipher implementations, use the actual implementations (e.g. Kyber from here or the deprecated ciphers we now fake here.

Distribution

Important

• Design Node.JS interface
• Setup Node.JS bindings, release npm package with prebuilt binaries
• Design Python interface
• Setup Python bindings, release npm package with prebuilt binaries

Tip

• Figure out MacOS code signing for the CLI tool.