Fix broken lists.linuxfoundation.org URLs

[kanzure]

This pull request updates lists.linuxfoundation.org urls to gnusha.org/url.

Background

Unfortunately, lists.linuxfoundation.org is no longer hosting the bitcoin-dev static email archives, nor any others for that matter. This has caused many links around the web to become broken.

Redirect web service

gnusha.org/url is a web service that redirects old style archive urls to a new location where the content can be viewed.

For example, a bitcoin-dev mailing list archive url like:

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

using https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

is redirected to:

https://gnusha.org/pi/bitcoindev/[email protected]/

The redirect mappings were generated from my archives and you can review the scripts I made to do that here:
https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b

Alternative

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

Those links would look like:

https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/
https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/

If this is the preferred solution for the BIPs, then let me know how I can help. Take a look here.

Finally, one other alternative is to wait and do nothing. Maybe someone can convince Linux Foundation to put up the static HTML archives for posterity?

More background

For more information see:

• https://gnusha.org/url
• doc: Update the developer mailing list address. bitcoin#29782 (comment)
• doc: Fix dead links to mailing list archives bitcoin#31240
• Fix broken URLs to lists.linuxfoundation.org bitcoinops/bitcoinops.github.io#2018
• chore(transcripts): fix all lists.linuxfoundation.org urls bitcointranscripts/bitcointranscripts#566
• https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/
• https://x.com/kanzure/status/1853779672514826334

[jlopp]

tACK

[jonatack]

Nice, concept ack.

Looks like the http urls need updating.

bip-0099.mediawiki:11:  Post-History: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/008936.html
bip-0101.mediawiki:90:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/008820.html Simulations show] that with the current peer-to-peer protocol, miners behind high-latency or low-bandwidth links are at a disadvantage compared to miners connected to a majority of hashpower via low-latency, high-bandwidth links. Larger blocks increase the advantage of miners with high-bandwidth connections, although that advantage can be minimized with changes to the way new blocks are announced (e.g. http://bitcoinrelaynetwork.org/ ).
bip-0106.mediawiki:55:These two proposals have been derived after discussion on [https://bitcointalk.org/index.php?topic=1154536.0 BitcoinTalk] and [http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010285.html bitcoin-dev mailing list]. The original idea and its evolution in the light of various arguments can be found [http://upalc.com/maxblocksize.php here].
bip-0111.mediawiki:30:to no privacy<ref>http://eprint.iacr.org/2014/763</ref>, as well as being a large DoS risk on some nodes<ref>[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2013-July/003044.html] is one example where the issues were found, though others independently discovered issues as well. Sample DoS exploit code available at https://github.com/petertodd/bloom-io-attack.</ref>.
bip-0112.mediawiki:381:[http://lists.linuxfoundation.org/pipermail/lightning-dev/2015-July/000021.html HTLCs using OP_CHECKSEQUENCEVERIFY/OP_LOCKTIMEVERIFY and revocation hashes]
bip-0112.mediawiki:389:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010396.html Softfork deployment considerations]
bip-0113.mediawiki:119:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010396.html Softfork deployment considerations]
bip-0124.mediawiki:13:  Post-History: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011795.html

[kanzure]

@jonatack, fixed that. thanks.

I am seeking feedback on gnusha.org/url/ vs resolving everything to urls like https://gnusha.org/pi/bitcoindev/[email protected]/

With the resolved urls, the IDs are more verbose-- instead of an integer ID for the message, it's the Message-ID from the actual email. Then in the future someone can use that Message-ID to find the email on a mirror, like https://mailing-list.bitcoindevs.xyz/bitcoindev/[email protected]/.

This seems to be the likely direction that bitcoin.git is going in. They only have four urls in the codebase.

[kanzure]

I'll elaborate on gnusha.org/url vs gnusha.org/pi/bitcoindev style urls.

It is nice and simple to use a find-and-replace with gnusha.org/url/ because anyone can easily repeat that exercise and verify the git diff before merging the pull request. However, the actual routing is not something that is verified by just checking that the git diff looks sane.

If I was to provide my mapping (I haven't uploaded this yet, but I did publish the scripts I used to make it), then someone could write a script and/or cross-verify that all of the replacements are valid according to the mapping, if we were to use the destination urls instead of the redirect service for this pull request. Then someone would be able to verify that kind of pull request.

There are other further issues though, like content malleability, wrong redirect link injection, service provider compromise... instead of referencing by Message-ID maybe in the future we can reference by email hash.... Just dreaming here.

[kanzure]

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

this is implemented here:
https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py

LMK if that's the preference instead of the current version of this pull request (gnusha.org/url inserts).

[jonatack]

Thanks @kanzure. No strong opinion, but I think I mildly prefer the destination urls over the redirects.

[murchandamus]

LGTM.

I also don’t have a strong opinion regarding the two approaches. I’d prefer a quicker reestablishing functionality over a delayed perfect solution.

[kanzure]

There would be no delay. Either will work now. See the gist.

[ajtowns]

The lists.linuxfoundation.org site seems to redirect automatically now:

$ curl -I https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html
HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html

(also applies to lightning-dev mails)

Is there still a good reason to churn a bunch of bip texts, rather than just restrict this to updating the mailing list location in bip 2?

[kanzure]

The lists.linuxfoundation.org site seems to redirect automatically now:

I haven't announced this yet. It's a strange situation because LF has not agreed to keep these redirects (perhaps they don't even know at the moment about these redirects), and LF is not running that server. Given the strangeness I haven't been sure what to say or announce precisely.