Home

Wget2 Introduction

The development of Wget2 started and everybody is invited to contribute, test, discuss, etc.
The codebase is hosted in the 'wget2' branch of wget's git repository and on github - both will be regularly synced.

Wget2 on Savannah (checkout branch 'wget2' afte cloning)

Wget2 on Github

The idea is to have a fresh and maintainable codebase with features like multithreaded downloads, HTTP2, OCSP, HSTS, Metalink, IDNA2008, Public Suffix List, Multi-Proxies, Sitemaps, Atom/RSS Feeds, compression (gzip, deflate, lzma, bzip2), support for local filenames, etc.
Some of these feature have been built into Wget in the meantime, but some other are really hard to implement into the old codebase.

Most of the functionality is exposed via library API (libwget), to allow external programs make use of it. E.g. have a look at examples/print_css_urls.c - just a few lines of C to parse and print out all URLs from a CSS file.

Wget2 will stay as an own executable separate from Wget.
So you can install and test Wget2 without endangering your existing architecture and scripts.

What is missing

• FTP(S) support
• WARC support
• Several Wget options are missing.
• HTTP/2 pipelining is missing
• API documentation

New options

--force-css         Treat input file as CSS. (default: off)
--force-sitemap     Treat input file as Sitemap. (default: off)
--force-atom        Treat input file as Atom Feed. (default: off)
--force-rss         Treat input file as RSS Feed. (default: off)
--max-threads       Max. concurrent download threads. (default: 5)
--gnutls-options    Custom GnuTLS priority string. Interferes with --secure-protocol. (default: none)
--ocsp-stapling     Use OCSP stapling to verify the server's certificate. (default: on)
--ocsp              Use OCSP server access to verify server's certificate. (default: on)
--ocsp-file         Set file for OCSP chaching. (default: .wget_ocsp)
--http2             Use HTTP/2 protocol if possible. (default: on)
--input-encoding    Character encoding of the file contents read with --input-file. (default: local encoding)
--cookie-suffixes   Load public suffixes from file. They prevent 'supercookie' vulnerabilities.
--chunk-size        Download large files in multithreaded chunks. (default: 0 (=off))
                    Example: wget --chunk-size=1M
--check-hostname    Check the server's certificate's hostname. (default: on)

• new 'include' statement for config files, e.g. to load /etc/wget/conf.d/*.conf
• --input-file - (reading URLs from stdin) starts downloading with the first URL to allow slow URL generators feed Wget2
• check HTTP 'ETag' to avoid parsing doublettes
• use HTTP 'Accept-Encoding': gzip, deflate, lzma, bzip2
• CLI string options can be set to NULL by prepending a --no-, e.g. --no-user-agent
• boolean CLI options can all be set to true or false
• $WGETRC is not read so far

Differing CLI options Wget/Wget2

Option	Wget	Wget2	Comment
--accept-regex	✓
--ask-password	✓
--auth-no-challenge	✓
--background	✓
--body-data	✓
--body-file	✓
--check-hostname		✓
--chunk-size		✓
--config	✓
--content-on-error	✓
--continue	✓
--convert-file-only	✓
--continue-download		✓
--cookie-suffixes		✓
--default_page		✓
--dns-caching		✓
--exclude-directories	✓
--egd-file	✓	✓	A Noop for compatibility (GnuTLS can be compiled/configured to use EGD)
--follow-ftp	✓
--force-atom		✓
--force-css		✓
--ftp-password	✓
--ftps-clear-data-connection	✓
--ftps-fallback-to-ftp	✓
--ftps-implicit	✓
--ftps-resume-ssl	✓
--ftp-user	✓
--glob	✓
--header	✓
--force-rss		✓
--force-sitemap		✓
--gnutls-options		✓
--http2		✓
--http-proxy		✓
--https-proxy		✓
--if-modified-since	✓		Wget2 uses If-Modified-Since when timestamping is turned on
--ignore-length	✓
--include-directories	✓
--input-encoding		✓
--limit-rate	✓
--method	✓
--num-threads		✓
--ocsp		✓
--ocsp-file		✓
--ocsp-stapling		✓
--passive-ftp	✓
--preserve-permissions	✓
--proxy-password	✓
--proxy-user	✓
--random-file		✓
--regex-type	✓
--rejected-log	✓
--reject-regex	✓
--relative	✓
--remove-listing	✓
--report-speed	✓
--retr-symlinks	✓
--retry-connrefused	✓
--robots		✓
--show-progress	✓
--start-pos	✓
--tcp-fastopen		✓
--unlink	✓
--warc-cdx	✓
--warc-compression	✓
--warc-dedup	✓
--warc-digests	✓
--warc-file	✓
--warc-header	✓
--warc-keep-log	✓
--warc-max-size	✓
--warc-tempdir	✓