Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

www.php.net blocks "fetch libfetch/2.0" user-agent #483

Open
CyberCr33p opened this issue Dec 26, 2021 · 7 comments
Open

www.php.net blocks "fetch libfetch/2.0" user-agent #483

CyberCr33p opened this issue Dec 26, 2021 · 7 comments

Comments

@CyberCr33p
Copy link

Description

This is what I get when I try to upgrade PHP from 7.4.26 to 7.4.27 using FreeBSD ports:

=> php-7.4.27.tar.xz doesn't seem to exist in /usr/ports/distfiles//.
=> Attempting to fetch https://www.php.net/distributions/php-7.4.27.tar.xz
fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz
fetch: http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles// and try again.

FreeBSD ports uses "fetch" to download https://www.php.net/distributions/php-7.4.27.tar.xz but looks like www.php.net web-server blocks user agent "fetch libfetch/2.0" and returns "Service Unavailable".

Can you whitelist this user agent?

PHP Version

PHP 7.4

Operating System

FreeBSD
@nikic nikic transferred this issue from php/php-src Dec 26, 2021
@cmb69
Copy link
Member

cmb69 commented Dec 26, 2021

curl -A "fetch libfetch/2.0" --output php-7.4.27.tar.xz https://www.php.net/distributions/php-7.4.27.tar.xz works for me. What am I missing?

@cmb69
Copy link
Member

cmb69 commented Dec 26, 2021

Is this possibly certificate related?

@dsh2dsh
Copy link

dsh2dsh commented Dec 26, 2021

I can confirm:

❯ fetch https://www.php.net/distributions/php-7.4.27.tar.xz
fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable

@olegsidorkin
Copy link

olegsidorkin commented Dec 26, 2021 •
edited
Loading

wget https://www.php.net/distributions/php-7.4.27.tar.xz - does work
fetch -vvvv https://www.php.net/distributions/php-7.4.27.tar.xz - doesn't. It successfully establishes connection and then fails:

Security CheckOur systems have detected unusual traffic from your computer network. This page checks to see if it is really you sending the requests and not a bot.Please click here to verify that you are not a bot. Browser Working Myra Working Host WorkingWhat happened?You ran into a security check to verify the validity of your request.What can I do?If you are a visitor of this website:You must confirm that you are human.If you are the owner of this website:Please check your security settings.

Headers are:
`resolving server address: www.php.net:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.php.net
Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
requesting https://www.php.net/distributions/php-7.4.27.tar.xz

GET /distributions/php-7.4.27.tar.xz HTTP/1.1
Host: www.php.net
Accept: /
User-Agent: fetch libfetch/2.0
Connection: close

<<< HTTP/1.1 503 Service Temporarily Unavailable
<<< Server: myracloud
<<< Date: Sun, 26 Dec 2021 16:05:06 GMT
<<< Content-Type: text/html
<<< Transfer-Encoding: chunked
<<< Connection: close
<<< cache-control: no-cache, no-store, max-age=0
<<< X-Frame-Options: SAMEORIGIN
<<< X-Content-Type-Options: nosniff
<<< X-XSS-Protection: 1; mode=block`

@cmb69
Copy link
Member

cmb69 commented Dec 26, 2021

Ah, so this would be a systems issue. Maybe @saschaschumann can have a look?

@5u623l20
Copy link

5u623l20 commented Dec 26, 2021 •
edited
Loading

I have tried changing user agent of fetch too with no results:
fetch -vvvvv --user-agent="curl/7.80.0" https://www.php.net/distributions/php-8.1.1.tar.xz
scheme: "https"
user: ""
password: ""
host: "www.php.net"
port: "0"
document: "/distributions/php-8.1.1.tar.xz"
---> www.php.net:443
resolving server address: www.php.net:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.php.net
Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
requesting https://www.php.net/distributions/php-8.1.1.tar.xz

GET /distributions/php-8.1.1.tar.xz HTTP/1.1
Host: www.php.net
Accept: /
User-Agent: curl/7.80.0
Connection: close

<<< HTTP/1.1 503 Service Temporarily Unavailable
<<< Server: myracloud
<<< Date: Sun, 26 Dec 2021 16:30:08 GMT
<<< Content-Type: text/html
<<< Transfer-Encoding: chunked
<<< Connection: close
<<< cache-control: no-cache, no-store, max-age=0
<<< X-Frame-Options: SAMEORIGIN
<<< X-Content-Type-Options: nosniff
<<< X-XSS-Protection: 1; mode=block
<<<
offset 0, length -1, size -1, clength -1
<<< 7738

freebsd-git pushed a commit to freebsd/freebsd-ports that referenced this issue Dec 26, 2021
@5u623l20
Fix fetch for php*
529117f 
- Primary hosting provider of php has introduced some sort of checking
  which is blocking fetch although curl/wget works perfectly. fetch with
  other --user-agent is also not working at the moment. An issue has been
  opened with the upstream which can be found at:
  php/web-php#483
- Untill this is fixed add php Github distributions URL to MASTER_SITES
- Modify MASTER_SITES for php[73|74|80] and remove SUBDIR as it affects
  DIST_SUBDIR
- Add DIST_SUBDIR as Github is IPv4 only which affects IPv6 only
  builders. In case someone is running IPv6 only builders distcache will
  pickup and serve.
- Although distinfo has been updated as we have added DIST_SUBDIR but
  SHA256 and SIZE are unchanged. In case someone needs to verify those
  are available from https://www.php.net/downloads
- We might have done some drastic changes but we are on the brink of
  quarterly build and we do not want to start a new year and a new
  quarter with BROKEN php and thousands of other ports.

PR: 260699
Reported by: [email protected]
Approved by: portmgr (blanket infrastructure)
Sponsored by: Bounce Experts
@5u623l20
Copy link

In case someone is already investigating I believe the problem is with http1.1 vs http2. In case we try curl with --http1.1 it also fails with the same errors.

netgate-git-updates pushed a commit to pfsense/FreeBSD-ports that referenced this issue Jan 4, 2022
@5u623l20
Netgate note:
26f4a58 
	Cherry-pick 529117f.  Revert this before next upstream sync

Fix fetch for php*

- Primary hosting provider of php has introduced some sort of checking
  which is blocking fetch although curl/wget works perfectly. fetch with
  other --user-agent is also not working at the moment. An issue has been
  opened with the upstream which can be found at:
  php/web-php#483
- Untill this is fixed add php Github distributions URL to MASTER_SITES
- Modify MASTER_SITES for php[73|74|80] and remove SUBDIR as it affects
  DIST_SUBDIR
- Add DIST_SUBDIR as Github is IPv4 only which affects IPv6 only
  builders. In case someone is running IPv6 only builders distcache will
  pickup and serve.
- Although distinfo has been updated as we have added DIST_SUBDIR but
  SHA256 and SIZE are unchanged. In case someone needs to verify those
  are available from https://www.php.net/downloads
- We might have done some drastic changes but we are on the brink of
  quarterly build and we do not want to start a new year and a new
  quarter with BROKEN php and thousands of other ports.

PR: 260699
Reported by: [email protected]
Approved by: portmgr (blanket infrastructure)
Sponsored by: Bounce Experts
@yoanm yoanm mentioned this issue Mar 19, 2022
Closed
@TinfoilSubmarine TinfoilSubmarine mentioned this issue Sep 30, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dsh2dsh @cmb69 @5u623l20 @olegsidorkin @CyberCr33p