opensubtitles.org blocks some downloads when using a VPN

[CristianKerr]

Doesn't matter if I download single file or whole directory, lot of subtitle files end up empty (not all).
e.g.
OpenSubtitlesDownload.py -u ****** -p ***** "Agents of SHIELD S02E11 Aftershocks.mp4"

but if I search with VLsub (tool in VLC) it finds and downloads them correctly.

I found out that when I force cli mode, i get this

>> OpenSubtitlesDownload encountered an unknown error, sorry about that...
Error: [type 'exceptions.IOError']
Line: 856

running with python3 gets me this:

>> OpenSubtitlesDownload encountered an unknown error, sorry about that...
Error: [class 'urllib.error.HTTPError']
Line: 846

What more details can I give you to find out what is happening?
Thank you

[emericg]

Well that's not good.
You can try to set opt_force_utf8 to False and see if the situation evolves.

It seems that the files aren't downloaded successfully, so maybe add the line print(subURL) around the line 843 and let me know what is printed.

else: # CLI
    print(">> Downloading '" + subtitlesResultList['data'][subIndex]['LanguageName'] + "' subtitles for '" + videoTitle + "'")
    print(">> Downloading '" + subURL + "'")

[CristianKerr]

I tried it and this is what I got

Downloading 'English' subtitles for '"Doctor Who" The Return of Doctor Mysterio'
>> Downloading 'https://dl.opensubtitles.org/en/download/src-api/vrf-19b80c53/sid-5ioridRGe5w49EHySi5Si8XXOh6/filead/1955415660.gz'
>> OpenSubtitlesDownload encountered an unknown error, sorry about that...
Error: [class 'urllib.error.HTTPError']
Line: 847

When I tried to go to the link in browser, I downloaded the subtitles correctly.

Then I realized I could try turn off VPN and it suddenly works even via your script. But how is it possible? With the VPN ON I was still able to download the link manually. And i am logged in so there should not be any limits. More over, some subs CAN be downloaded with your script over VPN every time, and some just don't.

Edit:
I tried aslo wget and this is the result

$ wget 'https://dl.opensubtitles.org/en/download/src-api/vrf-19b80c53/sid-5ioridRGe5w49EHySi5Si8XXOh6/filead/1955415660.gz'
--2021-02-22 20:42:21--  https://dl.opensubtitles.org/en/download/src-api/vrf-19b80c53/sid-5ioridRGe5w49EHySi5Si8XXOh6/filead/1955415660.gz
Resolving dl.opensubtitles.org (dl.opensubtitles.org)... ****, ...
Connecting to dl.opensubtitles.org (dl.opensubtitles.org)|***|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.opensubtitles.org/en/captcha2/redirect-%7Cen%7Cdownload%7Csrc-api%7Cvrf-19b80c53%7Csid-5ioridRGe5w49EHySi5Si8XXOh6%7Cfilead%7C1955415660.gz [following]
--2021-02-22 20:42:22--  http://www.opensubtitles.org/en/captcha2/redirect-%7Cen%7Cdownload%7Csrc-api%7Cvrf-19b80c53%7Csid-5ioridRGe5w49EHySi5Si8XXOh6%7Cfilead%7C1955415660.gz
Resolving www.opensubtitles.org (www.opensubtitles.org)... ***, ...
Connecting to www.opensubtitles.org (www.opensubtitles.org)|***|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: /en/captcha/redirect-%7Cen%7Cdownload%7Csrc-api%7Cvrf-19b80c53%7Csid-5ioridRGe5w49EHySi5Si8XXOh6%7Cfilead%7C1955415660.gz [following]
--2021-02-22 20:42:22--  http://www.opensubtitles.org/en/captcha/redirect-%7Cen%7Cdownload%7Csrc-api%7Cvrf-19b80c53%7Csid-5ioridRGe5w49EHySi5Si8XXOh6%7Cfilead%7C1955415660.gz
Reusing existing connection to www.opensubtitles.org:80.
HTTP request sent, awaiting response... 429 Unknown HTTP Status
2021-02-22 20:42:22 ERROR 429: Unknown HTTP Status.

[emericg]

Good catch about the VPN.
Your wget trace is interesting, the first url is modified by the server after the first download try, and then serves a Captcha. Maybe that's what happen with osd too, but the second url is not shown, only the error.
I googled and found this thread, https://forum.opensubtitles.org/viewtopic.php?f=1&t=14559 but no solution. Maybe try to validate the captcha at least once from your browser? It will probably whitelist your IP (well the one from your VPN, I don't know how often it changes). But maybe it was blacklisted by someone else using the VPN, I don't know...

I'm changing the title of the issue, maybe some people with similar situation can weigh in.

[Tony4dev]

Hi, I noticed this a bit ago, but didn't have time to look into this until now. I have created a PR #71 which changes download method from hard-coded URL to XmlRpc request.

[CristianKerr]

It doesn't help to do the captcha in browser.
i tried the changes from Tony and it seems to work for me now.

[emericg]

Yes it does seem to work, but there will be a bit of work to integrate this cleanly.
By the way what OS are you using?

[CristianKerr]

I am using Pop!_OS 20.10