Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix tiktok download #20

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

GreyAlien502
Copy link

Currently, tiktok downloads fail with status 403.
This request fixes that by downloading the page twice before trying the video.
This seems like a bug on their end since i noticed the same problem in the webapp:
the request for the video file fails with 403 when opening a video in a fresh incognito session, but it works when i reload.

Copy link

@merval merval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the approach, but I don't think you need to assign the webpage variable twice. simply calling self._download_webpage does the trick

youtube_dlc/extractor/tiktok.py Outdated Show resolved Hide resolved
@blackjack4494
Copy link
Owner

Unfortunately there is no retry option for downloading webpages or info files like in json.
However the mentioned approach here is poorly implemented.
I don't have any problems even trying multiple times. You could at most implement a solution that checks the http status and retries again after a certain timeout.
Currently you force even users without problems to download the page twice.
Also have you tried something like --force-ipv4 or --force-generic?

@merval
Copy link

merval commented Oct 27, 2020

@GreyAlien502 We should look at implementing what @blackjack4494 is referring to. Check the page status code, and call a second time if we get 503 back.

@GreyAlien502
Copy link
Author

@blackjack4494, We aren't retrying anything here,
especially not the webpage/info file download, which never fails,
The second call is preemptive, to avoid getting a 403 later, when we get the video.

I don't have any problem

Oh, i wasn't aware that the current code was working for anyone (without providing cookies).
It always fails for me.

have you tried something like --force-ipv4 or --force-generic

I just tried --force-generic, and it sometimes works for me.
Since it makes multiple requests for each resource it downloads,
it's not too surprising that it would.

@GreyAlien502
Copy link
Author

GreyAlien502 commented Oct 27, 2020

@merval, an InfoExtractor will not know that there's a problem
since the error doesn't occur during the information extraction, right?

Comment on lines +137 to 138
self._download_webpage(url, video_id)
webpage = self._download_webpage(url, video_id, note='Downloading video webpage')
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change this so that it won't download the webpage twice if not necessary. You can check the result and implement some try again but only once if response isn't satisfying.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If anyone can provide a way of determining if the response is satisfying, i can add such a check.
I don't see any meaningful difference between the first and second responses i receive.
Is there a specific reason why you think such a check would be possible?

@shiwei87tan
Copy link

tiktok download is down again.

U:\youtube-dlc -i -o "%(id)s.mp4" -v https://www.tiktok.com/@tiktok_stage/video/6885152542095887617
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-i', '-o', '%(id)s.mp4', '-v', 'https://www.tiktok.com/@tiktok_stage/video/6885152542095887617']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-08-07-fab00b0, ffprobe git-2020-08-07-fab00b0
[debug] Proxy map: {}
[TikTok] 6885152542095887617: Downloading video webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-web.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/9b3784d6dec84c75b0233ca0d90572c5/?a=1988&br=1272&bt=636&cr=0&cs=0&cv=1&dr=3&ds=3&er=&expire=1604852024&l=202011081013320101151511041701C791&lr=tiktok&mime_type=video_mp4&policy=2&qs=0&rc=M2x3OWhzaTtpeDMzaTgzM0ApZjhoZzczPGU7NzdnNGc5O2dtLmhoaGtxZzNfLS02LzRzc2JeNGM2MzUxLjAzXzVjMDM6Yw%3D%3D&signature=7773bd3019bd845e9f8ee935e9ef478b&tk=tt_webid_v2&vl=&vr='
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "youtube_dlc\YoutubeDL.py", line 2011, in process_info
File "youtube_dlc\YoutubeDL.py", line 1866, in dl
File "youtube_dlc\downloader\common.py", line 373, in download
File "youtube_dlc\downloader\http.py", line 349, in real_download
File "youtube_dlc\downloader\http.py", line 114, in establish_connection
File "youtube_dlc\downloader\http.py", line 110, in establish_connection
File "youtube_dlc\YoutubeDL.py", line 2316, in urlopen
File "urllib\request.py", line 523, in open
File "urllib\request.py", line 632, in http_response
File "urllib\request.py", line 561, in error
File "urllib\request.py", line 494, in _call_chain
File "urllib\request.py", line 641, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

@GreyAlien502
Copy link
Author

btw, this was fixed in ytdl-org/youtube-dl by adding the extra call to tiktok in the _real_initialize method.

@pukkandan pukkandan mentioned this pull request Jan 7, 2021
pukkandan referenced this pull request in yt-dlp/yt-dlp Jan 7, 2021
Authored by GreyAlien502
@pukkandan
Copy link
Contributor

pukkandan commented Jan 14, 2021

@tanliang1 Provide the full verbose output (with -v)

Also, you might want to try it on the master of yt-dlp. I have applied a tiny patch on top of this PR

pukkandan added a commit to yt-dlp/yt-dlp that referenced this pull request Jan 14, 2021
siikamiika pushed a commit to siikamiika/yt-dlc that referenced this pull request Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants