RedgifsRipper: Implemented redgifs.com Ripper #1672

borderline232 · 2020-05-23T18:49:34Z

Description

Using similar functionality found in the gfycat ripper the same was done for the redgifs ripper
A user's profile can be ripped, by retrieving all gfycats from their user api call
The categories and search ripper is also implemented but uses and incrementing start and count in the api url paramters as the cursor requires cookies (I believe) to work and this one just increments until there are no gfycats and returns error (there are no more gfycats)
Differentiated between getting profile and search urls using regex matching

COMMIT-2

Changed redgifs api URL to reflect updated change
Added check in getNextPage to see if page has any URLs else finishes rather than throwing error
Added gifdeliverynetwork.com as known host as it redirects to redgifs page
Added reddit compatibility by adding as known host in RipUtils reddit ripper thanks to @kepler-421b
Unit test to cover above changes

Testing

Required verification:

I've verified that there are no regressions in mvn test (there are no new failures or errors).
I've verified that this change works as intended. (Stops at error when ripping a search/category)
- Downloads all relevant content.
- Downloads content from multiple pages (as necessary or appropriate).
- Saves content at reasonable file names (e.g. page titles or content IDs) to help easily browse downloaded content.
I've verified that this change did not break existing functionality (especially in the Ripper I modified).

Optional but recommended:

I've added a unit test to cover my change.

- Using similar functionality found in the gfycat ripper the same was done for the redgifs ripper - A user's profile can be ripped, by retrieving all gfycats from their user api call - The categories and search ripper is also implemented but uses and incrementing start and count in the api url paramters as the cursor requires cookies (I believe) to work and this one just increments until there are no gfycats and returns error (there are no more gfycats) - Differentiated between getting profile and search urls using regex matching

borderline232 · 2020-05-23T19:11:01Z

Seems like there are build issues with other rippers, which is breaking maven

Addresses - #1654

borderline232 · 2020-05-30T06:01:08Z

@cyian-1756 @felixaufreisen could you guys take a look at my PR and let me know, if there's anything I can fix. Thanks!

revanwolf · 2020-06-02T19:54:08Z

Someone implemented your solution on a fork apparently.

I compiled it and it work (but some files are corrupted but it seems it's redgif fault).

borderline232 · 2020-06-03T01:39:36Z

@revanwolf Thanks, for checking it out, yeah it seems that person just added a RipUtils logging commit not sure what that does, aside from logging.

Also, it seems that redgifs has updated some of their api urls, so I'll update my PR to reflect that, thanks for catching that.

I'm still not sure who merges the pull requests into master, I'm assuming its @cyian-1756 as that's the name usually associated with the merged PRs, so hopefully he see's this and can merge it.

revanwolf · 2020-06-03T05:14:09Z

@revanwolf Thanks, for checking it out, yeah it seems that person just added a RipUtils logging commit not sure what that does, aside from logging.

I think it's for the reddit ripper so it can handle redgif link and he only copy/paste the Gfycat part.

Anon1337Elite · 2020-06-05T19:39:40Z

Some gfy links redirect to gifdeliverynetwork.com. Which makes RipMe to fail.
#1680

borderline232 · 2020-06-05T21:22:48Z

Some gfy links redirect to gifdeliverynetwork.com. Which makes RipMe to fail.

@BestRestUp3300 Could you give me an example?

Anon1337Elite · 2020-06-06T01:44:28Z

Some gfy links redirect to gifdeliverynetwork.com. Which makes RipMe to fail.

@BestRestUp3300 Could you give me an example?

Warning, NSFW: Reddit thread:
https://www.reddit.com/r/NSFW_GIF/comments/ennwsa/gorgeous_tits/

Link:

https://gfycat.com/foolishelasticchimpanzee

- Changed redgifs api URL to reflect updated change - Added check in getNextPage to see if page has any URLs else finishes rather than throwing error - Added gifdeliverynetwork.com as knwon host as it redirects to redgifs page - Added reddit compatibility by adding as known host in RipUtils reddit ripper - Unit test to cover above changes

borderline232 · 2020-06-07T04:21:10Z

I've updated the PR, if anyone wants to test it out. Also @BestRestUp3300 it looks like the gfy redirect issue should be another PR related more towards gfycat rather than redgifs.

revanwolf · 2020-06-09T05:58:56Z

I've updated the PR, if anyone wants to test it out.

I've tested with the Reddit ripper and it seems to work most of the time but when the download fail it seem to be Redgifs fault.

borderline232 · 2020-06-13T19:11:43Z

I've tested with the Reddit ripper and it seems to work most of the time but when the download fail it seem to be Redgifs fault.

Yeah it seems that the redgifs api isn't very stable so it'll kick you out if you make too many calls, which I think is what ends up happening as they just start rejecting all the api calls we are making. (i.e. - you'll notice it reject one vid with a 403 error and then every video after will also give a 403).

Not much we can do about that atm, until redgifs gets a more stable api we can scrape through.

revanwolf · 2020-06-13T20:19:03Z

One solution is to download by default the webm version of the file (the file is usually lighter than the mp4).

With Chrome i got" network error" randomly and a lot of retry on ripme(no 403 error only corrupt file).

cyian-1756 · 2020-06-19T17:40:07Z

LGTM

avekifes · 2020-06-20T06:19:06Z

Is there a possibility of this not being added due to the API issue outlined above?

borderline232 · 2020-06-20T18:03:27Z

@avekifes It isn't currently in the newest release https://github.com/ripmeapp/ripme/releases but it is in the updated master of the repo, so if you want to use it atm then you can clone the repo and create a jar file by running the application.

@cyian-1756 When will you be adding a new release for the ripper?

CaptRussia · 2020-06-26T22:17:41Z

@borderline232 Regarding your pull request and implementation, I was unable to rip redgif links from a redditor's profile (Individual redgif links do work however). I built and tested the functionality on the current master branch. Have you come across this issue as well?

avekifes · 2020-06-26T23:08:23Z

@borderline232 What's the profile? Seems to be working fine for me.

CaptRussia · 2020-06-27T08:21:43Z

I'm experiencing this issue with this account (NSFW): https://www.reddit.com/user/supersecretself/

The url's direct to gifdeliverynetwork.com which I believe is used for gyfcat videos that are moved to redgif network when there's no redgif user associated with it. If you could confirm it's working and is an issue on my end I'd appreciate it.

Did you build from the master or from borderline's branch?

avekifes · 2020-06-27T16:36:26Z

Doesn't work for me either. I wasn't even aware that older NSFW Gfycat stuff would automatically redirect to Redgifs.

CaptRussia · 2020-06-27T16:49:31Z

Thanks for validating the issue @avekifes

@borderline232 is this something you'll be able to fix?

There's actually a current issue related to this that's open. #1680

borderline232 · 2020-06-27T19:46:05Z

@CaptRussia So this issue is that gfycat redirects the links to gifdeliverynetwork.com which I have an individual redirect for (i.e. you rip using the gifdeliverynetowrk link and it downloads the associated redgifs video),

the thing with ripping a reddit profile which contains gfycat links that redirect to gifdeliverynetwork requires somehow modifying either RipUtils.java file or the DownloadVideoThread.java file where I believe? it handles the redirects when downloading videos, not too sure tbh. I can take a look at it again to see if there's a fix for it.

revanwolf · 2020-06-27T21:25:22Z

Funny thing is the old version 1.7.67 can download gifdeliverynetwork.com link without much issues but this version can't download Gfycat(change on the API/format IIRC) and of course RedGifs.

borderline232 · 2020-06-27T21:43:59Z

Oh damn, you're right. Thanks I'll take a look at it.

New fix available in #1695 depending on how long @cyian-1756 takes to merge it, it's best to clone my fork https://github.com/borderline232/ripme if you want the changes immediately. Let me know if there's any issues.

revanwolf · 2020-07-01T06:32:40Z

Let me know if there's any issues.

It works and the only issues is their website is running on a potato.

CaptRussia · 2020-07-01T10:08:11Z

Thanks for rolling out a fix so quickly @borderline232

I just compiled your fork and am sadly getting errors. I tried the same URL (reddit profile) I posted before and managed to download only 2 videos. The logs seems to suggest it's still trying to download from redgifs. I assume it redirects to redgif and then redirects again to gifdeliverynetwork.com.

Here's the output:

Downloading https://thcf7.redgifs.com/ThoroughAdorableDanishswedishfarmdog.mp4
Downloading https://thcf4.redgifs.com/FormalImmaterialDrake.mp4
Downloading https://thcf7.redgifs.com/SoreThirdGnat.mp4
Downloading https://thcf6.redgifs.com/PlayfulKeyFirecrest.mp4
Downloading https://thcf8.redgifs.com/LegitimateHighAmericangoldfinch.mp4
Downloading https://thcf7.redgifs.com/ThoroughAdorableDanishswedishfarmdog.mp4
Downloading https://thcf4.redgifs.com/FormalImmaterialDrake.mp4
Downloading https://thcf7.redgifs.com/SoreThirdGnat.mp4
Downloading https://thcf6.redgifs.com/PlayfulKeyFirecrest.mp4
Downloading https://thcf8.redgifs.com/LegitimateHighAmericangoldfinch.mp4
https://thcf7.redgifs.com/ThoroughAdorableDanishswedishfarmdog.mp4 : Failed to download https://thcf7.redgifs.com/ThoroughAdorableDanishswedishfarmdog.mp4
Downloading https://thcf3.redgifs.com/SeveralVictoriousAfghanhound.mp4
https://thcf7.redgifs.com/SoreThirdGnat.mp4 : Failed to download https://thcf7.redgifs.com/SoreThirdGnat.mp4
https://thcf4.redgifs.com/FormalImmaterialDrake.mp4 : Failed to download https://thcf4.redgifs.com/FormalImmaterialDrake.mp4
https://thcf6.redgifs.com/PlayfulKeyFirecrest.mp4 : Failed to download https://thcf6.redgifs.com/PlayfulKeyFirecrest.mp4
https://thcf8.redgifs.com/LegitimateHighAmericangoldfinch.mp4 : Failed to download https://thcf8.redgifs.com/LegitimateHighAmericangoldfinch.mp4
Downloading https://i.redd.it/6ru60nv6m1q21.jpg
Downloading https://thcf7.redgifs.com/RecentPrestigiousApisdorsatalaboriosa.mp4
Downloading https://i.redd.it/27ct7jdr0yf21.jpg
Downloading https://thcf4.redgifs.com/ScalyEvergreenHarvestmouse.mp4
Downloaded .\rips\reddit_user_supersecretself\apgdi2-I_m_a_creep__f_-27ct7jdr0yf21.jpg
Downloading https://thcf2.redgifs.com/DescriptivePracticalBlacklab.mp4
Downloaded .\rips\reddit_user_supersecretself\b8xble-Good_morning_____f_-6ru60nv6m1q21.jpg
Downloading https://i.redd.it/njnx85n1k1421.jpg
Downloaded .\rips\reddit_user_supersecretself\a5t6xc-Grab_a_hand_f_ul___I_dare_you_-njnx85n1k1421.jpg
Downloading https://thcf7.redgifs.com/RecentPrestigiousApisdorsatalaboriosa.mp4
Downloading https://thcf4.redgifs.com/ScalyEvergreenHarvestmouse.mp4
Downloading https://thcf2.redgifs.com/DescriptivePracticalBlacklab.mp4
https://thcf7.redgifs.com/RecentPrestigiousApisdorsatalaboriosa.mp4 : Failed to download https://thcf7.redgifs.com/RecentPrestigiousApisdorsatalaboriosa.mp4
https://thcf4.redgifs.com/ScalyEvergreenHarvestmouse.mp4 : Failed to download https://thcf4.redgifs.com/ScalyEvergreenHarvestmouse.mp4
Downloading https://i.redd.it/t0d9n3tqst221.jpg
Downloading https://thcf2.redgifs.com/PepperyOrdinaryJaeger.mp4
https://thcf2.redgifs.com/DescriptivePracticalBlacklab.mp4 : Failed to download https://thcf2.redgifs.com/DescriptivePracticalBlacklab.mp4
Downloading https://thcf2.redgifs.com/ThickFailingCat.mp4
Downloading https://i.redd.it/26nlmaqbrr401.jpg
Downloaded .\rips\reddit_user_supersecretself\7kpb0l-Panties_on_or_of_f__-26nlmaqbrr401.jpg
Downloading https://i.redd.it/vi0z49qjnr401.jpg
Downloaded .\rips\reddit_user_supersecretself\a3yedk-My_ass_is_getting__f_at_but_it_s_all_good__-t0d9n3tqst221.jpg
Downloading https://i.vidble.com//pS5RG2ixZP.jpeg
Downloaded .\rips\reddit_user_supersecretself\7kp6dz-_f__Mondays_can_suck_my_tit-vi0z49qjnr401.jpg
Downloading https://i.vidble.com//Sm3fp7r5Hm.jpeg
Downloading https://thcf2.redgifs.com/ThickFailingCat.mp4
Downloaded .\rips\reddit_user_supersecretself\bs1l26-Sorry_substitute__f_or_a_cock-SeveralVictoriousAfghanhound.mp4
Downloading https://i.vidble.com//a0EvZXEpqT.jpeg
Downloading https://thcf2.redgifs.com/PepperyOrdinaryJaeger.mp4
https://thcf2.redgifs.com/PepperyOrdinaryJaeger.mp4 : Failed to download https://thcf2.redgifs.com/PepperyOrdinaryJaeger.mp4
Downloading https://thcf3.redgifs.com/SimplisticObeseDikkops.mp4
https://thcf2.redgifs.com/ThickFailingCat.mp4 : Failed to download https://thcf2.redgifs.com/ThickFailingCat.mp4
Downloading https://i.redd.it/a2m7eb2xaz301.jpg
Downloaded .\rips\reddit_user_supersecretself\Veri_f_ication_album\7k2nqa003-a0EvZXEpqT.jpeg
Downloading https://thcf7.redgifs.com/ShamelessSnarlingBaboon.mp4
Downloaded .\rips\reddit_user_supersecretself\7jvxyd-My_pussy_is__f_eeling_awfully_neglected-a2m7eb2xaz301.jpg
Downloading https://i.redd.it/z8ikfls3ko301.jpg
Downloaded .\rips\reddit_user_supersecretself\Veri_f_ication_album\7k2nqa001-pS5RG2ixZP.jpeg
Downloading https://i.redd.it/1e6lh4m2yn301.jpg
Downloaded .\rips\reddit_user_supersecretself\Veri_f_ication_album\7k2nqa002-Sm3fp7r5Hm.jpeg
Downloading https://i.redd.it/b5rmq81awn301.jpg
Downloading https://thcf3.redgifs.com/SimplisticObeseDikkops.mp4
Downloaded .\rips\reddit_user_supersecretself\7ji9kq-_f__I_did_this_to_myself-1e6lh4m2yn301.jpg
Downloading https://thcf7.redgifs.com/ShamelessSnarlingBaboon.mp4
Downloaded .\rips\reddit_user_supersecretself\7jirjg-Okay__one_more__f_or_good_measure-z8ikfls3ko301.jpg
Downloaded .\rips\reddit_user_supersecretself\7ji858-A__f_resh_wax-b5rmq81awn301.jpg
https://thcf7.redgifs.com/ShamelessSnarlingBaboon.mp4 : Failed to download https://thcf7.redgifs.com/ShamelessSnarlingBaboon.mp4
Downloaded .\rips\reddit_user_supersecretself\7jw3vw-_f__one_more__because_I_want_you_to_watch_me_cum-SimplisticObeseDikkops.mp4
Rip complete, saved to E:\Downloads\ripme-fix\ripme\target\rips\reddit_user_supersecretself

revanwolf · 2020-07-01T10:18:16Z

I just compiled your fork and am sadly getting errors. I tried the same URL (reddit profile) I posted before and managed to download only 2 videos. The logs seems to suggest it's still trying to download from redgifs. I assume it redirects to redgif and then redirects again to gifdeliverynetwork.com.

It's Redgif fault, their servers is very bad, try to open the link with your internet browser (Chrome/Firefox/etc) and it won't load the video without issues and the download will stop with "network error".

CaptRussia · 2020-07-01T11:17:29Z

Oh, you're correct! My bad for not checking the links. 😬

I also verified that the links on the reddit profile don't work so it's completely on their end.

borderline232 · 2020-07-01T15:02:18Z

I've also noticed that sometimes the .mp4 redgifs URL doesn't work as well as the .webm URL in the redirect (which you've mentioned before @revanwolf ) so I'll probably just do a quick change on that and it should hopefully work? I'll update this if it doesn't.

Edit: It's able to scrape a couple of the vids that still render that weren't able to show the mp4, however it still misses a couple. I'm thinking about just scraping the mobile.mp4 files which is a bit more consistent but yields a lower resolution video. Thoughts?

Keep mp4Source (mildly reliable + high res)
Keep webmSource (medium reliable + medium-high res)
Keep mobile.mp4 (highly reliable + low res)

avekifes · 2020-07-01T18:46:38Z

@borderline232 Is there a solution for those of us who only want the highest-quality version (.mp4)? If I can't get a video in its highest quality, then I don't want it at all.

How does your current fork behave, anyway? Does it download lower-quality .webm when it can't grab the .mp4?

revanwolf · 2020-07-01T19:23:55Z

How does your current fork behave, anyway? Does it download lower-quality .webm when it can't grab the .mp4?

Actually it download MP4Source by default.

for the format choice maybe use the option for "prefer MP4 over Gif" to give choice? or add an option on rip.properties ?

Anon1337Elite · 2020-07-02T12:54:16Z

I've also noticed that sometimes the .mp4 redgifs URL doesn't work as well as the .webm URL in the redirect (which you've mentioned before @revanwolf ) so I'll probably just do a quick change on that and it should hopefully work? I'll update this if it doesn't.

Edit: It's able to scrape a couple of the vids that still render that weren't able to show the mp4, however it still misses a couple. I'm thinking about just scraping the mobile.mp4 files which is a bit more consistent but yields a lower resolution video. Thoughts?

Keep mp4Source (mildly reliable + high res)
Keep webmSource (medium reliable + medium-high res)
Keep mobile.mp4 (highly reliable + low res)

can we not add some way for it to fall back once it fails ? Try mp4source, Doesnt work ? do webm instead. Doesnt work ? do mobile instead.

CaptRussia · 2020-07-02T13:49:34Z

I agree @BestRestUp3300 I suggested that solution in #1695

Hopefully @borderline232 will be able to implement such a solution!

revanwolf · 2020-07-02T17:58:39Z

I agree @BestRestUp3300 I suggested that solution in #1695

Hopefully @borderline232 will be able to implement such a solution!

I don't think it will really work with the current state of RedGifs servers, if it want to fail it will fail with every type of file (WebM/MP4/MP4-mobile), the only change is with smaller file it has a higher chance to download a valid file.

Redgifs admin said they will upgrade their architecture to better handle the traffic, until then this site is not reliable.

Anon1337Elite · 2020-07-05T23:19:35Z

I agree @BestRestUp3300 I suggested that solution in #1695
Hopefully @borderline232 will be able to implement such a solution!

I don't think it will really work with the current state of RedGifs servers, if it want to fail it will fail with every type of file (WebM/MP4/MP4-mobile), the only change is with smaller file it has a higher chance to download a valid file.

Redgifs admin said they will upgrade their architecture to better handle the traffic, until then this site is not reliable.

I would say this is not entirely true. Sometimes it will always fail on others except MP4 Mobile. No matter how many times you try or you open in browser. The gif just doesn't exist other than in one version. So making the site handle more traffic won't fix this issue.

revanwolf · 2020-07-06T19:53:06Z

For the past few days only thcf1.redgifs.com work for HD version of the video.

ChildishGiant mentioned this pull request May 28, 2020

Add RedGIF Support #1654

Open

Anon1337Elite mentioned this pull request Jun 6, 2020

Add support to gfycat.com Links that redirect to gifdeliverynetwork.com now & redgifs.com shadowmoose/RedditDownloader#140

Open

borderline232 mentioned this pull request Jun 7, 2020

added redgifs support to reddit ripper borderline232/ripme#2

Closed

12 tasks

cyian-1756 merged commit adc352e into RipMeApp:master Jun 19, 2020

RedgifsRipper: Implemented redgifs.com Ripper #1672

RedgifsRipper: Implemented redgifs.com Ripper #1672

Conversation

borderline232 commented May 23, 2020 • edited Loading

Category

Description

Testing

borderline232 commented May 23, 2020 • edited Loading

borderline232 commented May 30, 2020

revanwolf commented Jun 2, 2020

borderline232 commented Jun 3, 2020

revanwolf commented Jun 3, 2020

Anon1337Elite commented Jun 5, 2020

borderline232 commented Jun 5, 2020 • edited Loading

Anon1337Elite commented Jun 6, 2020

borderline232 commented Jun 7, 2020

revanwolf commented Jun 9, 2020

borderline232 commented Jun 13, 2020

revanwolf commented Jun 13, 2020

cyian-1756 commented Jun 19, 2020

avekifes commented Jun 20, 2020

borderline232 commented Jun 20, 2020

CaptRussia commented Jun 26, 2020 • edited Loading

avekifes commented Jun 26, 2020

CaptRussia commented Jun 27, 2020 • edited Loading

avekifes commented Jun 27, 2020

CaptRussia commented Jun 27, 2020

borderline232 commented Jun 27, 2020 • edited Loading

revanwolf commented Jun 27, 2020

borderline232 commented Jun 27, 2020 • edited Loading

revanwolf commented Jul 1, 2020

CaptRussia commented Jul 1, 2020

revanwolf commented Jul 1, 2020

CaptRussia commented Jul 1, 2020 • edited Loading

borderline232 commented Jul 1, 2020 • edited Loading

avekifes commented Jul 1, 2020

revanwolf commented Jul 1, 2020 • edited Loading

Anon1337Elite commented Jul 2, 2020

CaptRussia commented Jul 2, 2020

revanwolf commented Jul 2, 2020

Anon1337Elite commented Jul 5, 2020 • edited Loading

revanwolf commented Jul 6, 2020 • edited Loading

borderline232 commented May 23, 2020 •

edited

Loading

borderline232 commented May 23, 2020 •

edited

Loading

borderline232 commented Jun 5, 2020 •

edited

Loading

CaptRussia commented Jun 26, 2020 •

edited

Loading

CaptRussia commented Jun 27, 2020 •

edited

Loading

borderline232 commented Jun 27, 2020 •

edited

Loading

borderline232 commented Jun 27, 2020 •

edited

Loading

CaptRussia commented Jul 1, 2020 •

edited

Loading

borderline232 commented Jul 1, 2020 •

edited

Loading

revanwolf commented Jul 1, 2020 •

edited

Loading

Anon1337Elite commented Jul 5, 2020 •

edited

Loading

revanwolf commented Jul 6, 2020 •

edited

Loading