Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download all images on the page #480

Open
argyleink opened this issue Dec 3, 2020 · 34 comments · May be fixed by #481
Open

Download all images on the page #480

argyleink opened this issue Dec 3, 2020 · 34 comments · May be fixed by #481
Assignees
Labels
🔥 enhancement New feature or request good first issue 👍 Good for newcomers

Comments

@argyleink
Copy link
Member

/download-images command in the search bar runs the following plugin script

const run = async () => {
  const dirHandle = await window.showDirectoryPicker();
  const imgs = document.querySelectorAll("img");
  let i = 0;

  imgs.forEach(async (img) => {
    const url = img.src;
    const name = `img-${i}.png`;
    i++;

    try {
      console.log(`Fetching ${url}`);
      const response = await fetch(url);

      console.log(`Saving to ${name}`);

      const file = await dirHandle.getFileHandle(name, { create: true });
      const writable = await file.createWritable();
      await response.body.pipeTo(writable);
    } catch (err) {
      console.log(err);
    }
  });
};

run();

https://paul.kinlan.me/bookmarklet-to-download-all-images-on-a-page-with-the-file-system-api/

@argyleink argyleink added 🔥 enhancement New feature or request good first issue 👍 Good for newcomers labels Dec 3, 2020
@argyleink argyleink linked a pull request Dec 3, 2020 that will close this issue
@argyleink
Copy link
Member Author

@aashu0148 here's another one that needs finished if you're lookin for another plugin to hack on!? the #481 has the work started but needs a few tweaks before it's ready. 🙂

@aashu0148
Copy link
Contributor

@aashu0148 here's another one that needs finished if you're lookin for another plugin to hack on!? the #481 has the work started but needs a few tweaks before it's ready. 🙂

I'll check for sure.
But now i have a function in my family 🥳🥳
I'll check later

@aashu0148
Copy link
Contributor

@argyleink I'll be happy to work on this issue but can you please tell me where the javascript file is located from which i can access the value entered in the search box.

@argyleink
Copy link
Member Author

@aashu0148
Copy link
Contributor

Hey @argyleink I'm really sorry for this delay but first i was busy in my family functions and then i got my exams but i thought i can easily make a good percentage in my exams but after giving my first exam i realized that this will not going until i dedicatedly study for it... so I decided to study for my exams and forgot about informing you..... yesterday was my last exam. I'll be working on this issue from now itself.

Again So sorry for accepting this issue and not active for such a long period :(

@aashu0148
Copy link
Contributor

hey @argyleink I run it on my machine and it is downloading all the images.
What's for me in it ?

@argyleink
Copy link
Member Author

yes, sorry, I see I never listed the missing aspects of the feature!

Currently, the user picks a folder to save the images to, and all the images download, but there's many that are broken and the original names of the images is lost. A file named logo.svg should save as logo.svg, where right now it saves as img-2.png. Both the name and src type are not brought along in the download.

  • file should save as the same name the src file is https://a.io/images/foo.svg -> foo.svg
  • file should save as it's natural src type
  • remove console logs
  • should feature check for showDirectoryPicker support and alert if not available
  • test on various websites

Thanks for the help!

@argyleink
Copy link
Member Author

also, would be cool if it downloaded all images that were referenced via CSS too! background images, etc. want to investigate level of effort to adding that too?

@aashu0148
Copy link
Contributor

okay @argyleink we will move to CSS images also later on but now let me work on javascript to complete the above mentioned tasks :)

@aashu0148
Copy link
Contributor

  • /
    Screenshot (40)

hey @argyleink first task is completed i think. If not please let me know;

@aashu0148
Copy link
Contributor

hey @argyleink to check for showDirectoryPicker can we put it into try and add a alert in catch.
Will this work ?Because i dont know how to test this as my machine support showDirectoryPicker

@argyleink
Copy link
Member Author

nice work on the names and file extensions 👍🏻

you could try catch for feature testing yeah. i just tested this and it works too:

if (window.showDirectoryPicker === undefined)
  console.log('no directory support')

@argyleink
Copy link
Member Author

here's some JS that grabs the external urls from stylesheets on the page:

let css_urls = [...document.styleSheets]
  .filter(sheet => {
    try { return sheet.cssRules }
    catch {}
  })
  .flatMap(sheet => Array.from(sheet.cssRules))
  .filter(rule => rule.style)
  .filter(rule => rule.style.backgroundImage !== '')
  .filter(rule => rule.style.backgroundImage !== 'initial')
  .filter(rule => rule.style.backgroundImage.includes("url"))
  .reduce((urls, {style}) => {
    urls.push(style.backgroundImage);
    return urls;
  }, []);
  
console.log(css_urls);

output example:

["url("../media/foo.jpg")", "url("../media/bar.svg")", "url("../media/baz.png")"]

seems doable with this concept proof yeah!?

@aashu0148
Copy link
Contributor

here's some JS that grabs the external urls from stylesheets on the page:

let css_urls = [...document.styleSheets]
  .filter(sheet => {
    try { return sheet.cssRules }
    catch {}
  })
  .flatMap(sheet => Array.from(sheet.cssRules))
  .filter(rule => rule.style)
  .filter(rule => rule.style.backgroundImage !== '')
  .filter(rule => rule.style.backgroundImage !== 'initial')
  .filter(rule => rule.style.backgroundImage.includes("url"))
  .reduce((urls, {style}) => {
    urls.push(style.backgroundImage);
    return urls;
  }, []);
  
console.log(css_urls);

output example:

["url("../media/foo.jpg")", "url("../media/bar.svg")", "url("../media/baz.png")"]

seems doable with this concept proof yeah!?

ya it seems you already done most of my work.
Thanks :)
I'll update you soon

@aashu0148
Copy link
Contributor

hey @argyleink I'm stuck with an awkward issue :/
can't say why is this showing. The issue is that i have changed the JS file and in the console it is still showing the console logs and not implementing my changes. I have closed vs code 3-4 times and re run the localhost dozen of times but still it is not showing the console according to the current code.
I'm attaching the screenshot of that please help me with that. check there in the console it says at line no 48,53,54 but in my code there is no console logs on that lines.. there are some before but i had deleted them but it is not showing the updated result.

can you please take a look at it. I'll also continue to find the problem and update you if it is solved.

Screenshot (41)

@aashu0148
Copy link
Contributor

hey @argyleink I'm stuck with an awkward issue :/
can't say why is this showing. The issue is that i have changed the JS file and in the console it is still showing the console logs and not implementing my changes. I have closed vs code 3-4 times and re run the localhost dozen of times but still it is not showing the console according to the current code.
I'm attaching the screenshot of that please help me with that. check there in the console it says at line no 48,53,54 but in my code there is no console logs on that lines.. there are some before but i had deleted them but it is not showing the updated result.

can you please take a look at it. I'll also continue to find the problem and update you if it is solved.

Screenshot (41)

Also in the source tab of dev tools the code is not what i have in my vs code.

Screenshot (42)

@argyleink
Copy link
Member Author

hm! I can try to help 🙂

  • do you have devtools local overrides turned on? perhaps you did some work inside of the sources tab on that file and hit save / ctrl + s? if so, then devtools will reload a modified file that it maintains, instead of the source file from your server. to me, this is the most likely given the screenshots and clues you've provided so far
  • when you run npm start in the visbug project, are there js errors? the site may continue to work and serve an older, previously built file and isnt producing a new one due to a syntax issue somewhere. tldr; it may be that a new js file isnt getting bundled, and therefore not showing your new lines of code

@aashu0148
Copy link
Contributor

hm! I can try to help 🙂

  • do you have devtools local overrides turned on? perhaps you did some work inside of the sources tab on that file and hit save / ctrl + s? if so, then devtools will reload a modified file that it maintains, instead of the source file from your server. to me, this is the most likely given the screenshots and clues you've provided so far
  • when you run npm start in the visbug project, are there js errors? the site may continue to work and serve an older, previously built file and isnt producing a new one due to a syntax issue somewhere. tldr; it may be that a new js file isnt getting bundled, and therefore not showing your new lines of code

Thanks @argyleink there are some errors while running npm start now it is working.

but can you tell me what to put in the name when we download css images because we fetch them from a url and hence can't find the name.

so should i name them like css-1,css-2,... ?

@aashu0148
Copy link
Contributor

@argyleink ??🤔

@argyleink
Copy link
Member Author

Can you give me examples of what you mean? URLs with no names?

@aashu0148
Copy link
Contributor

aashu0148 commented Dec 25, 2020

Can you give me examples of what you mean? URLs with no names?

yaa sure
like if a image was fetched from this url https://images.unsplash.com/photo-1608806947629-da2ae50be954?ixid=MXwxMjA3fDB8MHxlZGl0b3JpYWwtZmVlZHwyfHx8ZW58MHx8fA%3D%3D&ixlib=rb-1.2.1&auto=format&fit=crop&w=400&q=60

then we don't have a name for this image then how can i name such images... I have currently named them as css-1 , css-2 , css-3 ....

@aashu0148
Copy link
Contributor

@argyleink 🤔🤔🤔

@aashu0148
Copy link
Contributor

hey @argyleink where are you ??

@argyleink
Copy link
Member Author

Vacation 😁

@aashu0148
Copy link
Contributor

Vacation 😁

oh :)
when will it end ??

@argyleink
Copy link
Member Author

early january! but i can give some feedback real quick 👍🏻

some url rules?

  • remove all params after the ? on urls, as they're parameters and not filename values
  • limit filenames to 40 characters

that way really long ones can be cropped, and if they're still too long, they can be maxed out then saved. thoughts?

@aashu0148
Copy link
Contributor

early january! but i can give some feedback real quick 👍🏻

some url rules?

  • remove all params after the ? on urls, as they're parameters and not filename values
  • limit filenames to 40 characters

that way really long ones can be cropped, and if they're still too long, they can be maxed out then saved. thoughts?

ya its great i'll implement it and update you

@aashu0148
Copy link
Contributor

Its done @argyleink Please review it :)

Screenshot (43)
Screenshot (44)

@argyleink
Copy link
Member Author

Looks ready to share! Submit a pr?

@aashu0148
Copy link
Contributor

aashu0148 commented Dec 30, 2020

Looks ready to share! Submit a pr?

already made a pr. please review it

@aashu0148
Copy link
Contributor

hey @argyleink can you please tell me how can i use this project locally with any website?
like the chrome extension has only features which are developed and as we are developing it then how can i see this project working on other websites ?

@argyleink
Copy link
Member Author

Here's a wiki page with some instruction: https://github.com/GoogleChromeLabs/ProjectVisBug/wiki/Local-Extension-Dev-Testing

your github repo extension/ folder will allow you to load the plugin from your computer, it'll be called DevBug in Chrome. Visit any webpage, then launch DevBug and you'll be working with your local branch and plugin against any webpage. Depending on what code changes, you may need to rebundle the extension.

Let me know if you need more help 🙂

@aashu0148
Copy link
Contributor

ya @argyleink I need some more help!

First of all i really like this project idea and i want to contribute more to it until i have time from my studies and all stuff.
and here at github comment section is too much slow like i usually work at night i when i see your message i reply to it at night and then i'll get your reply next night and this process goes on like it takes 2-3 days to even understand this issue.
Frankly speaking i'm new to github and don't know if github have another method of communicating rather then commenting on the issue.
It would be great if you tell me where can we communicate faster then what we are currently doing.

Also i saw the link and in the download section of it i saw this error page.
Screenshot (52)

@argyleink
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔥 enhancement New feature or request good first issue 👍 Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants