Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scrape files with exact same names when they include french characters #466

Closed
LucasDemea opened this issue Nov 22, 2021 · 4 comments
Closed

Scrape files with exact same names when they include french characters #466

LucasDemea opened this issue Nov 22, 2021 · 4 comments
Labels
bug

Comments

@LucasDemea
Copy link

Hi,
I need the scraped files to have the exact same names as the original files. Instead, when the original filenames include french characters, I get:
Capture-d’écran-2020-09-01-à-15.58.14.png
becomes
Capture-dâ__écran-2020-09-01-à-15.58.14.png
Is there a way to preserve the exact original filenames ?
@s0ph1e
Copy link
Member

s0ph1e commented Nov 25, 2021

Hi @LucasDemea

Sounds like a bug, I need to check this

I suggest to customize generateFilename action as described here and try to receive correct filenames, hope it helps

@s0ph1e s0ph1e added the bug label Dec 28, 2021
@s0ph1e
Copy link
Member

s0ph1e commented Dec 28, 2021

I can confirm that this is a bug, root cause and possible workarounds are described in #454

We'll think how to fix it in future versions

@aivus aivus mentioned this issue Dec 29, 2021
Merged
2 tasks
s0ph1e pushed a commit that referenced this issue Jun 9, 2022
@aivus
Fix encoding issue for non-English websites, closes #454 #466 (#482)
5a58f48
@s0ph1e
Copy link
Member

s0ph1e commented Jun 9, 2022

The issue should be fixed by #482, the fix will be released in next version 5.1.0

@s0ph1e s0ph1e closed this as completed Jun 9, 2022
@s0ph1e s0ph1e mentioned this issue Jun 11, 2022
Merged
@s0ph1e
Copy link
Member

s0ph1e commented Jun 11, 2022

Reopen because a version with a fix 5.1.0 was deprecated and reverted

@s0ph1e s0ph1e reopened this Jun 11, 2022
@s0ph1e s0ph1e closed this as completed in e8c6f9d Jun 23, 2022
yamanyadav8810 added a commit to yamanyadav8810/node-website-scraper that referenced this issue Feb 14, 2023
@yamanyadav8810
Squashed commit of the following:
8aff1bc 
commit 65b8056
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Dec 6 01:50:13 2022 +0200

    Bump fs-extra from 10.1.0 to 11.1.0 (website-scraper#523)

    Bumps [fs-extra](https://github.com/jprichardson/node-fs-extra) from 10.1.0 to 11.1.0.
    - [Release notes](https://github.com/jprichardson/node-fs-extra/releases)
    - [Changelog](https://github.com/jprichardson/node-fs-extra/blob/master/CHANGELOG.md)
    - [Commits](jprichardson/node-fs-extra@10.1.0...11.1.0)

    ---
    updated-dependencies:
    - dependency-name: fs-extra
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 52a341a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Dec 5 20:36:07 2022 +0000

    Bump sinon from 14.0.2 to 15.0.0 (website-scraper#524)

commit faa8bab
Author: Illia Antypenko <[email protected]>
Date:   Thu Nov 17 00:11:58 2022 +0200

    Update stale.yml

commit 5886558
Author: Illia Antypenko <[email protected]>
Date:   Wed Nov 16 23:24:22 2022 +0200

    Update stale.yml

commit 2797213
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 21:09:28 2022 +0000

    Bump actions/setup-node from 2 to 3 (website-scraper#519)

commit d9d147a
Author: Illia Antypenko <[email protected]>
Date:   Wed Nov 16 23:06:02 2022 +0200

    Delete no-response.yml

commit 7b6cbb5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 20:54:40 2022 +0000

    Bump paambaati/codeclimate-action from 3.0.0 to 3.2.0 (website-scraper#520)

commit 28d6b40
Author: Illia Antypenko <[email protected]>
Date:   Wed Nov 16 22:40:53 2022 +0200

    Delete codeql-analysis.yml

commit 145f31d
Author: Illia Antypenko <[email protected]>
Date:   Wed Nov 16 22:40:44 2022 +0200

    Create codeql.yml

commit 4237388
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 17:15:31 2022 +0000

    Bump normalize-url from 7.2.0 to 8.0.0 (website-scraper#517)

commit 9dc9ef6
Author: Illia Antypenko <[email protected]>
Date:   Wed Nov 16 19:04:39 2022 +0200

    Auto-update of Github Actions

commit b169e86
Author: Sofiia Antypenko <[email protected]>
Date:   Sun Oct 9 09:56:22 2022 +0200

    Fix ENOENT when running from working directory without package.json (website-scraper#516)

commit de75674
Author: ryanolee <[email protected]>
Date:   Wed Sep 21 21:28:24 2022 +0100

    Remove plugin exports and update docs (website-scraper#511), closes website-scraper#510

    * chore(website-scraper#510): Remove plugin exports and update docs

    * core(website-scraper#510) Update messaging on internal plugins

    Co-authored-by: ryanolee <[email protected]>

commit 6c5cfc5
Author: Sofiia Antypenko <[email protected]>
Date:   Thu Sep 15 21:55:48 2022 +0200

    Revert "RFC: Refactor builtin plugins to expose class properties (website-scraper#506)" (website-scraper#509)

    This reverts commit 5dde8ec.

commit b845fb0
Author: Sofiia Antypenko <[email protected]>
Date:   Mon Sep 12 10:01:34 2022 +0200

    Remove lodash from dependencies (website-scraper#508)

commit 5dde8ec
Author: ryanolee <[email protected]>
Date:   Mon Sep 12 08:55:31 2022 +0100

    RFC: Refactor builtin plugins to expose class properties (website-scraper#506)

    * core: Refactor builtin plugins to expose class properties

    * chore: EsLint

    Co-authored-by: ryan lee <[email protected]>

commit c6ff84e
Author: Illia Antypenko <[email protected]>
Date:   Thu Sep 1 11:55:54 2022 +0300

    Remove demo link from the README

commit 8d6b5aa
Author: s0ph1e <[email protected]>
Date:   Tue Aug 30 12:23:56 2022 +0200

    5.3.0

commit 6bb4e20
Author: Sophia Antipenko <[email protected]>
Date:   Mon Aug 29 21:56:22 2022 +0200

    Use encoding from resource text (website-scraper#504)

commit 07c4c02
Author: Illia Antypenko <[email protected]>
Date:   Sat Jul 2 00:36:57 2022 +0300

    Use codeclimate project id instead of secret

commit 525dc50
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Jul 1 09:20:14 2022 +0200

    Bump cheerio from 1.0.0-rc.11 to 1.0.0-rc.12 (website-scraper#499)

    Bumps [cheerio](https://github.com/cheeriojs/cheerio) from 1.0.0-rc.11 to 1.0.0-rc.12.
    - [Release notes](https://github.com/cheeriojs/cheerio/releases)
    - [Changelog](https://github.com/cheeriojs/cheerio/blob/main/History.md)
    - [Commits](cheeriojs/cheerio@v1.0.0-rc.11...v1.0.0-rc.12)

    ---
    updated-dependencies:
    - dependency-name: cheerio
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 07e6858
Author: Sophia Antipenko <[email protected]>
Date:   Thu Jun 23 08:57:14 2022 +0200

    Bump version 5.2.0

commit e8c6f9d
Author: Paul Hawxby <[email protected]>
Date:   Thu Jun 23 07:24:45 2022 +0100

    fix: non-english char encoding, close website-scraper#454, close website-scraper#466 (website-scraper#496)

    * fix: non-english char encoding

    * fix: optional chaining not supported

    * fix: various tests

    not finished yet

    * fix: tests now pass

    * fix: testing and code complexity

    * fix: code climate

    * fix: change comments

    * fix: simplify code complexity

    * refactor: simplify logic and add more testing

    * fix: reduce cognitive complexity

    * fix: more cognitive complexity

    * chore: damn space

    * fix: broke a test

    * chore: update docs

    * test: re-enable utf8 test

    * chore: update request test

commit 3bd2555
Author: Sophia Antipenko <[email protected]>
Date:   Sat Jun 11 14:43:32 2022 +0200

    Revert encoding fix website-scraper#482 (website-scraper#495)

commit c6f60b8
Author: Sophia Antipenko <[email protected]>
Date:   Thu Jun 9 14:18:08 2022 +0200

    Remove link to gitter from CONTRIBUTING.md

commit 087d943
Author: Sophia Antipenko <[email protected]>
Date:   Thu Jun 9 14:07:10 2022 +0200

    Bump version 5.1.0

commit 5a58f48
Author: Illia Antypenko <[email protected]>
Date:   Thu Jun 9 14:04:51 2022 +0200

    Fix encoding issue for non-English websites, closes website-scraper#454 website-scraper#466 (website-scraper#482)

commit d80e9b0
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat May 28 08:57:53 2022 +0200

    Bump cheerio from 1.0.0-rc.10 to 1.0.0-rc.11 (website-scraper#491)

    Bumps [cheerio](https://github.com/cheeriojs/cheerio) from 1.0.0-rc.10 to 1.0.0-rc.11.
    - [Release notes](https://github.com/cheeriojs/cheerio/releases)
    - [Changelog](https://github.com/cheeriojs/cheerio/blob/main/History.md)
    - [Commits](cheeriojs/cheerio@v1.0.0-rc.10...v1.0.0-rc.11)

    ---
    updated-dependencies:
    - dependency-name: cheerio
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9397b07
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 10 10:58:35 2022 +0200

    Bump sinon from 13.0.2 to 14.0.0 (website-scraper#488)

    Bumps [sinon](https://github.com/sinonjs/sinon) from 13.0.2 to 14.0.0.
    - [Release notes](https://github.com/sinonjs/sinon/releases)
    - [Changelog](https://github.com/sinonjs/sinon/blob/main/docs/changelog.md)
    - [Commits](sinonjs/sinon@v13.0.2...v14.0.0)

    ---
    updated-dependencies:
    - dependency-name: sinon
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 66b1e49
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue May 3 14:49:33 2022 +0200

    Bump mocha from 9.2.2 to 10.0.0 (website-scraper#486)

    Bumps [mocha](https://github.com/mochajs/mocha) from 9.2.2 to 10.0.0.
    - [Release notes](https://github.com/mochajs/mocha/releases)
    - [Changelog](https://github.com/mochajs/mocha/blob/master/CHANGELOG.md)
    - [Commits](mochajs/mocha@v9.2.2...v10.0.0)

    ---
    updated-dependencies:
    - dependency-name: mocha
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 848ca1a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Feb 6 19:26:26 2022 +0200

    Bump sinon from 12.0.1 to 13.0.0 (website-scraper#483)

    Bumps [sinon](https://github.com/sinonjs/sinon) from 12.0.1 to 13.0.0.
    - [Release notes](https://github.com/sinonjs/sinon/releases)
    - [Changelog](https://github.com/sinonjs/sinon/blob/master/docs/changelog.md)
    - [Commits](sinonjs/sinon@v12.0.1...v13.0.0)

    ---
    updated-dependencies:
    - dependency-name: sinon
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@s0ph1e @LucasDemea