Finds the price of retail items online by scraping the web page.
import PriceFinder from 'price-finder';
const priceFinder = new PriceFinder();
// Led Zeppelin II vinyl (from Amazon)
const uri = 'https://www.amazon.com/Led-Zeppelin-II/dp/B00IXHBUG0';
const price = await priceFinder.findItemPrice(uri);
console.log(price); // 22.97When creating a new PriceFinder object, a configuration object can be specified. The following options are configurable:
retryStatusCodes: An array of status codes (Numbers) which when returned from the page scrape request, will trigger a retry request (meaning it will attempt to scrape the page again). Defaults to[503].retrySleepTime: If a retry status code is returned from a page scrape request, this is the amount of time (in milliseconds) that the code will sleep prior to re-issuing the request. Defaults to1000(ms).
For example:
import PriceFinder from 'price-finder';
const priceFinder = new PriceFinder({
retrySleepTime: 2000,
});Given a uri (that is for a supported site), this function will scrape the page and attempt to find the current price listed on the page. The result will be returned asynchronously.
If problems occur during processing, an Error will be thrown.
The pino package is used within price-finder to output debugging information useful in tracking down any potential issues.
The current supported sites are listed below.
- Amazon
- Crutchfield (requires Node 16)
- Home Depot
Don't see your site listed? Please consider contributing to the project!
The price-finder project is a Node.js module, so before cloning the repository make sure node is installed. Once cloned, install dependencies by issuing:
$ yarn
The project uses Jest for tests (please add tests for any new features).
To run the unit tests execute:
$ yarn test
These tests can be run in watch mode, listening for any file changes and re-running when that occurs. To do so execute:
$ yarn test:watch
End-to-end tests exist which will test the price-finder module using real URIs, scraping the pages to verify the code works correctly.
Note that these tests should be run on a limited basis while coding since some sites have been known to throw up CAPTCHA's after repeated, automated page requests.
To execute the end to end tests run:
$ yarn test:e2e
To execute a specific end to end test run:
$ yarn test:e2e:single test/e2e/<test file>
This project was built to easily drop in support for new sites. The site-manager iterates over all files contained within the sites directory, and adds it to the list of available sites. When a request is issued to price-finder to look up a price, it asks each site if the uri is supported by the site, and if so, uses that site to find the price.
- Create a new
Siteinsrc/sites:
export default class MySite implements Site {
...
}- Add unit and e2e tests for the site
-
unit:
test/unit/sites/MySite.test.ts -
e2e:
test/e2e/MySite.e2e.test.ts
-
Add site to Supported Sites
-
Create pull request to submit!
(These notes require admin permissions)
-
Update
package.jsonwith new version. -
Update
CHANGELOG.mdwith new version along with included PRs and short description. -
Run
yarn build -
Create pull request for release with updated files. Merge.
-
Git tag:
$ git tag -a < version >
(enter in message "Tag < version >" and include changelog message)
$ git show < version > (shows data on specific tag)
$ git push origin —tags (push all tags to remote)
- GitHub release:
- Go to releases, click "Draft a new release"
- Select the tag from the drop down list
- Release title is version: "< version >"
- Description is changelog message
- npm publish:
$ npm publish
