Fetch underlyings (#70)

* Add tests

* Clean up fetch

* Fix action

* Refactor set actions

* Clean up code

* Test fetch underlyings

* Add underlying urls to settings

* Update url

* Refactor url

* Add underlying url

* Fix tests

* Update changelog

* Refactor underlying

* Select underlying

* Select underlying

* Implement bookmark processor

* Update readme

* Fix serialize error

* Improve debugging

* Fix domain

* Add todo

* Add underlyings

* Convert underlyings

* Rename urls

* Refactor unwrap

* Format code

* Fix clippy warnings

CHANGELOG.md

@@ -6,6 +6,7 @@
   - Add `action` to `TargetBookmark`
   - Add benchmarks for fetching
   - Take ignored urls into account in `bogrep import`
+  - Fetch underlying urls
 - changed
   - Update to rust 1.75
   - Fix duplicate cache files for `bogrep fetch --urls`

Cargo.toml

@@ -39,6 +39,8 @@ futures = "0.3.28"
 html2md = "0.2.14"
 readability = "0.2.0"
 async-trait = "0.1.73"
+scraper = "0.18.1"
+parking_lot = "0.12.1"
 
 [dev-dependencies]
 assert_cmd = "2.0.12"

README.md

@@ -27,6 +27,7 @@ bogrep -i "reed-solomon code"
   - [Edge](#edge)
 - [Specify bookmark folders](#specify-bookmark-folders)
 - [Ignore URLs](#ignore-urls)
+- [Fetch underlying urls](#fetch-underlying-urls)
 - [Diff websites](#diff-websites)
 - [Manage internal bookmarks](#manage-internal-bookmarks)
 - [Request throttling](#request-throttling)
@@ -184,6 +185,17 @@ usually don't include relevant text to grep.
 bogrep config --ignore <url1> <url2> ...
 ```
 
+## Fetch underlying urls
+
+Fetch the underlying urls of supported websites:
+
+``` bash
+bogrep config --underlying <url1> <url2> ...
+```
+
+For example, if a specific url for <https://news.ycombinator.com> is
+bookmarked, the article of discussion will be fetched and cached.
+
 ## Diff websites
 
 Fetch difference between cached and fetched website for multiple urls, and display changes: