Improving Data Tracking with 'Crawl Once' Filter in Scrapy

[ZubairShahzad]

Hi everyone,

I’m using the 'crawl once' filter in Scrapy to avoid scraping the same link more than once, which helps reduce overall proxy usage. Is there a way to adjust the middleware so that, when 'crawl once' is active and detects a previously scraped listing, it can still yield the UID and ScrapeTime for that listing? If not, that's okay, but I’m hoping to improve data tracking on the backend.

Thank you!