docs: add gmaps scraping blog #2772
Conversation
There was a problem hiding this comment.
this is huge, isn't there a more adequate format than gif?
There was a problem hiding this comment.
i don't know. any suggestions? gif kinda fits here
There was a problem hiding this comment.
Could this be webp as well?
| @crawler.router.default_handler | ||
| async def default_handler(context): | ||
| await scrape_google_maps(context) |
There was a problem hiding this comment.
the default_handler does not make much sense here...
| @crawler.router.default_handler | |
| async def default_handler(context): | |
| await scrape_google_maps(context) | |
| crawler.router.default_handler(scrape_google_maps) |
This should be enough if you want to keep the handler definition outside of the main function.
| """ | ||
| page = context.page | ||
| await page.goto(context.request.url) | ||
| print("Connected to:", context.request.url) |
There was a problem hiding this comment.
| print("Connected to:", context.request.url) | |
| print("Processing: ", context.request.url) |
| # Pretty-print the data | ||
| print(json.dumps(data, indent=4)) | ||
| print("\n") |
There was a problem hiding this comment.
This is not how it's supposed to be done - it'd be better to use context.push_data(data)
| with open('google_maps_data.json', 'w', encoding='utf-8') as f: | ||
| json.dump(all_data, f, ensure_ascii=False, indent=2) |
There was a problem hiding this comment.
If you use the default dataset for this, you can simply do crawler.export_data_json('path', ensure_ascii=False, indent=2)
| First, we need a function that can handle the scrolling and detect when we've hit the bottom. Copy-paste this new function in the `gmap_scraper.py` file: | ||
|
|
||
| ```python | ||
| async def load_more_items(page) -> bool: |
There was a problem hiding this comment.
- The article does not mention where the function should be called.
- Crawlee already has
context.infinite_scroll()- does it not work in this case?
approved by adam and marketing.