Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up the purging logic for crawlee projects #590

Open
B4nan opened this issue Jul 3, 2024 · 0 comments
Open

Clean up the purging logic for crawlee projects #590

B4nan opened this issue Jul 3, 2024 · 0 comments
Assignees
Labels
t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@B4nan
Copy link
Member

B4nan commented Jul 3, 2024

Crawlee projects (both JS and python) are purging default storages automatically (at the framework level), if we detect those, apify run shouldn't do anything and let crawlee do the purge. apify run -p should behave the same for them, still no manual purging at the CLI level. apify run --no-purge (with --resurrect alias) should disable the purging via CRAWLEE_PURGE_ON_START=0 env var. This also applies for JS SDK v3 projects, since there is crawlee behind the scenes doing the same. Similarly, the upcoming python SDK v2 needs to behave the same, since again, there will be crawlee behind the scenes.

Projects that are not using crawlee or JS SDK v3 or python SDK v2 still need to keep the purging on CLI level. apify run -p will handle purging in the CLI. apify run without any flags shouldn't do anything either, like apify run --no-purge.

Also remove the --purge-xxx flags, and mention all of this in the migration guide https://github.com/apify/apify-cli/blob/master/MIGRATIONS.md

@B4nan B4nan added the t-tooling Issues with this label are in the ownership of the tooling team. label Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

2 participants