Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rate limit option #155

Open
ocervell opened this issue Apr 25, 2024 · 2 comments
Open

Add rate limit option #155

ocervell opened this issue Apr 25, 2024 · 2 comments
Labels
enhancement New feature or request Go help wanted Extra attention is needed

Comments

@ocervell
Copy link
Contributor

Add a rate limit option for controlling crawling speed (-rl, -rate-limit). You maybe can use https://github.com/projectdiscovery/ratelimit for this.

@edoardottt
Copy link
Owner

Hi @ocervell , there is this option available in cariddi, it's not a rate limiter..but similar:

  -d int
     Delay between a page crawled and another.

Seems this is in colly the way to provide rate limiting (https://go-colly.org/docs/examples/rate_limit/).

@ocervell
Copy link
Contributor Author

Argh ... this is clearly not enough since it depends on the number of threads as well... adding a delay between requests is okay but not as precise as setting a global rate limit. I will investigate to see what we can do.

@edoardottt edoardottt added enhancement New feature or request help wanted Extra attention is needed Go labels Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Go help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants