You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If a response is identified as a ban by response_is_ban(self, request, response), it'll currently reach the spider's parse() method after the final retry attempt by your middleware, because you don't raise an exception or otherwise stop the response after more than ROTATING_PROXY_PAGE_RETRY_TIMES banned attempts.
This is somewhat inconvenient as it requires the user to call response_is_ban(self, request, response) again in his parse() implementation.
Apart from that I also noticed that ROTATING_PROXY_PAGE_RETRY_TIMES = 1 generally results in 2 retries rather than just 1 (it's always 1 more than ROTATING_PROXY_PAGE_RETRY_TIMES).
The text was updated successfully, but these errors were encountered:
If a response is identified as a ban by
response_is_ban(self, request, response)
, it'll currently reach the spider'sparse()
method after the final retry attempt by your middleware, because you don't raise an exception or otherwise stop the response after more thanROTATING_PROXY_PAGE_RETRY_TIMES
banned attempts.This is somewhat inconvenient as it requires the user to call
response_is_ban(self, request, response)
again in hisparse()
implementation.Apart from that I also noticed that
ROTATING_PROXY_PAGE_RETRY_TIMES = 1
generally results in 2 retries rather than just 1 (it's always 1 more thanROTATING_PROXY_PAGE_RETRY_TIMES
).The text was updated successfully, but these errors were encountered: