Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added endpoint to get latest ads from a cursor (last_id) #1

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

eydine
Copy link

@eydine eydine commented May 10, 2024

No description provided.

@Lejo1
Copy link
Owner

Lejo1 commented May 10, 2024

This wont do what you would like to. This allows you to go through all ads in inverse order of the ad_id. This ad_id is in no way related to the time of the ads being published.
The kinda dirty way the latest ads are returned is by using the natural order in reverse, just meaning: Reading the database from the end. Therefore new or largely changed ads will always be there, allowing the query to return the current ads.
I think the best way would be some query by the ad_creation_time, it also got a indice.
I quickly created an endpoint for it: GET /adsbydate/date?offset=0&limit=0
As of the amount of ads per day, the offset shouldn't get to high to hit the timeout. Also you shouldn't have such the problem of the moving position of the cursor when crawling older dates.
Let me know, if this works for you!

FYI, these are the current indices:
grafik
New ones are of course possible!

@eydine
Copy link
Author

eydine commented May 11, 2024

Thanks man, this is actually insanely helpful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants