Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial data load takes a long time #4

Open
quasistoic opened this issue Feb 24, 2023 · 4 comments
Open

Initial data load takes a long time #4

quasistoic opened this issue Feb 24, 2023 · 4 comments

Comments

@quasistoic
Copy link
Owner

Initial data download takes a very long time because it has to make a ton of API calls, one per item in 1Password. I plan to switch the first call to “item list” to provide json output, which includes enough details that I can get rid of all the per-item calls when composing the list view, so the initial loading will change from many minutes to a few seconds. This should obviate the need for me to do on-disk caching. Also just makes the whole experience more pleasant.

quasistoic added a commit that referenced this issue Feb 26, 2023
Was an attempt to fix Issue #4, but I had forgotten that the initial list view sorts by difference_score, which requires fetching full details for every item anyway... At the very least this does mean we don't have to fetch details for non-duplicate items.
quasistoic referenced this issue Mar 7, 2023
@quasistoic
Copy link
Owner Author

This is as fast as we can reasonably get the initial load without getting rid of the score-based sorting.

@quasistoic
Copy link
Owner Author

Reopening. Turns out I'm not actually being rate-limited yet, but the calls themselves are I/O-bound, which means I should be able to parallelize. Additionally passing the --vault argument for each get call reportedly would improve latency.

@quasistoic
Copy link
Owner Author

quasistoic commented Mar 8, 2023

With some experimentation, I was able to infer that the 300th item get call in a single minute trips me over whatever actual qpm limits the 1Password API uses, so this allows me to reduce what used to take about 10-15 minutes down to about 3 minutes (with N~800 items).

So, significant speedup, but still not quite fast enough to ditch the on-disk cache.

Relevant commits:

  1. 04e5b32
  2. 30413d8
  3. a8ca0fd

@quasistoic
Copy link
Owner Author

I was hoping that 5cc958c would result in fewer backend API calls made by the op command line when running an item get, so I attempted to raise my throttled fetch rate to around 400qpm, but after exactly 300 successful fetches, I was blocked for an hour, so I suppose we're stuck with under 300qpm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant