Page responses times

[humitos]

I've been having a feeling that some pages were behaving slow. In particular project dashboard and build listing. Today, I wanted to remove doubts about this and I checked these times on New Relic:

• project dashboard: 3.5s (https://onenr.io/0Zw0Dl5v2wv)
• build listing: 1s (https://onenr.io/0OQMEqYNvQG)

I haven't explored too much yet because I don't have good connection from the train, but I'm almost sure this is because we are doing many queries to the db to show all the data it's currently showing. With a simple look, I can tell they are the builds_build queries the ones that are consuming most of the time in these pages to render.

[agjohnson]

Yeah, I've noticed both, but the project dashboard is the one I notice the most. This page used to be really slow in production as well, I'm not sure what we would have done there. I see the same timings on my side, and those timings are backend response times (though there is some additional DOM render time too). I haven't looked into this either though.

It would help to have the debug toolbar on the beta instance in our private beta probably, to help debug this a bit quicker. But at least we have NR, so that's pretty easy too.

Just a guess, but maybe pagination is be affecting this (though build listing is already paginated in prod and isn't slow)?

This would be a great place to jump in if you'd like. There is some additional DOM load time, but it seems the backend is the main culprit at the moment. It would be great to get this tuned more either way. If the issue is expensive Python code, I lean towards loading this data from the API async -- I've used this pattern to avoid resolving URLs as the view iterates over Version instances, etc.

[humitos]

Yeah. This is 99% backend for sure because those times come from NR.

However, we may be trying to "show too much in the frontend" in just one request. We could either, consider removing some elements or using async as you suggested 👍

[agjohnson]

Yeah, looks like select on builds, which is almost certainly the project listing showing data about the most recent build or build commit -- this could be dropped or moved async.

A sample transaction:
https://onenr.io/0OwvvBLvowv

[agjohnson]

Also, I opened #31 a while ago as I don't actually think the listing page has the best information on it already. The non-beta dashboard shows the latest build status, which is ... weird. So, I'm avoiding that logic.

[agjohnson]

Yeah, as suspected, the project listing uses a few model methods that aren't optimized. I'll want to revisit this later, to optimize and cache some new model fields. I'll reuse get_latest_build for now, but I don't think that's the most helpful information for the listing page.

[agjohnson]

Odd, I've removed basically everything from the template except the get_latest_build call, and timing hasn't changed at all. 😞

This is the same thing as the current templates though. However, I'm also now seeing the same slow load time at readthedocs.org, 3s for the project listing to show. The query that is taking the longest still looks like the last build query, but without some metadata on the page, the view is way too empty.

The build listing page performance is no better either.

I'm still happy with the clean up of stubbed in metadata/fields though, and quite like the layout now. I'm going to move on from this, this would be good to focus on during the public beta, or before if we get ambitious.

[agjohnson]

Think I might have found a clue here. I set up pyinstrument on the beta dashboard in production and did some profiling. I was trying to resolve #209 but having trouble even getting the page to load.

But I gave a try to the project listing and saw this:

So that already seems like a lot of time for is_admin.

I commented out the conditional block calling is_admin (this is on projects/partials/project_list.html) in production and page load times look great!

So, this can be resolved through some different template logic, hopefully.

[agjohnson]

Ayup. Same on build listing page too:

I'm going to bump this one up in priority, it should hopefully be a quick fix. Unfortunately, readthedocs/readthedocs.org#7277 doesn't seem usable in this case, where I am calling this in a listing view, once per project in the loop.

@humitos are you familiar with the correct query to use here? I could expose a is_project_admin:project style filter, but I'm not so sure.

[humitos]

@humitos are you familiar with the correct query to use here? I could expose a is_project_admin:project style filter, but I'm not so sure.

I posted my proposed solution in #49 (comment). I think the correct pattern is keep using is_admin:project everywhere and add a small ~5s cache on that filter tag.

[agjohnson]

Maybe I'm not understanding what you're suggesting correctly, but wouldn't a 5s cache invalidate before it would be useful?

The problem on at least the project listing page is that the is_admin conditional runs for each project in the listing page, not a single project. A cache on these checks would almost certainly invalidate before the next time the project listing page was loaded. Maybe a long cache timeout could work here, but I would start to worry about parity between our database and our cache.

The build listing page can likely just use the context variable approach, I'm guessing that view is duplicating the same check against a single version.

[humitos]

Here, I'm talking about the is_admin filter and is_project_admin context variable confusion. IMO, we should only use the is_admin filter and add a ~5s cache to solve the general issue.

The problem on at least the project listing page is that the is_admin conditional runs for each project in the listing page, not a single project

The cache won't solve the problem of iterating over a list of projects. It solves the general problem of using is_admin filter multiple times in the same template for the same project (similar to what readthedocs/readthedocs.org#7277 does but globally instead of just one page)

Iterating over multiple projects on the same template may require a specific solution, not a general one. I don't think caching the function/query for more time is a good solution --mainly because we are talking about permissions. We should improve the SQL query to solve this case by adding an index or similar approaches to make those queries faster.

[agjohnson]

Okay, maybe the fix needs to go in a different direction entirely then.

I understand the conflict between is_admin and is_project_admin, but I'll save that for another issue. I'm not yet looking to address general usage of is_admin, the issue here seems to be more that we can't really use is_admin while iterating over objects. I would agree that being able to use is_admin in templates keeps everything consistent, and another approach like caching could help some cases. Also though, adding a context variable is a simple approach, so there's that.

Alternative solutions for this issue could include doing a permission check from the front end code right before the user clicks the button for settings. This is the primary place these checks were added. I would only need some attribute in the API responses to indicate that the user has the correct permissions.

This is not ideal though. The buttons would need some default state, and really just shouldn't show for users that don't have permissions.

[agjohnson]

Actually, it doesn't seem like there are any attributes in the API response that I can use to check for authorization. I'm looking at both API v2 and v3 responses for a project.

@humitos is there another endpoint or something else we can use to inspect project authorization for a user in either API currently?

[agjohnson]

Build list performance is addressed with #244, but the project list performance will require something else, given we run checks against up to 15 projects per paginated page view.

Options I see remaining here:

• Add authorization information into an API response and use this on the frontend side to disable the configuration button. We need more API response or endpoints
• Cache the authorization check for some amount of minutes. This might be unsafe, timeout would need to be high to be usable.
• Always link to the configuration page, regardless of admin permissions. Let backend handle authorization UX. This is obviously confusing though.
• Don't link to the settings page from the listing UI 😞

[humitos]

@humitos is there another endpoint or something else we can use to inspect project authorization for a user in either API currently?

What data exactly do you need here?

The projects list endpoint (https://docs.readthedocs.io/en/stable/api/v3.html#projects-list) returns all the project the user has admin access to: https://github.com/readthedocs/readthedocs.org/blob/621fa5d5f2ad07934c0c3c8e7eb234a688158aa8/readthedocs/api/v3/views.py#L154-L159

I understand that a logic like "if the project is in the response list, then, show the link to settings page" should work.

[agjohnson]

I understand that a logic like "if the project is in the response list, then, show the link to settings page" should work.

For community, the answer is even simpler. The dashboard project list for an authenticated user doesn't even need the is_admin check, technically, as they would be a maintainer on every project in that list.

But on commercial, I don't think we can rely on the project list. Users can have different permission levels and we also show read only projects in that list.

I think always having the is_admin check is better for consistency though.

I was looking for that description of permissions in the project API response -- what level of permission does the requesting user have on the project in the API response. If we don't have this in the response already, the approach of conditionally disabling the UI is a bit hacky already and I don't yet feel great about it.

I may have to think more and come back to this.

[agjohnson]

Also worth mentioning that the API addition of the missing permission information would be usable in other API driven UI, like project/version/build/organization popup cards. Right now, these are a mix of template and API driven UI as the API doesn't have parity or the features are implemented as model methods only accessible to templates and application code.

[agjohnson]

Thinking about this more, I think the correct approach is to add a project API response field that brings in the is_admin logic for the request user. @humitos do you see any issues with this approach? Do you have concerns with slowing this query down with the this check? Maybe it's an expand field?

[humitos]

Hrm, adding a permission check for all the projects returned will definitely slow down the response. We can give this a try and see how it performs if you want.

I did a quick PR that implements ?expand=permissions on the project listing/detail API endpoints that you can use to try this out. It returns a new field in the form:

"permissions": {
  "admin": true
}

Note: the structure is based on GitHub's API, https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#get-a-repository. In case we need to expand to more permissions in the future, we already have the field.

The PR is at readthedocs/readthedocs.org#10978.

[agjohnson]

I've never paid attention to load time using expand fields. Will normal, unexpanded requests trigger the permission check code? I'm assuming it doesn't, which seems like a reasonable middle ground.

I'll pull that PR locally and try it out

[agjohnson]

So, page load time did indeed improve dramatically using the API over the template is_admin. Locally, the load time on the project listing dropped to 1s, where it was 5-8s normally. The change overall was medium:

• Improve page load performance on listing views #404

---

Code changes were overall pretty minor, but I did have to switch APIv2 to APIv3 endpoints though.

So this puts us back at:

• API: using private API endpoints on public listing pages #154

That is, we need APIv3 to support unauthenticated, public requests in order to support this functionality for public project views.

---

Needed to continue:

• Solution for public project views that doesn't involve private API data/endpoints -- addressing in API: using private API endpoints on public listing pages #154
• Add organization is_admin permissions to API v3 response
• Use project APIv3 response on other views where is_admin is used now
• Build listing API is_admin response too?

[agjohnson]

Yesterday, I was trying to replicate this change to the organization listing, but found it was still very slow to load. There might be something else happening on that view that is causing the slow down on the initial request.

[agjohnson]

So I actually think the organization listing view is not the is_admin filter but is instead something taking a lot of query time in the base queryset fro the view. I opened https://github.com/readthedocs/readthedocs-corporate/issues/1829

• https://github.com/readthedocs/readthedocs-corporate/issues/1829