Add metrics endpoints #665
Conversation
skoeva
force-pushed
the
skoeva/endpoints
branch
2 times, most recently
from
7cb2fd7
to
274f205
Compare
Pull request #656 adds the necessary instance_stats database table as well as functions for querying instance counts (by channel, version, and architecture) from the groups, instance, and instance_application tables and storing the results in the new table. To make this data accessible outside Nebraska, we create the following: - Prometheus metrics endpoint: A new HTTP endpoint (instance-metrics/prometheus) that serves only the latest snapshot from the instance_stats table. As Prometheus is a time-series database, this serves instance counts with the latest timestamp only. - JSON metrics data endpoint: A new HTTP endpoint that emits all instance_stats data in JSON format. The difference here is that we query for all data, and we emit one JSON document per row.
skoeva
force-pushed
the
skoeva/endpoints
branch
from
2e6dc8b
to
00ddefe
Compare
| /instance-metrics/json: | ||
| get: | ||
| description: Get instance stats | ||
| operationId: getInstanceStats | ||
| security: | ||
| - oidcBearerAuth: [] | ||
| - oidcCookieAuth: [] | ||
| - githubCookieAuth: [] | ||
| responses: | ||
| "200": | ||
| description: Get instance stats success response | ||
| content: | ||
| application/x-ndjson: | ||
| schema: | ||
| $ref: "#/components/schemas/instanceStats" | ||
| "500": | ||
| description: Get instance stats error response |
There was a problem hiding this comment.
Why is this a response stream? And do we want to return (via stream) all the instance stats in this endpoint? I would suggest to use some form pagination instead of stream.
There was a problem hiding this comment.
Yeah the goal is to output all instance stats here, and I agree with the pagination suggestion
| @@ -49,7 +49,7 @@ const ( | |||
|
|
|||
| const ( | |||
| validityInterval postgresDuration = "1 days" | |||
| defaultInterval time.Duration = time.Hour | |||
| defaultInterval time.Duration = 2 * time.Hour | |||
There was a problem hiding this comment.
Is this change intentional here?
There was a problem hiding this comment.
Yup! This was just an issue with rebasing, see #666
| @@ -651,7 +651,7 @@ func (api *API) GetDefaultInterval() time.Duration { | |||
| // that have been checked in during a given duration from a given time. | |||
| func (api *API) instanceStatsQuery(t *time.Time, duration *time.Duration) *goqu.SelectDataset { | |||
| if t == nil { | |||
| now := time.Now() | |||
| now := time.Now().UTC() | |||
There was a problem hiding this comment.
This change is done in #664, please rebase with main to avoid redoing this part.
| latestInstanceStatsSQL string = ` | ||
| SELECT channel_name, version, arch, timestamp, instances AS instances_count | ||
| FROM instance_stats | ||
| WHERE timestamp = (SELECT MAX(timestamp) FROM instance_stats) |
There was a problem hiding this comment.
Instead of doing raw SQL here, could we add a new function GetInstanceStatsLatest in pkg/api/instances.go and use that? This would also allow to add a test for this new function.
There was a problem hiding this comment.
I had this logic initially in pkg/api/instances.go but decided to follow the convention above, though I agree with this suggestion
|
I have left an update in my project doc for you guys to go through regarding my progress as my internship has concluded and I have started the fall semester at my university I'm unable to make changes with my current setup but happy to follow along with the work on Nebraska! |
Pull request #656 adds the necessary
instance_statsdatabase table as well as functions for querying instance counts (by channel, version, and architecture) from thegroups,instance, andinstance_applicationtables and storing the results in the new table.To make this data accessible outside Nebraska, we create the following:
instance_statstable. As Prometheus is a time-series database, this serves instance counts with the latest timestamp only.instance_statsdata in JSON format. The difference here is that we query for all data, and we emit one JSON document per row.