Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CRDB storage performance #1033

Open
BenjaminPelletier opened this issue May 16, 2024 · 1 comment
Open

Improve CRDB storage performance #1033

BenjaminPelletier opened this issue May 16, 2024 · 1 comment
Labels
deployment Related to deploying a DSS instance rather than application logic or behavior feature Issue would improve software P1 High priority

Comments

@BenjaminPelletier
Copy link
Member

Is your feature request related to a problem? Please describe.
A production GCP deployment was observed having relatively high request completion latency and was root-caused to likely be due to the storage type used for CockroachDB.

Cockroach Labs recommends the use of pd-ssd storage to avoid performance bottlenecks due to slow random-access storage. The pd-standard disks used for the standard storage class in GCP are intended for more traditional append-only / sequential read workloads, but we currently use the standard storage class in our deployment.

Describe the solution you'd like
Add a deployment configuration option to select the storage class to be used for CockroachDB nodes (all other deployment components can probably keep the standard storage class), default to premium-rwo (which selects pd-ssd disks), and document that standard or standard-rwo can be selected for cost savings.
image

Describe alternatives you've considered
We could hardcode the recommended storage solution to match Cockroach Labs' recommendation, but a configurable variable would allow us to serve more use cases, especially R&D instances with low performance needs but high cost sensitivity.

@BenjaminPelletier BenjaminPelletier added P1 High priority feature Issue would improve software deployment Related to deploying a DSS instance rather than application logic or behavior labels May 16, 2024
@barroco
Copy link
Contributor

barroco commented May 17, 2024

The default is indeed standard for Google Cloud (GKE) at the moment. I will prepare a PR which will change the default storage class to the type recommended by cockroach per cloud provider.

In the meantime, this can be configured depending on the deployment strategy:

Terraform

The storage class can be set using the terraform variable *_kubernetes_storage_class:

Tanka

As documented here, the storage class can be configured using the field cockroach.storage in the main.jsonnet file.

Helm

The storage class can be defined using Helm Chart value: cockroachdb.storage.persistentVolume.storageClass. Schema - Example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployment Related to deploying a DSS instance rather than application logic or behavior feature Issue would improve software P1 High priority
Projects
None yet
Development

No branches or pull requests

2 participants