Upgrade a cluster in Kubernetes

​

Before you begin

1. Verify the overall health of your cluster using the :
   • Under Node Status, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as SUSPECT or DEAD, identify why the nodes are offline and either restart them or them before beginning your upgrade. If there are DEAD and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. If any node is not fully decommissioned, try the following:
     1. First, reissue the . The second command typically succeeds within a few minutes.
     2. If the second decommission command does not succeed, and then decommission it again. Before continuing the upgrade, the node must be marked as decommissioned.
   • Under Replication Status, make sure there are 0 under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it’s important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
   • In the Node List, make sure all nodes are on the same version. Upgrade them to the cluster’s current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized.
   • In the Metrics dashboards, make sure , , and capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider to your cluster before beginning your upgrade.
2. Make sure your cluster is behind a , or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail.
3. By default, the uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the COCKROACH_COMPACTION_CONCURRENCY environment variable. This may help to reshape the LSM more quickly in scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs strongly recommends testing your workload against non-default values of this setting.
4. CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for planning. Refer to .
5. Review the , as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized.
6. Optionally disable auto-finalization to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized.

​

Ensure you have a valid license key

• The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the .
• The cluster is at the end of the grace period if no before then.

SET CLUSTER SETTING diagnostics.reporting.enabled = true;

• The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
• The cluster is if telemetry is not received before the end of the grace period.

​

Perform a patch upgrade

1. Change the container image in the custom resource:
   image:
     name: cockroachdb/cockroach:v24.3.29
   
2. Apply the new settings to the cluster:
   kubectl apply -f example.yaml
   
   The Operator will perform the staged update.
3. To check the status of the rolling upgrade, run kubectl get pods.
4. Verify that all pods have been upgraded:
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   
   You can also check the CockroachDB version of each node in the .

​

Roll back a patch upgrade

​

Perform a major-version upgrade

1. Change the container image image in the custom resource:
   image:
     name: cockroachdb/cockroach:v24.3.29
   
2. Apply the new settings to the cluster:
   kubectl apply -f example.yaml
   
   The Operator will perform the staged update.
3. To check the status of the rolling upgrade, run kubectl get pods.
4. Verify that all pods have been upgraded:
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   
   You can also check the CockroachDB version of each node in the .
5. Before beginning a major-version upgrade, the Operator disables auto-finalization by setting the cluster setting cluster.preserve_downgrade_option to the cluster’s current major version. Before finalizing an upgrade, follow your organization’s testing procedures to decide whether to finalize or roll back the upgrade. After finalization begins, you can no longer roll back to the cluster’s previous major version.

​

Finalize a major-version upgrade manually

1. Connect to the cluster using the SQL shell:
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   
2. Run the following command:
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   
   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command SHOW CLUSTER SETTING version will return the previous version`. You can monitor the process of the migration in the DB Console . Migration jobs have names in the format 24.3-{migration-id}. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs. The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load. When all migration jobs have completed, the upgrade is complete.
3. To confirm that finalization has completed, check the cluster version:
   > SHOW CLUSTER SETTING version;
   
   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of where decommission has stalled. In most cases, issuing the decommission command again resolves the issue. If you have trouble upgrading, contact Support.

​

Roll back a major-version upgrade

1. Change the container image in the custom resource to use the previous major version:
   image:
     name: cockroachdb/cockroach:v24.3
   
2. Apply the new settings to the cluster:
   kubectl apply -f example.yaml
   
   The Operator will perform the staged rollback.
3. To check the status of the rollback, run kubectl get pods.
4. Verify that all pods have been rolled back:
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   

​

Disable auto-finalization