getAccount(int id) {
return mapper.findAccountById(id);
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public int transferFunds(int fromId, int toId, int amount) {
return mapper.transfer(fromId, toId, amount);
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public long findCountOfAccounts() {
return mapper.findCountOfAccounts();
}
@Override
@Transactional(propagation = Propagation.REQUIRES_NEW)
public int deleteAllAccounts() {
return mapper.deleteAllAccounts();
}
}
```
Note that the public methods (i.e., the methods to be called by other classes in the project) are annotated as [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) methods. This ensures that all of the SQL statements executed in the data access layer are run within the context of a database transaction
`@Transactional` takes a number of parameters, including a `propagation` parameter that determines the transaction propagation behavior around an object (i.e., at what point in the stack a transaction starts and ends). `propagation=REQUIRES_NEW` for the methods in the service layer, meaning that a new transaction must be created each time a request is made to the service layer. With this propagation behavior, the application follows the [entity-control-boundary (ECB) pattern](https://wikipedia.org/wiki/Entity-control-boundary), as the service boundaries determine where a transaction starts and ends rather than the lower-level query methods of the [mapper interfaces](#mappers).
For more details on aspect-oriented transaction management in this application, [see below](#transaction-management).
### Models
Instances of the `Account` class, defined in `src/main/java/com/example/cockroachdemo/model/Account.java`, represent rows in the `accounts` table:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo.model;
public class Account {
private int id;
private int balance;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public int getBalance() {
return balance;
}
public void setBalance(int balance) {
this.balance = balance;
}
}
```
Instances of the `BatchResults` class, defined in `src/main/java/com/example/cockroachdemo/model/BatchResults.java`, hold metadata about a batch write operation and its results:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo.model;
public class BatchResults {
private int numberOfBatches;
private int totalRowsAffected;
public BatchResults(int numberOfBatches, int totalRowsAffected) {
this.numberOfBatches = numberOfBatches;
this.totalRowsAffected = totalRowsAffected;
}
public int getNumberOfBatches() {
return numberOfBatches;
}
public int getTotalRowsAffected() {
return totalRowsAffected;
}
}
```
### Transaction management
MyBatis-Spring supports Spring's [declarative, aspect-oriented transaction management syntax](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative), including the [`@Transactional`](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-annotations) annotation and [AspectJ's AOP annotations](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#transaction-declarative-aspectj).
Transactions may require retries if they experience deadlock or transaction contention that cannot be resolved without allowing serialization anomalies. To handle transactions that are aborted due to transient serialization errors, we highly recommend writing client-side transaction retry logic into applications written on CockroachDB. In this application, transaction retry logic is written into the methods of the `RetryableTransactionAspect` class, defined in `src/main/java/com/example/cockroachdemo/RetryableTransactionAspect.java`:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package com.example.cockroachdemo;
import java.lang.reflect.UndeclaredThrowableException;
import java.util.concurrent.atomic.AtomicLong;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.dao.ConcurrencyFailureException;
import org.springframework.dao.TransientDataAccessException;
import org.springframework.stereotype.Component;
import org.springframework.transaction.TransactionSystemException;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import org.springframework.util.Assert;
/**
* Aspect with an around advice that intercepts and retries transient concurrency exceptions.
* Methods matching the pointcut expression (annotated with @Transactional) are retried.
*
* This advice needs to run in a non-transactional context, which is before the underlying
* transaction advisor (@Order ensures that).
*/
@Component
@Aspect
// Before TX advisor
@Order(Ordered.LOWEST_PRECEDENCE - 1)
public class RetryableTransactionAspect {
protected final Logger logger = LoggerFactory.getLogger(getClass());
@Pointcut("@annotation(transactional)")
public void anyTransactionBoundaryOperation(Transactional transactional) {
}
@Around(value = "anyTransactionBoundaryOperation(transactional)",
argNames = "pjp,transactional")
public Object retryableOperation(ProceedingJoinPoint pjp, Transactional transactional)
throws Throwable {
final int totalRetries = 30;
int numAttempts = 0;
AtomicLong backoffMillis = new AtomicLong(150);
Assert.isTrue(!TransactionSynchronizationManager.isActualTransactionActive(), "TX active");
do {
try {
numAttempts++;
return pjp.proceed();
} catch (TransientDataAccessException | TransactionSystemException ex) {
handleTransientException(ex, numAttempts, totalRetries, pjp, backoffMillis);
} catch (UndeclaredThrowableException ex) {
Throwable t = ex.getUndeclaredThrowable();
if (t instanceof TransientDataAccessException) {
handleTransientException(t, numAttempts, totalRetries, pjp, backoffMillis);
} else {
throw ex;
}
}
} while (numAttempts < totalRetries);
throw new ConcurrencyFailureException("Too many transient errors (" + numAttempts + ") for method ["
+ pjp.getSignature().toLongString() + "]. Giving up!");
}
private void handleTransientException(Throwable ex, int numAttempts, int totalAttempts,
ProceedingJoinPoint pjp, AtomicLong backoffMillis) {
if (logger.isWarnEnabled()) {
logger.warn("Transient data access exception (" + numAttempts + " of max " + totalAttempts + ") "
+ "detected (retry in " + backoffMillis + " ms) "
+ "in method '" + pjp.getSignature().getDeclaringTypeName() + "." + pjp.getSignature().getName()
+ "': " + ex.getMessage());
}
if (backoffMillis.get() >= 0) {
try {
Thread.sleep(backoffMillis.get());
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
backoffMillis.set(Math.min((long) (backoffMillis.get() * 1.5), 1500));
}
}
}
```
The [`@Aspect` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-at-aspectj) declares `RetryableTransactionAspect` an [aspect](https://wikipedia.org/wiki/Aspect_\(computer_programming\)), with [pointcut](https://wikipedia.org/wiki/Pointcut) and [advice](https://wikipedia.org/wiki/Advice_\(programming\)) methods.
#### Transactional pointcut
The [`@Pointcut` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts) declares the `anyTransactionBoundaryOperation` method the pointcut for determining when to execute the aspect's advice. The `@annotation` [designator](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-pointcuts-designators) passed to the `@Pointcut` annotation limits the matches (i.e., [join points](https://wikipedia.org/wiki/Join_point)) to method calls with a specific annotation, in this case, `@Transactional`.
#### Transaction retry advice
`retryableOperation` handles the application retry logic, with [exponential backoff](https://wikipedia.org/wiki/Exponential_backoff), as the advice to execute at an `anyTransactionBoundaryOperation(transactional)` join point. Spring supports [several different annotations to declare advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-advice). The [`@Around` annotation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-around-advice) allows an advice method to work before and after the join point. It also gives the advice method control over executing any other matching advisors.
`retryableOperation` first verifies that there is no active transaction. It then increments the retry count and attempts to proceed to the next advice method with the `ProceedingJoinPoint.proceed()` method. If the underlying data access layer method (i.e., the mapper interface method annotated with `@Transactional`) succeeds, the results are returned and the application flow continues. If the method fails, then the transaction is retried. The time between each retry grows with each retry until the maximum number of retries is reached.
#### Advice ordering
Spring automatically adds [transaction management advice](https://docs.spring.io/spring/docs/current/spring-framework-reference/data-access.html#tx-decl-explained) to all methods annotated with `@Transactional`. Because the pointcut for `RetryableTransactionAspect` also matches methods annotated with `@Transactional`, there will always be two advisors that match the same pointcut. When multiple advisors match at the same pointcut, an `@Order` annotation on an advisor's aspect can specify the order in which the advice should be evaluated.
To control when and how often a transaction is retried, the transaction retry advice must be executed outside the context of a transaction (i.e., it must be evaluated before the primary transaction management advisor). By default, the primary transaction management advisor is given the lowest level of precedence. The `@Order` annotation on `RetryableTransactionAspect` is passed `Ordered.LOWEST_PRECEDENCE-1`, which places this aspect's advice at a level of precedence above the primary transaction advisor, which results in the retry logic being evaluated before the transaction management advisor.
For more details about advice ordering in Spring, see [Advice Ordering](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-ataspectj-advice-ordering) on the Spring documentation site.
## See also
Spring documentation:
* [Spring Boot website](https://spring.io/projects/spring-boot)
* [Spring Framework Overview](https://docs.spring.io/spring/docs/current/spring-framework-reference/overview.html#overview)
* [Spring Core documentation](https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#spring-core)
* [MyBatis documentation](https://mybatis.org/mybatis-3/)
* [MyBatis Spring integration](https://mybatis.org/spring/)
CockroachDB documentation:
* Learn CockroachDB SQL
* Client Connection Parameters
* CockroachDB Developer Guide
* Transactions
# Build a TypeScript App with CockroachDB and TypeORM
Source: https://docs.cockroachlabs.com/docs/v26.3/build-a-typescript-app-with-cockroachdb
This tutorial shows you how run a simple application built with [TypeORM](https://typeorm.io/#/).
## Step 1. Start CockroachDB
### Choose your installation method
You can create a CockroachDB Basic cluster using either the CockroachDB Cloud Console, a web-based graphical user interface (GUI) tool, or `ccloud`, a command-line interface (CLI) tool.
### Create a free cluster
Organizations without billing information on file can only create one CockroachDB Basic cluster.
1. If you haven't already, [sign up for a CockroachDB Cloud account](https://cockroachlabs.cloud/signup?referralId=docs_typescript).
2. [Log in](https://cockroachlabs.cloud/) to your CockroachDB Cloud account.
3. On the **Clusters** page, click **Create cluster**.
4. On the **Select a plan** page, select **Basic**.
5. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS) in the **Cloud provider** section.
6. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Basic clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
7. Click **Next: Capacity**.
8. On the **Capacity** page, select **Start for free**. Click **Next: Finalize**.
9. On the **Finalize** page, click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
### Create a SQL user
The **Create SQL user** dialog allows you to create a new SQL user and password.
1. Enter a username in the **SQL user** field or use the one provided by default.
2. Click **Generate & save password**.
3. Copy the generated password and save it in a secure location.
4. Click **Next**.
Currently, all new SQL users are created with admin privileges. For more information and to change the default settings, see Manage SQL users on a cluster.
### Get the connection string
The **Connect to cluster** dialog shows information about how to connect to your cluster.
1. Select **General connection string** from the **Select option** dropdown.
2. Open the **General connection string** section, then copy the connection string provided and save it in a secure location.
The sample application used in this tutorial uses system CA certificates for server certificate verification, so you can skip the **Download CA Cert** instructions.
The connection string is pre-populated with your username, password, cluster name, and other details. Your password, in particular, will be provided *only once*. Save it in a secure place (Cockroach Labs recommends a password manager) to connect to your cluster in the future. If you forget your password, you can reset it by going to the **SQL Users** page for the cluster, found at `https://cockroachlabs.cloud/cluster//users`.
Follow these steps to create a CockroachDB Basic cluster using the `ccloud` CLI tool.
The `ccloud` CLI tool is in Preview.
### Install `ccloud`
You can install `ccloud` using either Homebrew or by downloading the binary.
#### Use Homebrew
1. [Install Homebrew](http://brew.sh/).
2. Install using the `ccloud` tap:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
brew install cockroachdb/tap/ccloud
```
#### Download the binary
In a terminal, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_darwin-amd64_.tar.gz | tar -xJ && cp -i ccloud /usr/local/bin/
```
Use the ARM 64 binary if you have an M-series Mac:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_darwin-arm64_.tar.gz | tar -xJ && cp -i ccloud /usr/local/bin/
```
In a terminal, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https://binaries.cockroachdb.com/ccloud/ccloud_linux-amd64_.tar.gz | tar -xz && cp -i ccloud /usr/local/bin/
```
In a PowerShell window, enter the following command to download and extract the `ccloud` binary and add it to your `PATH`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ErrorActionPreference = "Stop"; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; $ProgressPreference = 'SilentlyContinue'; $null = New-Item -Type Directory -Force $env:appdata/ccloud; Invoke-WebRequest -Uri https://binaries.cockroachdb.com/ccloud/ccloud_windows-amd64_.zip -OutFile ccloud.zip; Expand-Archive -Force -Path ccloud.zip; Copy-Item -Force ccloud/ccloud.exe -Destination $env:appdata/ccloud; $Env:PATH += ";$env:appdata/ccloud"; # We recommend adding ";$env:appdata/ccloud" to the Path variable for your system environment. See https://docs.microsoft.com/powershell/module/microsoft.powershell.core/about/about_environment_variables#saving-changes-to-environment-variables for more information.
```
### Run `ccloud quickstart` to create a new cluster, create a SQL user, and retrieve the connection string.
The easiest way of getting started with CockroachDB Cloud is to use `ccloud quickstart`. The `ccloud quickstart` command guides you through logging in to CockroachDB Cloud, creating a new CockroachDB Basic cluster, and connecting to the new cluster. Run `ccloud quickstart` and follow the instructions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ccloud quickstart
```
The `ccloud quickstart` command will open a browser window to log you in to CockroachDB Cloud. If you are new to CockroachDB Cloud, you can register using one of the single sign-on (SSO) options, or create a new account using an email address.
The `ccloud quickstart` command will prompt you for the cluster name, cloud provider, and cloud provider region, then ask if you want to connect to the cluster. Each prompt has default values that you can select, or change if you want a different option.
Select **General connection string**, then copy the connection string displayed and save it in a secure location. The connection string is the line starting `postgresql://`.
```
? How would you like to connect? General connection string
Retrieving cluster info: succeeded
Downloading cluster cert to /Users/maxroach/.postgresql/root.crt: succeeded
postgresql://maxroach:ThisIsNotAGoodPassword@blue-dog-147.6wr.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert=%2FUsers%2Fmaxroach%2F.postgresql%2Froot.crt
```
1. If you haven't already, download the CockroachDB binary.
2. Run the `cockroach start-single-node` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start-single-node --advertise-addr 'localhost' --insecure
```
This starts an insecure, single-node cluster.
3. Take note of the following connection information in the SQL shell welcome text:
```
CockroachDB node starting at 2021-08-30 17:25:30.06524 +0000 UTC (took 4.3s)
build: CCL v21.1.6 @ 2021/07/20 15:33:43 (go1.15.11)
webui: http://localhost:8080
sql: postgresql://root@localhost:26257?sslmode=disable
```
You'll use the `sql` connection string to connect to the cluster later in this tutorial.
The `--insecure` flag used in this tutorial is intended for non-production testing only. To run CockroachDB in production, use a secure cluster instead.
## Step 2. Get the code
1. Clone [the code's GitHub repository](https://github.com/cockroachlabs/example-app-typescript-typeorm):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ git clone git@github.com:cockroachlabs/example-app-typescript-typeorm.git
```
2. Navigate to the repo directory and install the application dependencies:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cd example-app-typescript-typeorm
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ npm install
```
## Step 3. Configure your CockroachDB connection
1. Open the `datasource.ts` file, and comment out the `ssl: true`, `extra` and `options` configuration properties.
2. In the `datasource.ts` file, uncomment `ssl: { rejectUnauthorized: false }`.
Only use `ssl: { rejectUnauthorized: false }` in development, for insecure connections.
The `DataSource` configuration should look similar to the following:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: { rejectUnauthorized: false }, // For insecure connections only
synchronize: true,
logging: false,
entities: ["src/entity/**/*.ts"],
migrations: ["src/migration/**/*.ts"],
subscribers: ["src/subscriber/**/*.ts"],
})
```
1. Set the `DATABASE_URL` environment variable to the connection string provided in the `cockroach` welcome text:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="postgresql://root@localhost:26257/defaultdb?sslmode=disable"
```
2. Set the `DATABASE_URL` environment variable to a CockroachDB connection string compatible with TypeORM.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="CockroachDB Cloud MCP server.
* Cluster-level read and write access for self-hosted deployments via the CockroachDB MCP Server.
* Operational workflows encoded as Agent Skills for CockroachDB.
* Command-line automation with the `ccloud` CLI, designed to be compatible with AI agents.
* Access to the Cockroach Labs public documentation via the CockroachDB Docs MCP server.
The following sections provide an overview of these tools.
### CockroachDB MCP servers
CockroachDB provides Model Context Protocol (MCP) servers that expose a set of tools for inspecting and querying your clusters from your AI tools. These tools let your AI assistants list databases and tables, describe schemas and indexes, inspect cluster health and running queries, and run read-only SQL and `EXPLAIN` statements. When explicitly enabled, they can also create databases, create tables, and insert rows.
This allows you to manage and modify a CockroachDB cluster using natural language prompts. For example, you could use the following prompt to instruct your AI tool to interface with the cluster:
```
List all of the tables in the demo_db database.
```
An MCP server is available for both deployment types: CockroachDB Cloud clusters use the managed CockroachDB Cloud MCP server, and self-hosted clusters use the CockroachDB MCP Server, which you run yourself.
### Agent Skills for CockroachDB
Agent Skills for CockroachDB are small, structured capabilities that encode CockroachDB operational expertise in a machine-executable format. These skills live in the public [cockroachdb-skills](https://github.com/cockroachlabs/cockroachdb-skills) repository and follow the [Agent Skills Specification](https://agentskills.io/specification), with defined inputs, outputs, and safety guardrails.
Each skill focuses on a specific task, such as auditing user privileges, triaging live SQL activity, validating production readiness, or checking backup and disaster recovery posture. Skills are organized into domains like onboarding and migrations, application development, performance and scaling, operations and lifecycle, resilience and disaster recovery, observability and diagnostics, security and governance, integrations, and cost management.
Your AI tools can consume these skills directly, so you can reuse the same operational workflows across different toolchains.
### `ccloud` command-line interface
The `ccloud` CLI is the command-line interface for CockroachDB Cloud. You can use the `ccloud` CLI to create clusters, manage networking (for example, IP allowlists), create SQL users, retrieve connection information, and more.
Because `ccloud` is text-based and follows a stable command structure, it is well-suited for AI tools and automations. An AI assistant can generate or run `ccloud` commands to set up clusters, rotate credentials, or retrieve connection URLs, while you keep access mediated through the CLI and existing Cloud authentication.
### CockroachDB Docs MCP server
The CockroachDB Docs MCP server exposes the published CockroachDB documentation to your MCP-compatible tools over HTTP. After you add the server configuration to your client, your AI assistant can answer questions using the official documentation without leaving your editor. For example, you could use your AI tool to ask the following:
```
How do table statistics get refreshed in CockroachDB?
```
An AI tool using the documentation MCP server can provide a response to this prompt informed by documentation pages such as Cost-Based Optimizer and `CREATE STATISTICS`.
This integration only provides access to CockroachDB product documentation. It does not connect to your clusters or data. It is intended to give your AI tools reliable product information while you develop and operate CockroachDB and CockroachDB Cloud.
## CockroachDB as a data store for AI applications
CockroachDB provides the database features needed to store and query AI-related data, including vector embeddings and agent state, with the same transactional guarantees as your other workloads.
### Vector search and RAG
AI applications often represent text, images, and other content as vector embeddings. These are numerical representations that capture semantic meaning. To find relevant information, AI applications need to search for vectors that are similar to a query vector, typically using distance metrics. This similarity search is fundamental to retrieval-augmented generation (RAG), semantic search, and recommendation systems.
CockroachDB has a `VECTOR` data type for storing fixed-length floating-point embeddings and supports similarity operators such as L2 distance (`<->`), inner product (`<#>`), and cosine distance (`<=>`). You can index vectors using `CREATE VECTOR INDEX` and combine them with other indexed columns.
You can store vector embeddings, relational data, and JSON metadata in the same table and query them together. For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SELECT id, name
FROM items
WHERE customer_id = 1
ORDER BY embedding <-> $query_vector
LIMIT 5;
```
RAG systems depend on fast and accurate retrieval of contextual data. CockroachDB's vector support enables you to implement RAG and semantic search patterns so that you can use CockroachDB as the data store for AI applications. Vector indexes enable efficient similarity searches over large datasets. You can store document embeddings in a cluster alongside the source documents, metadata, and access control information associated with those documents. When a user query comes in, you can use vector similarity operators to retrieve semantically relevant documents, filter by permissions, and return both the relevant context and any associated metadata in a single transaction.
See the [RAG tutorial](https://www.cockroachlabs.com/blog/tutorial-rag-with-cockroachdb/) for a complete implementation example.
### AI agent state and workflow coordination
AI agents that perform autonomous operations require durable storage for execution state, workflow metadata, and operational history. These agents must track state transitions across multi-step processes, coordinate activities between concurrent executions, and ensure that operations can be safely retried or resumed after failures.
CockroachDB's transactional model provides a foundation for storing agent state, execution history, and coordination metadata. Serializable isolation ensures that state transitions occur correctly even when multiple agents or processes attempt concurrent updates. The database's high availability design allows agents to continue operating during node or region failures without requiring external coordination services.
### Scale, consistency, and governance
AI applications typically generate high data volumes, serve globally distributed users, and require both transactional correctness and operational durability. Conversation histories, vector embeddings, feature tables, and agent state accumulate quickly and are accessed across regions. These characteristics make AI workloads well-suited to CockroachDB's core design:
* CockroachDB scales horizontally by adding nodes to increase capacity.
* Data is automatically replicated and rebalanced across the cluster, so node failures do not require application-level failover.
* Multi-region deployments place data closer to users and can enforce data locality or residency requirements while maintaining strong consistency.
CockroachDB provides serializable transactions by default. Vector indexes participate in the same transaction and index maintenance model as other secondary indexes, so similarity search results remain aligned with the underlying data. This applies to tables storing embeddings, relational records, and agent state.
## See also
* [Tutorial: Augment your AI use case with RAG on CockroachDB](https://www.cockroachlabs.com/blog/tutorial-rag-with-cockroachdb/)
* [Real-Time Indexing for Billions of Vectors](https://www.cockroachlabs.com/blog/cspann-real-time-indexing-billions-vectors/)
* [How CockroachDB's AI Assistance Boosts Developer Productivity](https://www.cockroachlabs.com/blog/cockroachdb-ai-assistance-for-developers/)
* [Agent Development with CockroachDB using the LangChain Framework](https://www.cockroachlabs.com/blog/agent-development-cockroachdb-langchain/)
* [CockroachDB Plugin for Claude Code](https://github.com/cockroachdb/claude-plugin)
* [CockroachDB Plugin for Cursor](https://github.com/cockroachdb/cursor-plugin)
* [Zero to AI Hero](https://youtu.be/PQdeDj9QSaU)
* [Fraud Detection via Agentic AI](https://youtu.be/YzIv9yd5l_M)
* [Vector Deep Dive](https://youtu.be/rR2G35NHdZQ)
# Deploy a CockroachDB Cloud Application with Google Cloud Run
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-gcr
This tutorial shows you how to use Google Cloud Run to deploy a containerized Django application that communicates with a CockroachDB Standard cluster.
## Before you begin
Before starting the tutorial, do the following:
1. Create a [Google Cloud](https://cloud.google.com/) account and log in.
2. Install the [Google Cloud SDK](https://cloud.google.com/sdk).
3. Install [Docker Desktop](https://www.docker.com/products/docker-desktop).
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
## Step 2. Set up your cluster connection
Once your cluster is created, the **Connect to cluster-name** dialog displays. Use the information provided in the dialog to set up your cluster connection for the SQL user that was created by default:
1. In your terminal, run the second command from the dialog to create a new `certs` directory on your local machine and download the CA certificate to that directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --create-dirs -o ~/.postgresql/root.crt -O https://cockroachlabs.cloud/clusters/{cluster-id}/cert
```
Your `cert` file will be downloaded to `~/.postgresql/root.crt`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --create-dirs -o ~/.postgresql/root.crt -O https://cockroachlabs.cloud/clusters/{cluster-id}/cert
```
Your `cert` file will be downloaded to `~/.postgresql/root.crt`.
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir -p $env:appdata\.postgresql\; Invoke-WebRequest -Uri https://cockroachlabs.cloud/clusters/{cluster-id}/cert -OutFile $env:appdata\.postgresql\root.crt
```
Your `cert` file will be downloaded to `%APPDATA%/.postgresql/root.crt`.
2. Copy the connection string provided, which will be used in the next steps (and to connect to your cluster in the future).
This connection string contains your password, which will be provided only once. If you forget your password, you can reset it by going to the **SQL Users** page for the cluster, found at `https://cockroachlabs.cloud/cluster//users`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url 'postgresql://download the CockroachDB SQL Shell binary.
2. Start the built-in SQL shell using the connection string you got from the CockroachDB Cloud Console earlier:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql \
--url='postgres:// CREATE DATABASE bank;
```
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 4. Get the application code
1. Clone the example code's GitHub repo:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
git clone https://github.com/cockroachlabs/example-app-python-django/
```
2. Create a new directory named `certs` at the top level of the `example-app-python-django` project, and then copy the root certificate that you downloaded for your cluster to the new directory.
The project directory structure should look like this:
```
├── Dockerfile
├── README.md
├── certs
│ └── root.crt
├── cockroach_example
│ ├── cockroach_example
│ │ ├── __init__.py
│ │ ├── asgi.py
│ │ ├── migrations
│ │ │ ├── 0001_initial.py
│ │ │ └── __init__.py
│ │ ├── models.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ ├── views.py
│ │ └── wsgi.py
│ └── manage.py
└── requirements.txt
```
## Step 5. Initialize the database and test the app locally
1. At the top level of the app's project directory, create and then activate a virtual environment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
virtualenv env
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
source env/bin/activate
```
2. Install the required modules to the virtual environment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pip install -r requirements.txt
```
3. Set the `DATABASE_URL` environment variable to the connection string provided in the **Connection info** window of the CockroachDB Cloud Console, but with the root certificate located in the local `certs` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="postgresql://$USER:$PASSWORD@random-cluster-name-4300.6wr.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert=root.crt"
```
This Django app uses the `dj_database_url` module to configure the database connection from a connection URL. The module uses the value assigned to the `DATABASE_URL` environment variable for the connection.
In the Cloud Run deployment, we use the Google Cloud Secret Manager to define the `DATABASE_URL` environment variable for the deployment.
1. Execute the initial database schema migration:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
python3 cockroach_example/manage.py migrate
```
This migration initializes the `bank` database with the tables defined in `models.py`, in addition to some other tables for the admin functionality included with Django's starter application.
2. Run the app:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
python3 cockroach_example/manage.py runserver 0.0.0.0:8000
```
The output should look like this:
```
...
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
```
To perform simple reads and writes to the database, you can send HTTP requests to the application at [http://0.0.0.0:8000/](http://0.0.0.0:8000/).
3. In a new terminal, use `curl` to send a POST request to the application:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl --header "Content-Type: application/json" \
--request POST \
--data '{"name":"Carl"}' http://0.0.0.0:8000/customer/
```
This request inserts a new row into the `cockroach_example_customers` table.
4. Send a GET request to read from the `cockroach_example_customers` table:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl http://0.0.0.0:8000/customer/
```
```
[{"id": "bb7d6c4d-efb3-45f8-b790-9911aae7d8b2", "name": "Carl"}]
```
You can also query the table directly in the SQL shell to see the changes:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM bank.cockroach_example_customers;
```
```
id | name
---------------------------------------+-------
bb7d6c4d-efb3-45f8-b790-9911aae7d8b2 | Carl
(1 row)
```
5. Enter **Ctrl+C** to stop the application.
## Step 6. Configure GCP
1. In the terminal, authenticate the `gcloud` command-line tool with your Google Cloud account:
`gcloud` is included with the [Google Cloud SDK](https://cloud.google.com/sdk) installation.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud auth login
```
Follow the prompts to authenticate.
1. Create a Google Cloud project for the application deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud projects create {gcp_project_id}
```
You can specify a location for the project within your Google Cloud resources by using the `--organization` or `--folder` flags.
1. Configure the CLI to use your Google Cloud account and the new project ID by default:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud init
```
2. Set the `PROJECT_ID` environment variable:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export PROJECT_ID={gcp_project_id}
```
For the rest of the tutorial, we use `PROJECT_ID` to refer to the project ID.
## Step 7. Containerize the application and push it to the registry
1. Build the Docker image locally:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker build -t gcr.io/$PROJECT_ID/crdb-sample:v1 .
```
If there are no errors, the container built successfully.
2. Authenticate Docker with GCP's Container Registry:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud auth configure-docker
```
3. Enable the Container Registry API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable containerregistry.googleapis.com
```
4. Push the Docker image to the project's registry.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker push gcr.io/$PROJECT_ID/crdb-sample:v1
```
## Step 8. Create a secret for the database connection URI
1. Create a service account to manage the secrets for your project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud iam service-accounts create cockroach-labs
```
2. Enable the Secret Manager API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable secretmanager.googleapis.com
```
3. Create a secret for the connection string stored locally in the `DATABASE_URL` environment variable, and bind the new service account to the secret.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
echo $DATABASE_URL | gcloud secrets create cockroach-connection-uri --data-file=- --replication-policy=automatic
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud secrets add-iam-policy-binding cockroach-connection-uri \
--member=serviceAccount:cockroach-labs@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/secretmanager.secretAccessor
```
## Step 9. Deploy the application on Cloud Run
1. Enable the Cloud Run API for the project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud services enable run.googleapis.com
```
2. Create a [Cloud Run](https://console.cloud.google.com/run/) service for the application:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud alpha run deploy crl-app \
--region us-central1 \
--allow-unauthenticated \
--service-account "cockroach-labs@${PROJECT_ID}.iam.gserviceaccount.com" \
--set-secrets "DATABASE_URL=cockroach-connection-uri:latest" \
--image gcr.io/${PROJECT_ID}/crdb-sample:v1
```
Note the following:
* The `--region` flag specifies the region of the CockroachDB node targeted in the connection string.
* The `--service-account` flag specifies the `cockroach-labs` service account that you created earlier for the app deployment.
* The `--set-secrets` flag sets the `DATABASE_URL` environment variable to the `cockroach-connection-uri` secret that you created earlier.
* The `--image` flag specifies the container image URL for the `crdb-sample` image that you pushed to the container registry.
If prompted, select `Cloud Run (fully managed)`.
3. After the revision is deployed, verify that you can send requests to the application from a browser or a REST client:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl https:///customer/
```
```
[{"id": "bb7d6c4d-efb3-45f8-b790-9911aae7d8b2", "name": "Carl"}]
```
4. By default, the sample application allows all hosts/domain names to serve the application. After testing, we recommend that you update the [`ALLOWED_HOSTS` property in `settings.py`](https://docs.djangoproject.com/en/3.2/ref/settings/#allowed-hosts) to allow only a local testing URL and the Cloud Run service URL to serve the application.
## See also
* Build a Simple Django App with CockroachDB
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deploy a Netlify App Built on CockroachDB
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-netlify
This tutorial shows you how to deploy a [Netlify](https://www.netlify.com/) web application that communicates with a CockroachDB Standard cluster.
The sample app used in this tutorial simulates a [gaming leaderboard](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/) using [Netlify functions](https://www.netlify.com/products/functions/) written in TypeScript. The functions use [Prisma](https://www.prisma.io/) to connect to CockroachDB. The app's frontend, also written in TypeScript, uses React, bootstrapped with [Create React App](https://github.com/facebook/create-react-app).
You can view or download the [source code](https://github.com/cockroachdb/cockroachdb-typescript) for the example.
## Before you begin
Before starting the tutorial:
1. Create a Starter [Netlify](https://app.netlify.com/signup) account.
2. Install the `netlify` CLI locally and log in. Refer to [Get started with Netlify CLI](https://docs.netlify.com/cli/get-started/).
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
1. Open the code repository's [GitHub repository](https://github.com/cockroachdb/cockroachdb-typescript).
2. Fork it by clicking **Fork** at the top. Netlify requires you to own a Netlify app's repository.
* Set **Owner** to your GitHub identity or an organization.
* Disable **Copy the master branch only**.
* Click **Fork**.
You now have an exact copy of the code repository in the GitHub location you chose.
3. Copy the fork's address, which you will use to clone it locally. Click **Code**, then in the **Local** tab, click **SSH** if you have added SSH keys to GitHub, or **HTTPS** if not. Click the copy symbol to copy the address.
4. Clone your fork locally. Replace with the full address of your fork, which will end with `cockroachdb-typescript.git`.
The project has the following directory structure:
```
├── LICENSE.md
├── README.md
├── netlify
│ └── functions
│ ├── addScore.ts
│ ├── getPlayers.ts
│ └── getScores.ts
├── package-lock.json
├── package.json
├── prisma
│ ├── schema.prisma
│ └── seed.ts
├── public
├── src
└── tsconfig.json
```
This tutorial modifies the files in the `netlify` and `prisma` directories.
## Step 3. Initialize the database
1. Open the project's `.env` file in a text editor set the `DATABASE_URL` environment variable to [the connection string](#connection-string) you obtained earlier from the CockroachDB Cloud Console:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
DATABASE_URL=`SELECT` and `INSERT` queries against the database:
* [`getScores.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/getScores.ts) reads all rows from the `player_scores` table and returns values in the `id`, `name`, and `score` columns.
* [`getPlayers.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/getPlayers.ts) reads and returns all rows from the `players` table.
* [`addScore.ts`](https://raw.githubusercontent.com/cockroachdb/cockroachdb-typescript/master/netlify/functions/addScore.ts) writes new scores to the `player_scores` table.
2. Deploy your app with the Netlify CLI:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
netlify deploy
```
Choose to create a new site, and then select the default options for each of the subsequent prompts. You will be required to log in to Netlify using GitHub.
After the app is deployed, you should see the following message:
```
✔ Deploy is live!
```
3. Navigate to the admin URL for your site:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
netlify open
```
From the **Site overview** page, you can manage your site. The **Site overview** page also provides you with a public URL for your site.
## See also
* [How to build a Complete Webapp with React, TypeScript & CockroachDB](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/#deploy-the-application-to-netlify)
* Build a Simple CRUD Node.js App with CockroachDB and Prisma Client
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Deploy a Web App Built on CockroachDB with Vercel
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-app-vercel
This tutorial shows you how to use [Vercel](https://vercel.com/) to deploy a web application built with [Next.js](https://nextjs.org/), [Express](https://expressjs.com/), [Prisma](https://www.prisma.io/), and CockroachDB Standard.
## Before you begin
Before starting the tutorial, create a [Vercel](https://vercel.com/signup) account and sign in.
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
1. Open the code repository's [GitHub repository](https://github.com/cockroachdb/prisma-examples).
2. Fork it by clicking **Fork** at the top. Netlify requires you to own a Netlify app's repository.
* Set **Owner** to your GitHub identity or an organization.
* Disable **Copy the master branch only**.
* Click **Fork**.
You now have an exact copy of the code repository in the GitHub location you chose.
3. Copy the fork's address, which you will use to clone it locally. Click **Code**, then in the **Local** tab, click **SSH** if you have added SSH keys to GitHub, or **HTTPS** if not. Click the copy symbol to copy the address.
4. Clone your fork locally. Replace with the full address of your fork, which will end with `prisma-examples.git`.
## Step 3. Create a Vercel project
1. From the [Vercel dashboard](https://vercel.com/dashboard), click **Add new\...** > **Project**.
2. Select the `prisma-examples` repository you forked in [Step 2](#step-2-get-the-code) to import.
3. From the **Framework preset** dropdown, select **Next.js**.
4. In the **Root directory** field, click **Edit**.
5. Select the **typescript** > **rest-nextjs-api-routes** directory.
6. Click **Continue**.
7. Open the **Build and Output Settings** and toggle the build command's **Override** switch to on.
8. Enter `yarn prisma db push && yarn next build` as the **Build command**.
9. Click **Deploy**.
Your deployment will fail until you integrate CockroachDB in the next step, so you can leave this screen without waiting for it to finish.
## Step 4. Integrate your project with CockroachDB
1. Navigate to the [CockroachDB page](https://vercel.com/integrations/cockroachdb) of Vercel's integration marketplace.
2. Click **Add Integration**.
3. Select your Vercel account and click **Continue**.
4. Select the project you just created and click **Continue**.
5. Accept the permissions and click **Add Integration**.
6. When prompted, log into CockroachDB Cloud and select the organization and cluster to integrate with.
7. Click **Create** to finish the integration.
After a few seconds, the window will close automatically and your Vercel project will have the `DATABASE_URL` environment variable configured with the cluster's connection string.
## Step 5. Deploy the application
1. Navigate to the **Overview** page for your Vercel project.
2. Select the **Deployments** tab.
3. Click **Redeploy**.
4. Click **Redeploy** again from the pop-up dialog.
Your project will be deployed in a few seconds. You can view the final product by clicking on the **Domain** link on your project's **Overview** page.
## Step 3. Initialize the database
1. Save [the connection string](#connection-string) you obtained earlier from the CockroachDB Cloud Console to the `DATABASE_URL` environment variable in an `.env` file in your project:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
echo "DATABASE_URL=> .env
```
Prisma loads the variables defined in `.env` to the project environment. By default, Prisma uses the `DATABASE_URL` environment variable as the connection string to the database.
2. Run [Prisma Migrate](https://www.prisma.io/docs/concepts/components/prisma-migrate) to initialize the database with the schema defined in `prisma/prisma.schema`.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
prisma migrate dev --name init
```
You should see the following output:
```
Your database is now in sync with your schema.
✔ Generated Prisma Client (3.11.0 | library) to ./node_modules/@prisma/client in 87ms
```
This command also initializes [Prisma Client](https://www.prisma.io/docs/concepts/components/prisma-client) to communicate with your CockroachDB cluster, based on the configuration in the `prisma/schema.prisma` file.
## Step 4. Deploy the application
1. (Optional) Run the app server locally to preview your site:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
npm run dev
```
You should see the following output:
```
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
```
2. Run the `vercel` command to start deploying the app:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
vercel deploy --confirm
```
You will be asked to sign into your Vercel account. When you authenticate your credentials, you should see the following message:
```
> Success! GitHub authentication complete ...
```
The [`--confirm` flag](https://vercel.com/docs/cli#introduction/unique-options/confirm) instructs Vercel to deploy the app to the `example-app-typescript-vercel` Vercel project, using the default settings:
```
🔗 Linked to username/example-app-typescript-vercel (created .vercel and added it to .gitignore)
🔍 Inspect: https://vercel.com/username/example-app-typescript-vercel/123456789 [1s]
✅ Production: https://example-app-typescript-vercel-username.vercel.app [copied to clipboard] [47s]
📝 Deployed to production. Run `vercel --prod` to overwrite later (https://vercel.link/2F).
💡 To change the domain or build command, go to https://vercel.com/ericharmeling/example-app-typescript-vercel/settings
```
Follow the links provided to view and manage your deployed application.
## See also
* [Vercel's official How to Build a Fullstack App with Next.js, Prisma, and PostgreSQL guide](https://vercel.com/guides/nextjs-prisma-postgres)
* [How to build a Complete Webapp with React, TypeScript & CockroachDB](https://www.cockroachlabs.com/blog/react-typescript-cockroachdb-sample-app/#deploy-the-application-to-netlify)
* Build a Simple CRUD Node.js App with CockroachDB and Prisma Client
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Create and Deploy an AWS Lambda Function Built on CockroachDB
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-lambda-function
This tutorial shows you how to create an [AWS Lambda](https://aws.amazon.com/lambda) function that communicates with a CockroachDB Standard cluster.
## Before you begin
Before starting the tutorial, do the following:
1. Create an [AWS](https://aws.amazon.com/) account and log in, then:
1. Create an [AWS user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) with administrator permissions.
2. Install the [AWS CLI](https://aws.amazon.com/cli/) and log in locally.
## Step 1. Create a CockroachDB Standard cluster
1. Create a CockroachDB Cloud account. If this is your first CockroachDB Cloud organization, it will be credited with \$400 in free trial credits to get you started.
2. On the **Get Started** page, click **Create cluster**.
3. On the **Select a plan** page, select **Standard**.
4. On the **Cloud & Regions** page, select a cloud provider (GCP or AWS).
5. In the **Regions** section, select a region for the cluster. Refer to CockroachDB Cloud Regions for the regions where CockroachDB Standard clusters can be deployed. To create a multi-region cluster, click **Add region** and select additional regions.
6. Click **Next: Capacity**.
7. On the **Capacity** page, keep the **Provisioned capacity** at the default value of 2 vCPUs.
Click **Next: Finalize**.
8. On the **Finalize** page, name your cluster. If an active free trial is listed in the right pane, you will not need to add a payment method, though you will need to do this by the end of the trial to maintain your organization's clusters.
Click **Create cluster**.
Your cluster will be created in a few seconds and the **Create SQL user** dialog will display.
After the cluster is created, the **Connection info** window appears. Click the **Connection string** tab and copy the connection string to a secure location. You will use this connection string to connect to CockroachDB later in the tutorial.
The connection string is pre-populated with your SQL username and password, the cluster name, and other details. Save the password in a secure place such a password manager. You can find the connection string at any time from the CockroachDB Cloud Console. You can reset your SQL password for this cluster from the cluster's **SQL Users** page.
## Step 2. Get the code
In a terminal, clone the [sample code's GitHub repo](https://github.com/cockroachlabs/examples-aws-lambda):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
git clone https://github.com/cockroachlabs/examples-aws-lambda
```
This repo includes samples for Node.js and Python [Lambda runtimes](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html). Select either node.js or Python to continue.
The Node.js function code is located in the `examples-aws-lambda/node` directory and has the following structure:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
├── README.md
├── deployment-package.zip ## Lambda deployment package
├── index.js ## Lambda function source code
├── package-lock.json ## Dependencies
└── package.json ## Dependencies
```
The Lambda function uses the [node-postgres](https://node-postgres.com/) modules to connect to your cluster.
The Python function code is located in the `examples-aws-lambda/python` directory and has the following structure:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
├── README.md
├── deployment-package.zip ## Lambda deployment package
├── init_db.py ## Lambda function source code
├── package ## Psycopg dependencies
├── requirements.txt ## List of Python requirements
└── root.crt ## CA cert
```
The Lambda function uses the [Psycopg2](https://www.psycopg.org/) PostgreSQL adapter to connect to your cluster.
## Step 3. (Optional) Create the deployment package
Creating a deployment package to deploy the sample function is optional. The `examples-aws-lambda` repo includes deployment packages that are ready to deploy.
1. In the `node` directory, install the code dependencies:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cd node; npm install
```
2. Compress the project files to a ZIP file in the parent directory for deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
zip -r ../my-deployment-package.zip .
```
3. In the `python` directory, download and install the `psycopg2-binary` Python library:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cd python; python3 -m pip install \
--only-binary :all: \
--platform manylinux1_x86_64 \
--target ./my-package \
-r requirements.txt
```
To run on Amazon Linux distributions, `pscyopg2` dependencies must be compiled for Linux.
1. Compress the project files to a ZIP file in the parent directory for deployment:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
zip -r ../my-deployment-package.zip my-deployment-package ./init_db.py ./root.crt
```
## Step 4. Configure AWS
1. Configure the AWS CLI to authenticate with your AWS account:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws configure
```
Follow the prompts to authenticate as a user with administrator privileges. We do not recommend using the `root` user.
2. Create an execution role for the Lambda function and attach the `AWSLambdaBasicExecutionRole` policy to the role. The Lambda function needs this role to run.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws iam create-role \
--role-name lambda-ex \
--assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws iam attach-role-policy \
--role-name lambda-ex \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
```
## Step 5. Deploy the function to AWS Lambda
1. In the deployment package directory, use the AWS CLI to create a Lambda function:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws lambda create-function \
--function-name init-crdb \
--region \
--zip-file fileb://deployment-package.zip \
--handler index.handler \
--runtime nodejs14.x \
--role arn:aws:iam:: \
--zip-file fileb://deployment-package.zip \
--handler init_db.lambda_handler \
--runtime python3.9 \
--role arn:aws:iam::
To connect to a CockroachDB Standard cluster with Psycopg2, you must provide the client with a valid CA certificate. By default, Pscyopg2 searches for the certificate at `~/.postgresql/root.crt` and in the location point to by the environment variable `PGSSLROOTCERT`.
1. Invoke the function:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
aws lambda invoke \
--function-name init-crdb out \
--log-type Tail \
--query 'LogResult' \
--output text | base64 -d
```
```
START RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa Version: $LATEST
2022-02-16T19:24:49.569Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Initializing table...
2022-02-16T19:24:49.596Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Hey! You successfully connected to your CockroachDB cluster.
2022-02-16T19:24:49.635Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 320.
2022-02-16T19:24:49.655Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 593.
2022-02-16T19:24:49.660Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 277.
2022-02-16T19:24:49.675Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 844.
2022-02-16T19:24:49.680Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Created new account with balance 257.
2022-02-16T19:24:49.680Z 12232b98-daac-4a1e-80e9-f2ecaa6497aa INFO Database initialized.
END RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa
REPORT RequestId: 12232b98-daac-4a1e-80e9-f2ecaa6497aa Duration: 644.49 ms Billed Duration: 645 ms Memory Size: 128 MB Max Memory Used: 63 MB Init Duration: 198.77 ms
```
## See also
* Build a Simple Django App with CockroachDB
You might also be interested in the following pages:
* Client Connection Parameters
* Connection Pooling
* Data Replication
* CockroachDB Resilience
* Replication & Rebalancing
# Developer Basics
Source: https://docs.cockroachlabs.com/docs/v26.3/developer-basics
Before you start using CockroachDB, you should understand the following important mechanics of the database:
* [How transactions work in CockroachDB](#how-transactions-work-in-cockroachdb), including [serializability](#serializability-and-transaction-contention) and [transaction retries](#transaction-retries)
* [How applications interact with CockroachDB](#how-applications-interact-with-cockroachdb)
Note that the sections that follow were written for the purposes of orienting application developers. For more detailed technical documentation on CockroachDB's architecture, see our architecture documentation.
### How transactions work in CockroachDB
CockroachDB is designed to make your data scalable and consistent. All stored records are replicated across a distributed deployment of database instances, and all database transactions committed in CockroachDB satisfy [ACID properties](https://en.wikipedia.org/wiki/ACID).
To guarantee that database operations are [atomic](https://wikipedia.org/wiki/Atomicity_\(database_systems\)) (the "A" of the [ACID properties](https://en.wikipedia.org/wiki/ACID)), CockroachDB executes all statements in the context of an atomic [database transaction](https://wikipedia.org/wiki/Database_transaction). If a transaction succeeds, all data mutations are applied together simultaneously from the perspective of operations outside of the transaction. If any part of a transaction fails, the entire transaction is aborted, and the database is left unchanged.
Managing transactions is an important part of CockroachDB application development. When creating your application's persistence layer, you must understand how CockroachDB interacts with your application framework's transaction management system. You should also have some knowledge of CockroachDB's transaction syntax. We go into more detail about transaction management in Manage Transactions.
#### Serializability and transaction contention
By default, CockroachDB uses `SERIALIZABLE` transaction [isolation](https://wikipedia.org/wiki/Isolation_\(database_systems\)) (the "I" of ACID semantics). If transactions are executed concurrently, the final state of the database will appear as if the transactions were executed serially. `SERIALIZABLE` isolation, the strictest level of isolation, provides the highest level of data consistency and protects against concurrency anomalies.
To guarantee data consistency, CockroachDB locks the data targeted by an open transaction. If a separate transaction attempts to modify data that are locked by an open transaction, the newest transaction will not succeed, as committing it could result in incorrect data. This scenario is called *transaction contention*, and should be avoided when possible. For a more detailed explanation of transaction contention, and tips on how to avoid it, see Transaction contention.
#### Transaction retries
In some cases, transaction contention is unavoidable. If a transaction fails due to contention, CockroachDB will automatically retry the transaction. If the transaction is `SERIALIZABLE` and cannot automatically be retried without allowing serialization anomalies, CockroachDB will return a transaction retry error to the client.
Most official CockroachDB client libraries include a transaction-retrying wrapper function to make writing your persistence layer easier. If your framework's client library does not include a retry wrapper, you will need to write transaction retry logic in your application. We go into more detail about transaction retries later in the guide, in Retry Transactions.
#### Read Committed isolation
CockroachDB can be configured to execute transactions at `READ COMMITTED` instead of `SERIALIZABLE` isolation. `READ COMMITTED` permits some concurrency anomalies in exchange for minimizing transaction aborts and removing the need for client-side retries. Depending on your workload requirements, this may be desirable. For more information, see Read Committed Transactions.
### How applications interact with CockroachDB
CockroachDB supports the PostgreSQL wire protocol, and CockroachDB supports most PostgreSQL syntax and features, making most PostgreSQL client libraries compatible with CockroachDB. Cockroach Labs also maintains official CockroachDB adapters for some of the most popular PostgreSQL drivers and ORMs. Using an official CockroachDB adapter can help simplify transaction management and performance optimization in your application. If an official CockroachDB adapter does not yet exist for your application's data framework, we recommend just using the PostgreSQL client library that works for your application.
A growing number of popular third-party database tools offer full support for CockroachDB. Because CockroachDB uses the PostgreSQL wire protocol and supports PostgreSQL syntax, most PostgreSQL-compatible third-party tools are also compatible with CockroachDB. For a list of libraries and tools that we have tested against CockroachDB, see Third-party Database Tools.
# Install a FIPS-ready CockroachDB Runtime
Source: https://docs.cockroachlabs.com/docs/v26.3/fips
**What "FIPS support" means**
When CockroachDB documentation refers to "FIPS support" or "FIPS-ready" deployments, this means CockroachDB can be configured to use FIPS 140-3-approved cryptographic algorithms and operate in accordance with a FIPS 140-3 cryptographic module's Security Policy. It does not mean that CockroachDB itself is FIPS 140-3 validated.
CockroachDB uses the frozen Go Cryptographic Module v1.0.0, which is FIPS 140-3 validated under CMVP Certificate #5247.
## Overview of FIPS-ready CockroachDB
[Federal Information Processing Standards (FIPS) 140-3](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.140-3.pdf) is a U.S. government standard that specifies security requirements for cryptographic modules. FIPS 140-3 provides measurable security guidelines for protecting sensitive but unclassified information. The standard is applicable to all federal agencies that use cryptographic-based security systems to protect sensitive information in computer and telecommunication systems. U.S. and Canadian governments, as well as organizations working with them, may be subject to FIPS 140-3 requirements.
The Cryptographic Module Validation Program (CMVP) validates cryptographic modules to FIPS 140-3 and other cryptography-based standards. When a cryptographic module or library has a FIPS 140-3 certificate, it has been tested and formally validated under the CMVP as meeting the requirements for FIPS 140-3.
FIPS-ready CockroachDB binaries and Docker images are available for CockroachDB v23.1.0 and later. FIPS-ready CockroachDB runtimes run on Intel 64-bit Linux systems.
Starting with v26.1, FIPS-ready CockroachDB binaries are built using Go's native FIPS 140-3 support. The cryptographic operations are performed by Go's built-in cryptographic modules, which are independent of the host operating system's libraries. This represents a significant architectural change from previous versions (v25.4 and earlier), which used Red Hat's golang-fips toolchain with OpenSSL.
**Migration from FIPS 140-2 to FIPS 140-3**
Previous versions of CockroachDB (v25.4 and earlier) supported FIPS 140-2. Starting with v26.1, CockroachDB uses a cryptographic module designed for FIPS 140-3 support. Starting with v26.2, CockroachDB uses `GOFIPS140=v1.0.0`, the frozen module version that is FIPS 140-3 validated under CMVP Certificate #5247.
FIPS 140-2 will transition to historical status on September 22, 2026, per [NIST's FIPS 140-3 Transition Effort](https://csrc.nist.gov/Projects/fips-140-3-transition-effort).
For details about cryptographic algorithms and key lengths used by CockroachDB, refer to [Details About Cryptographic Algorithms](#details-about-cryptographic-algorithms).
### FIPS-ready features
When you use a FIPS-ready CockroachDB runtime, Cockroach Labs has verified that cryptographic operations in the following contexts meet the requirements of FIPS 140-3:
* Encryption At Rest
* Encrypted Backups
* Change Data Capture to Kafka over TLS
* Certificate-based Node-to-Node and Client-to-Node Authentication
* SASL SCRAM-SHA-256 Password Authentication
* SQL Cryptographic Built-in Functions
When running a FIPS-ready runtime, Cockroach Labs recommends that you avoid using cryptographic operations that are not supported by FIPS 140-3. For example, generating an MD5 hash is not compatible with FIPS 140-3, because MD5 is not a FIPS-validated algorithm. Use algorithms and functions that do not comply with the standard at your own risk.
This page shows how to install and configure a FIPS-ready CockroachDB runtime using Go's native FIPS 140-3 support.
### Performance considerations
When comparing performance of the same workload in a FIPS-ready CockroachDB runtime to a standard CockroachDB runtime, some performance difference may be observed. The amount of performance impact depends upon the workload, cluster configuration, query load, and other factors.
### Upgrading to a FIPS-ready CockroachDB runtime
Upgrading an existing CockroachDB cluster's binaries in-place to be FIPS-ready is not supported.
## Operating system requirements
FIPS-ready CockroachDB uses Go's native cryptographic module, which is independent of the host operating system's libraries. The FIPS-ready binary can run on any Intel 64-bit Linux system.
The FIPS-ready CockroachDB Docker images are based on [Red Hat's Universal Base Image 10](https://catalog.redhat.com/software/containers/ubi10/ubi-minimal/). To [use the FIPS-ready CockroachDB Docker image](#use-the-fips-ready-cockroachdb-docker-image), skip directly to that section of this page.
### Extend Red Hat's Universal Base Image 10 Docker image
If you do not want to use the FIPS-ready CockroachDB Docker image directly, you can create a custom Docker image based on [Red Hat's Universal Base Image 10](https://catalog.redhat.com/software/containers/ubi10/ubi-minimal/):
* The FIPS-ready binary includes the FIPS 140-3 Go Cryptographic Module and does not require additional system libraries to be installed.
## Download FIPS-ready runtimes
FIPS-ready runtimes are not yet available for CockroachDB v26.3. Download links will be added when v26.3.0 becomes available for self-hosted deployments.
## Install the FIPS-ready CockroachDB runtime
When FIPS-ready CockroachDB v26.3 binaries become available, install them in the same way as standard binaries. Refer to Install CockroachDB on Linux.
**Upgrade from v25.4 or v26.1 FIPS to v26.2 or later**
CockroachDB v26.1 and later use Go's native FIPS cryptographic module, a significant architectural change from the OpenSSL-based approach used in v25.4 and earlier. FIPS support was Preview in v26.1, so deployments using FIPS on v25.4 or v26.1 should upgrade to v26.2 or a later version.
Upgrading an existing CockroachDB cluster's binary in-place from non-FIPS to FIPS is not supported. Instead, you can restore your cluster to a new FIPS-ready cluster.
### Verify that CockroachDB is FIPS-ready
To verify that the CockroachDB binary is FIPS-ready, use the `cockroach version` command and check for the `FIPS enabled` field:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach version | grep -i fips
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
FIPS enabled: true
```
This indicates that CockroachDB was built with Go's native FIPS 140-3 support.
**Change from previous versions:** In v25.4 and earlier, FIPS-ready binaries showed `fips` appended to the Go version (e.g., `go1.19.5fips`). Starting with v26.1, FIPS status is indicated by the `FIPS enabled: true` field.
## Use the FIPS-ready CockroachDB Docker image
FIPS-ready CockroachDB Docker images are not yet available for CockroachDB v26.3. Image tags and usage instructions will be added when v26.3.0 becomes available for self-hosted deployments.
## Details about cryptographic algorithms
This section provides more information about the cryptographic algorithms and key lengths used by FIPS-ready CockroachDB.
### Authentication
#### Inter-node and node identity
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
Refer to Using Digital Certificates with CockroachDB.
#### Client identity
##### Password authentication
**Algorithm**: `bcrypt` or `scram-sha-256`.
Refer to SASL/SCRAM-SHA-256 Secure Password-based Authentication.
##### Client certificates
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
Refer to Using Digital Certificates with CockroachDB.
##### GSSAPI / Kerberos
Not supported for FIPS-ready deployments.
##### SASL / SCRAM password authentication
**Algorithm**: `scram-sha-256`.
Refer to SASL/SCRAM-SHA-256 Secure Password-based Authentication.
##### JSON Web Tokens (JWTs)
**Algorithms**: Specified by the `server.jwt_authentication.jwks` cluster setting.
Refer to Cluster Single Sign-on (SSO) using a JSON web token (JWT).
##### DB Console authentication via OIDC
**Algorithm**: Specified by the identity provider (IdP) as part of the OIDC handshake process.
Refer to Single Sign-on (SSO) for DB Console.
##### HTTP API access via login tokens
**Algorithm**: `sha256` ([RFC 6234](https://datatracker.ietf.org/doc/html/rfc6234)).
### Encryption
#### In flight
**Algorithm**: [TLS 1.3 (RFC 8446)](https://www.rfc-editor.org/rfc/rfc8446).
**Key sizes**: Depends upon the cipher suite in use:
* TLS 1.2:
* `tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`
* `tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`
* If the `COCKROACH_TLS_ENABLE_OLD_CIPHER_SUITES` environment variable is set:
* `tls.TLS_RSA_WITH_AES_128_GCM_SHA256`
* `tls.TLS_RSA_WITH_AES_256_GCM_SHA384`
* TLS 1.3:
* `TLS_AES_128_GCM_SHA256`
* `TLS_AES_256_GCM_SHA384`
#### At rest
##### Customer-managed backups
##### AWS
Default encryption provided by the [AWS Encryption SDK](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html).
##### GCP
Default encryption provided by [Google Cloud](https://cloud.google.com/docs/security/encryption/default-encryption).
##### Data Encryption at Rest (EAR)
**Algorithm**: Advanced Encryption Standard (AES) encryption, in counter (CTR) mode.
**Key sizes**: The store key is specified by the user, and can be of length 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). Data keys are the same length as the store key.
## See also
* Install CockroachDB
* Releases
# Build a GraphQL Application Using Hasura
Source: https://docs.cockroachlabs.com/docs/v26.3/hasura-getting-started
This tutorial shows you how to create a GraphQL application using [Hasura Cloud](https://cloud.hasura.io) and CockroachDB Standard.
The Hasura GraphQL Engine creates [GraphQL](https://graphql.org/) schemas and resolvers based on the tables and views in your CockroachDB cluster, allowing you to submit GraphQL queries to access and manipulate your data.
Hasura GraphQL Engine v2.16 and later supports CockroachDB v22.2 and later.
## Overview
This tutorial will show you how to configure a Hasura project with a CockroachDB data source, allow connections to your CockroachDB cluster from Hasura Cloud, and create a table that is then used with Hasura GraphQL Engine to retrieve and modify data.
## Before you begin
Before you start this tutorial, you need:
* An existing CockroachDB Cloud cluster, running CockroachDB v22.2 or later.
* A [Hasura Cloud account](https://hasura.io/docs/latest/getting-started/getting-started-cloud/).
## Create a new project in Hasura Cloud
1. In the Hasura Cloud console, select **Projects**, then click **New Project**.
2. In the **Create Project** panel select the cloud infrastructure provider and region.
The cloud infrastructure provider and region should match your cluster. For example, if you created a CockroachDB Standard cluster in GCP's `us-east1` region, choose a GCP region closest to `us-east1`.
3. Click **Create Free Project**.
### Add environment variables to your project
You will store your cluster root certificate and connection string in your Hasura project's environment variables. Storing this connection information in environment variables is considered a best practice.
Create a `SSL_ROOT_CERT` environment variable for your cluster's CA cert.
1. Select **Env vars** in your project settings, and click **New Env Var**.
2. Under **Key** type `SSL_ROOT_CERT`, then press **Enter**.
3. Copy the contents of your cluster's CA certificate file you downloaded earlier.
For example, on Mac you can copy the contents of the CA certificate in a terminal using `pbcopy`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pbcopy < $HOME/.postgresql/root.crt
```
4. Paste the contents of the CA certificate file under **Value**.
5. Click **Add**.
6. Select **Env vars** in your project settings, and click **New Env Var**.
7. Under **Key** type `SSL_ROOT_CERT`, then press **Enter**.
8. Copy the contents of your CockroachDB Standard cluster's CA certificate file you downloaded earlier.
For example, on Mac you can copy the contents of the CA certificate in a terminal using `pbcopy`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pbcopy < $HOME/Library/CockroachCloud/certs/fake-puppy-ca.crt
```
9. Paste the contents of the CA certificate file under **Value**.
![Adding the SSL_ROOT_CERT environment variable to the Hasura project]()
10. Click **Add**.
Create a `CRDB_URL` environment variable to store the connection string.
1. Click **New Env Var**.
2. Under **Key** type `CRDB_URL`, then press **Enter**.
3. Paste the connection string for your cluster you copied earlier. Add your SQL user password if necessary to the connection string.
4. Remove the `sslmode` and `sslrootcert` query parameters from the end of the connection string. You will add the SSL settings later.
For example, if your connection string in the CockroachDB Cloud console is:
```
postgresql://maxroach:NotAGoodPassword@fake-puppy-595.g95.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full
```
You would modify the connection string to:
```
postgresql://maxroach:NotAGoodPassword@fake-puppy-595.g95.cockroachlabs.cloud:26257/defaultdb
```
5. Click **Add**.
### Create a Data Source in Hasura Cloud
1. In the overview page in Hasura Cloud, select **Projects**, find the project you created earlier, and click **Launch Console**.
2. Click **Data** in the console header to open the **Data Manager**.
3. Click **Connect Database**.
4. In the **Connect Existing Database** tab configure the connection to your cluster.
1. Under **Database Display Name** type a name for your connection. Your cluster name is a good option. For example, `fake-puppy`.
2. Set **Data Source Driver** to **CockroachDB**.
3. Select **Environment Variable**, then enter `CRDB_URL` in the **Environment Variable** input box.
![Adding the connection string to the Hasura Data Manager]()
5. Expand **Connection Settings**, then expand **SSL Certificates Settings** and set **SSL Mode** to `verify-full`.
6. Enter `SSL_ROOT_CERT` under **SSL Root Certificate**, the environment variable you configured earlier.
7. Click **Connect Database**.
## Add the Hasura Cloud network to your cluster allowlist
Your CockroachDB Standard cluster needs to be configured to allow incoming client connections from Hasura Cloud.
1. In the Hasura Cloud overview page select **Projects**, then click the **Config** icon for your project.
![Click the project Config icon]()
2. In the **General** section, copy the IP address listed under **Hasura Cloud IP**.
3. In the CockroachDB Cloud console select your cluster, click **Networking**, then **Add Network**.
4. In the **Add Network** dialog, set **Network name** to `Hasura Cloud`.
5. Under **Network**, select **New Network**. Paste the IP address of your Hasura Cloud instance you copied earlier.
6. Check **CockroachDB Client to access the databases**, then click **Apply**.
## Create a table in your CockroachDB cluster
1. Connect to your cluster using the SQL client.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach sql --url "{connection string}"
```
Where is the connection string for your cluster you copied earlier.
2. In the SQL client, create an `accounts` table.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE TABLE IF NOT EXISTS accounts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
balance INT8
);
```
3. Insert some data into the `accounts` table.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
UPSERT INTO accounts (balance) VALUES (1000), (250);
```
This will create two accounts, one with $1000 and another with $250.
## Track the `accounts` table in the Hasura Cloud Data Manager
1. In the Hasura Cloud Data Manager click `public` under your connection name.
2. Click the **Track All** button.
3. Click **Yes** when prompted to expose the table to the GraphQL API.
### Create a GraphQL query in the Hasura Cloud API Explorer
In the Hasura Cloud Explorer, you will see the `accounts` table. You can expand `accounts` to construct a GraphQL query against the `accounts` table.
For example, to create a query that finds all accounts with balances greater than or equal to \$500:
1. Enter `GetLargeAccounts` as the query name in the Explorer.
2. Expand `accounts` in the Explorer.
3. Check `balance` and `id`, then expand `where`, then `balance`.
4. Check `_gte` to create a "greater than or equal to" where clause for `balance`, then enter `500`.
This creates the following GraphQL query:
```graphql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
query GetLargeAccounts {
accounts(where: {balance: {_gte: "500"}}) {
balance
id
}
}
```
Click the **Execute Query** button to run the `GetLargeAccounts` GraphQL query. The returned data should look like the following:
```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{
"data": {
"accounts": [
{
"balance": 1000,
"id": "2ed41abf-e831-42df-aff4-f88a6f2b0afc"
}
]
}
}
```
![Executing the GraphQL query in the Hasura Cloud API Explorer]()
## Next steps
Now that you have connected your cluster to Hasura GraphQL Engine, you can use the Explorer to create GraphQL queries to access and modify data in your cluster's tables. You can then use these queries in your application. For example, you can create [Hasura Actions](https://hasura.io/docs/latest/actions/index/?utm_source=cockroachdb) that use these queries. See the [Hasura documentation](https://hasura.io/docs/latest/index/?utm_source=cockroachdb) for detailed information on integrating and deploying Hasura in your applications.
# CockroachDB Docs
Source: https://docs.cockroachlabs.com/docs/v26.3/index
CockroachDB is the SQL database for building global, scalable cloud services that survive disasters.
![icon browser]()
Start a Cloud
cluster
![MCP server]()
Use the Docs
MCP server
Get Started
- Install CockroachDB
- Start a Local Cluster
- Connect to the Database
- Learn CockroachDB SQL
- Explore Capabilities
CockroachDB and AI
- CockroachDB and AI Overview
- Vector Indexes
- Agent Skills
- Docs MCP Server
Deploy
- Production Checklist
- CockroachDB Cloud Deployment
- Kubernetes Overview
- Multi-Region Capabilities
- Cluster Maintenance
Migrate
- Migration Overview
- Migrate from PostgreSQL
- Migrate from MySQL
- Migrate from Oracle
- MOLT Fetch
Security
- Security Overview
- Authentication
- Authorization
- Encryption
Reference
- Architecture
- SQL
- CLI
- DB Console
- Third-Party Tools
Learning Resources
Cockroach University
Free online learning platform covering distributed databases, cloud-native applications, general purpose SQL databases & much more!
Start free course ![icon arrow right]()
Report
Cockroach Labs
October 29, 2025
# Install CockroachDB on Linux
Source: https://docs.cockroachlabs.com/docs/v26.3/install-cockroachdb-linux
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
**This page refers to CockroachDB v26.3.** The latest supported production release is CockroachDB v26.1.
Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to Upgrade to v26.3.
* To install a FIPS-compliant CockroachDB binary, refer to Install a FIPS-compliant build of CockroachDB instead of this page.
* For limitations of CockroachDB on ARM, refer to [ARM limitations](#arm).
* For limitations of CockroachDB on a NUMA architecture, refer to [NUMA limitations](#numa).
## Download the binary
The CockroachDB binary for Linux requires `glibc`, `libncurses`, and `tzdata`, which are found by default on nearly all Linux distributions.
1. Visit Releases to download the CockroachDB archive for the architecture of your Linux host. The archive contains the `cockroach` binary and the supporting libraries that are used to provide spatial features. Follow the steps shown below to install the `cockroach` binary into your `PATH` so you can execute cockroach commands from any shell.
1. Download the binary you want from the Releases page. In the following commands, replace `VERSION` with the version of CockroachDB you are installing (e.g., v26.3), and replace `ARCHITECTURE` with `linux-amd64` for Intel, or with `linux-arm64` for ARM.
2. Extract the archive:
```
tar-xz cockroach-VERSION.linux-ARCHITECTURE.tgz
```
3. Copy the `cockroach` binary into your `PATH`:
```
cp -i cockroach-VERSION.linux-ARCHITECTURE/cockroach /usr/local/bin/
```
If you plan to use CockroachDB's spatial features, you must complete the following steps. Otherwise, your installation is now complete.
2. CockroachDB uses custom-built versions of the GEOS libraries. Copy these libraries to one of the locations where CockroachDB expects to find them.
By default, CockroachDB looks for external libraries in `/usr/local/lib/cockroach` or a `lib` subdirectory of the CockroachDB binary's current directory. If you place these libraries in another location, you must pass the location in the `--spatial-libs` flag to `cockroach start`. The instructions below assume the `/usr/local/lib/cockroach` location.
1. Create the directory where the external libraries will be stored:
```
mkdir -p /usr/local/lib/cockroach
```
2. Copy the library files to the directory. In the following commands, replace `VERSION` with the version of CockroachDB you are installing, and replace `ARCHITECTURE` with `linux-amd64` for Intel, or with `linux-arm64` for ARM.
```
cp -i cockroach-VERSION.linux-ARCHITECTURE/lib/libgeos.so /usr/local/lib/cockroach/
```
```
cp -i cockroach-VERSION.linux-ARCHITECTURE/lib/libgeos_c.so /usr/local/lib/cockroach/
```
If you get a permissions error, prefix the command with `sudo`.
3. Verify that CockroachDB can execute spatial queries.
1. Make sure the `cockroach` binary you just installed is the one that runs when you type `cockroach` in your shell:
```
which cockroach
```
```
/usr/local/bin/cockroach
```
2. Start a temporary, in-memory cluster using `cockroach demo`:
```
cockroach demo
```
* In the demo cluster's interactive SQL shell, run the following command to test that the spatial libraries have loaded properly:
```
> SELECT ST_IsValid(ST_MakePoint(1,2));
```
You should see the following output:
```
st_isvalid
--------------
true
(1 row)
```
If your `cockroach` binary is not properly accessing the dynamically linked C libraries in `/usr/local/lib/cockroach`, it will output an error message like the one below.
```
ERROR: st_isvalid(): geos: error during GEOS init: geos: cannot load GEOS from dir "/usr/local/lib/cockroach": failed to execute dlopen
Failed running "sql"
```
* Keep up-to-date with CockroachDB releases and best practices:
## Use Kubernetes
To orchestrate CockroachDB using [Kubernetes](https://kubernetes.io/), use the official CockroachDB operator.
You can evaluate a CockroachDB Kubernetes deployment on a local machine with [minikube](https://minikube.sigs.k8s.io/docs/start/).
## Use Docker
Deploy a local container in Docker
Deploy a local container in Docker
Deploy a local container in Docker
Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.
CockroachDB's Docker images are [multi-platform images](https://docs.docker.com/build/building/multi-platform/) that contain binaries for both Intel and ARM. Multi-platform images do not take up additional space on your Docker host.
Experimental images are not qualified for production use and not eligible for support or uptime SLA commitments.
1. Install a container runtime, such as [Docker Desktop](https://docs.docker.com/desktop/).
2. Verify that the runtime service is installed correctly and running in the background. Refer to the runtime's documentation. For Docker, start a terminal and run `docker version`. If you get an error, verify your installation and try again.
1\. Visit [Docker Hub](https://hub.docker.com/r/cockroachdb/cockroach) and decide which image tag to pull. Releases are rolled out gradually. Docker images for a new release are published when other binary artifacts are published. The following tag formats are commonly used, although other tags are available.
| Tag | Example | Description |
| - | - | - |
| An exact patch | \`\` | Pins a cluster to an exact patch. The cluster is upgraded to a newer patch or major version only when you pull a newer tag. |
| Latest patch within a major version | `latest-v26.3` | Automatically updates a cluster to the latest patch of the version you specify. This tag is recommended in production, because it keeps your cluster updated within a major version but does not automatically upgrade your cluster to a new major version. |
| `latest` | The latest patch within the latest major version. This is the default if you do not specify a tag. It updates your cluster automatically to each new patch and major version, and is not recommended in production. | |
Copy the tag you want to pull.
1. Pull the image. Replace with the tag from the previous step.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker pull cockroachdb/cockroach:{TAG}
```
1. Start a cluster by starting the container on each node using `docker start`. The default command is `cockroach start`. Pass your desired flags as the final argument. For details, refer to .
For CockroachDB v22.2.beta-5 and above, Docker images are [multi-platform images](https://docs.docker.com/build/building/multi-platform/) that contain binaries for both Intel and ARM. Multi-platform images do not take up additional space on your Docker host.
Docker images for previous releases contain Intel binaries only. Intel binaries can run on ARM systems, but with a significant reduction in performance.
CockroachDB on ARM is in **Limited Access** in v22.2.13, and is **experimental** in all other versions. Experimental images are not qualified for production use and not eligible for support or uptime SLA commitments.
1. Install [Docker for Linux](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/). Please carefully check that you meet all prerequisites.
2. Confirm that the Docker daemon is running in the background:
```
$ docker version
```
If you do not see the server listed, start the **Docker** daemon.
3. Pull the image for the release of CockroachDB from [Docker Hub](https://hub.docker.com/r//):
```
$ sudo docker pull :
```
4. Keep up-to-date with CockroachDB releases and best practices:
## Build from source
See the [public wiki](https://wiki.crdb.io/wiki/spaces/CRDB/pages/181338446/Getting+and+building+CockroachDB+from+source) for guidance. When building on the ARM architecture, refer to [Limitations](#limitations).
## Limitations
### ARM
CockroachDB runtimes built for the ARM architecture have the following limitations:
* Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add). Validate workloads that rely on floating point operations or FMA instrincs before migrating those workloads to ARM in production.
* When [building from source](#install-source) on ARM, it is not currently possible to disable FMA intrinsics in Go. To track the status of this feature request, refer to [GoLang issue #36971](https://github.com/golang/go/issues/36971).
* In production, Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, CPU architecture, hardware, and software.
* A mix of Intel and ARM nodes is supported as a temporary transitional state during the migration only. Cockroach Labs recommends that you test and validate your workload ahead of the migration to ensure that the workload and your application work as expected in a cluster with both Intel and ARM nodes, especially with respect to floating-point arithmetic.
### NUMA
In a [NUMA (non-uniform memory access) architecture](https://en.wikipedia.org/wiki/Non-uniform_memory_access), the system’s memory is physically distributed across multiple memory banks or "nodes", and each node is assigned to a processor. A processor can access its local memory much faster than non-local memory. This non-uniform memory access can lead to performance differences depending on data location.
CockroachDB is written in Go, which has no process-level support for NUMA scheduling or pinning. Instead, you must manage NUMA at the operating system level. To run multiple CockroachDB clusters on a NUMA architecture:
* Assign no more than 64 cores to the node for each instance. Refer to the documentation for [`numactl --membind`](https://man7.org/linux/man-pages/man8/numactl.8.html).
* Ensure that your orchestration framework, process manager, or startup scripts start each CockroachDB in a separate NUMA node.
* If multiple CockroachDB nodes for the same CockroachDB cluster run on the same physical host, ensure that they are in the same `--locality` to ensure that data is distributed across different physical hosts.
* Ensure that each CockroachDB node writes to a unique set of storage volumes (block devices).
## What's next?
* If you're just getting started with CockroachDB:
* [Create a CockroachDB Cloud account](https://cockroachlabs.cloud/signup?experience=enterprise) where you can generate and manage licenses for CockroachDB installations
* Start a cluster locally and talk to it via the built-in SQL client
* Learn more about CockroachDB SQL
* Explore CockroachDB features like automatic replication, rebalancing, and fault tolerance
* If you're ready to run CockroachDB in production:
* Review the Production Checklist
* Deploy CockroachDB manually or use an orchestration system like Kubernetes
# Install CockroachDB on Mac
Source: https://docs.cockroachlabs.com/docs/v26.3/install-cockroachdb-mac
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
**This page refers to CockroachDB v26.3.** The latest supported production release is CockroachDB v26.1.
For CockroachDB v22.2.x and above, Homebrew installs binaries for your system architecture, either Intel or ARM ([Apple Silicon](https://support.apple.com/HT211814)).
For previous releases, Homebrew installs Intel binaries. Intel binaries can run on ARM systems, but with a significant reduction in performance. CockroachDB on ARM for macOS is **experimental** and is not yet qualified for production use and not eligible for support or uptime SLA commitments.
For CockroachDB v22.2.x and above, download the binaries for your system architecture, either Intel or ARM ([Apple Silicon](https://support.apple.com/en-us/HT211814)).
For previous releases, download Intel binaries. Intel binaries can run on ARM systems, but with a significant reduction in performance. CockroachDB on ARM for macOS is **experimental** and is not yet qualified for production use and not eligible for support or uptime SLA commitments.
CockroachDB on macOS is experimental and not suitable for production deployments.
Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to Upgrade to v26.3. For limitations specific to geospatial features, refer to [Limitations](#limitations).
## Use Homebrew
1. [Install Homebrew](http://brew.sh/).
2. Instruct Homebrew to install CockroachDB:
```
$ brew install cockroachdb/tap/cockroach
```
3. Keep up-to-date with CockroachDB releases and best practices:
If you previously installed CockroachDB via Homebrew, you can upgrade the CockroachDB binary to the next major version or to a patch version using HomeBrew. After updating the binary on each node, restart the `cockroach` process on the node. When upgrading to a new major version, you must complete additional steps to finalize the upgrade. If you need to upgrade through multiple major versions, you must complete each major-version upgrade separately, including finalizing the upgrade, before beginning the next one.
Before starting the upgrade, review the release notes, including temporary limitations during the upgrade.
To upgrade CockroachDB via HomeBrew:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
brew update
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
brew update
brew upgrade cockroach
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
xattr -d com.apple.quarantine lib/libgeos\*
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
st_isvalid
--------------
true
(1 row)
```
## Download the binary
1. Visit Releases to download the CockroachDB archive for the architecture of your macOS host. The archive contains the `cockroach` binary and the supporting libraries that are used to provide spatial features.
You can download the binary using a web browser or you can copy the link and use a utility like `curl` to download it. If you download the ARM binary using a web browser and you plan to use CockroachDB's spatial features, an additional step is required before you can install the library, as outlined in the next step.
Extract the archive and optionally copy the `cockroach` binary into your `PATH` so you can execute cockroach commands from any shell. If you get a permission error, use `sudo`.
If you plan to use CockroachDB's spatial features, you must complete all of the following steps. Otherwise, your installation is now complete.
2. CockroachDB uses custom-built versions of the GEOS libraries. To install those libraries:
1. Note that spatial features are currently disabled for Mac ARM users, for whom these steps do not apply. For an upcoming patch release where this functionality is reenabled, if you downloaded the CockroachDB ARM binary archive using a web browser, macOS flags the GEOS libraries in the extracted archive as quarantined. This flag must be removed before CockroachDB can use the libraries. To remove the quarantine flag from the libraries:
This step is not required for Intel systems.
2. Copy these libraries to one of the locations where CockroachDB expects to find them. By default, CockroachDB looks for external libraries in `/usr/local/lib/cockroach` or a `lib` subdirectory of the CockroachDB binary's current directory. If you place these libraries in another location, you must pass the location in the `--spatial-libs` flag to `cockroach start`. The instructions below assume the `/usr/local/lib/cockroach` location.
1. Create the directory where the external libraries will be stored:
```
mkdir -p /usr/local/lib/cockroach
```
2. Copy the library files to the directory:
```
cp -i cockroach-v26.3.darwin-10.9-amd64/lib/libgeos.dylib /usr/local/lib/cockroach/
```
```
cp -i cockroach-v26.3.darwin-10.9-amd64/lib/libgeos_c.dylib /usr/local/lib/cockroach/
```
If you get a permissions error, prefix the command with `sudo`.
3. Verify that CockroachDB can execute spatial queries.
1. Make sure the `cockroach` binary you just installed is the one that runs when you type `cockroach` in your shell:
```
which cockroach
```
```
/usr/local/bin/cockroach
```
2. Start a temporary, in-memory cluster using `cockroach demo`:
```
cockroach demo
```
3. In the demo cluster's interactive SQL shell, run the following command to test that the spatial libraries have loaded properly:
```
> SELECT ST_IsValid(ST_MakePoint(1,2">Releases);
```
You should see the following output:
If your `cockroach` binary is not properly accessing the dynamically linked C libraries in `/usr/local/lib/cockroach`, it will output an error message like the one below.
```
ERROR: st_isvalid(): geos: error during GEOS init: geos: cannot load GEOS from dir "/usr/local/lib/cockroach": failed to execute dlopen
Failed running "sql"
```
* Keep up-to-date with CockroachDB releases and best practices:
## Use Kubernetes
To orchestrate CockroachDB using [Kubernetes](https://kubernetes.io/), use the official CockroachDB operator.
You can evaluate a CockroachDB Kubernetes deployment on a local machine with [minikube](https://minikube.sigs.k8s.io/docs/start/).
## Use Docker
Deploy a local container in Docker
Deploy a local container in Docker
Deploy a local container in Docker
Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.
CockroachDB's Docker images are [multi-platform images](https://docs.docker.com/build/building/multi-platform/) that contain binaries for both Intel and ARM. Multi-platform images do not take up additional space on your Docker host.
Experimental images are not qualified for production use and not eligible for support or uptime SLA commitments.
1. Install a container runtime, such as [Docker Desktop](https://docs.docker.com/desktop/).
2. Verify that the runtime service is installed correctly and running in the background. Refer to the runtime's documentation. For Docker, start a terminal and run `docker version`. If you get an error, verify your installation and try again.
1\. Visit [Docker Hub](https://hub.docker.com/r/cockroachdb/cockroach) and decide which image tag to pull. Releases are rolled out gradually. Docker images for a new release are published when other binary artifacts are published. The following tag formats are commonly used, although other tags are available.
| Tag | Example | Description |
| - | - | - |
| An exact patch | \`\` | Pins a cluster to an exact patch. The cluster is upgraded to a newer patch or major version only when you pull a newer tag. |
| Latest patch within a major version | `latest-v26.3` | Automatically updates a cluster to the latest patch of the version you specify. This tag is recommended in production, because it keeps your cluster updated within a major version but does not automatically upgrade your cluster to a new major version. |
| `latest` | The latest patch within the latest major version. This is the default if you do not specify a tag. It updates your cluster automatically to each new patch and major version, and is not recommended in production. | |
Copy the tag you want to pull.
1. Pull the image. Replace with the tag from the previous step.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker pull cockroachdb/cockroach:{TAG}
```
1. Start a cluster by starting the container on each node using `docker start`. The default command is `cockroach start`. Pass your desired flags as the final argument. For details, refer to .
## Build from source
See the [public wiki](https://wiki.crdb.io/wiki/spaces/CRDB/pages/181338446/Getting+and+building+CockroachDB+from+source) for guidance. When building on the ARM architecture, refer to [Limitations](#limitations).
## Limitations
CockroachDB runtimes built for the ARM architecture have the following limitations:
* CockroachDB on ARM for macOS is **experimental** and is not yet qualified for production use and not eligible for support or uptime SLA commitments.
* Clusters with a mix of Intel and ARM nodes are untested. Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, hardware, and software.
* Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://developer.arm.com/documentation/dui0375/g/Compiler-specific-Features/Fused-Multiply-Add--FMA--intrinsics).
* When [building from source](#install-source) on ARM, consider disabling FMA intrinsics in your compiler. For GCC, refer to [Options That Control Optimization](https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html) in the GCC documentation.
## What's next?
* If you're just getting started with CockroachDB:
* [Create a CockroachDB Cloud account](https://cockroachlabs.cloud/signup?experience=enterprise) where you can generate and manage licenses for CockroachDB installations
* Start a cluster locally and talk to it via the built-in SQL client
* Learn more about CockroachDB SQL
* Explore CockroachDB features like automatic replication, rebalancing, and fault tolerance
* If you're ready to run CockroachDB in production:
* Review the Production Checklist
* Deploy CockroachDB manually or use an orchestration system like Kubernetes
## Limitations
On macOS ARM systems, spatial features are disabled due to an issue with macOS code signing for the [GEOS](https://libgeos.org/) libraries. Users needing spatial features on an ARM Mac may instead [run the Intel binary](#install-the-binary) or use the[Docker container image](#use-docker).
# Install CockroachDB on Windows
Source: https://docs.cockroachlabs.com/docs/v26.3/install-cockroachdb-windows
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
**This page refers to CockroachDB v26.3.** The latest supported production release is CockroachDB v26.1.
CockroachDB executable for Windows is experimental and not suitable for production deployments. Windows 8 or higher is required.
Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to Upgrade to v26.3.
## Download the executable
You can download and install CockroachDB for Windows in two ways. Either:
* **Recommended**: Visit Releases to download CockroachDB. The archive contains the `cockroach.exe` binary. Extract the archive and optionally copy the `cockroach.exe` binary into your `PATH` so you can execute cockroach commands from any shell. Releases are rolled out gradually, so the latest version may not yet be downloadable.
* Instead of downloading the binary directly, you can use PowerShell to download and install CockroachDB:
1. Visit Releases and make a note of the full version of CockroachDB to install, such as . Releases are rolled out gradually, so the latest version may not yet be downloadable.
2. Save the following PowerShell script and replace the following:
* `VERSION`: the full version of CockroachDB to download, such as \`\`. Replace this value in both the `Invoke-WebRequest` statement and the `Copy-Item` statement.
* `{ INSTALL_DIRECTORY }`: the local file path where the `cockroachdb.exe` executable will be installed. Replace the value in both the `Destination` argument and the `$Env:PATH` statement, which adds the destination directory to your `PATH`.
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ErrorActionPreference = "Stop";
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;$ProgressPreference = 'SilentlyContinue'; $null = New-Item -Type Directory -Force $env:appdata/cockroach;
Invoke-WebRequest -Uri https://binaries.cockroachdb.com/cockroach-VERSION.windows-6.2-amd64.zip -OutFile cockroach.zip;
Expand-Archive -Force -Path cockroach.zip;
Copy-Item -Force "cockroach/cockroach-VERSION.windows-6.2-amd64/cockroach.exe" -Destination $env:{ INSTALL_DIRECTORY };$Env:PATH += ";$env:{ INSTALL_DIRECTORY }"
```
1. Run the PowerShell script. To run a PowerShell script from a file, use syntax like:
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
powershell.exe -Command "{path_to_script}"
```
1. Check that the installation succeeded and that you can run `cockroach` commands:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach version
```
## Use Kubernetes
To orchestrate CockroachDB using [Kubernetes](https://kubernetes.io/), use the official CockroachDB operator.
You can evaluate a CockroachDB Kubernetes deployment on a local machine with [minikube](https://minikube.sigs.k8s.io/docs/start/).
## Use Docker
This section shows how to install CockroachDB on a Windows host using Docker. On a Linux or Windows Docker host, the image creates a Linux container.
Running a stateful application like CockroachDB in Docker is more complex and error-prone than most uses of Docker. Unless you are very experienced with Docker, we recommend starting with a different installation and deployment method.
Docker images are [multi-platform images](https://docs.docker.com/build/building/multi-platform/) that contain binaries for both Intel and ARM. Multi-platform images do not take up additional space on your Docker host.
Intel binaries can run on ARM systems, but with a significant reduction in performance.
1. Install [Docker for Windows](https://docs.docker.com/docker-for-windows/install/).
Docker for Windows requires 64-bit Windows 10 Pro and Microsoft Hyper-V. Refer to the [official documentation](https://docs.docker.com/docker-for-windows/install/#what-to-know-before-you-install) for more details. If your system does not satisfy the stated requirements, you can try using [Docker Toolbox](https://docs.docker.com/toolbox/overview/).
2. In PowerShell, confirm that Docker is running in the background:
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker version
```
If you see an error, verify your Docker for Windows installation, then try again.
1. [Enable synchronized drive sharing](https://docs.docker.com/desktop/synchronized-file-sharing/) so you can mount local directories as data volumes to persist node data after containers are stopped or deleted.
1\. Visit [Docker Hub](https://hub.docker.com/layers//) and make a note of the full version of CockroachDB to pull. Releases are rolled out gradually, so the latest version may not yet be available. Using the `latest` tag is not recommended; to pull the latest release within a major version, use a tag like `latest-v26.3`. The following command always pulls the \`\` image.
```powershell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
docker pull :
```
Keep up-to-date with CockroachDB releases and best practices:
## What's next?
* If you're just getting started with CockroachDB:
* [Create a CockroachDB Cloud account](https://cockroachlabs.cloud/signup?experience=enterprise) where you can generate and manage licenses for CockroachDB installations
* Start a cluster locally and talk to it via the built-in SQL client
* Learn more about CockroachDB SQL
* Explore CockroachDB features like automatic replication, rebalancing, and fault tolerance
* If you're ready to run CockroachDB in production:
* Review the Production Checklist
* Deploy CockroachDB manually or use an orchestration system like Kubernetes
# MovR
Source: https://docs.cockroachlabs.com/docs/v26.3/movr
MovR is a fictional vehicle-sharing company created to demonstrate CockroachDB's features.
## Overview
The MovR example consists of the following:
* The `movr` dataset, which contains rows of data that populate tables in the `movr` database. The `movr` dataset is built into `cockroach demo` and `cockroach workload`.
* The MovR application, a fully-functional vehicle-sharing application, written in Python. All of MovR application source code is open-source, and available on the [movr](https://github.com/cockroachdb/movr) GitHub repository.
## The `movr` database
The six tables in the `movr` database store user, vehicle, and ride data for MovR:
| Table | Description |
| - | - |
| `users` | People registered for the service. |
| `vehicles` | The pool of vehicles available for the service. |
| `rides` | When and where users have rented a vehicle. |
| `promo_codes` | Promotional codes for users. |
| `user_promo_codes` | Promotional codes in use by users. |
| `vehicle_location_histories` | Vehicle location history. |
![Geo-partitioning schema]()
## Generating schemas and data for MovR
You can use the `cockroach demo` and `cockroach workload` commands to load the `movr` database and dataset into a CockroachDB cluster.
`cockroach demo` opens a SQL shell to a temporary, in-memory cluster. To open a SQL shell to a demo cluster with the `movr` database preloaded and set as the current database, use the following command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach demo movr
```
`cockroach workload` loads sample datasets and workloads into running clusters. To load the `movr` database and some sample data into a running cluster, do the following:
1. Start a secure local cluster.
2. Use `cockroach workload` to load the `movr` dataset:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init movr 'postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt'
```
3. Use `cockroach sql` to open an interactive SQL shell and set `movr` as the current database:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host=localhost:26257
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> USE movr;
```
1. Start an insecure local cluster.
2. Use `cockroach workload` to load the `movr` dataset:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init movr 'postgresql://root@localhost:26257?sslmode=disable'
```
3. Use `cockroach sql` to open an interactive SQL shell and set `movr` as the current database:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> USE movr;
```
## How the MovR application works
The workflow for MovR is as follows:
1. A user loads the app and sees the 25 closest vehicles.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT id, city, status FROM vehicles WHERE city='amsterdam' limit 25;
```
2. The user signs up for the service.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO users (id, name, address, city, credit_card)
VALUES ('66666666-6666-4400-8000-00000000000f', 'Mariah Lam', '88194 Angela Gardens Suite 60', 'amsterdam', '123245696');
```
3. In some cases, the user adds their own vehicle to share.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO vehicles (id, city, type, owner_id,creation_time,status, current_location, ext)
VALUES ('ffffffff-ffff-4400-8000-00000000000f', 'amsterdam', 'skateboard', '66666666-6666-4400-8000-00000000000f', current_timestamp(), 'available', '88194 Angela Gardens Suite 60', '{"color": "blue"}');
```
4. More often, the user reserves a vehicle and starts a ride, applying a promo code, if available and valid.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT code FROM user_promo_codes WHERE user_id ='66666666-6666-4400-8000-00000000000f';
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE vehicles SET status = 'in_use' WHERE id='bbbbbbbb-bbbb-4800-8000-00000000000b';
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO rides (id, city, vehicle_city, rider_id, vehicle_id, start_address,end_address, start_time, end_time, revenue)
VALUES ('cd032f56-cf1a-4800-8000-00000000066f', 'amsterdam', 'amsterdam', '66666666-6666-4400-8000-00000000000f', 'bbbbbbbb-bbbb-4800-8000-00000000000b', '70458 Mary Crest', '', TIMESTAMP '2020-10-01 10:00:00.123456', NULL, 0.0);
```
5. During the ride, MovR tracks the location of the vehicle.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO vehicle_location_histories (city, ride_id, timestamp, lat, long)
VALUES ('amsterdam', 'cd032f56-cf1a-4800-8000-00000000066f', current_timestamp(), -101, 60);
```
6. The user ends the ride and releases the vehicle.
For example:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE vehicles SET status = 'available' WHERE id='bbbbbbbb-bbbb-4800-8000-00000000000b';
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE rides SET end_address ='33862 Charles Junctions Apt. 49', end_time=TIMESTAMP '2020-10-01 10:30:00.123456', revenue=88.6
WHERE id='cd032f56-cf1a-4800-8000-00000000066f';
```
## See also
* Features in Preview
# Serverless Function Best Practices
Source: https://docs.cockroachlabs.com/docs/v26.3/serverless-function-best-practices
This page provides best practices for optimizing the performance of serverless functions that connect to CockroachDB, such as [AWS Lambda functions](https://aws.amazon.com/lambda) and [Google Cloud Functions](https://cloud.google.com/functions).
## Use connection pools that persist across function invocations
Use connection pools to manage the lifecycle of database connections established by serverless functions. Connection pools check connection health and re-establish broken connections in the event of a communication error.
When creating connection pools in serverless functions:
* Set the maximum connection pool size to 1, unless your function is multi-threaded and establishes multiple concurrent requests to your database within a single function instance.
* Do not set a minimum idle connection count. The connection pool should be free to open connections as needed.
* If supported by your pooling library, set the maximum lifetime on the connection pool to 30 minutes.
If you plan to invoke a serverless function frequently, configure the function to persist connection pools across function invocations. This helps to limit the number of new connection attempts to the cluster. One way to do this is to initialize the connection pool variable outside the scope of the serverless function definition.
For example if an AWS Lambda function uses `INSERT` to add data to a table and runs every few seconds, initialize the connection pool variable outside of the [handler function](https://docs.aws.amazon.com/lambda/latest/dg/foundation-progmodel.html) definition, then define the connection pool in the handler only if the pool does not already exist.
Select either node.js or Python to continue.
The following node.js code implements this pattern:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
const { Pool } = require('pg')
let pool
const insertRows = async (p) => {
const client = await p.connect()
try {
await client.query('INSERT INTO table (col1, col2) VALUES (val1, val2)')
} catch (err) {
console.log(err.stack)
} finally {
client.release()
}
}
exports.handler = async (context) => {
if (!pool) {
const connectionString = process.env.DATABASE_URL
pool = new Pool({
connectionString,
max: 1
})
}
await insertRows(pool)
}
```
The following Python code implements this pattern:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
from psycopg2.pool import SimpleConnectionPool
pool = None
def query(p):
conn = p.getconn()
with conn.cursor() as cur:
cur.execute("INSERT INTO table (col1, col2) VALUES (val1, val2)")
conn.commit()
def lambda_handler(context):
global pool
if not pool:
pool = SimpleConnectionPool(0, 1, dsn=os.environ['DATABASE_URL'])
query(pool)
return
```
## Use CockroachDB Standard
As a database-as-a-service, CockroachDB Standard abstracts away the complexity of deploying, scaling, and load-balancing your database.
To create a free CockroachDB Standard cluster:
## Deploy serverless functions in the same region as your cluster
To minimize network latency, you should deploy your serverless functions in the same region as your cluster. If your serverless function provider does not offer deployments in the same region as your database deployment, choose the region nearest to the region where your database is deployed.
## See also
* Create and Deploy an AWS Lambda Function Built on CockroachDB
* Connect to your Cluster
# Authenticate to CockroachDB Self-Hosted Clusters
Source: https://docs.cockroachlabs.com/docs/v26.3/authentication
Authentication refers to the act of verifying the identity of the other party in communication. CockroachDB requires TLS 1.3 digital certificates for inter-node authentication and accepts TLS 1.2 and TLS 1.3 certificates for client-node authentication. This document discusses how CockroachDB uses digital certificates and what your options are for configuring user authentication for SQL clients and the DB Console UI. It also offers a [conceptual overview](#background-on-public-key-cryptography-and-digital-certificates) of public key cryptography and digital certificates.
* If you are familiar with public key cryptography and digital certificates, then reading the [Using digital certificates with CockroachDB](#using-digital-certificates-with-cockroachdb) section should be enough.
* If you are unfamiliar with public key cryptography and digital certificates, you might want to skip over to the [conceptual overview](#background-on-public-key-cryptography-and-digital-certificates) first and then come back to the [Using digital certificates with CockroachDB](#using-digital-certificates-with-cockroachdb) section.
* If you want to know how to create CockroachDB security certificates, see Create Security Certificates.
## Connecting to a CockroachDB cluster
Users may connect with CockroachDB self-hosted clusters in 2 main ways:
* SQL clients connections, including the CockroachDB CLI client and the various supported drivers and ORMs, connect directly to CockroachDB clusters using the CockroachDB SQL interface.
* A read-only monitoring service, which provides cluster and database details, and information useful for troubleshooting and performance tuning. Each CockroachDB node also acts as an HTTP server, providing both a browser UI DB console and the cluster API, which provides much of the same information as the DB console, but as a rest API suitable for programmatic access.
## Using digital certificates with CockroachDB
Each CockroachDB node in a secure cluster must have a **node certificate**, which is a TLS 1.3 certificate. This certificate is multi-functional; the same certificate is presented irrespective of whether the node is acting as a server or a client. The nodes use these certificates to establish secure connections with clients and with other nodes. Node certificates have the following requirements:
* The hostname or address (IP address or DNS name) used to reach a node, either directly or through a load balancer, must be listed in the **Common Name** or **Subject Alternative Names** fields of the certificate:
* The values specified in `--listen-addr` and `--advertise-addr` flags, or the node hostname and fully qualified hostname if not specified
* Any host addresses/names used to reach a specific node
* Any load balancer addresses/names or DNS aliases through which the node could be reached
* `localhost` and local address if connections are made through the loopback device on the same host
* CockroachDB must be configured to trust the certificate authority that signed the certificate.
Based on your security setup, you can use the `cockroach cert` commands, `openssl` commands, or a custom CA to generate all the keys and certificates.
A CockroachDB cluster consists of multiple nodes and clients. The nodes can communicate with each other, with the SQL clients, and the DB Console. In client-node SQL communication and client-UI communication, the node acts as a server, but in inter-node communication, a node may act as a server or a client. Hence authentication in CockroachDB involves:
* Node authentication using [TLS 1.3](https://wikipedia.org/wiki/Transport_Layer_Security) digital certificates.
* Client authentication using TLS digital certificates, passwords, or GSSAPI authentication (for Enterprise users).
If you manage your own Certificate Authority (CA) infrastructure, CockroachDB supports mapping between the Subject field of your [X.509 certificates](https://en.wikipedia.org/wiki/X.509) and SQL roles. For more information, see Certificate-based authentication using multiple values from the X.509 Subject field.
### Node authentication
To set up a secure cluster without using an existing certificate authority, you'll need to generate the following files:
* CA certificate
* Node certificate and key
* (Optional) UI certificate and key
### Client authentication
CockroachDB offers the following methods for client authentication:
* **Client certificate and key authentication**, which is available to all users. To ensure the highest level of security, we recommend only using client certificate and key authentication.
Example:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --user=jpointsman
```
* **Password authentication**, which is available to users and roles who you've created passwords for. Password creation is supported only in secure clusters.
Example:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --user=jpointsman
```
```
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
Enter password:
```
The client still needs the CA certificate to validate the certificate of the node.
For improved performance, CockroachDB securely caches password authentication information for users. To limit the authentication latency of users logging into a new session, we recommend that you run bulk `ROLE` operations (`CREATE ROLE`, `ALTER ROLE`, `DROP ROLE`) inside a transaction, and run any regularly-scheduled `ROLE` operations together, rather than at different times throughout the day.
* **Password authentication without TLS**
For deployments where transport security is already handled at the infrastructure level (e.g., IPSec with DMZ), and TLS-based transport security is not possible or not desirable, CockroachDB now supports delegating transport security to the infrastructure with the flag `--accept-sql-without-tls` for `cockroach start`. The `--accept-sql-without-tls` flag is in preview.
With this flag, SQL clients can establish a session over TCP without a TLS handshake. They still need to present valid authentication credentials, for example a password in the default configuration. Different authentication schemes can be further configured as per `server.host_based_authentication.configuration`.
Example:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --user=jpointsman --insecure
```
```
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
Enter password:
```
* **Single sign-on authentication to DB Console**. CockroachDB supports OpenID Connect (OIDC) for authenticating users to the DB Console. You can enable automatic user provisioning and automatic role synchronization based on group memberships from your identity provider.
* **JWT authentication for SQL clients**. CockroachDB supports JSON Web Token (JWT) authentication for SQL client connections. You can enable automatic user provisioning and automatic role synchronization based on group claims in JWT tokens.
* **GSSAPI authentication**.
### Using `cockroach cert` or `openssl` commands
You can use the `cockroach cert` commands or `openssl` commands to create the CA certificate and key, and node and client certificates and keys.
Note that the node certificate created using `cockroach cert` or`openssl` is multi-functional, which means that the same certificate is presented irrespective of whether the node is acting as a server or a client. Thus all nodes must have the following:
* `CN=node` for the special user `node` when the node acts as a client.
* All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name` field for when the node acts as a server. CockroachDB also supports [wildcard notation in DNS names](https://wikipedia.org/wiki/Wildcard_certificate).
**Node key and certificates**
A node must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate created using the `cockroach cert` command. |
| `node.crt` | Server certificate created using the `cockroach cert` command.
`node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in the `Subject Alternative Name` field. CockroachDB also supports [wildcard notation in DNS names](https://wikipedia.org/wiki/Wildcard_certificate).
Must be signed by the CA represented by `ca.crt`. |
| `node.key` | Server key created using the `cockroach cert` command. |
**Client key and certificates**
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate created using the `cockroach cert` command. |
| `client.
Each `client.
Must be signed by the CA represented by `ca.crt`. |
| `client.
All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name`. CockroachDB also supports [wildcard notation in DNS names](https://wikipedia.org/wiki/Wildcard_certificate).
Must be signed by the CA represented by `ca.crt`. |
| `node.key` | Server key corresponding to `node.crt`. |
| `client.node.crt` | Node certificate for when node acts as client.
Must have `CN=node`.
Must be signed by the CA represented by `ca.crt`. |
| `client.node.key` | Client key corresponding to `client.node.crt`. |
Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag.
**Client key and certificates**
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate issued by the public CA or your organizational CA. |
| `client.
Each `client.
Must be signed by the CA represented by `ca.crt`. |
| `client.
`node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in the `Subject Alternative Name` field. CockroachDB also supports [wildcard notation in DNS names](https://wikipedia.org/wiki/Wildcard_certificate).
Must be signed by the CA represented by `ca.crt`. |
| `node.key` | Server key created using the `cockroach cert` command. |
| `ui.crt` | UI certificate signed by the public CA. `ui.crt` must have the IP addresses and DNS names used to reach the DB Console listed in the `Subject Alternative Name`. |
| `ui.key` | UI key corresponding to `ui.crt`. |
**Client key and certificates**
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate created using the `cockroach cert` command. |
| `client.
Each `client.
Must be signed by the CA represented by `ca.crt`. |
| `client.
We do not recommend you use split CA certificates unless your organizational security practices mandate you to do so.
You might encounter situations where you need separate CAs to sign and verify node and client certificates. In that case, you would need two CAs and their respective certificates and keys: `ca.crt` and `ca-client.crt`.
**Node key and certificates**
A node must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate to verify node certificates. |
| `ca-client.crt` | CA certificate to verify client certificates. |
| `node.crt` | Node certificate for when node acts as server.
All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name`. CockroachDB also supports [wildcard notation in DNS names](https://wikipedia.org/wiki/Wildcard_certificate).
Must be signed by the CA represented by `ca.crt`. |
| `node.key` | Server key corresponding to `node.crt`. |
| `client.node.crt` | Node certificate for when node acts as client. This certificate must be signed by the CA represented by `ca-client.crt`.
Must have `CN=node`. |
| `client.node.key` | Client key corresponding to `client.node.crt`. |
Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag.
**Client key and certificates**
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate. |
| `client.
Each `client.
Must be signed by the CA represented by `ca-client.crt`. |
| `client.Use Cloud Storage for Bulk Operations.
## Authentication best practice
As a security best practice, we recommend that you rotate the node, client, or CA certificates in the following scenarios:
* The node, client, or CA certificates are expiring soon.
* Your organization's compliance policy requires periodical certificate rotation.
* The key (for a node, client, or CA) is compromised.
* You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.
For details about when and how to change security certificates without restarting nodes, see Rotate Security Certificates.
## Background on public key cryptography and digital certificates
CockroachDB supports the [TLS 1.3 and TLS 1.2](https://wikipedia.org/wiki/Transport_Layer_Security) encryption for SQL clients. Starting in CockroachDB v22.2, only cipher suites currently recommended by the IETF ([RFC 8447](https://datatracker.ietf.org/doc/html/rfc8447)) are enabled by default. In v22.2.4 and above, the environment variable `COCKROACH_TLS_ENABLE_OLD_CIPHER_SUITES` can be used to revert to the cipher suite configuration used in version 22.1. This environment variable is not available in v22.2.0-v22.2.3. You should set this environment variable only if you cannot use one of the default cipher suites, but you can use one of the disabled ones.
CockroachDB supports the [TLS 1.3 and TLS 1.2](https://wikipedia.org/wiki/Transport_Layer_Security) encryption for SQL clients. Only cipher suites currently recommended by the IETF ([RFC 8447](https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4)) are enabled by default. The environment variable `COCKROACH_TLS_ENABLE_OLD_CIPHER_SUITES` can be used to revert to the cipher suite configuration used prior to v22.2. You should set this environment variable only if you cannot use one of the default cipher suites, but you can use one of the disabled ones.
SQL clients, intermediate proxies, or load balancers that do not support any cipher suites that CockroachDB supports will be unable to connect to CockroachDB clusters. For the full list of supported cipher suites, refer to [Supported cipher suites](https://cockroachlabs.com/docs/stable/authentication#supported-cipher-suites).
Authentication refers to the act of verifying the identity of the other party in communication. CockroachDB uses TLS 1.3 digital certificates for inter-node authentication, and your choice of TLS 1.2 and TLS 1.3 certificates for client-node authentication (defaulting to TLS 1.3, if supported by the client). These authentication methods require a certificate authority (CA) as well as keys and certificates for nodes, clients, and, optionally, the [DB Console](#using-a-public-ca-certificate-to-access-the-db-console-for-a-secure-cluster).
To understand how CockroachDB uses digital certificates, let's first understand what each of these terms means.
Consider two people: Amy and Rosa, who want to communicate securely over an insecure computer network. The traditional solution is to use symmetric encryption that involves encrypting and decrypting a plaintext message using a shared key. Amy encrypts her message using the key and sends the encrypted message across the insecure channel. Rosa decrypts the message using the same key and reads the message. This seems like a logical solution until you realize that you need a secure communication channel to send the encryption key.
To solve this problem, cryptographers came up with **asymmetric encryption** to set up a secure communication channel over which an encryption key can be shared.
### Asymmetric encryption
Asymmetric encryption involves a pair of keys instead of a single key. The two keys are called the **public key** and the **private key**. The keys consist of very long numbers linked mathematically in a way such that a message encrypted using a public key can only be decrypted using the private key and vice versa. The message cannot be decrypted using the same key that was used to encrypt the message.
So going back to our example, Amy and Rosa both have their own public-private key pairs. They keep their private keys safe with themselves and publicly distribute their public keys. Now when Amy wants to send a message to Rosa, she requests Rosa's public key, encrypts the message using Rosa’s public key, and sends the encrypted message. Rosa uses her own private key to decrypt the message.
But what if a malicious imposter intercepts the communication? The imposter might pose as Rosa and send their public key instead of Rosa’s. There's no way for Amy to know that the public key she received isn’t Rosa’s, so she would end up using the impostor's public key to encrypt the message and send it to the imposter. The imposter can use their own private key and decrypt and read the message, thus compromising the secure communication channel between Amy and Rosa.
To prevent this security risk, Amy needs to be sure that the public key she received was indeed Rosa’s. That’s where the Certificate Authority (CA) comes into the picture.
### Certificate authority
Certificate authorities are established entities with their own public and private key pairs. They act as a root of trust and verify the identities of the communicating parties and validate their public keys. CAs can be public and paid entities (e.g., GeoTrust and Comodo), or public and free CAs (e.g., Let’s Encrypt), or your own organizational CA (e.g., CockroachDB CA). The CAs' public keys are typically widely distributed (e.g., your browser comes preloaded with certs from popular CAs like DigiCert, GeoTrust, and so on).
Think of the CA as the passport authority of a country. When you want to get your passport as your identity proof, you submit an application to your country's passport authority. The application contains important identifying information about you: your name, address, nationality, date of birth, and so on. The passport authority verifies the information they received and validates your identity. They then issue a document - the passport - that can be presented anywhere in the world to verify your identity. For example, the TSA agent at the airport does not know you and has no reason to trust you are who you say you are. However, they trust the passport authority and thus accept your identity as presented on your passport because it has been verified and issued by the passport authority.
Going back to our example and assuming that we trust the CA, Rosa needs to get her public key verified by the CA. She sends a CSR (Certificate Signing Request) to the CA that contains her public key and relevant identifying information. The CA will verify that it is indeed Rosa’s public key and information, *sign* the CSR using the CA's own private key, and generate a digital document called the **digital certificate**. In our passport analogy, this is Rosa's passport containing verified identifying information about her and trusted by everyone who trusts the CA. The next time Rosa wants to establish her identity, she will present her digital certificate.
### Digital certificate
A public key is shared using a digital certificate signed by a CA using the CA's private key. The digital certificate contains:
* The certificate owner’s public key
* Information about the certificate owner
* The CA's digital signature
### Digital signature
The CA's digital signature works as follows: The certificate contents are put through a mathematical function to create a **hash value**. This hash value is encrypted using the CA's private key to generate the **digital signature**. The digital signature is added to the digital certificate. In our example, the CA adds their digital signature to Rosa's certificate validating her identity and her public key.
As discussed [earlier](#certificate-authority), the CA's public key is widely distributed. In our example, Amy already has the CA's public key. Now when Rosa presents her digital certificate containing her public key, Amy uses the CA's public key to decrypt the digital signature on Rosa's certificate and gets the hash value encoded in the digital signature. Amy also generates the hash value for the certificate on her own. If the hash values match, then Amy can be sure that the certificate and hence the public key it contains indeed belongs to Rosa; otherwise, she can determine that the communication channel has been compromised and refuse further contact.
### How it all works together
Let's see how the digital certificate is used in client-server communication: The client (e.g., a web browser) has the CA certificate (containing the CA's public key). When the client receives a server's certificate signed by the same CA, it can use the CA certificate to verify the server's certificate, thus validating the server's identity, and securely connect to the server. The important thing here is that the client needs to have the CA certificate. If you use your own organizational CA instead of a publicly established CA, you need to make sure you distribute the CA certificate to all the clients.
## Supported cipher suites
CockroachDB supports the [TLS 1.3 and TLS 1.2](https://wikipedia.org/wiki/Transport_Layer_Security) encryption for SQL clients. However, only cipher suites currently recommended by the IETF ([RFC 8447](https://datatracker.ietf.org/doc/html/rfc8447)) are enabled by default. The environment variable `COCKROACH_TLS_ENABLE_OLD_CIPHER_SUITES` can be used to revert to the cipher suite configuration used prior to version 22.2. You should set this environment variable only if you cannot use one of the default cipher suites, but you can use one of the disabled ones.
By default, CockroachDB negotiates TLS 1.3 for client connections when supported by the client. If the client does not support TLS 1.3, the connection will fall back to TLS 1.2.
To restrict your cluster to only allow connections using specific cipher suites, use the `--tls-cipher-suites` startup flag with the `cockroach start` command. This setting applies to all incoming SQL, RPC, and HTTP connections.
The following cipher suites are enabled by default:
* `TLS_AES_128_GCM_SHA256`,
* `TLS_AES_256_GCM_SHA384`,
* `TLS_CHACHA20_POLY1305_SHA256`
* `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`
* `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`
* `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`
* `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`
* `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256`
* `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256`
The following cipher suites are rejected by default because they are not recommended by the IETF ([RFC 8447](https://datatracker.ietf.org/doc/html/rfc8447)):
* `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`
* `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA`
* `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA`
* `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA`
* `TLS_RSA_WITH_AES_128_GCM_SHA256`
* `TLS_RSA_WITH_AES_256_GCM_SHA384`
* `TLS_RSA_WITH_AES_128_CBC_SHA`
* `TLS_RSA_WITH_AES_256_CBC_SHA`
## Supported key exchange mechanisms
For information about supported TLS 1.3 key exchange mechanisms, including post-quantum cryptography support, see Post-quantum cryptography support in CockroachDB.
## See also
* Client Connection Parameters
* Manual Deployment
* Orchestrated Deployment
* Local Deployment
* `cockroach` Commands Overview
* Certificate-based authentication using multiple values from the X.509 Subject field
* `ALTER ROLE ... SUBJECT`
* `CREATE ROLE ... SUBJECT`
* `cockroach cert`
* `cockroach auth-session`
* GSSAPI Authentication
* Single Sign-on (SSO) for DB Console
* Cluster Single Sign-on (SSO) using JWTs
* JWT Authorization
* OIDC Authorization
* LDAP Authorization
* SQL Authentication
* Cloud Storage Authentication
# Manage Cluster User Authorization
Source: https://docs.cockroachlabs.com/docs/v26.3/authorization
This page documents procedures for managing the lifecycle of SQL users and roles on CockroachDB clusters.
Procedures for managing access are covered in Managing Access in CockroachDB Cloud.
The concept of *Organization user* is distinct from Authorization of *SQL user/role* on clusters is distinct from authorization of CockroachDB Cloud users within CockroachDB Cloud organizations.
Learn more: Overview of the CockroachDB Cloud authorization model
For reference documentation and explanation of related concepts, see Security Reference — Authorization.
## Create CockroachDB users
Use the `CREATE USER` and `DROP USER` statements to create and remove users, the `ALTER USER` statement to add or change a user's password and role options, the `GRANT` and `REVOKE` statements to manage the user’s privileges, and the `SHOW USERS` statement to list users.
A new user must be granted the required privileges for each database and table that the user needs to access.
By default, a new user belongs to the `public` role and has no privileges other than those assigned to the `public` role.
## Create and manage roles
For examples showing how to create and manage your cluster's users and roles, see the following documentation:
| Statement | Description |
| - | - |
| `CREATE ROLE` | Create SQL roles. |
| `DROP ROLE` | Remove one or more SQL roles. |
| `ALTER ROLE` | Change passwords, role options, and default session variables for a role. |
| `CREATE USER` | Create SQL users. |
| `DROP USER` | Remove one or more SQL users. |
| `ALTER USER` | Change passwords and role options for a user. |
| `GRANT` | Manage each role or user's SQL privileges for interacting with specific databases and tables, or add a role or user as a member to a role. |
| `REVOKE` | Revoke privileges from users and/or roles, or revoke a role or user's membership to a role. |
| `SHOW ROLES` | List the roles for all databases. |
| `SHOW GRANTS` | List the privileges granted to users. |
## See also
* Client Connection Parameters
* SQL Statements
* `CREATE USER`
* `ALTER USER`
* `DROP USER`
* `SHOW USERS`
* `CREATE ROLE`
* `ALTER ROLE`
* `DROP ROLE`
* `SHOW ROLES`
* `GRANT`
* `REVOKE`
* `SHOW GRANTS`
# Certificate-based authentication using multiple values from the X.509 Subject field
Source: https://docs.cockroachlabs.com/docs/v26.3/certificate-based-authentication-using-the-x509-subject-field
For customers that need to use their established Certificate Authority (CA) infrastructure to manage SQL user authentication, CockroachDB supports mapping of SQL user roles to values in the Subject field of the [X.509 certificate](https://en.wikipedia.org/wiki/X.509) used for TLS authentication.
This mapping (hereafter referred to as "Subject mapping") can be used to automate the synchronization of SQL user roles with specific certificate attributes, leading to improved scalability of access control mechanisms.
Subject mapping is useful if:
* You run your own Certificate Authority (CA) infrastructure.
* You need to use your existing CA infrastructure to manage SQL user authentication.
* You need to use the same CA for multiple CockroachDB clusters.
For instructions showing how to map SQL user roles to values in the Subject field of the X.509 certificate, see the [Example](#example).
## Related functionality
Prior to CockroachDB v24.1, it was only safe to use certificates generated by CAs that placed restrictions on the `CommonName` field (so that only authorized users could get a cert with `CommonName=root`, for example). That meant that mapping the certificate principals to IP addresses, DNS names, and SQL users required using the `--cert-principal-map` flag, which was first added in v20.1. That approach still works, but is superseded by the functionality described on this page, which should be much easier to use since you do not need to restart nodes [when you add or remove users](#step-3-set-the-subject-role-option-using-create-role-or-alter-role-to-match-your-certificates).
The Subject mapping functionality described on this page makes it possible to use CAs that do not place restrictions on the `CommonName` field but do place restrictions on other subject fields (for example, that every cert has an `OU` field based on the user that created it).
The default workflow using `cockroach cert` only sets the `CommonName` field in its certificates, so the functionality described in this page is unnecessary. It is not possible to share a single CA managed by `cockroach cert` across multiple CockroachDB clusters.
## Example
This example shows how to set up a mapping between the Subject field of an [X.509 certificate](https://en.wikipedia.org/wiki/X.509) and a SQL user or role.
### Step 1. Create certificates using your infrastructure
These instructions assume that you have already created certificates using your own infrastructure for the following users:
* The `root` user.
* The `node` user.
* Any SQL user who needs to authenticate with your cluster. In this example, we will call the user `maxroach`.
General guidelines for certificate creation:
* The cluster name and SQL user name will generally both appear somewhere in the certificate's Subject.
* The cluster name will usually go in the `OU` or `DC` fields, and the user name in `UID` or `CN` fields. You can also map the user name to a `SAN` field by enabling [Subject Alternative Name mapping](#optional-enable-the-cluster-setting-to-map-users-to-subject-alternative-name-san-fields).
* For example, the Subject might look like `O=Acme Inc,OU=movr-prod,UID=root`.
### Step 2. Start your cluster with root and node certificate flags
At cluster startup, you'll need to pass the `cockroach start` flags `--node-cert-distinguished-name` and `--root-cert-distinguished-name`.
The argument to each flag is a string with a comma separated list of distinguished name (DN) mappings in `{attribute-type}={attribute-value}` format in accordance with [RFC4514](https://www.rfc-editor.org/rfc/rfc4514). When each of these flags are set, the argument needs to be an exact match with the DN subject in the client certificate provided. By exact match, we mean that the order of attributes in the argument must match the order of attributes in the DN subject in the certificate.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach start --certs-dir=/path/to/certs --node-cert-distinguished-name="O=Acme Inc,OU=movr-prod,UID=node" --root-cert-distinguished-name="O=Acme Inc,OU=movr-prod,UID=root" # other startup flags, etc.
```
The DN mappings in the certificates you create should match what you pass in at cluster start time. One way to create a certificate matching the `node` user in the example above is:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
openssl req -new -key /path/to/certs/client.node.key -out client.node.csr -batch -subj '/O=Acme Inc/OU=movr-prod/UID=node'
```
A similar command can be used to generate a certificate for the `root` user.
### Step 3. Set the `SUBJECT` role option using `CREATE ROLE` or `ALTER ROLE` to match your certificates
You can associate an [X.509](https://en.wikipedia.org/wiki/X.509) certificate's Subject with a role as shown below. Note that the Subject fields in the certificate have to be an exact match with what you pass in via the SQL statement. By exact match, we mean that the order of attributes passed in via the SQL statement must match the order of attributes in the certificate.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE ROLE maxroach WITH SUBJECT 'CN=myName,OU=myOrgUnit,O=myOrg,L=myLocality,ST=myState,C=myCountry' LOGIN;
```
Alternatively, you can update existing users with `ALTER ROLE ... SUBJECT`.
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER ROLE maxroach WITH SUBJECT 'CN=myName2,OU=myOrgUnit2,O=myOrg2,L=myLocality2,ST=myState2,C=myCountry2' LOGIN;
```
Note that `ALTER ROLE ... SUBJECT` cannot be applied to the `root` user. You must use `cockroach start --root-cert-distinguished-name` instead.
```
ERROR: role "root" cannot have a SUBJECT%!(EXTRA string=use the --%s CLI flag to configure root, string=root-cert-distinguished-name)
SQLSTATE: 22023
```
If you do not have an enterprise license, the following error is signaled:
```
ERROR: use of SUBJECT role option requires an enterprise license. see https://cockroachlabs.com/pricing for details on how to enable enterprise features
SQLSTATE: XXC02
```
### Step 4. Enable the cluster setting to require the Subject field in certificates
Once the cluster is started, enable the following cluster setting:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING security.client_cert.subject_required.enabled = true;
```
When this setting is enabled, users who don't have a `SUBJECT` role option set will be unable to log in with certificates.
Regardless of this setting's value, CockroachDB will verify the following during user authentication provided the Subject options or DN flags or both are set:
* For the `root` user, that the distinguished name in the certificate Subject matches the distinguished name fields passed in via `cockroach start --root-cert-distinguished-name`.
* For the `node` user, that the distinguished name in the certificate Subject matches the distinguished name fields passed in via `cockroach start --node-cert-distinguished-name`.
* For all other SQL users, that the values in the Subject field of the [X.509 certificate](https://en.wikipedia.org/wiki/X.509) match the values attached to the user or role with `CREATE ROLE ... SUBJECT` or `ALTER ROLE ... SUBJECT`.
#### (Optional) Enable the cluster setting to map users to Subject Alternative Name (`SAN`) fields
If your organization uses Subject Alternative Name (`SAN`) fields for user identity mapping, enable the following cluster setting to map user identity to `SAN` fields instead of the default `CN` lookup:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING security.client_cert.san_required.enabled = true;
```
## See also
* `ALTER ROLE ... SUBJECT`
* `CREATE ROLE ... SUBJECT`
* Authenticate to CockroachDB self-hosted Clusters
* GSSAPI Authentication
* SQL Authentication
* Cloud Storage Authentication
* Cluster Settings
* `cockroach start`
* `cockroach cert`
* `cockroach auth-session`
# CockroachDB MCP Server
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-mcp-server
The CockroachDB MCP Server is a [Model Context Protocol (MCP)](https://modelcontextprotocol.io) server that lets your AI coding tools and AI agents explore live schemas, run queries, and write data on a CockroachDB cluster. The server runs as a local subprocess or a shared HTTP service and exposes a set of typed tools that agents invoke on your behalf. Once connected, you can interact with your cluster using natural language prompts to perform operations such as listing tables, executing `SELECT` statements, and inserting rows.
This page explains how the CockroachDB MCP Server works, and what functionality it enables. To learn how to set it up, refer to Install, Configure, and Connect to the CockroachDB MCP Server. For the full list of configuration options, refer to the CockroachDB MCP Server Reference.
For CockroachDB Cloud deployments, refer to Connect to the CockroachDB Cloud MCP Server.
## Requirements
* An existing CockroachDB cluster with a SQL user whose privileges are suitable for the operations that you want your AI agents to perform.
* Installing with `go install` or building from source requires [Go 1.26 or later](https://go.dev/dl/). Alternatively, you can install with pre-built binaries (for Linux and Windows), and a Docker image is also available.
## How it works
The server registers a set of [typed MCP tools](#tools) that AI agents invoke on your behalf. These tools trigger SQL queries against the connected cluster using a CockroachDB SQL user.
The server supports two transports:
* **stdio (default)**: Your AI tool launches the server as a local subprocess and communicates with it over standard input/output. Each user runs their own server instance, authenticated to the cluster with their own SQL credentials. This is the simplest deployment and the recommended starting point.
* **HTTP**: The server runs as a long-lived process that multiple MCP clients connect to over HTTPS with a bearer token. This is best for shared or remote deployments.
## Security
The server is configured with secure defaults and requires explicit opt-in to relax them.
* **Read-only by default.** Unless you explicitly enable write tools with the `CRDB_MCP_ENABLE_WRITE_QUERIES` environment variable, only read tools are registered, and every SQL session is additionally forced read-only with `default_transaction_read_only=true`. For per-operation control, grant the connecting SQL role only the privileges you want: for example, `SELECT`, `INSERT`, and `UPDATE` but not `DELETE` to allow writes without deletes.
* **SQL grants are the permission boundary.** Security ultimately comes from what the connecting SQL user is allowed to do in CockroachDB (via `GRANT`), not from which environment variables you set on the MCP server. The server configuration controls which tools are registered, but even with write tools enabled, the SQL user can only do what its grants allow. Least-privilege grants are also the containment for prompt injection: the server cannot distinguish a tool call you intended from one induced by malicious content that the agent has read.
* **Password authentication is rejected by default.** In stdio mode the server runs as a subprocess of the AI agent host, which can read passwords from the process environment or `~/.pgpass`. Certificate-based authentication is recommended; password-based authentication requires an explicit opt-in.
* **TLS by default, in both directions.** Connections to the cluster require `sslmode` of `require` or stronger, and HTTP mode refuses to start without a TLS certificate and key. Running without TLS—for local development or behind a TLS-terminating reverse proxy—requires explicit opt-in.
* **Background quality of service.** By default, queries run at `default_transaction_quality_of_service=background`, so agent traffic does not contend with latency-sensitive foreground workloads.
## Tools
For the full tool specifications including parameters, refer to the CockroachDB MCP Server Reference.
The server registers the following read-only tools by default:
| Tool | Description |
| - | - |
| `list_databases` | List all databases in the cluster. |
| `list_tables` | List all tables in a database. |
| `get_table_schema` | Get the schema for a table, including columns and indexes. |
| `get_cluster` | Return cluster identity and version metadata. |
| `list_sql_users` | List SQL users defined in the cluster. |
| `list_cluster_nodes` | List cluster nodes with address, liveness, and locality. |
| `show_running_queries` | List in-flight SQL statements on the cluster. |
| `select_query` | Execute a `SELECT` statement. |
| `explain_query` | Return the `EXPLAIN` plan for a statement without executing it. |
| `show_statement` | Execute a `SHOW` statement such as `SHOW SCHEMAS`, `SHOW INDEXES`, or `SHOW JOBS`. |
The following write tools are registered only when `CRDB_MCP_ENABLE_WRITE_QUERIES``=true` is set:
| Tool | Description |
| - | - |
| `create_database` | Create a database. |
| `create_table` | Execute a `CREATE TABLE` statement. |
| `insert_rows` | Execute an `INSERT` statement. |
| `update_rows` | Execute an `UPDATE` statement. A `WHERE` clause is mandatory. |
| `delete_rows` | Execute a `DELETE` statement. A `WHERE` clause is mandatory. |
## Example usage
Use natural language prompts to read from and write to the cluster. You likely do not have to invoke the preceding tools explicitly. Most AI tools should be able to decide which functions to call based on natural language prompts provided by the user.
Prompts can be simple, for example:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
List all of the tables in the movr database.
```
They can also be complex and conversational, for example:
```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
I need to add a reviews table to the movr database so that riders can rate their rides.
For the schema, I'm thinking we need:
- Review ID (primary key)
- Rider reference (foreign key to movr.users)
- Ride reference (foreign key to movr.rides)
- A 1–5 star rating
- Optional text feedback
- Timestamp
Let's include a secondary index on the rating column.
Show me the CREATE TABLE statement so that we can talk it through before you actually create the table.
```
## See also
* Install, Configure, and Connect to the CockroachDB MCP Server
* CockroachDB MCP Server Reference
* Connect to the CockroachDB Cloud MCP Server
* CockroachDB and AI
* Agent Skills for CockroachDB
* Docs MCP Server
# CockroachDB MCP Server Reference
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-mcp-server-reference
This page lists the configuration options for the MCP Server that connects to self-hosted CockroachDB deployments. To set up the server, refer to Install, Configure, and Connect to the CockroachDB MCP Server.
## Configuration
All configuration is managed via environment variables. In stdio mode, set them in the `env` block of your MCP client configuration; in HTTP mode, export them in the server's environment.
### Authentication
Certificate-based authentication is recommended in stdio mode. The server runs as a subprocess of the AI agent host, which can read `CRDB_PWD`, a password embedded in `CRDB_DATABASE_URL`, `PGPASSWORD`, or `~/.pgpass` from the process's environment. To protect those credentials, password-based authentication is rejected by default; set [`CRDB_MCP_ALLOW_PASSWORD_AUTH`](#crdb-mcp-allow-password-auth)`=true` to opt in.
You can authenticate with individual variables (recommended) or with a full connection string. [`CRDB_DATABASE_URL`](#crdb-database-url) takes precedence when both are set.
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_HOST` | Hostname. For a multi-node cluster, use a load balancer address; for development, any node's address. | Yes | |
| `CRDB_PORT` | Port. | No | `26257` |
| `CRDB_USERNAME` | SQL user. | Yes | |
| `CRDB_PWD` | Password. Requires [`CRDB_MCP_ALLOW_PASSWORD_AUTH`](#crdb-mcp-allow-password-auth)`=true`. Certificate-based authentication is recommended instead. | No | |
| `CRDB_SSL_MODE` | `require`, `verify-ca`, or `verify-full`. TLS-optional modes (`disable`, `allow`, `prefer`) require [`CRDB_MCP_ALLOW_INSECURE_DB`](#crdb-mcp-allow-insecure-db)`=true`. | No | `verify-full` |
| `CRDB_SSL_CA_PATH` | CA certificate path. | When `CRDB_SSL_MODE` is `verify-ca` or `verify-full` (the default) | |
| `CRDB_SSL_CERTFILE` | Client certificate path. | When `CRDB_SSL_MODE` is `require`, `verify-ca`, or `verify-full` (the default) | |
| `CRDB_SSL_KEYFILE` | Client key path. | When `CRDB_SSL_MODE` is `require`, `verify-ca`, or `verify-full` (the default) | |
Alternatively, use a full connection string:
| Variable | Description |
| - | - |
| `CRDB_DATABASE_URL` | A full libpq connection string (for example, `postgresql://user@host:26257/defaultdb?sslmode=verify-full&sslcert=...`). When set, the individual variables above are ignored. A `CRDB_DATABASE_URL` that omits `sslmode` is rejected at startup. |
### Query behavior
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_QUERY_TIMEOUT` | Per-query timeout, as a Go duration (for example, `30s`). The server cancels a query that exceeds it. | No | `30s` |
| `CRDB_MCP_MAX_ROWS_COUNT` | Cap on the maximum `LIMIT` that list-style tools issue to CockroachDB. | No | `10000` |
| `CRDB_MCP_MAX_CONNS` | Upper bound on the connection pool size (up to 100). | No | `10` |
| `CRDB_MCP_TXN_QOS` | Default transaction quality of service: `background`, `regular`, or `critical`. Refer to the precedence notes below the table. | No | `background` |
| `CRDB_MCP_ENABLE_WRITE_QUERIES` | Enable the [write tools](#write-tools). For per-operation control, grant the connecting role only the SQL privileges you want, for example `SELECT`, `INSERT`, and `UPDATE` but not `DELETE` to allow writes without deletes. | No | `false` |
| `CRDB_MCP_ALLOW_PASSWORD_AUTH` | Opt in to password-based authentication. The server refuses to start if a password is detected from any source ([`CRDB_PWD`](#crdb-pwd), a password in [`CRDB_DATABASE_URL`](#crdb-database-url), `PGPASSWORD`, or `~/.pgpass`) unless this is set to `true`. | No | `false` |
| `CRDB_MCP_ALLOW_INSECURE_DB` | Opt in to the `sslmode` values that can run without TLS: `disable`, `allow`, `prefer`. Certificate variables are not required in these modes. Development only. | No | `false` |
MCP traffic runs at `default_transaction_quality_of_service=background` by default, so it does not contend with latency-sensitive foreground workloads. The value is chosen with the following precedence:
1. [`CRDB_MCP_TXN_QOS`](#crdb-mcp-txn-qos), if set (`background`, `regular`, or `critical`).
2. Otherwise, a `default_transaction_quality_of_service=...` query parameter in [`CRDB_DATABASE_URL`](#crdb-database-url), if present.
3. Otherwise, `background`.
### Transport
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_TRANSPORT` | `stdio` or `http`. | No | `stdio` |
| `CRDB_MCP_HTTP_LISTEN_ADDR` | Listen address in HTTP mode, any `host:port`. Omit the host to bind all interfaces. Examples: `:8080`, `0.0.0.0:8443`, `127.0.0.1:9090`. | No | `:8080` |
| `CRDB_MCP_BEARER_TOKEN` | Bearer token that clients must send in `Authorization: Bearer {token}`. Must be at least 16 characters. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_NO_BEARER`](#crdb-mcp-allow-no-bearer)`=true` | |
| `CRDB_MCP_TLS_CERT` | PEM-encoded server certificate path. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_INSECURE_HTTP`](#crdb-mcp-allow-insecure-http)`=true` | |
| `CRDB_MCP_TLS_KEY` | PEM-encoded private key path. | Required for `http` mode, unless [`CRDB_MCP_ALLOW_INSECURE_HTTP`](#crdb-mcp-allow-insecure-http)`=true` | |
| `CRDB_MCP_ALLOW_INSECURE_HTTP` | Opt in to cleartext HTTP (for reverse-proxy deployments). | No | `false` |
| `CRDB_MCP_ALLOW_NO_BEARER` | Opt in to skipping bearer authentication (for upstream-auth deployments). A startup warning is logged. Mutually exclusive with [`CRDB_MCP_BEARER_TOKEN`](#crdb-mcp-bearer-token): the server refuses to start if both are set. | No | `false` |
### Logging
| Variable | Description | Required | Default |
| - | - | - | - |
| `CRDB_MCP_LOG_LEVEL` | `debug`, `info`, `warn`, or `error`. | No | `info` |
| `CRDB_MCP_LOG_PATH` | Log file path, or `-` for stderr. No rotation; use logrotate or your orchestrator. | No | |
### Tracing (OpenTelemetry)
Tracing is opt-in. When neither of the following variables is set, no exporter is installed. When enabled, tool calls and their SQL statements are exported as spans, and server logs are exported as OpenTelemetry log records. Query text and errors are redacted before export, so literals and user data never leave the server.
| Variable | Description | Required |
| - | - | - |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP gRPC endpoint; the standard `OTEL_*` environment variables are honored. | No |
| `CRDB_MCP_OTEL_FILE` | Write traces and logs as JSON lines to this file instead (takes precedence). | No |
## Tools
The following sections list the tools that the MCP server makes available to agents.
You likely do not have to invoke these functions explicitly. Most AI tools should be able to decide which functions to call based on natural language prompts provided by the user.
### Read-only tools
The following tools are registered by default. They do not modify cluster data.
#### `list_databases`
List all databases in the cluster.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `list_tables`
List all tables in a database.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `database` | Database whose tables are listed. | string | Yes | |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `get_table_schema`
Get detailed schema information for a table, including columns and indexes. Returns the `CREATE TABLE` statement.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `database` | Database containing the table. | string | Yes | |
| `table` | Table name. | string | Yes | |
| `schema` | Schema name. | string | No | `public` |
#### `get_cluster`
Return cluster identity and version metadata: `cluster_id`, `cluster_name`, `binary_version`, `active_version`.
The identity fields (`cluster_id`, `cluster_name`, and `active_version`) require access to `crdb_internal` built-in functions, which are gated by default on CockroachDB v25.4 and later. When these fields cannot be accessed, they are returned as `null`, with the reason reported under an `unavailable` key. `binary_version` is always returned.
This tool takes no parameters.
#### `list_sql_users`
List SQL users defined in the cluster.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `list_cluster_nodes`
List cluster nodes with address, liveness, and locality. Requires `admin` or the `VIEWCLUSTERMETADATA` system privilege.
This tool takes no parameters.
#### `show_running_queries`
List in-flight SQL statements on the cluster, ordered by start time descending. Viewing other users' statements requires `admin` or the `VIEWACTIVITY` or `VIEWACTIVITYREDACTED` system privilege; otherwise, only the connecting user's own statements are returned.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
#### `select_query`
Execute a single `SELECT` statement. Non-`SELECT` statements are rejected. A default `LIMIT` of `100` is appended when none is supplied, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). An explicit `LIMIT` that exceeds [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count), or `LIMIT ALL`, is rejected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `query` | A single `SELECT` statement. | string | Yes |
#### `explain_query`
Return the `EXPLAIN` plan for a single SQL statement without executing it. `EXPLAIN ANALYZE` (and `EXPLAIN ANALYZE (DEBUG)`) is rejected because it executes the statement. Display options (`VERBOSE`, `DISTSQL`, `TYPES`, `OPT`, and so on) pass through.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `query` | A single SQL statement. | string | Yes |
#### `show_statement`
Execute a single `SHOW` statement such as `SHOW SCHEMAS`, `SHOW INDEXES`, `SHOW REGIONS`, or `SHOW JOBS`. A default `LIMIT` is appended, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count).
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `query` | A single `SHOW` statement. | string | Yes | |
| `limit` | Maximum rows to return, capped at [`CRDB_MCP_MAX_ROWS_COUNT`](#crdb-mcp-max-rows-count). | integer | No | `100` |
| `offset` | Rows to skip before returning results. | integer | No | `0` |
### Write tools
The following tools are registered only when [`CRDB_MCP_ENABLE_WRITE_QUERIES`](#crdb-mcp-enable-write-queries)`=true` is set. Any write tool that includes a `RETURNING` clause returns the affected rows instead of just the count.
#### `create_database`
Create a database.
| Parameter | Description | Type | Required | Default |
| - | - | - | - | - |
| `name` | Database name to create. | string | Yes | |
| `if_not_exists` | When `true`, the statement is a no-op if the database already exists. | boolean | No | `false` |
#### `create_table`
Execute a single `CREATE TABLE` statement.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `CREATE TABLE` statement. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `insert_rows`
Execute a single `INSERT` statement. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `INSERT` statement. A `RETURNING` clause returns the inserted rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `update_rows`
Execute a single `UPDATE` statement. A `WHERE` clause is mandatory. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `UPDATE` statement. A `WHERE` clause is mandatory. A `RETURNING` clause returns the updated rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
#### `delete_rows`
Execute a single `DELETE` statement. A `WHERE` clause is mandatory. Returns the number of rows affected.
| Parameter | Description | Type | Required |
| - | - | - | - |
| `statement` | A single `DELETE` statement. A `WHERE` clause is mandatory. A `RETURNING` clause returns the deleted rows. Qualify the table with a database when not using the connection's default database. | string | Yes |
## See also
* CockroachDB MCP Server
* Install, Configure, and Connect to the CockroachDB MCP Server
* [CockroachDB MCP Server on GitHub](https://github.com/cockroachdb/cockroachdb-mcp-server)
# CockroachDB Operator Overview
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-operator-overview
The CockroachDB operator is a fully-featured [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that allows you to deploy and manage CockroachDB self-hosted clusters.
## CockroachDB operator
This section describes how to:
* Deploy a CockroachDB cluster using the CockroachDB operator.
* Migrate from an existing CockroachDB Kubernetes deployment using Helm or the Public operator.
* Operate a CockroachDB cluster:
* Manage pod scheduling.
* Manage cluster resources.
* Manage certificates.
* Scale a cluster.
* Monitor a cluster.
* Upgrade a cluster.
* Override deployment templates.
* Improve cluster performance.
## Kubernetes terminology
Before starting, review some basic Kubernetes terminology. Note that CockroachDB nodes are distinct from Kubernetes "nodes" or "worker nodes".
| Feature | Description |
| - | - |
| [node](https://kubernetes.io/docs/concepts/architecture/nodes/) | A physical or virtual machine. In the deployment guide, you'll create instances and join them as worker nodes into a single Kubernetes cluster. |
| [pod](http://kubernetes.io/docs/user-guide/pods/) | A pod is a group of one of more Docker containers. In the deployment guide, each pod will run on a separate Kubernetes worker node and include one Docker container running a single CockroachDB node, reflecting our topology recommendations. |
| [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) | An operator is an extension to Kubernetes that uses custom resources to efficiently manage specific applications. The CockroachDB operator includes two custom resource definitions to manage CockroachDB, `CrdbCluster` and `CrdbNode`. Unlike the older [](https://github.com/cockroachdb/cockroach-operator), the CockroachDB operator does not use StatefulSets and is designed to simplify multi-region deployments. |
| [persistent volume](http://kubernetes.io/docs/user-guide/persistent-volumes/) | A persistent volume is a piece of networked storage (Persistent Disk on GCE, Elastic Block Store on AWS) mounted into a pod. The lifetime of a persistent volume is decoupled from the lifetime of the pod that's using it, ensuring that each CockroachDB node binds back to the same storage on restart.
The deployment guide assumes that dynamic volume provisioning is available. When that is not the case, [persistent volume claims](http://kubernetes.io/docs/user-guide/persistent-volumes/#persistentvolumeclaims) need to be created manually. |
| [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) | RBAC, or Role-Based Access Control, is the system Kubernetes uses to manage permissions within the cluster. In order to take an action (e.g., `get` or `create`) on an API resource (e.g., a `pod`), the client must have a `Role` that allows it to do so. The deployment guide creates the RBAC resources necessary for CockroachDB to create and access certificates. |
# Performance with the CockroachDB Operator
Source: https://docs.cockroachlabs.com/docs/v26.3/cockroachdb-operator-performance
Kubernetes provides many useful abstractions for deploying and operating distributed systems, but some of the abstractions come with a performance overhead and an increase in underlying system complexity. This page outlines potential bottlenecks when running CockroachDB in Kubernetes and how to optimize performance.
## Before you begin
Before you focus on optimizing a Kubernetes-orchestrated CockroachDB cluster:
1. Before deploying on Kubernetes, ensure that performance is optimized for your workload on identical hardware. You may find that you first need to modify your workload or use different machine specs to achieve the performance you need.
2. Read the documentation for deploying CockroachDB on a Kubernetes cluster to familiarize yourself with the necessary Kubernetes terminology and deployment abstractions.
## Performance factors
A number of independent factors affect performance when running CockroachDB on Kubernetes. Most are easiest to change before you create your CockroachDB cluster. If you need to modify a CockroachDB cluster that is already running on Kubernetes, extra care and testing is strongly recommended.
The following sections show how to modify excerpts from the configuration YAML file. You can generate this file locally for modification with the following command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
```
### Version of CockroachDB
Because CockroachDB is under very active development, there are typically substantial performance gains in each release. If you are not experiencing optimal performance and aren't running the latest release, consider upgrading.
### Client workload
Your workload is the single most important factor in database performance. Read through SQL performance best practices and determine whether you can make workload changes to speed up your application.
### Machine size
The size of the machines you're using is not a Kubernetes-specific concern, but is a good place to start if you want more performance. Using machines with more CPU will almost always allow for greater throughput. Because Kubernetes runs a set of processes on every machine in a cluster, it is typically more efficient to use fewer large machines than more small machines. For specific suggestions, refer to Hardware.
### Disk type
CockroachDB makes heavy use of the disks you provide it, so using faster disks is an easy way to improve your cluster's performance. For the best performance, SSDs are strongly recommended.
The Cockroach Labs-provided configuration does not specify disk type, so in most environments Kubernetes will auto-provision disks of the default type. In the common cloud environments (AWS, GCP, Azure) this means you'll get slow disks that aren't optimized for database workloads (e.g., HDDs on GCE, SSDs without provisioned IOPS on AWS).
#### Create a different disk type
Kubernetes exposes the disk types used by its volume provisioner via its [`StorageClass` API object](https://kubernetes.io/docs/concepts/storage/storage-classes/). Each cloud environment has a default `StorageClass`, but you can easily change the default or create a new named class that you can specify later.
To do this, pick a volume provisioner from the list in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/), modify the example YAML file to specify the disk type you want, then run `kubectl create -f {your-storage-class-file}.yaml`. For example, in order to use the `pd-ssd` disk type on Google Compute Engine or Google Kubernetes Engine, you can use a `StorageClass` file like the following:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: {your-ssd-class-name}
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-ssd
```
You may also want to set additional parameters as documented in the list of Kubernetes [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/), such as configuring the `iopsPerGB` if you're creating a `StorageClass` for AWS's `io1` Provisioned IOPS volume type.
You can configure this new disk type to only be used by CockroachDB nodes or as the default for all volumes in your cluster:
#### Configure the disk type used by CockroachDB
To use a new `StorageClass` without making it the default in your cluster, modify your application's YAML file to ask for it. In the CockroachDB configuration, that means adding `storageClassName` to `cockroachdb.crdbCluster.dataStore.volumeClaimTemplates`:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
storageClassName:
```
When running `kubectl create -f` on your modified YAML file, Kubernetes should create volumes using the specified `storageClassName`.
#### Change the default disk type
To make a new `StorageClass` the default for all volumes in your cluster, run the following `kubectl` commands.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl get storageclasses
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
NAME PROVISIONER
ssd kubernetes.io/gce-pd
standard (default) kubernetes.io/gce-pd
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
storageclass "standard" patched
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl patch storageclass ssd -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
storageclass "ssd" patched
```
### Disk size
On some cloud providers, including all GCP disks and the AWS `io1` disk type, the number of IOPS available to a disk is directly correlated to the size of the disk. In such cases, increasing the size of your disks can significantly improve CockroachDB performance, and decrease the risk of filling them up. Before you create your CockroachDB cluster, modify the `cockroachdb.crdbCluster.dataStore.volumeClaimTemplate` in the CockroachDB YAML file to ask for more space. The following example sets this value to 1TB:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 1024Gi
```
Since [GCE disk IOPS scale linearly with disk size](https://cloud.google.com/compute/docs/disks/performance#type_comparison), a 1TiB disk gives 1024 times as many IOPS as a 1GiB disk, which can make a very large difference for write-heavy workloads.
### Local disks
The examples thus far assume the use of auto-provisioned, remotely attached disks. However, local disks typically provide better performance than remotely attached disks. For example, SSD Instance Store Volumes outperform EBS Volumes on AWS, and Local SSDs outperform Persistent Disks on GCE. As of v1.14, Kubernetes supports [local volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local).
When using local disks, consider using replication controls to increase the replication factor of your data from 3 (default) to 5. This is because local disks have a greater chance of experiencing a disk failure than a cloud provider's network-attached disks, which are often replicated underneath the covers.
### Resource requests and limits
When you ask Kubernetes to run a pod, you can tell it to reserve certain amounts of CPU or memory for each container in the pod, or to limit the CPU or memory of each container. Setting resource [requests](#resource-requests) or [limits](#resource-limits) can have different implications, depending on your Kubernetes cluster's resource utilization. For the authoritative information on this topic, refer to the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/).
#### Resource requests
Resource requests reserve a certain amount of CPU or memory for your container. If you add resource requests to your CockroachDB YAML file, Kubernetes will schedule each CockroachDB pod onto a node with sufficient unreserved resources and ensure the pods are guaranteed the reserved resources using the applicable Linux container primitives. If you are running other workloads in your Kubernetes cluster, setting resource requests is strongly recommended to ensure good performance. If you do not set resource requests, CockroachDB could be starved of CPU cycles or OOM-stopped before less important processes.
To determine how many resources are usable on your Kubernetes nodes, you can run:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl describe nodes
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
Name: gke-perf-default-pool-aafee20c-k4t8
[...]
Capacity:
cpu: 4
memory: 15393536Ki
pods: 110
Allocatable:
cpu: 3920m
memory: 12694272Ki
pods: 110
[...]
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system kube-dns-778977457c-kqtlr 260m (6%) 0 (0%) 110Mi (0%) 170Mi (1%)
kube-system kube-proxy-gke-perf-default-pool-aafee20c-k4t8 100m (2%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
360m (9%) 0 (0%) 110Mi (0%) 170Mi (1%)
```
In the output, the `Allocatable` field shows the `cpu` and `memory` resources Kubernetes will provide to pods running on the machine. The difference between the machine's `Capacity` and its `Allocatable` resources is taken up by the operating system and Kubernetes' management processes. In the preceding output, `3920m` stands for 3920 "milli-CPUs", or "thousandths of a CPU".
Kubernetes runs additional pods in the `kube-system` namespace that are part of the cluster infrastructure. If you want to run CockroachDB on every node in your cluster, you must leave room for these processes, which are essential for the Kubernetes cluster's health. If you are only running CockroachDB on a subset of the Kubernetes machines, you can take up all the `Allocatable` space other than what is used by the `kube-system` pods that are on all the Kubernetes machines, such as `kube-proxy` or the `fluentd` logging agent.
On Kubernetes v1.10 or earlier, it is difficult to truly use all of the allocatable space, because you'd have to manually preempt the `kube-system` pods on each machine (by deleting them). When the Kubernetes [Pod Priority](https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/) feature is promoted from alpha to beta and becomes more widely available, you could set the CockroachDB pods to a higher priority, causing the Kubernetes scheduler to preempt and reschedule the `kube-system` pods onto other machines.
Once you've picked out an amount of CPU and memory to reserve for Cockroach, configure the resource requests in your CockroachDB YAML file. They should go underneath the `containers` heading. For example, to use most of the available resources on the machines described above, you'd configure the following lines of your values file:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
resources:
requests:
cpu: 3500m
memory: 12300Mi
```
When you initialize the cluster, check that all the CockroachDB pods are scheduled successfully. If you see any get stuck in the pending state, run `kubectl describe pod {podname}` and check the `Events` for information about why they're still pending. You may need to manually preempt pods on one or more nodes by running `kubectl delete pod` on them to make room for the CockroachDB pods. As long as the pods you delete were created by a higher-level Kubernetes object such as a `Deployment`, they'll be safely recreated on another node.
#### Resource limits
Resource limits cap the resources used by a pod to no more than the provided limit. This makes for more predictable performance because your pods will not be allowed to use any excess capacity on their machines. Pods will not have more resources available to them at some times (e.g., lulls in traffic) than others (e.g., busy periods where the other pods on a machine are also fully utilizing their reserved resources). Resource limits also increase the ["Quality of Service" guaranteed by the Kubernetes runtime](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md) on Kubernetes v1.8 and earlier, making the pods less likely to be preempted when a machine is oversubscribed. Finally, memory limits in particular define the amount of memory the container perceives as available, which is useful when specifying percentage-based values for the CockroachDB `--cache` and `--max-sql-memory` flags, as in our default configuration file.
To set resource limits, in addition to the [resource requests](#resource-requests) described in the preceding section, change the configuration as follows:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
resources:
requests:
cpu: 3500m
memory: 12300Mi
limits:
memory: 12300Mi
```
Pods will be limited to their reserved resources and are unlikely to be preempted, except in rare cases. This will not improve performance on an underutilized Kubernetes cluster, but provides more predictable performance as other workloads run.
While setting memory limits is strongly recommended, [setting CPU limits can hurt tail latencies as currently implemented by Kubernetes](https://github.com/kubernetes/kubernetes/issues/51135). Cockroach Labs recommends not setting CPU limits at all, unless you have explicitly enabled the non-default [Static CPU Management Policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy) when setting up your Kubernetes cluster. In this case, set CPU limits as integers and match memory limits exactly to their corresponding requests.
#### Default resource requests and limits
Even if you do not manually set resource requests, they are likely being applied. In many installations of Kubernetes, a [LimitRange](https://kubernetes.io/docs/tasks/administer-cluster/cpu-default-namespace/) is preconfigured for the `default` namespace that applies a default CPU request of `100m`, or one-tenth of a CPU. You can see this configuration by running the following command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl describe limitranges
```
Experimentally, this does not appear to have a noticeable effect on CockroachDB's performance when a Kubernetes cluster isn't heavily utilized, but do not be surprised if you see CPU requests on your pods that you didn't set.
### Other pods on the same machines as CockroachDB
As described in [Resource requests and limits](#resource-requests-and-limits), your Kubernetes cluster will always run pods other than CockroachDB. You can see them by running:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl get pods --all-namespaces
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system event-exporter-v0.1.7-5c4d9556cf-6v7lf 2/2 Running 0 2m
kube-system fluentd-gcp-v2.0.9-6rvmk 2/2 Running 0 2m
kube-system fluentd-gcp-v2.0.9-m2xgp 2/2 Running 0 2m
kube-system fluentd-gcp-v2.0.9-sfgps 2/2 Running 0 2m
kube-system fluentd-gcp-v2.0.9-szwwn 2/2 Running 0 2m
kube-system heapster-v1.4.3-968544ffd-5tsb8 3/3 Running 0 1m
kube-system kube-dns-778977457c-4s7vv 3/3 Running 0 1m
kube-system kube-dns-778977457c-ls6fq 3/3 Running 0 2m
kube-system kube-dns-autoscaler-7db47cb9b7-x2cc4 1/1 Running 0 2m
kube-system kube-proxy-gke-test-default-pool-828d39a7-dbn0 1/1 Running 0 2m
kube-system kube-proxy-gke-test-default-pool-828d39a7-nr06 1/1 Running 0 2m
kube-system kube-proxy-gke-test-default-pool-828d39a7-rc4m 1/1 Running 0 2m
kube-system kube-proxy-gke-test-default-pool-828d39a7-trd1 1/1 Running 0 2m
kube-system kubernetes-dashboard-768854d6dc-v7ng8 1/1 Running 0 2m
kube-system l7-default-backend-6497bcdb4d-2kbh4 1/1 Running 0 2m
```
These ["cluster add-ons"](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) provide a variety of basic services like managing DNS entries for services within the cluster, powering the Kubernetes dashboard UI, or collecting logs or metrics from all the pods running in the cluster. If you do not like having them take up space in your cluster, you can prevent some of them from running by configuring your Kubernetes cluster appropriately. For example, on GKE, you can create a cluster with the minimal set of add-ons by running:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
gcloud container clusters create --no-enable-cloud-logging --no-enable-cloud-monitoring --addons=""
```
However, some pods like `kube-proxy` and `kube-dns` are required for compliant Kubernetes clusters. Since there will always be pods other than CockroachDB running in your cluster, it's important to understand and account for the effects of having CockroachDB share a machine with other processes. The more processes there are on the same machine as a CockroachDB pod, the slower and less predictable its performance will likely be. To protect against this, it's strongly recommended to specify [resource requests](#resource-requests) on your CockroachDB pods to provide some level of CPU and memory isolation.
Even with resource requests, there can still be contention for shared resources like network I/O or, in [exceptional](https://sysdig.com/blog/container-isolation-gone-wrong/) cases, internal kernel data structures. For these reasons and because of the Kubernetes infrastructure processes running on each machine, CockroachDB running on Kubernetes cannot match the performance of running CockroachDB directly on dedicated machines, although it can get quite close with careful configuration.
If setting appropriate resource requests still isn't getting you the performance you expect, consider using [dedicated nodes](#dedicated-nodes).
#### Client applications on the same machines as CockroachDB
Client applications such as benchmarking applications running on the same machines as CockroachDB are likely to compete for resources. As application load increases, so does the load on CockroachDB processes. The best way to avoid this is to [set resource requests and limits](#resource-requests-and-limits). Alternatively, you can also set [anti-affinity scheduling policies](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) on your client applications:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- loadgen
topologyKey: kubernetes.io/hostname
- weight: 99
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- cockroachdb
topologyKey: kubernetes.io/hostname
```
The preceding configuration will first prefer to put the `loadgen` pods on different nodes from each other, which is important for the fault tolerance of the `loadgen` pods themselves. As a secondary priority, it will attempt to put the pods on nodes that do not already have a running `CockroachDB` pod. This will ensure the best possible balance of fault tolerance and performance for the load generator and CockroachDB cluster.
### Networking
[Kubernetes places significant demands on the underlying network](https://kubernetes.io/docs/concepts/cluster-administration/networking/) in order to provide each pod a routable IP address and isolated Linux network namespace, among other requirements. While the impact is heavily dependent on your Kubernetes cluster's network setup, Docker and Kubernetes' networking abstractions often introduce a performance penalty for high-throughput distributed applications such as CockroachDB.
Experimenting with networking can be a way to eke more performance out of your cluster. You can either replace your cluster's networking solution with a more performant one, or bypass most of the networking overhead by using the host machines' networks directly.
#### Networking solutions
If you aren't using a hosted Kubernetes service, you'll need to choose a [networking solution](https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-achieve-this) when creating a Kubernetes cluster. While Cockroach Labs does not endorse any specific networking solutions, note that your choice can meaningfully impact CockroachDB's performance compared to running it outside of Kubernetes.
### Dedicated nodes
If your Kubernetes cluster uses heterogeneous hardware, you will likely want to ensure that CockroachDB only runs on specific machines. To optimize performance, it can be beneficial to dedicate those machines exclusively to CockroachDB.
For more information, refer to Pod scheduling.
# Common Issues to Monitor
Source: https://docs.cockroachlabs.com/docs/v26.3/common-issues-to-monitor
This page summarizes how to configure and monitor your cluster to prevent issues commonly encountered with:
* [CPU](#cpu)
* [Memory](#memory)
* [Storage and disk I/O](#storage-and-disk-i/o)
## CPU
In our sizing and production guidance, 1 vCPU is considered equivalent to 1 core in the underlying hardware platform.
Issues with CPU most commonly arise when there is insufficient CPU to support the scale of the workload.
### CPU planning
Provision enough CPU to support your operational and workload concurrency requirements:
| Category | Recommendations |
| - | - |
| CPU | - For cluster stability, Cockroach Labs recommends a *minimum* of **8 vCPUs**, and strongly recommends no fewer than **4 vCPUs** per node. In a cluster with too few CPU resources, foreground client workloads will compete with the cluster's background maintenance tasks.
- Cockroach Labs does not extensively test clusters with more than **64 vCPUs** per node. This is the recommended *maximum* threshold.
- Use larger VMs to handle temporary workload spikes and processing hotspots.
- Use connection pooling to manage workload concurrency. The number of **active** connections across all connection pools **should not exceed 4 times the number of vCPUs** in the cluster by a large amount. A connection is "active" when it is actively executing a query. To monitor active connections, use the **Active SQL Statements** graph and `sql.statements.active` metric. For more details, refer to Size connection pools.
- For additional CPU recommendations, refer to Recommended Production Settings.
|
### CPU monitoring
Monitor possible signs of CPU starvation:
| Parameter | Description |
| - | - |
| [Service latency](#service-latency) | The time between when the cluster receives a query and finishes executing the query. |
| [CPU usage](#cpu-usage) | The CPU consumption by the CockroachDB node process. |
| [Workload concurrency](#workload-concurrency) | The number of SQL statements being executed on the cluster at the same time. |
| [LSM health](#lsm-health) | The health of the persistent stores. |
| [Node health](#node-health) | The operational status of the nodes. |
#### Service latency
Degradation in SQL response time is the most common symptom of CPU starvation. It can also be a symptom of [insufficient disk I/O](#storage-and-disk-i/o).
* The **Service Latency: SQL Statements, 99th percentile** and **Service Latency: SQL Statements, 90th percentile** graphs on the SQL dashboard show the time in nanoseconds between when the cluster receives a query and finishes executing the query. This time does not include returning results to the client.
If latencies are consistently high, check for:
* High [CPU usage](#cpu-usage).
* An [I/O bottleneck](#disk-iops).
#### CPU usage
Compaction on the storage layer uses CPU to run concurrent worker threads.
* The **CPU Utilization** graph on the Hardware and Overload dashboards shows the CPU consumption by the CockroachDB process, and excludes other processes on the node.
**Expected values for a healthy cluster**: CPU utilized by CockroachDB should not persistently exceed 80%. Because this metric does not reflect system CPU usage, values above 80% suggest that actual CPU utilization is nearing 100%.
If CPU usage is high, check whether [workload concurrency](#workload-concurrency) is exceeding CPU resources.
By default, the storage engine uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the `COCKROACH_COMPACTION_CONCURRENCY` environment variable. This may help to reshape the LSM more quickly in inverted LSM scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs **strongly recommends** testing your workload against non-default values of this setting.
#### Workload concurrency
The number of concurrent active SQL statements should be proportionate to your provisioned CPU.
* The **SQL Queries Per Second** graph on the Overview and SQL dashboards shows the 10-second moving average of `SELECT`, `UPDATE`, `INSERT`, and `DELETE` statements being executed per second on the cluster or node, as well as `Total queries`. The latest QPS value for the cluster is also displayed with the **Queries per second** counter on the Metrics page.
**Expected values for a healthy cluster**: At any time, the total number of connections actively executing SQL statements should not exceed 4 times the number of vCPUs in the cluster. You can find them in the Active Executions view in the DB Console or Cloud Console. You can find the number of open connections in the DB Console or Cloud Console. For more details on configuring connection pools, see Size connection pools.
If workload concurrency exceeds CPU resources, you will observe:
* High [CPU usage](#cpu-usage).
* Degradation in [SQL response time](#service-latency).
* Over time, an [unhealthy LSM](#lsm-health) and [cluster instability](#node-health).
To prevent issues with workload concurrency, provision sufficient CPU and use connection pooling for the workload.
#### LSM health
Issues at the storage layer, including an inverted LSM and high read amplification, can be observed when compaction falls behind due to insufficient CPU or excessively high recovery and rebalance rates.
* The **IO Overload** graph on the Overload dashboard shows the health of the persistent stores, which are implemented as log-structured merge (LSM) trees. Level 0 is the highest level of the LSM tree and consists of files containing the latest data written to the Pebble storage engine. For more information about LSM levels and how LSMs work, see Log-structured Merge-trees.
**Expected values for a healthy cluster**: An IO Overload value greater than 1.0 generally indicates an overload in the Pebble LSM tree. High values indicate heavy write load that is causing accumulation of files in level 0. These files are not being compacted quickly enough to lower levels, resulting in a misshapen LSM.
An unhealthy LSM can be caused by other factors, including [under-provisioned storage](#storage-and-disk-i/o). To correlate this symptom with CPU starvation, check for high [CPU usage](#cpu-usage) and excessive [workload concurrency](#workload-concurrency).
* The **Read Amplification** graph on the Storage Dashboard shows the average number of disk reads per logical SQL statement, also known as the read amplification factor.
**Expected values for a healthy cluster**: Read amplification factor should be in the single digits. A value exceeding 50 for 1 hour strongly suggests that the LSM tree has an unhealthy shape.
* The `STORAGE` logging channel indicates an unhealthy LSM with the following:
* Frequent `compaction` status messages.
* High-read-amplification warnings, e.g., `sstables (read amplification = 54)`.
If compaction has fallen behind and caused an inverted LSM, throttle your workload concurrency to allow compaction to catch up and restore a healthy LSM shape. The number of **active** connections across all connection pools **should not exceed 4 times the number of vCPUs** in the cluster by a large amount. A connection is "active" when it is actively executing a query. To monitor active connections, use the **Active SQL Statements** graph and `sql.statements.active` metric. If a node is severely impacted, you can start a new node and then decommission the problematic node. By default, the storage engine uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the `COCKROACH_COMPACTION_CONCURRENCY` environment variable. This may help to reshape the LSM more quickly in inverted LSM scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs **strongly recommends** testing your workload against non-default values of this setting.
#### Node health
If [issues at the storage layer](#lsm-health) remain unresolved, affected nodes will eventually become unresponsive.
* The **Node status** on the Cluster Overview page indicates whether nodes are online (`LIVE`) or have crashed (`SUSPECT` or `DEAD`).
* The `/health` endpoint of the Cluster API returns a `500` error when a node is unhealthy.
* A Prometheus alert can notify when a node has been down for 15 minutes or more.
If nodes have shut down, this can also be caused by [insufficient storage capacity](#storage-capacity).
If the cluster becomes unavailable, the DB Console and Cluster API will also become unavailable. You can continue to monitor the cluster via the Prometheus endpoint and logs.
## Memory
CockroachDB is resilient to node crashes. However, frequent node restarts caused by out-of-memory (OOM) crashes can impact cluster stability and performance.
### Memory planning
Provision enough memory and allocate an appropriate portion for data caching:
* Provision at least **4 GiB of RAM per vCPU**.
* Disable Linux memory swapping. Over-allocating memory on production machines can lead to unexpected performance issues when pages have to be read back into memory.
* The default value for `--cache` is 128 MiB. For production deployments, set `--cache` to `25%` or higher. To determine appropriate settings for `--cache` and `--max-sql-memory`, use the following formula: `(2 * --max-sql-memory) + --cache <= 80% of system RAM
`
To help guard against OOM events, CockroachDB sets a soft memory limit using mechanisms in Go. Depending on your hardware and workload, you may not need to manually tune `--max-sql-memory`.
Test the configuration with a reasonable workload before deploying it to production.
On startup, if CockroachDB detects that `--max-sql-memory` or `--cache` are set too aggressively, a warning is logged.
For additional memory recommendations, refer to Recommended Production Settings: Memory and Recommended Production Setting: Cache and SQL memory size.
### Memory monitoring
Monitor memory usage and node behavior for OOM errors:
| Metric or event | Description |
| - | - |
| [Node process restarts](#node-process-restarts) | Nodes restarting after crashes |
| [SQL memory usage](#sql-memory-usage) | The memory allocated to the SQL layer |
| [Database memory usage](#database-memory-usage) | The memory in use by CockroachDB processes |
#### Node process restarts
CockroachDB attempts to restart nodes after they crash. Nodes that frequently restart following an abrupt process exit may point to an underlying memory issue.
* The **Node status** on the Cluster Overview page indicates whether nodes are online (`LIVE`) or have crashed (`SUSPECT` or `DEAD`).
* When deploying on Kubernetes, the `kubectl get pods` output contains a `RESTARTS` column that tracks the number of restarts for each CockroachDB pod.
* The `OPS` logging channel will record a `node_restart` event whenever a node rejoins the cluster after being offline.
* A Prometheus alert can notify when a node has restarted more than once in the last 10 minutes.
##### Verify OOM errors
If you observe nodes frequently restarting, confirm that the crashes are caused by OOM errors:
* Monitor `dmesg` to determine if a node crashed because it ran out of memory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo dmesg | grep -iC 3 "cockroach"
```
The following output indicates the node crashed due to insufficient memory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ host kernel: Out of Memory: Killed process (cockroach).
```
* When deploying on Kubernetes, run `kubectl logs {pod-name}` and look for OOM errors in the log messages.
To prevent OOM crashes, provision sufficient memory. If all CockroachDB machines are provisioned and configured correctly, either run the CockroachDB process on another node with sufficient memory, or reduce the memory allocated to CockroachDB.
If you confirm that nodes are crashing due to OOM errors, also check whether [SQL queries](#sql-memory-usage) may be responsible.
#### SQL memory usage
An untuned SQL query can consume significant resources and impact the performance of other workloads.
* The **SQL Memory** graph on the SQL dashboard shows the current amount of memory in KiB allocated to the SQL layer.
**Expected values for a healthy cluster**: This value should not exceed the `--max-sql-memory` size. A healthy threshold is 75% of allocated `--max-sql-memory`.
* The "active query dump", enabled by default with the `diagnostics.active_query_dumps.enabled` cluster setting, is a record of anonymized active queries that is written to disk when a node is detected to be under memory pressure.
You can use the active query dump to correlate specific queries to OOM crashes. Active query dumps have the filename `activequeryprof.{date-and-time}.csv` and are found in the `heap_profiler` directory in the configured logging directory. They are also included when running `cockroach debug zip`.
* A `SHOW STATEMENTS` statement can identify long-running queries on the cluster that may be consuming excessive memory.
* A `memory budget exceeded` error in the logs indicates that `--max-sql-memory`, the memory allocated to the SQL layer, was exceeded by the operation referenced in the error. For guidance on resolving this issue, see Common Errors.
If you find queries that are consuming too much memory, cancel the queries to free up memory usage. For information on optimizing query performance, see SQL Performance Best Practices.
We **strongly recommend** adding size limits to all indexed columns, which includes columns in primary keys.
Values exceeding 1 MiB can lead to storage layer write amplification and cause significant performance degradation or even crashes due to OOMs (out of memory errors).
To add a size limit using `CREATE TABLE`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE TABLE name (first STRING(100), last STRING(100));
```
To add a size limit using `ALTER TABLE ... ALTER COLUMN`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER TABLE name ALTER first TYPE STRING(99);
```
#### Database memory usage
CockroachDB memory usage includes both accounted memory, such as the amount allocated to `--cache` and `--max-sql-memory`; and unaccounted memory, such as uncollected Go garbage and process overhead.
* The **Memory Usage** graph on the Runtime dashboard shows the total memory in use by CockroachDB processes. The RSS (resident set size) metric represents actual CockroachDB memory usage from the OS/Linux/pod point of view. The Go and CGo metrics represent memory allocation and total usage from a CockroachDB point of view.
**Expected values for a healthy cluster**: Go Allocated will depend on workload but should not exceed `--max-sql-memory` by more than 100%. CGo Allocated should not exceed the `--cache` size and CGo Total should not exceed the `--cache` size by more than 15%.
For more context on acceptable memory usage, see Suspected memory leak.
## Storage and disk I/O
The cluster will underperform if storage is not provisioned or configured correctly. This can lead to further issues such as disk stalls and node shutdown.
### Storage and disk planning
Provision enough storage capacity for CockroachDB data, and configure your volumes to maximize disk I/O:
| Category | Recommendations |
| - | - |
| Storage | - Provision volumes with **320 GiB per vCPU**. - Use dedicated volumes for the CockroachDB store. Do not share the store volume with any other I/O activity. - Determine where CockroachDB log files will be stored: either on the same volume as the main data store or on a separate volume from the main data store. Refer to Storage considerations for file sinks in Logging Best Practices. - See additional storage recommendations in the Recommended Production Settings. |
| Disk I/O | - Disks must be able to achieve **500 IOPS and 30 MB/s per vCPU**. - Do not use LVM in the I/O path. Dynamically resizing CockroachDB store volumes can result in significant performance degradation. Using LVM snapshots in lieu of CockroachDB backup and restore is also not supported. Use multiple stores per node instead. - See additional disk I/O recommendations in the Recommended Production Settings. |
### Storage and disk monitoring
Monitor storage capacity and disk performance:
| Metric or event | Description |
| - | - |
| [Storage capacity](#storage-capacity) | The available and used disk capacity in the CockroachDB store. |
| [Disk IOPS](#disk-iops) | The I/O requests per second. |
| [Node heartbeat latency](#node-heartbeat-latency) | The time between node liveness heartbeats. |
| [Command commit latency](#command-commit-latency) | The speed at which Raft commands are being committed by nodes in the cluster. |
#### Storage capacity
CockroachDB requires disk space in order to accept writes. When a node runs out of disk space, it [shuts down](#node-health) and cannot be restarted until space is freed up.
* The **Capacity** graph on the Overview and Storage dashboards shows the available and used disk capacity in the CockroachDB store.
**Expected values for a healthy cluster**: Used capacity should not persistently exceed 80% of the total capacity.
* A Prometheus alert can notify when a node has less than 15% of free space remaining.
Ensure that you provision sufficient storage. If storage is correctly provisioned and is running low, CockroachDB automatically creates an emergency ballast file that can free up space. For details, see Disks filling up.
For instructions on how to free up disk space as quickly as possible after dropping a table, see How can I free up disk space that was used by a dropped table?
#### Disk IOPS
Insufficient disk I/O can cause [poor SQL performance](#service-latency) and potentially disk stalls.
* The **Disk Ops In Progress** graph on the Hardware dashboard shows the number of disk reads and writes in queue.
**Expected values for a healthy cluster**: This value should be 0 or single-digit values for short periods of time. If the values persist in double digits, you may have an I/O bottleneck.
* The Linux tool `iostat` (part of `sysstat`) can be used to monitor IOPS. In the device status output, `avgqu-sz` corresponds to the **Disk Ops In Progress** metric. If service times persist in double digits on any node, this means that your storage device is saturated and is likely under-provisioned or misconfigured.
With insufficient disk I/O, you may also see:
* Degradation in [SQL response time](#service-latency).
* An [unhealthy LSM](#lsm-health).
#### Node heartbeat latency
* The **Node Heartbeat Latency: 99th percentile** and **Node Heartbeat Latency: 90th percentile** graphs on the Distributed Dashboard show the time elapsed between node heartbeats.
**Expected values for a healthy cluster**: Less than 100ms in addition to the network latency between nodes in the cluster.
For the purposes of Raft replication and determining the leaseholder of a range, node health is no longer determined by heartbeating a single "liveness range"; instead it is determined using Leader leases.
However, node heartbeats of a single range are still used to determine:
* Whether a node is still a member of a cluster (this is used by `cockroach node decommission`).
* Whether a node is dead (in which case its leases will be transferred away).
* How to avoid placing replicas on dead, decommissioning or unhealthy nodes, and to make decisions about lease transfers.
#### Command commit latency
* The **Command Commit Latency: 50th percentile** and **Command Commit Latency: 99th percentile** graphs on the Storage dashboard show how quickly Raft commands are being committed by nodes in the cluster. This is a good signal of I/O load.
**Expected values for a healthy cluster**: On SSDs (strongly recommended), this should be between 1 and 100 milliseconds. On HDDs, this should be no more than 1 second.
## See also
* Recommended Production Settings
* Monitoring and Alerting
* Common Errors and Solutions
* Operational FAQs
* Performance Tuning Recipes
* Troubleshoot Cluster Setup
* Troubleshoot SQL Behavior
* Admission Control
* Metrics
* Alerts Page (CockroachDB Advanced)
# Resource Management
Source: https://docs.cockroachlabs.com/docs/v26.3/configure-cockroachdb-kubernetes
This page explains how to configure Kubernetes cluster resources such as memory, CPU, and storage.
This page is for Kubernetes deployments that are not using the CockroachDB operator. For guidance specific to the CockroachDB operator, read Resource management with the CockroachDB operator.
This page describes a Kubernetes deployment method that is considered "legacy" and will not be supported in a future release of CockroachDB. New CockroachDB deployments on Kubernetes are recommended to use the newer, fully-featured CockroachDB operator that's easier to deploy and supports scaling of multi-region clusters. To migrate an existing deployment to use the CockroachDB operator, read the Helm and Public operator migration guides.
For more information on legacy Kubernetes deployments and support timelines, read the deprecation notice.
These settings override the defaults used when deploying CockroachDB on Kubernetes.
All `kubectl` steps should be performed in the namespace where you installed the operator. By default, this is `cockroach-operator-system`.
If you deployed CockroachDB on Red Hat OpenShift, substitute `kubectl` with `oc` in the following commands.
On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. We recommend that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster.
Run `kubectl describe nodes` to see the available resources on the instances that you have provisioned.
## Memory and CPU
You can set the CPU and memory resources allocated to the CockroachDB container on each pod.
1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, see the Production Checklist.
Specify CPU and memory values in `resources.requests` and `resources.limits` in the Public operator's custom resource, which is used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl apply -f example.yaml
```
Specify CPU and memory values in `resources.requests` and `resources.limits` in the StatefulSet manifest you used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
template:
containers:
- name: cockroachdb
resources:
requests:
cpu: "4"
memory: "16Gi"
limits:
cpu: "4"
memory: "16Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl apply -f {statefulset-manifest}.yaml
```
Specify CPU and memory values in `resources.requests` and `resources.limits` in the custom values file you created when deploying the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
statefulset:
resources:
limits:
cpu: "4"
memory: "16Gi"
requests:
cpu: "4"
memory: "16Gi"
```
Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
```
We recommend using identical values for `resources.requests` and `resources.limits`. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes.
If no resource limits are specified, the pods will be able to consume the maximum available CPUs and memory. However, to avoid overallocating resources when another memory-intensive workload is on the same instance, always set resource requests and limits explicitly.
For more information on how Kubernetes handles resources, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/).
## Cache and SQL memory size
Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.
Our Kubernetes manifests dynamically set cache size and SQL memory size each to 1/4 (the recommended fraction) of the available memory, which depends on the memory request and limit you [specified](#memory-and-cpu) for your configuration. If you want to customize these values, set them explicitly.
Specify `cache` and `maxSQLMemory` in the Public operator's custom resource, which is used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
cache: "4Gi"
maxSQLMemory: "4Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl apply -f example.yaml
```
Specifying these values is equivalent to using the `--cache` and `--max-sql-memory` flags with `cockroach start`.
## Cache and SQL memory size
Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.
Our Kubernetes manifests dynamically set cache size and SQL memory size each to 1/4 (the recommended fraction) of the available memory, which depends on the memory request and limit you [specified](#memory-and-cpu) for your configuration. If you want to customize these values, set them explicitly.
Specify `cache` and `maxSQLMemory` in the custom values file you created when deploying the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
conf:
cache: "4Gi"
max-sql-memory: "4Gi"
```
Apply the custom values to override the default Helm chart [values](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ helm upgrade {release-name} --values {custom-values}.yaml cockroachdb/cockroachdb
```
## Persistent storage
When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/).
The storage capacity of each volume is set in `pvc.spec.resources` in the Public operator's custom resource, which is used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
dataStore:
pvc:
spec:
resources:
limits:
storage: "60Gi"
requests:
storage: "60Gi"
```
The storage capacity of each volume is initially set in `volumeClaimTemplates.spec.resources` in the StatefulSet manifest you used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
volumeClaimTemplates:
spec:
resources:
requests:
storage: 100Gi
```
The storage capacity of each volume is initially set in the Helm chart's [values file](https://github.com/cockroachdb/helm-charts/blob/master/cockroachdb/values.yaml):
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
persistentVolume:
size: 100Gi
```
You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the Production Checklist.
### Expand disk size
If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often beneficial for CockroachDB performance.
Specify a new volume size in `resources.requests` and `resources.limits` in the Public operator's custom resource, which is used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
dataStore:
pvc:
spec:
resources:
limits:
storage: "100Gi"
requests:
storage: "100Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl apply -f example.yaml
```
The Public operator updates the StatefulSet and triggers a rolling restart of the pods with the new storage capacity.
To verify that the storage capacity has been updated, run `kubectl get pvc` to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated.
You can expand certain [types of persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes) (including GCE Persistent Disk and Amazon Elastic Block Store) by editing their persistent volume claims.
These steps assume you followed the tutorial Deploy CockroachDB on Kubernetes.
1. Get the persistent volume claims for the volumes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
datadir-cockroachdb-1 Bound pvc-75e143ca-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
datadir-cockroachdb-2 Bound pvc-75ef409a-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
```
2. In order to expand a persistent volume claim, `AllowVolumeExpansion` in its storage class must be `true`. Examine the storage class:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl describe storageclass standard
```
```
Name: standard
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/gce-pd
Parameters: type=pd-standard
AllowVolumeExpansion: False
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:
```
If necessary, edit the storage class:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl patch storageclass standard -p '{"allowVolumeExpansion": true}'
```
```
storageclass.storage.k8s.io/standard patched
```
3. Edit one of the persistent volume claims to request more space:
The requested `storage` value must be larger than the previous value. You cannot use this method to decrease the disk size.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl patch pvc datadir-cockroachdb-0 -p '{"spec": {"resources": {"requests": {"storage": "200Gi"}}}}'
```
```
persistentvolumeclaim/datadir-cockroachdb-0 patched
```
1. Check the capacity of the persistent volume claim:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc datadir-cockroachdb-0
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 18m
```
If the PVC capacity has not changed, this may be because `AllowVolumeExpansion` was initially set to `false` or because the [volume has a file system](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim) that has to be expanded. You will need to start or restart a pod in order to have it reflect the new capacity.
Running `kubectl get pv` will display the persistent volumes with their *requested* capacity and not their actual capacity. This can be misleading, so it's best to use `kubectl get pvc`.
1. Examine the persistent volume claim. If the volume has a file system, you will see a `FileSystemResizePending` condition with an accompanying message:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl describe pvc datadir-cockroachdb-0
```
```
Waiting for user to (re-)start a pod to finish file system resize of volume on node.
```
2. Delete the corresponding pod to restart it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl delete pod cockroachdb-0
```
The `FileSystemResizePending` condition and message will be removed.
3. View the updated persistent volume claim:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc datadir-cockroachdb-0
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 200Gi RWO standard 20m
```
4. The CockroachDB cluster needs to be expanded one node at a time. Repeat steps 3 - 6 to increase the capacities of the remaining volumes by the same amount.
You can expand certain [types of persistent volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes) (including GCE Persistent Disk and Amazon Elastic Block Store) by editing their persistent volume claims.
These steps assume you followed the tutorial Deploy CockroachDB on Kubernetes.
1. Get the persistent volume claims for the volumes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-my-release-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
datadir-my-release-cockroachdb-1 Bound pvc-75e143ca-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
datadir-my-release-cockroachdb-2 Bound pvc-75ef409a-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 17m
```
2. In order to expand a persistent volume claim, `AllowVolumeExpansion` in its storage class must be `true`. Examine the storage class:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl describe storageclass standard
```
```
Name: standard
IsDefaultClass: Yes
Annotations: storageclass.kubernetes.io/is-default-class=true
Provisioner: kubernetes.io/gce-pd
Parameters: type=pd-standard
AllowVolumeExpansion: False
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:
```
If necessary, edit the storage class:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl patch storageclass standard -p '{"allowVolumeExpansion": true}'
```
```
storageclass.storage.k8s.io/standard patched
```
3. Edit one of the persistent volume claims to request more space:
The requested `storage` value must be larger than the previous value. You cannot use this method to decrease the disk size.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl patch pvc datadir-my-release-cockroachdb-0 -p '{"spec": {"resources": {"requests": {"storage": "200Gi"}}}}'
```
```
persistentvolumeclaim/datadir-my-release-cockroachdb-0 patched
```
4. Check the capacity of the persistent volume claim:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc datadir-my-release-cockroachdb-0
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-my-release-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 100Gi RWO standard 18m
```
If the PVC capacity has not changed, this may be because `AllowVolumeExpansion` was initially set to `false` or because the [volume has a file system](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim) that has to be expanded. You will need to start or restart a pod in order to have it reflect the new capacity.
Running `kubectl get pv` will display the persistent volumes with their *requested* capacity and not their actual capacity. This can be misleading, so it's best to use `kubectl get pvc`.
5. Examine the persistent volume claim. If the volume has a file system, you will see a `FileSystemResizePending` condition with an accompanying message:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl describe pvc datadir-my-release-cockroachdb-0
```
```
Waiting for user to (re-)start a pod to finish file system resize of volume on node.
```
6. Delete the corresponding pod to restart it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl delete pod my-release-cockroachdb-0
```
The `FileSystemResizePending` condition and message will be removed.
7. View the updated persistent volume claim:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl get pvc datadir-my-release-cockroachdb-0
```
```
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-my-release-cockroachdb-0 Bound pvc-75dadd4c-01a1-11ea-b065-42010a8e00cb 200Gi RWO standard 20m
```
8. The CockroachDB cluster needs to be expanded one node at a time. Repeat steps 3 - 6 to increase the capacities of the remaining volumes by the same amount.
## Network ports
The Public operator separates network traffic into three ports:
| Protocol | Default | Description | Custom Resource Field |
| - | - | - | - |
| gRPC | 26258 | Used for node connections | `grpcPort` |
| HTTP | 8080 | Used to access the DB Console | `httpPort` |
| SQL | 26257 | Used for SQL shell access | `sqlPort` |
Specify alternate port numbers in the Public operator's custom resource (for example, to match the default port `5432` on PostgreSQL):
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
sqlPort: 5432
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl apply -f example.yaml
```
The Public operator updates the StatefulSet and triggers a rolling restart of the pods with the new port settings.
Currently, only the pods are updated with new ports. To connect to the cluster, you need to ensure that the `public` service is also updated to use the new port. You can do this by deleting the service with `kubectl delete service {cluster-name}-public`. When service is recreated by the operator, it will use the new port. This is a known limitation.
## Ingress
You can configure an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) object to expose an internal HTTP or SQL [`ClusterIP` service](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) through a hostname.
In order to use the Ingress resource, your cluster must be running an [Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) for load balancing. This is **not** handled by the Public operator and must be deployed separately.
Specify Ingress objects in `ingress.ui` (HTTP) or `ingress.sql` (SQL) in the Public operator's custom resource, which is used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
spec:
ingress:
ui:
ingressClassName: nginx
annotations:
key: value
host: ui.example.com
sql:
ingressClassName: nginx
annotations:
key: value
host: sql.example.com
```
* `ingressClassName` specifies the [`IngressClass`](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class) of the Ingress controller. This example uses the [nginx](https://kubernetes.github.io/ingress-nginx/) controller.
* The `host` must be made publicly accessible. For example, create a route in [Amazon Route 53](https://aws.amazon.com/route53/), or add an entry to `/etc/hosts` that maps the IP address of the Ingress controller to the hostname.
Multiple hosts can be mapped to the same Ingress controller IP.
* TCP connections for SQL clients must be enabled for the Ingress controller. For an example, see the [nginx documentation](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/).
Changing the SQL Ingress `host` on a running deployment will cause a rolling restart of the cluster, due to new node certificates being generated for the SQL host.
The [custom resource definition](https://github.com/cockroachdb/cockroach-operator/blob/v2.18.3/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml) details the fields supported by the operator.
# Resource Management with the CockroachDB Operator
Source: https://docs.cockroachlabs.com/docs/v26.3/configure-cockroachdb-operator
This page explains how to configure Kubernetes cluster resources such as memory, CPU, and storage.
On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. Cockroach Labs recommends that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster.
Run `kubectl describe nodes` to see the available resources on the instances that you have provisioned.
## Memory and CPU
You can set the CPU and memory resources allocated to the CockroachDB container on each pod.
1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the Production Checklist.
Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
podTemplate:
spec:
resources:
limits:
cpu: 4000m
memory: 16Gi
requests:
cpu: 4000m
memory: 16gi
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
Cockroach Labs recommends using identical values for `resources.requests` and `resources.limits`. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes.
If no resource limits are specified, the pods will be able to consume the maximum available CPUs and memory. However, to avoid overallocating resources when another memory-intensive workload is on the same instance, always set resource requests and limits explicitly.
For more information on how Kubernetes handles resources, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/).
### Cache and SQL memory size
Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.
The CockroachDB operator dynamically sets cache size and SQL memory size each to 25% (the recommended percentage) of the available memory, which depends on the memory request and limit you [specified](#memory-and-cpu) for your configuration. These values can be modified by adding the `cache` or `max-sql-memory` fields to `cockroachdb.crdbCluster.flags`, which is equivalent to appending `--cache` or `--max-sql-memory` as cockroach start flags.
## Persistent storage
When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/).
The storage capacity of each volume is set in `cockroachdb.crdbCluster.dataStore.volumeClaimTemplate.spec.resources` in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
spec:
resources:
requests:
storage: "10Gi"
```
You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the Production Checklist.
### Expand disk size
If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often beneficial for CockroachDB performance.
The volume size should only adjusted on disk types that can dynamically scale up, such as Amazon EBS volumes. Adjusting the volume size on non-dynamically scaling disks is not recommended, and instead you should horizontally scale the number of disks used.
Specify a new volume size in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
dataStore:
volumeClaimTemplate:
spec:
resources:
requests:
storage: "100Gi"
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new storage capacity.
To verify that the storage capacity has been updated, run `kubectl get pvc` to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated.
## Network ports
The CockroachDB operator separates network traffic into three ports:
| Protocol | Default Port | Description | Custom Resource Field |
| - | - | - | - |
| gRPC | 26258 | Used for node connections | service.ports.grpc |
| HTTP | 8080 | Used to access the DB Console | service.ports.http |
| SQL | 26257 | Used for SQL shell access | service.ports.sql |
Specify alternate port numbers in `cockroachdb.crdbCluster.service.ports` of the 's custom resource (for example, to match the default port `5432` on PostgreSQL):
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
service:
ports:
sql: 5432
```
Apply the new settings to the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
# Generate values.yaml only if it does not already exist locally, then edit it before upgrading
[ -f values.yaml ] || helm show values cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} > values.yaml
# Apply the edited values.yaml
helm upgrade $CRDBCLUSTER cockroachdb-v2/cockroachdb-chart --version {CHART_VERSION} -f values.yaml -n $NAMESPACE
```
The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new port settings.
## Ingress
You can configure an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) object to expose an internal HTTP or SQL [`ClusterIP` service](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) through a hostname.
In order to use the Ingress resource, your cluster must be running an [Ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) for load balancing. This is **not** handled by the CockroachDB operator and must be deployed separately.
Specify Ingress objects in `cockroachdb.crdbCluster.service.ingress`. Set `ingress.enabled` to `true` and specify `ingress.ui` (HTTP) or `ingress.sql` (SQL) in the values file used to deploy the cluster:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroachdb:
crdbCluster:
service:
ingress:
enabled: true
ui:
ingressClassName: nginx
annotations:
key: value
host: ui.example.com
sql:
ingressClassName: nginx
annotations:
key: value
host: sql.example.com
```
* `ingressClassName` specifies the [`IngressClass`](https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class) of the Ingress controller. This example uses the [nginx](https://kubernetes.github.io/ingress-nginx/) controller.
* The `host` must be made publicly accessible. For example, create a route in [Amazon Route 53](https://aws.amazon.com/route53/), or add an entry to `/etc/hosts` that maps the IP address of the Ingress controller to the hostname.
Multiple hosts can be mapped to the same Ingress controller IP.
* TCP connections for SQL clients must be enabled for the Ingress controller. For an example, see the [nginx documentation](https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/).
Changing the SQL Ingress `host` on a running deployment will cause a rolling restart of the cluster, due to new node certificates being generated for the SQL host.
# Connect to a CockroachDB Cluster
Source: https://docs.cockroachlabs.com/docs/v26.3/connect-to-the-database
This page documents the required connection configuration for fully-supported third-party tools.
For a list of all supported cluster connection parameters, see the `cockroach` Connection Parameters.
For a list of community-supported third-party tools, see Third-Party Tools Supported by the Community. CockroachDB supports both native drivers and the PostgreSQL wire protocol. Most client drivers and ORM frameworks connect to CockroachDB like they connect to PostgreSQL.
## Step 1. Select your deployment
To connect to a CockroachDB Cloud cluster, you need a general connection string or connection parameters, which include the username, host, database, and port. To find these, open the **Connect** dialog for your cluster in the [CockroachDB Cloud Console](https://cockroachlabs.cloud) and select either **General connection string** or **Parameters only** as the option.
To connect to a CockroachDB Cloud cluster, you need a general connection string or connection parameters, which include the username, host, database, and port. To find these, open the **Connect** dialog for your cluster in the [CockroachDB Cloud Console](https://cockroachlabs.cloud) and select either **General connection string** or **Parameters only** as the option.
To connect to a CockroachDB self-hosted cluster, you need the general connection string or connection parameters for your cluster.
The connection strings and parameters for your cluster are output when you start the cluster.
## Step 2. Select your language and driver
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [node-postgres](https://node-postgres.com), create a new [`Client`](https://node-postgres.com/apis/client) object with a connection string.
For example:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
const { Client } = require('pg')
const client = new Client(process.env.DATABASE_URL)
client.connect()
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with node-postgres, see the [official node-postgres documentation](https://node-postgres.com/features/connecting).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Sequelize](https://sequelize.org), create a `Sequelize` object with the [CockroachDB Sequelize adapter](https://github.com/cockroachdb/sequelize-cockroachdb).
For example:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
const Sequelize = require("sequelize-cockroachdb");
const connectionString = process.env.DATABASE_URL
const sequelize = new Sequelize(connectionString)
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Sequelize, you must install the [CockroachDB Sequelize adapter](https://github.com/cockroachdb/sequelize-cockroachdb).
For more information about connecting with Sequelize, see the [official Sequelize documentation](https://sequelize.org/docs/v6/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object (`ssl: true`).
```
postgresql://{username}:{password}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ssl: {
ca: process.env.CA_CERT
}
```
Where `CA_CERT` is an environment variable set to the root certificate downloaded from the CockroachDB Cloud Console.
```
postgresql://{username}@{host}:{port}/{database}
```
TypeORM configures SSL separately in the DataSource object:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ssl: {
ca: process.env.CA_CERT,
key: process.env.CLIENT_KEY,
cert: process.env.CLIENT_CERT
}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [TypeORM](https://typeorm.io), update your project's [`DataSource`](https://typeorm.io/data-source) with the required connection properties.
For example, suppose that you are defining the `DataSource` for your application in a file named `datasource.ts`:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: true,
...
});
```
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: {
ca: process.env.CA_CERT
},
...
});
```
Where `CA_CERT` is an environment variable set to the root certificate downloaded from the CockroachDB Cloud Console.
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export const AppDataSource = new DataSource({
type: "cockroachdb",
url: process.env.DATABASE_URL,
ssl: {
ca: process.env.CA_CERT,
key: process.env.CLIENT_KEY,
cert: process.env.CLIENT_CERT
},
...
});
```
Where:
* `CA_CERT` is an environment variable set to the root certificate. You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert.
* `CLIENT_KEY` is an environment variable set to the client key for the user connecting to the cluster. You can generate this key with `cockroach cert create-client`.
* `CLIENT_CERT` is an environment variable set to the client certificate for the user connecting to the cluster. You can generate this certificate with `cockroach cert create-client`.
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
You can then import the `AppDataSource` into any file in your project and call `AppDataSource.initialize()` to connect to CockroachDB:
```ts theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
AppDataSource.initialize()
.then(async () => {
// Execute operations
});
```
For more information about connecting with TypeORM, see the [official TypeORM documentation](https://typeorm.io/#/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Prisma](https://prisma.io/), set the `url` field of the `datasource` block in your Prisma schema to your database connection URL:
```js theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "cockroachdb"
url = env("DATABASE_URL")
}
model Widget {
id String @id @default(dbgenerated("gen_random_uuid()")) @db.Uuid
}
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Prisma, see the [official Prisma documentation](https://www.prisma.io/cockroachdb).
To connect to a CockroachDB Basic or Standard cluster from a Python application, you must have a valid CA certificate located at `~/.postgresql/root.crt`.
For instructions on downloading a CA certificate from the CockroachDB Cloud Console, see [Connect to a CockroachDB Basic Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-a-basic-cluster) or [Connect to a CockroachDB Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-your-cluster).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Psycopg2](https://www.psycopg.org), pass a connection string to the [`psycopg2.connect` function](https://www.psycopg.org/docs/connection.html).
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
import psycopg2
import os
conn = psycopg2.connect(os.environ['DATABASE_URL'])
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Psycopg, see the [official Psycopg documentation](https://www.psycopg.org/docs).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Psycopg3](https://www.psycopg.org), pass a connection string to the [`psycopg.connect` function](https://www.psycopg.org/psycopg3/docs/basic/usage.html).
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
import psycopg
import os
with psycopg.connect(os.environ['DATABASE_URL']) as conn:
# application logic here
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with Psycopg, see the [official Psycopg documentation](https://www.psycopg.org/psycopg3/docs/basic/index.html).
### Connection string format
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
cockroachdb://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [SQLAlchemy](http://docs.sqlalchemy.org/), [create an `Engine` object](https://docs.sqlalchemy.org/core/engines.html) by passing the connection string to the `create_engine` function.
For example:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
from sqlalchemy import create_engine
import os
engine = create_engine(os.environ['DATABASE_URL'])
engine.connect()
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with SQLAlchemy, you must install the [CockroachDB SQLAlchemy adapter](https://github.com/cockroachdb/sqlalchemy-cockroachdb).
For more information about connecting with SQLAlchemy, see the [official SQLAlchemy documentation](https://docs.sqlalchemy.org/core/engines_connections.html).
In order for SQLAlchemy to use the CockroachDB adapter, the connection string must begin with `cockroachdb://`. You can use the following code to modify the general connection string, which begins with `postgresql://`, to the format that works with SQLAlchemy and the CockroachDB adapter:
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
engine = create_engine(os.environ['DATABASE_URL'].replace("postgresql://", "cockroachdb://"))
engine.connect()
```
### Connection string format
Django does not use a connection string. Instead, you configure connection parameters in the `DATABASES` property in your project's `settings.py` file.
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Connect to the cluster
To connect to CockroachDB from a [Django](https://www.djangoproject.com) application, update the `DATABASES` property in the project's `settings.py` file.
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'PASSWORD': '{password}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full'
},
},
}
...
```
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'PASSWORD': '{password}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full',
'sslrootcert': os.path.expandvars('{root-cert}'),
},
},
}
...
```
```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
## settings.py
...
DATABASES = {
'default': {
'ENGINE': 'django_cockroachdb',
'NAME': '{database}',
'USER': '{username}',
'HOST': '{host}',
'PORT': '{port}',
'OPTIONS': {
'sslmode': 'verify-full',
'sslrootcert': os.path.expandvars('{root-cert}'),
'sslcert': os.path.expandvars('{client-cert}'),
'sslkey': os.path.expandvars('{client-key}'),
},
},
}
...
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
To connect to CockroachDB with Django, you must install the [CockroachDB Django adapter](https://github.com/cockroachdb/django-cockroachdb).
For more information about connecting with Django, see the [official Django documentation](https://docs.djangoproject.com/en/4.0/).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [pgx](https://github.com/jackc/pgx), use the `pgx.Connect` function.
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"context"
"log"
"os"
"github.com/jackc/pgx/v4"
)
func main() {
conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
defer conn.Close(context.Background())
}
```
For more information about connecting with pgx, see the [official pgx documentation](https://pkg.go.dev/github.com/jackc/pgx).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [pq](https://godoc.org/github.com/lib/pq), use the [`sql.Open` function](https://go.dev/doc/tutorial/database-access).
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"database/sql"
"log"
"os"
_ "github.com/lib/pq"
)
func main() {
db, err := sql.Open("postgres", os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
defer db.Close()
}
```
For more information about connecting with pq, see the [official pq documentation](https://pkg.go.dev/github.com/lib/pq).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [GORM](http://gorm.io), use the `gorm.Open` function, with the GORM `postgres` driver.
For example:
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
package main
import (
"log"
"os"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
func main() {
db, err := gorm.Open(postgres.Open(os.Getenv("DATABASE_URL")), &gorm.Config{})
if err != nil {
log.Fatal(err)
}
}
```
For more information about connecting with GORM, see the [official GORM documentation](https://gorm.io/docs).
### Connection string format
```
jdbc:postgresql://{host}:{port}/{database}?password={password}&sslmode=verify-full&user={username}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&password={password}&sslmode=verify-full&sslrootcert={root-cert}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the [PKCS#8](https://tools.ietf.org/html/rfc5208)-formatted client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client --also-generate-pkcs8-key`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `JDBC_DATABASE_URL` environment variable to your JDBC connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export JDBC_DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:JDBC_DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with the [JDBC](https://jdbc.postgresql.org) driver, create a `DataSource` object ([`PGSimpleDataSource` or `PGPoolingDataSource`](https://jdbc.postgresql.org/documentation/datasource/#applications-datasource)), and set the connection string with the `setUrl` class method.
For example:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
PGSimpleDataSource ds = new PGSimpleDataSource();
ds.setUrl(System.getenv("JDBC_DATABASE_URL"));
```
Where `JDBC_DATABASE_URL` is an environment variable set to a valid JDBC-compatible CockroachDB connection string.
For more information about connecting with JDBC, see the [official JDBC documentation](https://jdbc.postgresql.org/documentation/).
### Connection string format
```
jdbc:postgresql://{host}:{port}/{database}?password={password}&sslmode=verify-full&user={username}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&password={password}&sslmode=verify-full&sslrootcert={root-cert}
```
```
jdbc:postgresql://{host}:{port}/{database}?user={username}&sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The [URL-encoded](https://wikipedia.org/wiki/Percent-encoding) path to the [PKCS#8](https://tools.ietf.org/html/rfc5208)-formatted client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client --also-generate-pkcs8-key`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `JDBC_DATABASE_URL` environment variable to your JDBC connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export JDBC_DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$env:JDBC_DATABASE_URL = "{connection string}"
```
### Connect to the cluster
To connect to CockroachDB with [Hibernate](https://hibernate.org/orm) ORM, set the object's `hibernate.connection.url` property to a valid CockroachDB connection string.
For example, if you are bootstrapping your application with a [`ServiceRegistry`](https://docs.jboss.org/hibernate/orm/current/userguide/html_single/Hibernate_User_Guide.html#bootstrap-native-registry) object, first read from the Hibernate configuration file, and then set the `hibernate.connection.url` with the `applySetting` class method:
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
StandardServiceRegistry standardRegistry = new StandardServiceRegistryBuilder()
.configure( "hibernate.cfg.xml" ).applySetting("hibernate.connection.url", System.getenv("JDBC_DATABASE_URL"))
.build();
Metadata metadata = new MetadataSources( standardRegistry )
.getMetadataBuilder()
.build();
SessionFactory sessionFactory = metadata.getSessionFactoryBuilder()
.build();
```
Where `JDBC_DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Hibernate, you must specify the CockroachDB Hibernate dialect in your `hibernate.cfg.xml` configuration file.
For more information about connecting with Hibernate, see the [official Hibernate documentation](https://hibernate.org/orm/documentation).
To connect to a CockroachDB Basic or Standard cluster from a Ruby application, you must have a valid CA certificate located at `~/.postgresql/root.crt`.
For instructions on downloading a CA certificate from the CockroachDB Cloud Console, see [Connect to a CockroachDB Basic Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-a-basic-cluster) or [Connect to a CockroachDB Standard Cluster](https://www.cockroachlabs.com/docs/cockroachcloud/connect-to-your-cluster).
### Connection string format
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
postgresql://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with the [Ruby pg](https://rubygems.org/gems/pg) driver, use the `PG.connect` function.
For example:
```ruby theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
#!/usr/bin/env ruby
require 'pg'
conn = PG.connect(ENV['DATABASE_URL'])
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
For more information about connecting with pg, see the [official pg documentation](https://www.rubydoc.info/gems/pg).
### Connection string format
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full
```
```
cockroachdb://{username}:{password}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}
```
```
cockroachdb://{username}@{host}:{port}/{database}?sslmode=verify-full&sslrootcert={root-cert}&sslcert={client-cert}&sslkey={client-key}
```
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see Start a Local Cluster (Secure).
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{password}` | The password for the SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate that you downloaded from the CockroachDB Cloud Console. |
| Parameter | Description |
| - | - |
| `{username}` | The SQL user connecting to the cluster. |
| `{host}` | The host on which the CockroachDB node is running. |
| `{port}` | The port at which the CockroachDB node is listening. |
| `{database}` | The name of the (existing) database. |
| `{root-cert}` | The path to the root certificate.
You can generate this certificate with `cockroach cert create-ca`, or you can use a custom CA cert. |
| `{client-cert}` | The path to the client certificate for the user connecting to the cluster.
You can generate this certificate with `cockroach cert create-client`. |
| `{client-key}` | The path to the client key for the user connecting to the cluster.
You can generate this key with `cockroach cert create-client`. |
The connection information shown on this page uses client certificate and key authentication to connect to a secure, CockroachDB self-hosted cluster.
To connect to a CockroachDB self-hosted cluster with client certificate and key authentication, you must first generate server and client certificates.
For instructions on starting a secure cluster, see [Start a Local Cluster (Secure)](/docs/v26.2/secure-a-cluster).
### Set environment variable
Set a `DATABASE_URL` environment variable to your connection string:
```shell Mac or Linux theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
export DATABASE_URL="{connection string}"
```
```shell Windows theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
set DATABASE_URL={connection string}
```
### Connect to the cluster
To connect to CockroachDB with [Active Record](https://github.com/rails/rails/tree/main/activerecord) from a Rails app, update the database configuration in `config/database.yml`:
```yaml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
default: &default
adapter: cockroachdb
url: <%= ENV['DATABASE_URL'] %>
...
```
Where `DATABASE_URL` is an environment variable set to a valid CockroachDB connection string.
To connect to CockroachDB with Active Record, you must install the [Active Record CockroachDB adapter](https://rubygems.org/gems/activerecord-cockroachdb-adapter).
For more information about connecting with Active Record, see the [official Active Record documentation](https://guides.rubyonrails.org/active_record_querying.html).
## See also
* Install a Driver or ORM Framework
* Connection Pooling
* `cockroach` Connection Parameters
# Client Connection Parameters
Source: https://docs.cockroachlabs.com/docs/v26.3/connection-parameters
Client applications, including `cockroach` client
commands, work by establishing a network
connection to a CockroachDB cluster. The client connection parameters
determine which CockroachDB cluster they connect to, and how to
establish this network connection.
## Supported connection parameters
Most client apps, including `cockroach` client commands, determine
which CockroachDB server to connect to using a [PostgreSQL connection
URL](#connect-using-a-url). When using a URL, a client can also
specify additional SQL-level parameters. This mode provides the most
configuration flexibility.
In addition, all `cockroach` client commands also accept [discrete
connection parameters](#connect-using-discrete-parameters) that can
specify the connection parameters separately from a URL.
## When to use a URL and when to use discrete parameters
Specifying client parameters using a URL may be more convenient during
experimentation, as it facilitates copy-pasting the connection
parameters (the URL) between different tools: the output of `cockroach
start`, other `cockroach` commands, GUI database visualizer,
programming tools, etc.
Discrete parameters may be more convenient in automation, where the
components of the configuration are filled in separately from
different variables in a script or a service manager.
## Connect using a URL
A connection URL has the following format:
```
postgres://SQL user that will own the client session. | ✗ |
| `
Note that passwords with special characters must be passed as [query string parameters](#additional-connection-parameters) (e.g., `postgres://maxroach@localhost:26257/movr?password=
Find more detail about how CockroachDB handles passwords. | ✗ |
| `current database. Defaults to `defaultdb` when using `cockroach` client commands. Drivers and ORMs may have different defaults. | ✗ |
| `
For `cockroach` commands that accept a URL, you can specify the URL with the command-line flag `--url`.
If `--url` is not specified but
the environment variable `COCKROACH_URL` is defined, the environment
variable is used. Otherwise, the `cockroach` command will use
[discrete connection parameters](#connect-using-discrete-parameters)
as described below.
The ``cockroach`
commands other than `cockroach
sql`. A warning
is currently printed if it is mistakenly specified, and
future versions of CockroachDB may return an error in that case.
### Additional connection parameters
The following additional parameters can be passed after the `?` character in the URL. After the first parameter is specified, any additional parameters must be separated by an ampersand (`&`).
| Parameter | Description | Default value |
| - | - | - |
| `application_name` | An initial value for the `application_name` session variable.
Note: For Java JDBC, use `ApplicationName`. | Empty string. |
| `sslmode` | Which type of secure connection to use: `disable`, `allow`, `prefer`, `require`, `verify-ca` or `verify-full`. See [Secure Connections With URLs](#secure-connections-with-urls) for details. | `disable` |
| `sslrootcert` | Path to the CA certificate, when `sslmode` is not `disable`. | Empty string. |
| `sslcert` | Path to the client certificate, when `sslmode` is not `disable`. | Empty string. |
| `sslkey` | Path to the client private key, when `sslmode` is not `disable`. | Empty string. |
| `password` | The SQL user's password. It is not recommended to pass the password in the URL directly.
Note that passwords with special characters must be passed as [query string parameters](#additional-connection-parameters) (e.g., `postgres://maxroach@localhost:26257/movr?password=`sql.defaults.results_buffer.size` cluster setting. Can be set as a top-level query parameter or as an `options` parameter. | |
#### Supported `options` parameters
CockroachDB supports the following `options` parameters. After the first `options` parameter is specified, any additional parameters in the same connection string must be separated by a space.
| Parameter | Description |
| - | - |
| `--cluster==session variable for the SQL session. |
| `results_buffer_size` | Default size of the buffer that accumulates results for a statement or a batch of statements before they are sent to the client. Can also be set using the `sql.defaults.results_buffer.size` cluster setting. Can be set as a top-level query parameter or as an `options` parameter. |
Note that some drivers require certain characters to be properly encoded in URL connection strings. For example, spaces in [a JDBC connection string](https://jdbc.postgresql.org/documentation/use/#connection-parameters) must be specified as `%20`.
### Secure connections with URLs
The following values are supported for `sslmode`, although only the first and the last are recommended for use.
| Parameter | Description | Recommended for use |
| - | - | - |
| `sslmode=disable` | Do not use an encrypted, secure connection at all. | Use during development. |
| `sslmode=allow` | Enable a secure connection only if the server requires it.
**Not supported in all clients.** | |
| `sslmode=prefer` | Try to establish a secure connection, but accept an insecure connection if the server does not support secure connections.
**Not supported in all clients.** | |
| `sslmode=require` | Force a secure connection. An error occurs if the secure connection cannot be established. | |
| `sslmode=verify-ca` | Force a secure connection and verify that the server certificate is signed by a known CA. | |
| `sslmode=verify-full` | Force a secure connection, verify that the server certificate is signed by a known CA, and verify that the server address matches that specified in the certificate. | Use for secure deployments. |
Some client drivers and the `cockroach` commands do not support
`sslmode=allow` and `sslmode=prefer`. Check the documentation of your
SQL driver to determine whether these options are supported.
### Convert a URL for different drivers
The subcommand `cockroach convert-url` converts a connection URL, such as those printed out by `cockroach start` or included in the online documentation, to the syntax recognized by various client drivers.
The `cockroach encode-uri` command has been deprecated and merged into `cockroach convert-url`. Use `cockroach convert-url --format crdb` instead. The `encode-uri` command will be removed in a future release.
#### Basic usage
By default, `convert-url` outputs the URL in multiple formats:
```
$ ./cockroach convert-url --url "postgres://foo/bar"
```
```
# Connection URL for libpq (C/C++), psycopg (Python), lib/pq & pgx (Go), node-postgres (JS) and most pq-compatible drivers:
postgresql://root@foo:26257/bar
# Connection DSN (Data Source Name) for Postgres drivers that accept DSNs - most drivers and also ODBC:
database=bar user=root host=foo port=26257
# Connection URL for JDBC (Java and JVM-based languages):
jdbc:postgresql://foo:26257/bar?user=root
# Direct URL to CockroachDB:
postgresql://root@foo:26257/bar
```
#### Specify output format
Use the `--format` flag to output a specific format:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgres://foo/bar" --format jdbc
```
```
jdbc:postgresql://foo:26257/bar?user=root
```
Supported formats:
| Format | Description |
| - | - |
| `pq` | Connection URL for libpq (C/C++), psycopg (Python), lib/pq & pgx (Go), node-postgres (JS), and most pq-compatible drivers |
| `dsn` | Data Source Name for Postgres drivers that accept DSNs and ODBC |
| `jdbc` | Connection URL for JDBC (Java and JVM-based languages) |
| `crdb` | CockroachDB-specific URL format with inlined certificates for Physical Cluster Replication and Logical Data Replication |
#### Generate URLs for replication
To generate a connection URL for Physical Cluster Replication or Logical Data Replication, use the `--format crdb` flag with the `--inline` flag to inline certificates directly into the URL:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgresql://{user}:{password}@{node IP}:26257" --format crdb --ca-cert {path to CA certificate} --inline
```
The `--inline` flag automatically sets `--format crdb`, so you can also use:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach convert-url --url "postgresql://{user}:{password}@{node IP}:26257" --ca-cert {path to CA certificate} --inline
```
#### Additional flags
| Flag | Description |
| - | - |
| `--url` | Connection URL to convert (required) |
| `--format` | Output format: `pq`, `dsn`, `jdbc`, or `crdb` |
| `--inline` | Inline certificates into the URL (automatically sets `--format crdb`) |
| `--database` | Database name |
| `--user` | Username |
| `--password` | Password |
| `--cluster` | Cluster name (sets `options=-ccluster=Create Security Certificates and
Rotate Certificates.
### Example URI for a Unix domain socket
The following URI is suitable to connect to a CockroachDB cluster listening for Unix domain socket connections at `/path/to/client`:
```
postgres://root@?host=/path/to/client&port=26257
```
This specifies a connection for the `root` user to an insecure cluster listening for a socket connection (e.g., a cluster started with the `--socket-dir` flag) at `/path/to/client`, and on port 26257.
### Example URI for connecting to a database with a user-defined schema
The following URI connects to a CockroachDB cluster with a user-defined schema named `max_schema` in the `movr` database using the [`options` parameter](#supported-options-parameters).
```
postgres://maxroach@db.example.com:26257/movr?sslmode=verify-full&options%3D-c%20search_path%3Dmax_schema
```
The `options=-c search_path=max_schema` parameter is URL-encoded in the example above.
## Connect using discrete parameters
Most `cockroach` commands accept connection
parameters as separate, discrete command-line flags, in addition (or
in replacement) to `--url` which [specifies all parameters as a
URL](#connect-using-a-url).
For each command-line flag that directs a connection parameter,
CockroachDB also recognizes an environment variable. The environment
variable is used when the command-line flag is not specified.
| Flag | Description |
| - | - |
| `--url` | A connection URL to use instead of the other arguments. To convert a connection URL to the syntax that works with your client driver, run `cockroach convert-url`.
**Env Variable:** `COCKROACH_URL`
**Default:** no URL |
| `--host` | The server host and port number to connect to. This can be the address of any node in the cluster.
**Env Variable:** `COCKROACH_HOST`
**Default:** `localhost:26257` |
| `--port`
`-p` | The server port to connect to. Note: The port number can also be specified via `--host`.
**Env Variable:** `COCKROACH_PORT`
**Default:** `26257` |
| `--user`
`-u` | The SQL user that will own the client session.
**Env Variable:** `COCKROACH_USER`
**Default:** `root` |
| `--insecure` | Use an insecure connection.
**Env Variable:** `COCKROACH_INSECURE`
**Default:** `false` |
| `--cert-principal-map` | A comma-separated list of `certificate directory containing the CA and client certificates and client key.
**Env Variable:** `COCKROACH_CERTS_DIR`
**Default:** `${HOME}/.cockroach-certs/` |
### Example command-line flags for an insecure connection
The following command-line flags establish an insecure connection:
```
--user=root \
--host=
When using discrete connection parameters, the file names of the CA
and client certificates and client key are derived automatically from
the value of `--certs-dir`.
## Using both URL and client parameters
Most `cockroach` commands accept both a URL and client parameters.
The information contained therein is combined in the order it appears
in the command line.
This combination is useful so that discrete command-line flags can
override settings not otherwise set in the URL.
### Example override of the current database
The `cockroach start` command prints out the following connection URL, which connects to the `defaultdb` database:
```
postgres://root@servername:26257/?sslmode=disable
```
To specify `mydb` as the current database using `cockroach sql`, run the following command:
```
cockroach sql \
--url "postgres://root@servername:26257/?sslmode=disable" \
--database mydb
```
This is equivalent to:
```
cockroach sql --url "postgres://root@servername:26257/mydb?sslmode=disable"
```
## See also
* `cockroach` Commands Overview
* Create Security Certificates
* Secure a Cluster
* Create and Manage Users
# Connection Pooling
Source: https://docs.cockroachlabs.com/docs/v26.3/connection-pooling
This page has information on planning, configuring, and using connection pools when using drivers or frameworks with CockroachDB.
In our sizing and production guidance, 1 vCPU is considered equivalent to 1 core in the underlying hardware platform.
## About connection pools
A typical database operation consists of several steps.
1. The driver uses a configuration to start a connection to the database server.
2. A network socket is opened on the client that connects to the database server.
3. Data is read or written through the network socket.
4. The connection is closed down.
5. The network socket is closed down and its resources are freed.
For simple database operations, these steps are not expensive, but as an application scales up, the performance of the application will suffer as each connection is created and destroyed. One pattern for improving performance is a connection pool, a group of already configured and established network connections between the client and database that can be reused for data operations within an application.
Each time an application reads or writes data, it will request one of the connections in the pool. After the data operation completes, the connection is returned to the pool so other operations can use it.
Connection pooling can be a enabled as a feature of the driver, a separate library used in conjunction with a driver, a feature of an application server, or a proxy server that acts as a gateway to the database server.
Sizing a connection pool is critical to maximizing application performance. Too few connections and the application will be blocked, waiting for the connection pool to establish a new connection, or for an existing connection to become available. Conversely, a connection pool with too many connections actively executing a query at the same time can also result in high latency as the cluster cycles through every connection concurrently, decreasing efficiency as the cluster incurs a cost every time it context-switches through every query. For example, the time it takes for many threads to complete many queries in parallel is typically higher than that for a smaller number of threads scaled to the number of available vCPUs or IOPs.
To read more about connection pooling, see our [What is Connection Pooling, and Why Should You Care](https://www.cockroachlabs.com/blog/what-is-connection-pooling/) blog post.
## Recommended settings for connection pools
These are Cockroach Labs recommendations for settings common to most connection pooling software. If your application has multiple workers, each worker should be configured based on the kinds of queries required for its workload.
### Set the maximum lifetime of connections
The maximum lifetime of a connection should be set to between 5 and 30 minutes. CockroachDB Cloud clusters support 30 minutes as the maximum connection lifetime. When a node is shut down or restarted, client connections can be reset after 30 minutes, causing a disruption to applications.
Cockroach Labs recommends starting with a 5 minute maximum connection lifetime and increasing the connection lifetime if there is an impact on tail latency, normally seen when there are large numbers of connections to a cluster. Setting the connection lifetime below 5 minutes is possible, but there is little benefit, and comes at a cost of increased CPU usage for clients and servers. The maximum connection lifetime changes the [rate of new connections per second](#monitor-new-connections) (i.e., average new connections per second = total connections / maximum connection age).
Configure your connection pooling with [connection jitter](#avoid-spikes-in-new-connections) to prevent connection storms.
### Set the maximum lifetime of idle connections
Set the maximum lifetime of idle connections to the same value as the [maximum lifetime of connections](#set-the-maximum-lifetime-of-connections).
### Set the maximum number of open connections
The maximum number of open connections should be set to a value that is greater than the maximum number of concurrent queries that your application worker expects to make. Doing so minimizes the need to establish new connections.
### Set the maximum number of idle connections
Set the maximum number of idle connections to the same value as the [maximum number of open connections](#set-the-maximum-number-of-open-connections).
## Optimize your connection pool
Follow these additional recommendations optimize the performance of your connection pool software.
### Avoid spikes in new connections
If possible configure your connection pool to avoid periodic spikes in new connections, also known as "thundering herds" or "connection storms." Most connection pools offer the ability to stagger their connection age through the use of a connection lifetime "jitter," where the wait time between new connections is randomized. Using a connection jitter of 10% of the maximum connection lifetime smooths out the rate of new connections to a cluster. For example, for a maximum connection lifetime of 1800 seconds (30 minutes), set the connection jitter to 500 seconds (3 minutes).
### Validate connections in a pool
After a connection pool initializes connections to CockroachDB clusters, those connections can occasionally break. This could be due to changes in the cluster topology, or rolling upgrades and restarts, or network disruptions.
Validating connections is typically handled automatically by the connection pool. For example, in HikariCP the connection is validated whenever you request a connection from the pool, and the [`keepaliveTime` property](https://github.com/brettwooldridge/HikariCP#frequently-used) allows you to configure an interval to periodically check if the connections in the pool are valid. Whatever connection pool you use, make sure connection validation is enabled when running your application.
## Size connection pools
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
The **SQL Connection** graph shows how many new connections are being created each second. The **Open SQL Sessions** graph shows the total number of SQL client connections across the cluster. To determine if your connection pool is correctly configured use the metrics from these graphs in the following formula:
**SQL Connection Attempts \< SQL Open Sessions/100**
That is, divide the number of open SQL sessions by 100. This result should be less than the number of new SQL connections per second.
Configure the minimum number of connections to equal to the maximum number of connections, creating a fixed pool size.
Configure the maximum connection lifetime to 30 minutes.
Multi-region Basic and Standard clusters have the same recommendations as single-region clusters.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
Use the following formula as a starting point to size the connection pool:
**active connections = (number of cores \* 4)**
If you have a large number of services connecting to the same cluster, make sure the number of [concurrent active connections](#monitor-active-connections) across all the services does not exceed this recommendation. If each service has its own connection pool, ensure the total number of concurrent active connections across all services stays within this limit.
For multi-region clusters, create a connection pool per region, and apply the same active connection limits for each region in your cluster. For example, if you have 3 regions in your cluster, and each region has 12 vCPUs, the concurrent active connection limit **per region** is 48 (12 \[processor cores] \* 4).
[Connection pool sizes](#set-the-maximum-number-of-open-connections) can be much larger than the active connections limit, since most connections will be idle and have almost no overhead.
To control the maximum number of non-superuser (`root` user or other `admin` role) connections a gateway node can have open at one time, use the `server.max_connections_per_gateway` cluster setting. If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
If your connection pool is properly configured, the total number of open connections to your cluster will typically be at least 100 times larger than the number of new connections per second to your cluster.
Idle connections in CockroachDB do not consume many resources compared to PostgreSQL. Unlike PostgreSQL, which has a hard limit of 5000 connections, CockroachDB can safely support tens of thousands of connections.
Use the following formula as a starting point to size the connection pool:
**active connections = (number of cores \* 4)**
If you have a large number of services connecting to the same cluster, make sure the number of [concurrent active connections](#monitor-active-connections) across all the services does not exceed this recommendation. If each service has its own connection pool, ensure the total number of concurrent active connections across all services stays within this limit.
For multi-region clusters, create a connection pool per region, and apply the same active connection limits for each region in your cluster. For example, if you have 3 regions in your cluster, and each region has 12 vCPUs, the concurrent active connection limit **per region** is 48 (12 \[processor cores] \* 4).
[Connection pool sizes](#set-the-maximum-number-of-open-connections) can be much larger than the active connections limit, since most connections will be idle and have almost no overhead.
To control the maximum number of non-superuser (`root` user or other `admin` role) connections a gateway node can have open at one time, use the `server.max_connections_per_gateway` cluster setting. If a new non-superuser connection would exceed this limit, the error message `"sorry, too many clients already"` is returned, along with error code `53300`.
Cockroach Labs strongly recommends that your application instance run in the same region as your CockroachDB cluster.
## Monitor SQL connections
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The **SQL Connections** graph shows the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The **Open SQL Sessions** graph shows the number of open connections on your cluster.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The `sql.new_conns` metric and **SQL Connection Rate** graph expose the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The `sql.conns` metric and **Open SQL Sessions** graph show the number of open connections on your cluster or node.
The `sql.statements.active` metric and **Active SQL Statements** graph show the number of **active** connections on your cluster or node. A connection is "active" when it is actively executing a query.
Using the following formula:
**active connections = (number of cores \* 4)**
If the number of concurrent active connections exceeds 4 times the number of cores in your cluster, your application is likely not achieving maximum throughput. You should reduce the concurrent workload across your applications, or scale the cluster by adding more nodes or cores per node. Reducing the number of active connections may increase overall throughput, possibly at the expense of increased tail latency for your queries.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
To validate that your connection pool is correctly configured, monitor the SQL connection metrics and graphs on your cluster.
### Monitor new connections
The `sql.new_conns` metric and **SQL Connection Rate** graph expose the number of new SQL connections per second.
A misconfigured connection pool will result in most database operations requiring a new connection to be established, which will increase query latency.
### Monitor active connections
The `sql.conns` metric and **Open SQL Sessions** graph show the number of open connections on your cluster or node.
The `sql.statements.active` metric and **Active SQL Statements** graph show the number of **active** connections on your cluster or node. A connection is "active" when it is actively executing a query.
Using the following formula:
**active connections = (number of cores \* 4)**
If the number of concurrent active connections exceeds 4 times the number of cores in your cluster, your application is likely not achieving maximum throughput. You should reduce the concurrent workload across your applications, or scale the cluster by adding more nodes or cores per node. Reducing the number of active connections may increase overall throughput, possibly at the expense of increased tail latency for your queries.
If your connection pool is properly configured, the total number of open connections to your cluster should be at least 100 times larger than the number of new connections per second to your cluster.
## Serverless functions
"Serverless" refers to stateless, programmatic functions deployed in a cloud environment that provides an execution framework to provision resources dynamically, such as Amazon Lambda functions.
If your application uses serverless functions to connect to CockroachDB, use a connection pool if you plan to invoke functions frequently. To ensure that the connection pool is reused across invocations of the same function instance, initialize the connection pool variable outside the scope of the serverless function definition. Set the maximum connection pool size to 1, unless your function is multi-threaded and establishes multiple concurrent requests to your database within a single function instance.
## Example
In this example, a Java application similar to the basic JDBC example uses the PostgreSQL JDBC driver and HikariCP as the connection pool layer to connect to a CockroachDB cluster. The database is being run on 10 cores across the cluster.
Using the connection pool formula above:
**connections = (10 \[processor cores] \* 4)**
The connection pool size should be 40.
The maximum lifetime of a connection is set to 300000 milliseconds, or 5 minutes.
```java theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
HikariConfig config = new HikariConfig();
config.setJdbcUrl("jdbc:postgresql://localhost:26257/bank");
config.setUsername("maxroach");
config.setPassword("password");
config.addDataSourceProperty("ssl", "true");
config.addDataSourceProperty("sslmode", "require");
config.addDataSourceProperty("reWriteBatchedInserts", "true");
config.setAutoCommit(false);
config.setMaximumPoolSize(40);
config.setMaxLifetime(300000);
config.setKeepaliveTime(150000);
HikariDataSource ds = new HikariDataSource(config);
Connection conn = ds.getConnection();
```
In this example, a Go application similar to the basic pgx example uses the pgxpool library to create a connection pool on a CockroachDB cluster. The database is being run on 10 cores across the cluster.
Using the connection pool formula above:
**connections = (10 \[processor cores] \* 4)**
The connection pool size should be 40, and is set using the `pool_max_conns` query parameter.
The `pool_max_conn_lifetime` query parameter sets the maximum age of a connection to 300 seconds, or 5 minutes.
The `pool_max_conn_lifetime_jitter` query parameter sets the connection jitter to 30 seconds, or 10% of the maximum connection age.
```go theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
// Set connection pool configuration, with maximum connection pool size.
config, err := pgxpool.ParseConfig("postgres://max:roach@127.0.0.1:26257/bank?sslmode=require&pool_max_conns=40&pool_max_conn_lifetime=300s&pool_max_conn_lifetime_jitter=30s")
if err != nil {
log.Fatal("error configuring the database: ", err)
}
// Create a connection pool to the "bank" database.
dbpool, err := pgxpool.ConnectConfig(context.Background(), config)
if err != nil {
log.Fatal("error connecting to the database: ", err)
}
defer dbpool.Close()
```
For a full list of connection pool configuration parameters for pgxpool, see the pgxpool documentation.
## Known limitations
* When a node has both a high number of client connections and running queries, the node may crash due to memory exhaustion. This is due to CockroachDB not accurately limiting the number of clients and queries based on the amount of available RAM on the node. To prevent memory exhaustion, monitor each node's memory usage and ensure there is some margin between maximum CockroachDB memory usage and available system RAM. For more details about memory usage in CockroachDB, see [this blog post](https://www.cockroachlabs.com/blog/memory-usage-cockroachdb/).
# Advanced Public Key Infrastructure (PKI) Certificate Scenarios
Source: https://docs.cockroachlabs.com/docs/v26.3/create-security-certificates-custom-ca
This document discusses the use of advanced Public Key Infrastructure (PKI) systems and certificates issued by them with CockroachDB.
security certificates with CockroachDB. PKI certificates are used in CockroachDB for TLS encryption and for node and client authentication.
| Approach | Use case description |
| - | - |
| [UI certificate and key](#accessing-the-db-console-for-a-secure-cluster) | When you want to access the DB Console for a secure cluster and avoid clicking through a warning message to get to the UI. |
| [Split-node certificate](#split-node-certificates) | When your organizational CA requires you to have separate certificates for the node's incoming connections (from SQL and DB Console clients, and from other CockroachDB nodes) and for outgoing connections to other CockroachDB nodes. |
| [Split-CA certificates](#split-ca-certificates) | When you have multiple CockroachDB clusters and need to restrict access to clients from accessing the other cluster. |
## Accessing the DB Console for a secure cluster
On accessing the DB Console for a secure cluster, your web browser will consider the CockroachDB-issued certificate invalid, because the browser hasn't been configured to trust the CA that issued the certificate.
For secure clusters, you can avoid getting the warning message by using a certificate issued by a public CA whose certificates are trusted by browsers, in addition to the CockroachDB-created certificates.
1. Request a certificate from a public CA (for example, [Let's Encrypt](https://letsencrypt.org/)). The certificate must have the IP addresses and DNS names used to reach the DB Console listed in the `Subject Alternative Name` field.
2. Rename the certificate and key files to `ui.crt` and `ui.key`.
3. Add the `ui.crt` and `ui.key` files to the trust store. `ui.key` must meet the permission requirements check on macOS, Linux, and other UNIX-like systems. If your cluster is deployed using containers, update the containers to include the new certificate and key.
4. The cockroach process reads certificates only when the process starts.
* In a manually-deployed cluster, load the `ui.crt` certificate without restarting the node by issuing a `SIGHUP` signal to the cockroach process:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
pkill -SIGHUP -x cockroach
```
The `SIGHUP` signal must be sent by the same user running the process or by a user with adequate privileges to send signals to processes owned by another user, such as a user with `sudo` access.
* In a cluster deployed using the Public operator, there is no way to send a `SIGHUP` signal to the individual `cockroach` process on each cluster node. Instead, perform a rolling restart of the cluster's pods.
### Node key and certificates
A node must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate issued by the public CA or your organizational CA. |
| `node.crt` | Server certificate created using the `cockroach cert` command.
`node.crt` must have `CN=node` and the list of IP addresses and DNS names listed in `Subject Alternative Name` field.
Must be signed by `ca.crt`. |
| `node.key` | Server key created using the `cockroach cert` command. |
| `ui.crt` | UI certificate signed by the public CA. `ui.crt` must have the IP addresses and DNS names used to reach the DB Console listed in `Subject Alternative Name`. |
| `ui.key` | UI key corresponding to `ui.crt`. |
### Client key and certificates
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate issued by the public CA or your organizational CA. |
| `client.
Each `client.
Must be signed by `ca.crt`. |
| `client.roles. For more information, see Certificate-based authentication using multiple values from the X.509 Subject field.
## Split node certificates
The node certificate discussed in the `cockroach cert` command documentation is multifunctional, which means the same certificate is presented for the node's incoming connections (from SQL and DB Console clients, and from other CockroachDB nodes) and for outgoing connections to other CockroachDB nodes. To make the certificate multi-functional, the `node.crt` created using the `cockroach cert` command has `CN=node` and the list of IP addresses and DNS names listed in `Subject Alternative Name` field. This works if you are also using the CockroachDB CA created using the `cockroach cert` command. However, if you need to use an external public CA or your own organizational CA, the CA policy might not allow it to sign a server certificate containing a CN that is not an IP address or domain name.
To get around this issue, you can split the node key and certificate into two:
* `node.crt` and `node.key`: `node.crt` is used as the server certificate when a node receives incoming connections from clients and other nodes. All IP addresses and DNS names for the node must be listed in the `Subject Alternative Name` field.
* `client.node.crt` and `client.node.key`: `client.node.crt` is used as the client certificate when making connections to other nodes. `client.node.crt` must have `CN=node`.
### Node key and certificates
A node must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate issued by the public CA or your organizational CA. |
| `node.crt` | Server certificate used when a node receives incoming connections from clients and other nodes.
All IP addresses and DNS names for the node must be listed in `Subject Alternative Name`.
Must be signed by `ca.crt`. |
| `node.key` | Server key corresponding to `node.crt`. |
| `client.node.crt` | Client certificate when making connections to other nodes.
Must have `CN=node`.
Must be signed by `ca.crt`. |
| `client.node.key` | Client key corresponding to `client.node.crt`. |
Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag.
### Client key and certificates
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate issued by the public CA or your organizational CA. |
| `client.
Each `client.
Must be signed by `ca.crt`. |
| `client.
We do not recommend you use split CA certificates unless your organizational security practices mandate you to do so.
If you need to use separate CAs to sign node certificates and client certificates, then you need two CAs and their respective certificates and keys: `ca.crt` and `ca-client.crt`.
### Node key and certificates
A node must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate to verify node certificates. |
| `ca-client.crt` | CA certificate to verify client certificates. |
| `node.crt` | Server certificate used when a node receives incoming connections from clients and other nodes.
All IP addresses and DNS names for the node must be listed in `Subject Alternative Name`.
Must be signed by `ca.crt`. |
| `node.key` | Server key corresponding to `node.crt`. |
| `client.node.crt` | Client certificate when making connections to other nodes. This certificate must be signed using `ca-client.crt`
Must have `CN=node`. |
| `client.node.key` | Client key corresponding to `client.node.crt`. |
Optionally, if you have a certificate issued by a public CA to securely access the DB Console, you need to place the certificate and key (`ui.crt` and `ui.key` respectively) in the directory specified by the `--certs-dir` flag.
### Client key and certificates
A client must have the following files with file names as specified in the table:
| File name | File usage |
| - | - |
| `ca.crt` | CA certificate. |
| `client.
Each `client.
Must be signed by `ca-client.crt`. |
| `client.Set the cluster setting `security.ocsp.mode` to `lax` (by default, the cluster setting is set to `off`).
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW CLUSTER SETTING security.ocsp.mode;
```
```
security.ocsp.mode
----------------------
off
(1 row)
Server Execution Time: 56µs
Network Latency: 181µs
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SET CLUSTER SETTING security.ocsp.mode = lax;
```
For production clusters, you might want to set the setting to `strict`.
In the `strict` mode, all certificates are presumed to be invalid if the OCSP server is not reachable. Setting the cluster setting `security.ocsp.mode` to `strict` will lock you out of your CockroachDB database if your OCSP server goes down.
## See also
* Public Key Infrastructure (PKI) and Transport Layer Security (TLS)
* Use the CockroachDB CLI to provision a development cluster.
* Manage PKI certificates for a CockroachDB deployment with HashiCorp Vault.
* Manual Deployment: Learn about starting a multi-node secure cluster and accessing it from a client.
* Start a Node: Learn more about the flags you pass when adding a node to a secure cluster
* Client Connection Parameters
# Create Security Certificates using OpenSSL
Source: https://docs.cockroachlabs.com/docs/v26.3/create-security-certificates-openssl
This tutorial shows how to provision a public key infrastructure (PKI) certificate authority (CA) for a CockroachDB self-hosted cluster deployed in Google Cloud Platform (GCP).
See also:
* Public Key Infrastructure (PKI) and Transport Layer Security (TLS)
* Use the CockroachDB CLI to provision a development cluster.
* Manage PKI certificates for a CockroachDB deployment with HashiCorp Vault.
To create these certificates and keys, use the `cockroach cert` commands with the appropriate subcommands and flags, use [`openssl` commands](https://wiki.openssl.org/index.php/), or use a custom CA (for example, a public CA or your organizational CA).
If you manage your own Certificate Authority (CA) infrastructure, CockroachDB supports mapping between the Subject field of your [X.509 certificates](https://en.wikipedia.org/wiki/X.509) and SQL roles. For more information, see Certificate-based authentication using multiple values from the X.509 Subject field.
## Subcommands
| Subcommand | Usage |
| - | - |
| [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) | Create an RSA private key. |
| [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) | Create CA certificate and CSRs (certificate signing requests). |
| [`openssl ca`](https://www.openssl.org/docs/manmaster/man1/ca.html) | Create node and client certificates using the CSRs. |
## Configuration files
To use [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) and [`openssl ca`](https://www.openssl.org/docs/manmaster/man1/ca.html) subcommands, you need the following configuration files:
| File name pattern | File usage |
| - | - |
| `ca.cnf` | CA configuration file |
| `node.cnf` | Server configuration file |
| `client.cnf` | Client configuration file |
## Certificate directory
To create node and client certificates using the OpenSSL commands, you need access to a local copy of the CA certificate and key. We recommend creating all certificates (node, client, and CA certificates), and node and client keys in one place and then distributing them appropriately. Store the CA key somewhere safe and keep a backup; if you lose it, you will not be able to add new nodes or clients to your cluster.
## Required keys and certificates
Use the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) and [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) subcommands to create all certificates, and node and client keys in a single directory, with the files named as follows:
### Node key and certificates
| File name pattern | File usage |
| - | - |
| `ca.crt` | CA certificate |
| `node.crt` | Server certificate |
| `node.key` | Key for server certificate |
### Client key and certificates
| File name pattern | File usage |
| - | - |
| `ca.crt` | CA certificate. |
| `client.permission requirements check on macOS, Linux, and other UNIX-like systems.
## Examples
### Step 1. Create the CA key and certificate pair
1. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs my-safe-directory
```
* `certs`: Create your CA certificate and all node and client certificates and keys in this directory and then upload the relevant files to the nodes and clients.
* `my-safe-directory`: Create your CA key in this directory and then reference the key when generating node and client certificates. After that, keep the key safe and secret; do not upload it to your nodes or clients.
2. Create the `ca.cnf` file and copy the following configuration into it.
You can set the CA certificate expiration period using the `default_days` parameter. We recommend using the CockroachDB default value of the CA certificate expiration period, which is 365 days.
```
# OpenSSL CA configuration file
[ ca ]
default_ca = CA_default
[ CA_default ]
default_days = 365
database = index.txt
serial = serial.txt
default_md = sha256
copy_extensions = copy
unique_subject = no
# Used to create the CA certificate.
[ req ]
prompt=no
distinguished_name = distinguished_name
x509_extensions = extensions
[ distinguished_name ]
organizationName = Cockroach
commonName = Cockroach CA
[ extensions ]
keyUsage = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign
basicConstraints = critical,CA:true,pathlen:1
# Common policy for nodes and users.
[ signing_policy ]
organizationName = supplied
commonName = optional
# Used to sign node certificates.
[ signing_node_req ]
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth
# Used to sign client certificates.
[ signing_client_req ]
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = clientAuth
```
The `keyUsage` and `extendedkeyUsage` parameters are vital for CockroachDB functions. You can modify or omit other parameters as per your preferred OpenSSL configuration and you can add additional usages, but do not omit `keyUsage` and `extendedkeyUsage` parameters or remove the listed usages.
1. Create the CA key using the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl genrsa -out my-safe-directory/ca.key 2048
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chmod 400 my-safe-directory/ca.key
```
2. Create the CA certificate using the [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl req \
-new \
-x509 \
-config ca.cnf \
-key my-safe-directory/ca.key \
-out certs/ca.crt \
-days 365 \
-batch
```
3. Reset database and index files:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -f index.txt serial.txt
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ touch index.txt
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo '01' > serial.txt
```
### Step 2. Create the certificate and key pairs for nodes
In the following steps, replace the placeholder text in the code with the actual username and node address.
1. Create the `node.cnf` file for the first node and copy the following configuration into it:
```
# OpenSSL node configuration file
[ req ]
prompt=no
distinguished_name = distinguished_name
req_extensions = extensions
[ distinguished_name ]
organizationName = Cockroach
[ extensions ]
subjectAltName = critical,DNS:
```
The `subjectAltName` parameter is vital for CockroachDB functions. You can modify or omit other parameters as per your preferred OpenSSL configuration, but do not omit the `subjectAltName` parameter.
1. Create the key for the first node using the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl genrsa -out certs/node.key 2048
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chmod 400 certs/node.key
```
2. Create the CSR for the first node using the [`openssl req`](https://www.openssl.org/docs/manmaster/man1/req.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl req \
-new \
-config node.cnf \
-key certs/node.key \
-out node.csr \
-batch
```
3. Sign the node CSR to create the node certificate for the first node using the [`openssl ca`](https://www.openssl.org/docs/manmaster/man1/ca.html) command.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl ca \
-config ca.cnf \
-keyfile my-safe-directory/ca.key \
-cert certs/ca.crt \
-policy signing_policy \
-extensions signing_node_req \
-out certs/node.crt \
-outdir certs/ \
-in node.csr \
-batch
```
4. Verify the values in the `Subject Alternative Name` field in the certificate:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl x509 -in certs/node.crt -text | grep "X509v3 Subject Alternative Name" -A 1
```
Sample output:
```
X509v3 Subject Alternative Name: critical
DNS:localhost, DNS:node.example.io, IP Address:127.0.0.1
```
### Step 3. Create the certificate and key pair for the first user
In the following steps, replace the placeholder text in the code with the actual username.
1. Create the `client.cnf` file for the first user and copy the following configuration into it:
```
[ req ]
prompt=no
distinguished_name = distinguished_name
req_extensions = extensions
[ distinguished_name ]
organizationName = Cockroach
commonName =
[ extensions ]
subjectAltName = DNS:root
```
The `commonName` and `subjectAltName` parameters are vital for CockroachDB functions. You can modify or omit other parameters as per your preferred OpenSSL configuration, but do not omit the `commonName` parameter or modify the `subjectAltName` parameter.
1. Create the key for the first client using the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl genrsa -out certs/client. create user ;
```
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
### Step 5. Create the certificate and key pair for a client
In the following steps, replace the placeholder text in the code with the actual username.
1. Edit the `client.cnf` file for the client and copy the following configuration into it:
```
[ req ]
prompt=no
distinguished_name = distinguished_name
[ distinguished_name ]
organizationName = Cockroach
commonName =
```
2. Create the key for the first client using the [`openssl genrsa`](https://www.openssl.org/docs/manmaster/man1/genrsa.html) command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ openssl genrsa -out certs/client.Manual Deployment: Learn about starting a multi-node secure cluster and accessing it from a client.
* Start a Node: Learn more about the flags you pass when adding a node to a secure cluster
* Client Connection Parameters
# Data Resilience
Source: https://docs.cockroachlabs.com/docs/v26.3/data-resilience
CockroachDB provides built-in [**high availability (HA)**](#high-availability) features and [**disaster recovery (DR)**](#disaster-recovery) tooling to achieve operational resiliency in various deployment topologies and use cases.
* HA features ensure continuous access to data without interruption even in the presence of failures or disruptions to maximize uptime.
* DR tools allow for recovery from major incidents to minimize downtime and data loss.
![Diagram showing how HA features and DR tools create a resilient CockroachDB deployment.]()
You can balance required SLAs and recovery objectives with the cost and management of each of these features to build a resilient deployment.
* **Recovery Point Objective (RPO)**: The maximum amount of data loss (measured by time) that an organization can tolerate.
* **Recovery Time Objective (RTO)**: The maximum length of time it should take to restore normal operations following an outage.
For a practical guide on how CockroachDB uses Raft to replicate, distribute, and rebalance data, refer to the CockroachDB Resilience demo.
## High availability
* **Multi-active availability**: CockroachDB's built-in Raft replication stores data safely and consistently on multiple nodes to ensure no downtime even during a temporary node outage. Replication controls allow you to configure the number and location of replicas to suit a deployment.
* For more detail on planning for single-region or multi-region recovery, refer to Single-region survivability planning or Multi-region survivability planning.
* **Advanced fault tolerance**: Capabilities built in to CockroachDB to perform routine maintenance operations with minimal impact to foreground performance. For example, online schema changes, write-ahead log failover, and Leader leases.
* **Logical data replication (LDR)** (Preview): A cross-cluster replication tool between active CockroachDB clusters, which supports a range of topologies. LDR provides eventually consistent, table-level replication between the clusters. Individually, each active cluster uses CockroachDB multi-active availability to achieve low, single-region write latency with transactionally consistent writes using Raft replication.
### Choose an HA strategy
| | Single-region replication (synchronous) | Multi-region replication (synchronous) | Logical data replication (asynchronous) |
| - | - | - | - |
| **RPO** | 0 seconds | 0 seconds | Immediate mode 0.5 seconds |
| **RTO** | Zero RTO Potential increased latency for 1-9 seconds | Zero RTO Potential increased latency for 1-9 seconds | Zero RTO Application traffic failover time |
| **Write latency** | Region-local write latency p50 latency \< 5ms (for multiple availability zones in us-east1) | Cross-region write latency p50 latency > 50ms (for a multi-region cluster in us-east1, us-east-2, us-west-1) | Region-local latency depending on design p50 latency \< 5ms (for multiple availability zones in us-east1) |
| **Recovery** | Automatic | Automatic | Semi-automatic |
| **Fault tolerance** | Zero RPO node, availability zone failures within a cluster | Zero RPO node, availability zone failures, region failures within a cluster | Zero RPO node, availability zone within a cluster, region failures with loss up to RPO in a two-region (or two datacenter) setup |
| **Minimum regions to achieve fault tolerance** | 1 | 3 | 2 |
For details on designing your cluster topology for HA with replication, refer to the Disaster Recovery Planning page.
## Disaster recovery
* **Backup and point-in-time restore**: Point-in-time backup and restore allows you to roll back to a specific point in time. Multiple supported cloud storage providers means that you can store backups in your chosen provider. Incremental backups allow you to configure backup frequency for lower RPO.
* **Physical cluster replication (PCR)**: A cross-cluster replication tool between an active primary CockroachDB cluster and a passive standby CockroachDB cluster. PCR provides transactionally consistent full-cluster replication.
### Choose a DR strategy
CockroachDB is designed to recover automatically; however, building backups or PCR into your DR plan protects against unforeseen incidents.
| | Point-in-time backup & restore | Physical cluster replication (asynchronous) |
| - | - | - |
| **RPO** | >=5 minutes | 10s of seconds |
| **RTO** | Minutes to hours, depending on data size and number of nodes | Seconds to minutes, depending on cluster size, and time of failover |
| **Write latency** | No impact | No impact |
| **Recovery** | Manual restore | Manual failover |
| **Fault tolerance** | Not applicable | Zero RPO node, availability zone within a cluster, region failures with loss up to RPO in a two-region (or two-datacenter) setup |
| **Minimum regions to achieve fault tolerance** | 1 | 2 |
## See also
* Architecture Overview
* Set Up Logical Data Replication
* Set Up Physical Cluster Replication
* Physical Cluster Replication Technical Overview
* Backup Architecture
* Backup and Restore Overview
* Managed Backups
# CockroachDB Resilience Demo
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-cockroachdb-resilience
This page describes how to perform a hands-on demonstration of CockroachDB's fault-tolerant design allowing services to remain available during a failure and recovery.
## Run a guided demo in CockroachDB Cloud
CockroachDB Cloud Advanced includes a built-in fault tolerance demo in the Cloud Console that automatically runs a sample workload and executes a controlled availability zone failure on your cluster, showing real-time metrics of query latency and failure rate during the outage and recovery.
The CockroachDB Cloud fault tolerance demo is in Preview.
The following prerequisites are needed to run the fault tolerance demo:
* You must have CockroachDB Advanced cluster with at least three nodes.
* All nodes must be healthy.
* The cluster's CPU utilization must be below 30%.
* The cluster must not currently be in a locked state as a result of maintenance such as scaling. Clusters that were recently created may be locked intermittently while post-provisioning setup completes. Monitor your cluster's status in the Cloud Console to make sure it is not in a maintenance state.
* You must have the Cluster Operator and/or Cluster Admin role assigned to start the demo.
* Another fault tolerance demo must not currently be running.
In total, the fault tolerance demo takes approximately 10 to 15 minutes to complete. The demo creates a temporary database and demo workload on your cluster, then cleans up after the demo is complete. If the demo becomes stuck, the system will automatically force a full cleanup after 30 minutes. You do not need to contact support before this window has elapsed.
The fault tolerance demo is not recommended to be run on a production cluster. The demo temporarily blocks network communication to all nodes in one availability zone. This disruption is real, and may affect cluster performance during the outage window. Instead, use a dedicated test or staging cluster for the fault tolerance demo.
To run the fault tolerance demo, open the Cloud Console and navigate to the **Overview** page, then select **Actions > Fault tolerance demo**.
### Known limitations
The following known limitations apply to the fault tolerance demo in CockroachDB Cloud:
* QPS may briefly appear to drop to zero on the chart during or after the disruption phase. This occurs because metrics from the nodes in the disrupted availability zone are temporarily unavailable during the network block, it does not reflect actual cluster behavior. The unaffected nodes continue serving traffic normally.
* If you cancel the fault tolerance demo and immediately try to start a new one, you may encounter an error. Wait a few minutes before starting again.
* Clean-up of the temporary workload and database may continue for a few minutes after the demo ends. Refresh the Console page if the message does not clear after a few minutes.
## Run a manual demo on a local machine
This guide walks you through a simple demonstration of CockroachDB's resilience on a local cluster deployment.
Starting with a 6-node local cluster with the default 3-way replication, you will perform the following steps:
1. Run a sample workload.
2. Terminate a node to simulate failure.
3. Observe the cluster continuing to serve query traffic uninterrupted during the failure state.
4. Observe the cluster repair itself by re-replicating missing data to other nodes.
5. Prepare the cluster for 2 simultaneous node failures by increasing to 5-way replication. ^.6. Take two nodes offline at the same time, and again observe how the cluster continues uninterrupted.
### Before you begin
Make sure you have already installed CockroachDB.
### Step 1. Start a 6-node cluster
1. In separate terminal windows, use the `cockroach start` command to start six nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node1 \
--listen-addr=localhost:26257 \
--http-addr=localhost:8080 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node2 \
--listen-addr=localhost:26258 \
--http-addr=localhost:8081 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node3 \
--listen-addr=localhost:26259 \
--http-addr=localhost:8082 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node4 \
--listen-addr=localhost:26260 \
--http-addr=localhost:8083 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node6 \
--listen-addr=localhost:26262 \
--http-addr=localhost:8085 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. Use the `cockroach init` command to perform a one-time initialization of the cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --insecure --host=localhost:26257
```
### Step 2. Set up load balancing
In this tutorial, you run a sample workload to simulate multiple client connections. Each node is an equally suitable SQL gateway for the load, but it's always recommended to spread requests evenly across nodes. This section shows how to set up the open-source [HAProxy](http://www.haproxy.org/) load balancer.
1. Install HAProxy.
If you're on a Mac and use Homebrew, run:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ brew install haproxy
```
If you're using Linux and use apt-get, run:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install haproxy
```
2. Run the `cockroach gen haproxy` command, specifying the port of any node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach gen haproxy --insecure --host=localhost --port=26257
```
This command generates an `haproxy.cfg` file automatically configured to work with the nodes of your running cluster.
3. In `haproxy.cfg`, change `bind :26257` to `bind :26000`. This changes the port on which HAProxy accepts requests to a port that is not already in use by a node.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
sed -i.saved 's/^ bind :26257/ bind :26000/' haproxy.cfg
```
4. Start HAProxy, with the `-f` flag pointing to the `haproxy.cfg` file:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ haproxy -f haproxy.cfg &
```
### Step 3. Run a sample workload
Use the `cockroach workload` command to run CockroachDB's built-in version of the YCSB benchmark, simulating multiple client connections, each performing mixed read/write operations.
1. Load the initial `ycsb` schema and data, pointing it at HAProxy's port:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init ycsb --splits=50 'postgresql://root@localhost:26000?sslmode=disable'
```
The `--splits` flag tells the workload to manually split ranges a number of times. This is not something you'd normally do, but for the purpose of this tutorial, it makes it easier to visualize the movement of data in the cluster.
2. Run the `ycsb` workload, pointing it at HAProxy's port:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run ycsb \
--duration=20m \
--concurrency=3 \
--max-rate=1000 \
--tolerate-errors \
'postgresql://root@localhost:26000?sslmode=disable'
```
This command initiates 3 concurrent client workloads for 20 minutes, but limits the total load to 1000 operations per second (since you're running everything on a single machine).
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1.0s 0 902.8 930.9 1.1 2.1 4.1 62.9 read
1.0s 0 46.5 48.0 3.4 4.7 6.0 6.0 update
2.0s 0 923.3 926.9 1.0 2.9 5.5 8.9 read
2.0s 0 38.0 43.0 3.0 6.0 7.6 7.6 update
3.0s 0 901.1 918.3 1.1 2.5 5.0 6.6 read
3.0s 0 55.0 47.0 3.4 7.9 9.4 11.5 update
4.0s 0 948.9 926.0 1.0 1.6 2.6 5.0 read
4.0s 0 46.0 46.7 3.1 5.2 16.8 16.8 update
5.0s 0 932.0 927.2 1.1 1.8 2.9 13.6 read
5.0s 0 56.0 48.6 3.0 4.2 5.2 5.5 update
...
```
After the specified duration (20 minutes in this case), the workload will stop and you'll see totals printed to standard output.
### Step 4. Check the workload
Initially, the workload creates a new database called `ycsb`, creates the table `public.usertable` in that database, and inserts rows into the table. Soon, the load generator starts executing approximately 95% reads and 5% writes.
1. Go to the DB Console at [http://localhost:8080](http://localhost:8080).
2. To check the SQL queries getting executed, click **Metrics** on the left, and hover over the **SQL Statements** graph at the top:
![DB Console SQL Statements]()
3. To check the client connections from the load generator, select the **SQL** dashboard and hover over the **Open SQL Sessions** graph:
![DB Console Open Sessions]()
You'll notice 3 client connections from the load generator. If you want to check that HAProxy balanced each connection to a different node, you can change the **Graph** dropdown from **Cluster** to specific nodes.
4. To see more details about the `ycsb` database and the `public.usertable` table, click **Databases** in the upper left and click **ycsb**:
![DB Console Databases]()
You can also view the schema and other table details of `public.usertable` by clicking the table name:
![DB Console usertable]()
5. By default, CockroachDB replicates all data 3 times and balances it across all nodes. To see this balance, click **Overview** and check the replica count across all nodes:
![DB Console Overview]()
### Step 5. Simulate a single node failure
When a node fails, the cluster waits for the node to remain offline for 5 minutes by default before considering it dead, at which point the cluster automatically repairs itself by re-replicating any of the replicas on the down nodes to other available nodes.
1. In a new terminal, edit the default replication zone to reduce the amount of time the cluster waits before considering a node dead to the minimum allowed of 1 minute and 15 seconds:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26000 \
--execute="SET CLUSTER SETTING server.time_until_store_dead = '1m15s';"
```
2. Get the process IDs of the nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ps -ef | grep cockroach | grep -v grep
```
```
503 53645 1 0 10:18AM ttys013 0:44.80 cockroach start --insecure --store=fault-node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
503 53651 1 0 10:18AM ttys013 0:38.81 cockroach start --insecure --store=fault-node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
503 53671 1 0 10:18AM ttys013 0:35.40 cockroach start --insecure --store=fault-node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
503 53684 1 0 10:18AM ttys013 0:38.44 cockroach start --insecure --store=fault-node4 --listen-addr=localhost:26260 --http-addr=localhost:8083 --join=localhost:26257,localhost:26258,localhost:26259
503 53708 1 0 10:18AM ttys013 0:39.49 cockroach start --insecure --store=fault-node5 --listen-addr=localhost:26261 --http-addr=localhost:8084 --join=localhost:26257,localhost:26258,localhost:26259
503 53730 1 0 10:18AM ttys013 0:33.37 cockroach start --insecure --store=fault-node6 --listen-addr=localhost:26262 --http-addr=localhost:8085 --join=localhost:26257,localhost:26258,localhost:26259
```
3. Gracefully shut down the node stored in `fault-node5`, specifying its process ID (in this example, `53708`):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM 53708
```
### Step 6. Check load continuity and cluster health
Go back to the DB Console, click **Metrics** on the left, and verify that the cluster as a whole continues serving data, despite one of the nodes being unavailable and marked as **Suspect**:
![DB Console Suspect Node]()
This shows that when all ranges are replicated 3 times (the default), the cluster can tolerate a single node failure because the surviving nodes have a majority of each range's replicas (2/3).
### Step 7. Watch the cluster repair itself
Click **Overview** on the left:
![DB Console Cluster Repair]()
Because you reduced the time it takes for the cluster to consider the down node dead, after 1 minute or so, the cluster will consider the down node "dead", and you'll see the replica count on the remaining nodes increase and the number of under-replicated ranges decrease to 0. This shows the cluster repairing itself by re-replicating missing replicas.
### Step 8. Prepare for two simultaneous node failures
At this point, the cluster has recovered and is ready to handle another failure. However, the cluster cannot handle two *near-simultaneous* failures in this configuration. Failures are "near-simultaneous" if they are closer together than the `server.time_until_store_dead` cluster setting plus the time taken for the number of replicas on the dead node to drop to zero. If two failures occurred in this configuration, some ranges would become unavailable until one of the nodes recovers.
To be able to tolerate 2 of 5 nodes failing simultaneously without any service interruption, ranges must be replicated 5 times.
1. In the terminal window where you started `fault-node5` initially, start it again using the same command you used to [start the node initially](#step-1-start-a-6-node-cluster):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--insecure \
--store=fault-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. Use the `ALTER RANGE ... CONFIGURE ZONE` command to change the cluster's `default` replication factor to 5:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26000 \
--execute="ALTER RANGE default CONFIGURE ZONE USING num_replicas=5;"
```
3. In the DB Console **Overview** dashboard, watch the replica count increase and even out across all 6 nodes:
![DB Console Cluster Restore]()
This shows the cluster up-replicating so that each range has 5 replicas, one on each node.
### Step 9. Simulate two simultaneous node failures
Gracefully shut down **2 nodes**, specifying the [process IDs you retrieved earlier](#step-5-simulate-a-single-node-failure):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM {process IDs}
```
### Step 10. Check cluster status and service continuity
1. Click **Overview** on the left, and verify the state of the cluster:
![DB Console Cluster Health]()
2. To verify that the cluster still serves data, use the `cockroach sql` command to interact with the cluster.
Count the number of rows in the `ycsb.public.usertable` table to see that it serves reads:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="SELECT count(*) FROM ycsb.public.usertable;"
```
```
count
+-------+
10000
(1 row)
```
Insert data to see that it serves writes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="INSERT INTO ycsb.public.usertable VALUES ('asdf', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);"
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost:26257 \
--execute="SELECT count(*) FROM ycsb.public.usertable;"
```
```
count
+-------+
10001
(1 row)
```
This shows that when all ranges are replicated 5 times, the cluster can tolerate 2 simultaneous node outages because the surviving nodes have a majority of each range's replicas (3/5).
### Step 11. Clean up
1. In the terminal where the YCSB workload is running, press **CTRL + c**.
2. Terminate HAProxy:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ pkill haproxy
```
3. Gracefully shut down the remaining **4 nodes**, specifying the [process IDs you retrieved earlier](#step-5-simulate-a-single-node-failure) and the new process ID for the node stored in `fault-node5`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM {process IDs}
```
For the final 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational.
1. To restart the cluster at a later time, run the same `cockroach start` commands as in [Step 1. Start a 6-node cluster](#step-1-start-a-6-node-cluster) from the directory containing the nodes' data stores.
2. If you do not plan to restart the cluster, you may want to remove the nodes' data stores and the HAProxy config files:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf fault-node1 fault-node2 fault-node3 fault-node4 fault-node5 fault-node6 haproxy.cfg haproxy.cfg.saved
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
# Replication and Rebalancing
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-replication-and-rebalancing
This page guides you through a simple demonstration of how CockroachDB replicates, distributes, and rebalances data. Starting with a 3-node local cluster, you'll write some data and verify that it replicates in triplicate by default. You'll then add 2 more nodes and watch how CockroachDB automatically rebalances replicas to efficiently use all available capacity.
## Before you begin
Make sure you have already installed CockroachDB.
## Step 1. Generate certificates
1. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs my-safe-directory
```
2. Create the CA (Certificate Authority) certificate and key pair:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
3. Create the certificate and key pair for your nodes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
localhost \
$(hostname) \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
Because you're running a local cluster and all nodes use the same hostname (`localhost`), you only need a single node certificate. Note that this is different than running a production cluster, where you would need to generate a certificate and key for each node, issued to all common names and IP addresses you might use to refer to the node as well as to any load balancer instances.
4. Create a client certificate and key pair for the `root` user:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-client \
root \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
## Step 2. Start a 3-node cluster
1. Use the `cockroach start` command to start node 1:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node1 \
--listen-addr=localhost:26257 \
--http-addr=localhost:8080 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. In a new terminal, start node 2:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node2 \
--listen-addr=localhost:26258 \
--http-addr=localhost:8081 \
--join=localhost:26257,localhost:26258,localhost:26259
```
3. In a new terminal, start node 3:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node3 \
--listen-addr=localhost:26259 \
--http-addr=localhost:8082 \
--join=localhost:26257,localhost:26258,localhost:26259
```
4. In a new terminal, use the `cockroach init` command to perform a one-time initialization of the cluster, sending the request to any node on the `--join` list:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=localhost:26257
```
You'll see the following message:
```
Cluster successfully initialized
```
At this point, each node also prints helpful startup details to its log.
## Step 3. Create a SQL user
You'll use a non-`root` user for running a client workload and accessing the DB Console.
1. In the same terminal, as the `root` user, open the built-in SQL shell against any node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql \
--certs-dir=certs \
--host=localhost:26257
```
2. Create the `maxroach` user with a password:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE USER maxroach WITH PASSWORD ' GRANT admin TO maxroach;
```
This role assignment is for convenience for this tutorial; it gives the user access to all data without the need for additional privileges. For more details, see Authorization.
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 4. Write data
1. In the same terminal, run the `cockroach workload` command to generate an example `intro` database. In the connection string, replace ` SHOW DATABASES;
```
```
database_name
+---------------+
defaultdb
intro
postgres
system
(4 rows)
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TABLES FROM intro;
```
```
table_name
--------------
mytable
(1 row)
```
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM intro.mytable WHERE (l % 2) = 0;
```
```
l | v
+----+------------------------------------------------------+
0 | !__aaawwmqmqmwwwaas,,_ .__aaawwwmqmqmwwaaa,,
2 | !"VT?!"""^~~^"""??T$Wmqaa,_auqmWBT?!"""^~~^^""??YV^
4 | ! "?##mW##?"-
6 | ! C O N G R A T S _am#Z??A#ma, Y
8 | ! _ummY" "9#ma, A
10 | ! vm#Z( )Xmms Y
12 | ! .j####mmm#####mm#m##6.
14 | ! W O W ! jmm###mm######m#mmm##6
16 | ! ]#me*Xm#m#mm##m#m##SX##c
18 | ! dm#||+*$##m#mm#m#Svvn##m
20 | ! :mmE=|+||S##m##m#1nvnnX##; A
22 | ! :m#h+|+++=Xmm#m#1nvnnvdmm; M
24 | ! Y $#m>+|+|||##m#1nvnnnnmm# A
26 | ! O ]##z+|+|+|3#mEnnnnvnd##f Z
28 | ! U D 4##c|+|+|]m#kvnvnno##P E
30 | ! I 4#ma+|++]mmhvnnvq##P` !
32 | ! D I ?$#q%+|dmmmvnnm##!
34 | ! T -4##wu#mm#pw##7'
36 | ! -?$##m####Y'
38 | ! !! "Y##Y"-
40 | !
(21 rows)
```
4. Exit the SQL shell:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
## Step 5. Verify replication
1. To understand replication in CockroachDB, it's important to review a few concepts from the architecture:
| Concept | Description |
| - | - |
| **Range** | CockroachDB stores all user data (tables, indexes, etc.) and almost all system data in a giant sorted map of key-value pairs. This keyspace is divided into "ranges", contiguous chunks of the keyspace, so that every key can always be found in a single range.
From a SQL perspective, a table and its secondary indexes initially map to a single range, where each key-value pair in the range represents a single row in the table (also called the primary index because the table is sorted by the primary key) or a single row in a secondary index. As soon as that range reaches the maximum range size, it splits into two ranges. This process continues for these new ranges as the table and its indexes continue growing. |
| **Replica** | CockroachDB replicates each range (3 times by default) and stores each replica on a different node. |
2. With those concepts in mind, open the DB Console at [http://localhost:8080](http://localhost:8080) and log in with the `maxroach` user.
3. On the **Overview** page, note that the **Replicas** count is the same on all three nodes. This indicates:
* There are this many "ranges" of data in the cluster. These are mostly internal "system" ranges since you haven't added much table data.
* Each range has been replicated 3 times (according to the CockroachDB default).
* For each range, each replica is stored on different nodes.
## Step 6. Add two more nodes
1. Back in the terminal, add a fourth node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node4 \
--listen-addr=localhost:26260 \
--http-addr=localhost:8083 \
--join=localhost:26257,localhost:26258,localhost:26259
```
2. In a new terminal, add a fifth node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--store=rep-node5 \
--listen-addr=localhost:26261 \
--http-addr=localhost:8084 \
--join=localhost:26257,localhost:26258,localhost:26259
```
## Step 7. Watch data rebalance
Back in the DB Console, you'll see that there are now 5 nodes listed.
At first, the replica count will be lower for nodes 4 and 5. Very soon, however, you'll see those numbers even out across all nodes, indicating that data is being automatically rebalanced to utilize the additional capacity of the new nodes.
## Step 8. Stop the cluster
1. When you're done with your test cluster, press `ctrl-c` in each terminal where a node is running.
For the last 2 nodes, the shutdown process will take longer (about a minute each) and will eventually force the nodes to stop. This is because, with only 2 of 5 nodes left, a majority of replicas are not available, and so the cluster is no longer operational. To speed up this process, you can press `ctrl-c` a second time.
1. To restart the cluster at a later time, run the same `cockroach start` commands as earlier from the directory containing the nodes' data stores.
If you do not plan to restart the cluster, you may want to remove the cluster's certificates and data stores:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf certs my-safe-directory rep-node1 rep-node2 rep-node3 rep-node4 rep-node5
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
# Serializable Transactions
Source: https://docs.cockroachlabs.com/docs/v26.3/demo-serializable
In contrast to most databases, CockroachDB offers `SERIALIZABLE` isolation by default, which is the strongest of the four [transaction isolation levels](https://wikipedia.org/wiki/Isolation_\(database_systems\)) defined by the SQL standard and is stronger than the `SNAPSHOT` isolation level developed later. `SERIALIZABLE` isolation guarantees that even though transactions may execute in parallel, the result is the same as if they had executed one at a time, without any concurrency. This ensures data correctness by preventing all "anomalies" allowed by weaker isolation levels.
In this tutorial, you'll work through a hypothetical scenario that demonstrates the effectiveness of `SERIALIZABLE` isolation for maintaining data correctness.
1. You'll start by reviewing the scenario and its schema.
2. You'll finish by executing the scenario at `SERIALIZABLE` isolation, observing how it guarantees correctness. You'll use CockroachDB for this portion.
For a deeper discussion of transaction isolation and the write skew anomaly, see the [Real Transactions are Serializable](https://www.cockroachlabs.com/blog/acid-rain/) and [What Write Skew Looks Like](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/) blog posts.
## Overview
### Scenario
* A hospital has an application for doctors to manage their on-call shifts.
* The hospital has a rule that at least one doctor must be on call at any one time.
* Two doctors are on-call for a particular shift, and both of them try to request leave for the shift at approximately the same time.
* In PostgreSQL, with the default `READ COMMITTED` isolation level, the [write skew](#write-skew) anomaly results in both doctors successfully booking leave and the hospital having no doctors on call for that particular shift.
* In CockroachDB, with the `SERIALIZABLE` isolation level, write skew is prevented, one doctor is allowed to book leave and the other is left on-call, and lives are saved.
#### Write skew
When write skew happens, a transaction reads something, makes a decision based on the value it saw, and writes the decision to the database. However, by the time the write is made, the premise of the decision is no longer true. Only `SERIALIZABLE` and some implementations of `REPEATABLE READ` isolation prevent this anomaly.
### Schema
![Schema for serializable transaction tutorial]()
## Step 1. Set up the scenario on PostgreSQL
1. If you haven't already, install PostgreSQL locally. On Mac, you can use [Homebrew](https://brew.sh/):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ brew install postgres
```
2. [Start PostgreSQL](https://www.postgresql.org/docs/10/static/server-start.html):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ postgres -D /usr/local/var/postgres &
```
3. Open a SQL connection to PostgreSQL:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ psql
```
4. Create the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE doctors (
id INT PRIMARY KEY,
name TEXT
);
```
5. Create the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE schedules (
day DATE,
doctor_id INT REFERENCES doctors (id),
on_call BOOL,
PRIMARY KEY (day, doctor_id)
);
```
6. Add two doctors to the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO doctors VALUES
(1, 'Abe'),
(2, 'Betty');
```
7. Insert one week's worth of data into the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO schedules VALUES
('2024-10-01', 1, true),
('2024-10-01', 2, true),
('2024-10-02', 1, true),
('2024-10-02', 2, true),
('2024-10-03', 1, true),
('2024-10-03', 2, true),
('2024-10-04', 1, true),
('2024-10-04', 2, true),
('2024-10-05', 1, true),
('2024-10-05', 2, true),
('2024-10-06', 1, true),
('2024-10-06', 2, true),
('2024-10-07', 1, true),
('2024-10-07', 2, true);
```
8. Confirm that at least one doctor is on call each day of the week:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT day, count(*) AS doctors_on_call FROM schedules
WHERE on_call = true
GROUP BY day
ORDER BY day;
```
```
day | doctors_on_call
------------+-----------------
2024-10-01 | 2
2024-10-02 | 2
2024-10-03 | 2
2024-10-04 | 2
2024-10-05 | 2
2024-10-06 | 2
2024-10-07 | 2
(7 rows)
```
## Step 2. Run the scenario on PostgreSQL
1. Doctor 1, Abe, starts to request leave for 10/5/18 using the hospital's schedule management application. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
2. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 1;
```
```
count
-------
1
(1 row)
```
3. Around the same time, doctor 2, Betty, starts to request leave for the same day using the hospital's schedule management application. In a new terminal, start a second SQL session:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ psql
```
4. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
5. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 2;
```
```
count
-------
1
(1 row)
```
6. In the terminal for doctor 1, since the previous check confirmed that another doctor is on call for 10/5/18, the application tries to update doctor 1's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 1;
```
7. In the terminal for doctor 2, since the previous check confirmed the same thing, the application tries to update doctor 2's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 2;
```
8. In the terminal for doctor 1, the application commits the transaction, despite the fact that the previous check (the `SELECT` query) is no longer true:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
9. In the terminal for doctor 2, the application commits the transaction, despite the fact that the previous check (the `SELECT` query) is no longer true:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
## Step 3. Check data correctness on PostgreSQL
So what just happened? Each transaction started by reading a value that, before the end of the transaction, became incorrect. Despite that fact, each transaction was allowed to commit. This is known as write skew, and the result is that 0 doctors are scheduled to be on call on 10/5/18.
To check this, in either terminal, run:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM schedules WHERE day = '2024-10-05';
```
```
day | doctor_id | on_call
------------+-----------+---------
2024-10-05 | 1 | f
2024-10-05 | 2 | f
(2 rows)
```
Again, this anomaly is the result of PostgreSQL's default isolation level of `READ COMMITTED`, but note that this would happen with any isolation level except `SERIALIZABLE` and some implementations of `REPEATABLE READ`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TRANSACTION_ISOLATION;
```
```
transaction_isolation
-----------------------
read committed
(1 row)
```
Exit each SQL shell with `\q` and then stop the PostgreSQL server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ pkill -9 postgres
```
## Step 4. Set up the scenario on CockroachDB
When you repeat the scenario on CockroachDB, you'll see that the anomaly is prevented by CockroachDB's `SERIALIZABLE` transaction isolation.
1. If you haven't already, install CockroachDB locally.
2. Use the `cockroach start-single-node` command to start a one-node CockroachDB cluster in insecure mode:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start-single-node \
--insecure \
--store=serializable-demo \
--listen-addr=localhost
```
3. In a new terminal window, open the built-in SQL client and connect to `localhost`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost
```
4. Create the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE doctors (
id INT PRIMARY KEY,
name TEXT
);
```
5. Create the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> CREATE TABLE schedules (
day DATE,
doctor_id INT REFERENCES doctors (id),
on_call BOOL,
PRIMARY KEY (day, doctor_id)
);
```
6. Add two doctors to the `doctors` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO doctors VALUES
(1, 'Abe'),
(2, 'Betty');
```
7. Insert one week's worth of data into the `schedules` table:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> INSERT INTO schedules VALUES
('2024-10-01', 1, true),
('2024-10-01', 2, true),
('2024-10-02', 1, true),
('2024-10-02', 2, true),
('2024-10-03', 1, true),
('2024-10-03', 2, true),
('2024-10-04', 1, true),
('2024-10-04', 2, true),
('2024-10-05', 1, true),
('2024-10-05', 2, true),
('2024-10-06', 1, true),
('2024-10-06', 2, true),
('2024-10-07', 1, true),
('2024-10-07', 2, true);
```
8. Confirm that at least one doctor is on call each day of the week:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT day, count(*) AS on_call FROM schedules
WHERE on_call = true
GROUP BY day
ORDER BY day;
```
```
day | on_call
-------------+----------
2024-10-01 | 2
2024-10-02 | 2
2024-10-03 | 2
2024-10-04 | 2
2024-10-05 | 2
2024-10-06 | 2
2024-10-07 | 2
(7 rows)
```
## Step 5. Run the scenario on CockroachDB
1. Doctor 1, Abe, starts to request leave for 10/5/18 using the hospital's schedule management application. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
2. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 1;
```
```
count
---------
1
(1 row)
```
3. Around the same time, doctor 2, Betty, starts to request leave for the same day using the hospital's schedule management application. In a new terminal, start a second SQL session:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --insecure --host=localhost
```
4. The application starts a transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> BEGIN;
```
5. The application checks to make sure at least one other doctor is on call for the requested date:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT count(*) FROM schedules
WHERE on_call = true
AND day = '2024-10-05'
AND doctor_id != 2;
```
```
count
---------
1
(1 row)
```
6. In the terminal for doctor 1, since the previous check confirmed that another doctor is on call for 10/5/18, the application tries to update doctor 1's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 1;
```
7. In the terminal for doctor 2, since the previous check confirmed the same thing, the application tries to update doctor 2's schedule:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> UPDATE schedules SET on_call = false
WHERE day = '2024-10-05'
AND doctor_id = 2;
```
8. In the terminal for doctor 1, the application tries to commit the transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
The transaction for doctor 1 is committed.
9. In the terminal for doctor 2, the application tries to commit the transaction:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> COMMIT;
```
Since CockroachDB uses `SERIALIZABLE` isolation, the database detects that the previous check (the `SELECT` query) is no longer true due to a concurrent transaction. It therefore prevents the transaction from committing, returning a retry error that indicates that the transaction must be attempted again.
```
ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to encountered recently written committed value /Table/105/1/20001/1/0 @1700513356.063385000,2): "sql txn" meta={id=10f4abbc key=/Table/105/1/20001/2/0 iso=Serializable pri=0.00167708 epo=0 ts=1700513366.194063000,2 min=1700513327.262632000,0 seq=1} lock=true stat=PENDING rts=1700513327.262632000,0 wto=false gul=1700513327.762632000,0
SQLSTATE: 40001
HINT: See: https://www.cockroachlabs.com/docs/v26.3/transaction-retry-error-reference.html#retry_serializable
```
For this kind of error, CockroachDB recommends a client-side transaction retry loop that would transparently observe that the one doctor cannot take time off because the other doctor already succeeded in asking for it. You can find generic transaction retry functions for various languages in the Develop with CockroachDB section.
For more information about the error message for the `RETRY_SERIALIZABLE` error type, see the Transaction Retry Error Reference.
## Step 6. Check data correctness on CockroachDB
1. In either terminal, confirm that one doctor is still on call for 10/5/18:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SELECT * FROM schedules WHERE day = '2024-10-05';
```
```
day | doctor_id | on_call
-------------+-----------+----------
2024-10-05 | 1 | f
2024-10-05 | 2 | t
(2 rows)
```
2. Again, the write skew anomaly was prevented by CockroachDB using the `SERIALIZABLE` isolation level:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW TRANSACTION_ISOLATION;
```
```
transaction_isolation
-------------------------
serializable
(1 row)
```
3. Exit the SQL shell in each terminal:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> \q
```
4. Exit each SQL shell with `\q` and then stop the node:
Get the process ID of the node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ps -ef | grep cockroach | grep -v grep
```
```
501 21691 1 0 6:19PM ttys001 0:01.15 cockroach start-single-node --insecure --store=serializable-demo --listen-addr=localhost
```
Gracefully shut down the node, specifying its process ID:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kill -TERM 21691
```
```
initiating graceful shutdown of server
server drained and shutdown completed
```
If you do not plan to restart the cluster, you may want to remove the node's data store:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ rm -rf serializable-demo
```
## What's next?
Explore other CockroachDB benefits and features:
* Replication & Rebalancing
* CockroachDB Resilience
* Serializable Transactions
You might also want to learn more about how transactions work in CockroachDB and in general:
* Transactions Overview
* [What Write Skew Looks Like](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/)
* Read Committed Transactions
# Deploy CockroachDB on AWS EC2
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-aws
This page shows you how to manually deploy a multi-node, self-hosted CockroachDB cluster on Amazon's AWS EC2 platform, using AWS's managed load-balancing service to distribute client traffic.
After setting up the AWS network, clock synchronization, and load balancing, it should take approximately 20 minutes to complete the deployment. This is based on initializing a three-node CockroachDB cluster in a single AWS region and running our sample workload.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
You need a license to use CockroachDB. Refer to the Licensing FAQ and [CockroachDB Pricing](https://www.cockroachlabs.com/pricing). [Contact us](https://cockroachlabs.com/contact-sales) about custom pricing through AWS Marketplace.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
CockroachDB is supported in all [AWS regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html).
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
* You should have familiarity with configuring the following AWS components:
* [Amazon VPC](https://docs.aws.amazon.com/vpc/index.html)
* [Subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html)
* [Internet gateways](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html)
* [Route tables](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html)
* [AWS VPN](https://aws.amazon.com/vpn/)
* [Virtual private gateways](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html)
* [Elastic Load Balancing](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html)
* All Amazon EC2 instances running CockroachDB should be members of the same [security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html). For an example, see [AWS architecture](#aws-architecture).
* Follow the [AWS IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) to harden the AWS environment. Use roles to grant access to the deployment, following a [policy of least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege).
* The [AWS root user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html) is *not* necessary.
* Review the [AWS Config service limits](https://docs.aws.amazon.com/config/latest/developerguide/configlimits.html) and contact AWS support to request a quota increase, if necessary.
### AWS architecture
In this basic deployment, 3 CockroachDB nodes are each deployed on an Amazon EC2 instance across 3 availability zones. These are grouped within a single VPC and security group. Users are routed to the cluster via [Amazon Route 53](https://aws.amazon.com/route53/) (which is not used in this tutorial) and a load balancer.
![Architecture diagram for a three-node CockroachDB cluster deployed on AWS]()
## Step 1. Create instances
Open the [Amazon EC2 console](https://console.aws.amazon.com/ec2/) and [launch an instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/launching-instance.html#launch-instance-console) for each node you plan to have in your cluster. If you plan to [run our sample workload](#step-9-run-a-sample-workload) against the cluster, create a separate instance for that workload.
* Run at least 3 nodes to ensure survivability.
* Your instances will rely on Amazon Time Sync Service for clock synchronization. When choosing an AMI, note that some machines are preconfigured to use Amazon Time Sync Service (e.g., Amazon Linux AMIs) and others are not.
* Use general-purpose [`m6i` or `m6a`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose-instances.html) VMs with SSD-backed [EBS volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html). For example, Cockroach Labs has used `m6i.2xlarge` for performance benchmarking. If your workload requires high throughput, use network-optimized `m5n` instances. To simulate bare-metal deployments, use `m5d` with [SSD Instance Store volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html).
* `m5` and `m5a` instances, and [compute-optimized `c5`, `c5a`, and `c5n`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compute-optimized-instances.html) instances, are also acceptable.
**Do not** use [burstable performance instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html), which limit the load on a single core.
* Note the ID of the VPC you select. You will need to look up its IP range when setting inbound rules for your security group.
* Make sure all your instances are in the same security group.
* If you are creating a new security group, add the [inbound rules](#step-2-configure-your-network) from the next step. Otherwise note the ID of the security group.
* When creating the instance, you will be prompted to specify an EC2 key pair. For more information on key pairs, see the [AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html). These are used to securely connect to your instances and should be encrypted (e.g., `ssh-keygen -p -f $keypairfile` in Linux).
For more details, see Hardware Recommendations and Cluster Topology.
## Step 2. Configure your network
[Add Custom TCP inbound rules](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html#adding-security-group-rule) to your security group to allow TCP communication on two ports:
* `26257` for inter-node and client-node communication. This enables the nodes to work as a cluster, the load balancer to route traffic to the nodes, and applications to connect to the load balancer.
* `8080` for exposing the DB Console to the user, and for routing the load balancer to the health check endpoint.
#### Inter-node and load balancer-node communication
| Field | Value |
| - | - |
| Port Range | **26257** |
| Source | The ID of your security group (e.g., *sg-07ab277a*) |
#### Application data
| Field | Value |
| - | - |
| Port Range | **26257** |
| Source | Your application's IP ranges |
#### DB Console
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | Your network's IP ranges |
You can set your network IP by selecting "My IP" in the Source field.
#### Load balancer-health check communication
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | The IP range of your VPC in CIDR notation (e.g., 10.12.0.0/16) |
To get the IP range of a VPC, open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and find the VPC listed in the section called Your VPCs.
#### Load balancer-health check communication
| Field | Value |
| - | - |
| Port Range | **8080** |
| Source | The IP range of your VPC in CIDR notation (e.g., 10.12.0.0/16) |
To get the IP range of a VPC, open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and find the VPC listed in the section called Your VPCs.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
AWS offers fully-managed load balancing to distribute traffic between instances.
1. [Add AWS load balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancer-getting-started.html). Be sure to:
* Select a **Network Load Balancer** and use the ports we specify below.
* Select the VPC and *all* availability zones of your instances. This is important, as you cannot change the availability zones once the load balancer is created. The availability zone of an instance is determined by its subnet, found by inspecting the instance in the Amazon EC2 Console.
* Set the load balancer port to **26257**.
* Create a new target group that uses TCP port **26257**. Traffic from your load balancer is routed to this target group, which contains your instances.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
* Register your instances with the target group you created, specifying port **26257**. You can add and remove instances later.
2. To test load balancing and connect your application to the cluster, you will need the provisioned internal (private) **IP address** for the load balancer. To find this, open the Network Interfaces section of the Amazon EC2 console and look up the load balancer by its name.
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test your cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
In the Target Groups section of the Amazon EC2 console, [check the health](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-health-checks.html) of your instances by inspecting your target group and opening the Targets tab.
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
Before adding a new node, [create a new instance](#step-1-create-instances) as you did earlier. Then [generate and upload a certificate and key](#step-5-generate-certificates) for the new node.
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Digital Ocean
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-digital-ocean
This page shows you how to deploy a secure multi-node CockroachDB cluster on Digital Ocean, using Digital Ocean's managed load balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
* If all of your CockroachDB nodes and clients will run on Droplets in a single region, consider using [private networking](https://docs.digitalocean.com/products/networking/vpc/how-to/create/).
## Step 1. Create Droplets
[Create Droplets](https://www.digitalocean.com/community/tutorials/how-to-create-your-first-digitalocean-droplet) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate Droplet for that workload.
* Run at least 3 nodes to ensure survivability.
* Use any [droplets](https://www.digitalocean.com/pricing/) except standard droplets with only 1 GB of RAM, which is below our minimum requirement. All Digital Ocean droplets use SSD storage.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 2. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 3. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
Digital Ocean offers fully-managed load balancers to distribute traffic between Droplets.
1. [Create a Digital Ocean Load Balancer](https://www.digitalocean.com/community/tutorials/an-introduction-to-digitalocean-load-balancers). Be sure to:
* Set forwarding rules to route TCP traffic from the load balancer's port **26257** to port **26257** on the node Droplets.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
2. Note the provisioned **IP Address** for the load balancer. You'll use this later to test load balancing and to connect your application to the cluster.
## Step 4. Configure your network
Set up a firewall for each of your Droplets, allowing TCP communication on the following two ports:
* **26257** (`tcp:26257`) for inter-node communication (i.e., working as a cluster), for applications to connect to the load balancer, and for routing from the load balancer to nodes
* **8080** (`tcp:8080`) for exposing your DB Console
For guidance, you can use Digital Ocean's guide to configuring firewalls based on the Droplet's OS:
* Ubuntu and Debian can use [`ufw`](https://www.digitalocean.com/community/tutorials/how-to-setup-a-firewall-with-ufw-on-an-ubuntu-and-debian-cloud-server).
* FreeBSD can use [`ipfw`](https://www.digitalocean.com/community/tutorials/recommended-steps-for-new-freebsd-10-1-servers).
* Fedora can use [`iptables`](https://www.digitalocean.com/community/tutorials/initial-setup-of-a-fedora-22-server).
* CoreOS can use [`iptables`](https://www.digitalocean.com/community/tutorials/how-to-secure-your-coreos-cluster-with-tls-ssl-and-firewall-rules).
* CentOS can use [`firewalld`](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-firewall-using-firewalld-on-centos-7).
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Google Cloud Platform GCE
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-google-cloud-platform
This page shows you how to manually deploy a secure multi-node CockroachDB cluster on Google Cloud Platform's Compute Engine (GCE), using Google's TCP Proxy Load Balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
* This article covers the use of Linux instances with GCE. You may wish to review the instructions for [connecting to Windows instances](https://cloud.google.com/compute/docs/instances/connecting-to-instance#windows).
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Configure your network
CockroachDB requires TCP communication on two ports:
* `26257` for inter-node communication (i.e., working as a cluster)
* `8080` for exposing your DB Console
To expose your DB Console and allow traffic from the TCP proxy load balancer and health checker to your instances, [create firewall rules](https://cloud.google.com/compute/docs/vpc/firewalls) for your project. When creating firewall rules, we recommend using Google Cloud Platform's **tag** feature to apply the rule only to instances with the same tag.
#### DB Console
| Field | Recommended Value |
| - | - |
| Name | **cockroachadmin** |
| Source filter | IP ranges |
| Source IP ranges | Your local network's IP ranges |
| Allowed protocols... | **tcp:8080** |
| Target tags | **cockroachdb** |
#### Application data
Applications will not connect directly to your CockroachDB nodes. Instead, they'll connect to GCE's TCP Proxy Load Balancing service, which automatically routes traffic to the instances that are closest to the user. Because this service is implemented at the edge of the Google Cloud, you'll need to create a firewall rule to allow traffic from the load balancer and health checker to your instances. This is covered in [Step 4](#step-4-set-up-load-balancing).
## Step 2. Create instances
[Create an instance](https://cloud.google.com/compute/docs/instances/create-start-instance) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate instance for that workload.
* Run at least 3 nodes to ensure survivability.
* Use general-purpose [`t2d-standard`, `n2-standard`, or `n2d-standard`](https://cloud.google.com/compute/pricing#predefined_machine_types) VMs, or use [custom VMs](https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type). For example, Cockroach Labs has used `t2d-standard-8`, `n2-standard-8`, and `n2d-standard-8` for performance benchmarking.
Do not use `f1` or `g1` [shared-core machines](https://cloud.google.com/compute/docs/machine-types#sharedcore), which limit the load on CPU resources.
* If you used a tag for your firewall rules, when you create the instance, click **Management, security, disks, networking, sole tenancy**. Under the **Networking** tab, in the **Network tags** field, enter **cockroachdb**.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
GCE offers fully-managed [TCP Proxy Load Balancing](https://cloud.google.com/load-balancing/docs/tcp/). This service lets you use a single IP address for all users around the world, automatically routing traffic to the instances that are closest to the user.
When using TCP Proxy Load Balancing, you cannot use firewall rules to control access to the load balancer. If you need such control, consider using [Network TCP Load Balancing](https://cloud.google.com/compute/docs/load-balancing/network/) instead, but note that it cannot be used across regions. You might also consider using the HAProxy load balancer (see the On-Premises tutorial for guidance).
To use GCE's TCP Proxy Load Balancing service:
1. For each zone in which you're running an instance, [create a distinct instance group](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances).
* To ensure that the load balancer knows where to direct traffic, specify a port name mapping, with `tcp26257` as the **Port name** and `26257` as the **Port number**.
2. [Add the relevant instances to each instance group](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances#addinstances).
3. [Configure TCP Proxy Load Balancing](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#configure_load_balancer).
* During backend configuration, create a health check, setting the **Protocol** to `HTTPS`, the **Port** to `8080`, and the **Request path** to path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
* If you want to maintain long-lived SQL connections that may be idle for more than tens of seconds, increase the backend timeout setting accordingly.
* During frontend configuration, reserve a static IP address and note the IP address and the port you select. You'll use this address and port for all client connections.
4. [Create a firewall rule](https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#config-hc-firewall) to allow traffic from the load balancer and health checker to your instances. This is necessary because TCP Proxy Load Balancing is implemented at the edge of the Google Cloud.
* Be sure to set **Source IP ranges** to `130.211.0.0/22` and `35.191.0.0/16` and set **Target tags** to `cockroachdb` (not to the value specified in the linked instructions).
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
By default, inter-node communication uses the internal IP addresses of your GCE instances.
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB on Microsoft Azure
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-microsoft-azure
This page shows you how to manually deploy a secure multi-node CockroachDB cluster on Microsoft Azure, using Azure's managed load balancing service to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Configure your network
CockroachDB requires TCP communication on two ports:
* **26257** (`tcp:26257`) for inter-node communication (i.e., working as a cluster), for applications to connect to the load balancer, and for routing from the load balancer to nodes
* **8080** (`tcp:8080`) for exposing your DB Console
To enable this in Azure, you must create a Resource Group, Virtual Network, and Network Security Group.
1. [Create a Resource Group](https://azure.microsoft.com/updates/create-empty-resource-groups/).
2. [Create a Virtual Network](https://docs.microsoft.com/azure/virtual-network/virtual-networks-create-vnet-arm-pportal) that uses your **Resource Group**.
3. [Create a Network Security Group](https://docs.microsoft.com/azure/virtual-network/virtual-networks-create-nsg-arm-pportal) that uses your **Resource Group**, and then add the following **inbound** rules to it:
* **DB Console support**:
| Field | Recommended Value |
| - | - |
| Name | **cockroachadmin** |
| Source | **IP Addresses** |
| Source IP addresses/CIDR ranges | Your local network’s IP ranges |
| Source port ranges | \* |
| Destination | **Any** |
| Destination port range | **8080** |
| Protocol | **TCP** |
| Action | **Allow** |
| Priority | Any value > 1000 |
* **Application support**:
| Field | Recommended Value |
| - | - |
| Name | **cockroachapp** |
| Source | **IP Addresses** |
| Source IP addresses/CIDR ranges | Your local network’s IP ranges |
| Source port ranges | \* |
| Destination | **Any** |
| Destination port range | **26257** |
| Protocol | **TCP** |
| Action | **Allow** |
| Priority | Any value > 1000 |
## Step 2. Create VMs
[Create Linux VMs](https://docs.microsoft.com/azure/virtual-machine-scale-sets/quick-create-portal) for each node you plan to have in your cluster. If you plan to run a sample workload against the cluster, create a separate VM for that workload.
* Run at least 3 nodes to ensure survivability.
* Use general-purpose [Dsv5-series](https://docs.microsoft.com/azure/virtual-machines/dv5-dsv5-series) and [Dasv5-series](https://docs.microsoft.com/azure/virtual-machines/dasv5-dadsv5-series) or memory-optimized [Ev5-series](https://docs.microsoft.com/azure/virtual-machines/ev5-esv5-series) and [Easv5-series](https://docs.microsoft.com/azure/virtual-machines/easv5-eadsv5-series#easv5-series) VMs. For example, Cockroach Labs has used `Standard_D8s_v5`, `Standard_D8as_v5`, `Standard_E8s_v5`, and `Standard_e8as_v5` for performance benchmarking.
* Compute-optimized [F-series](https://docs.microsoft.com/azure/virtual-machines/fsv2-series) VMs are also acceptable.
Do not use ["burstable" B-series](https://docs.microsoft.com/azure/virtual-machines/linux/b-series-burstable) VMs, which limit the load on CPU resources. Also, Cockroach Labs has experienced data corruption issues on A-series VMs, so we recommend avoiding those as well.
* When creating the VMs, make sure to select the **Resource Group**, **Virtual Network**, and **Network Security Group** you created.
For more details, see Hardware Recommendations and Cluster Topology.
## Step 3. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 4. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
Microsoft Azure offers fully-managed load balancing to distribute traffic between instances.
1. [Add Azure load balancing](https://docs.microsoft.com/azure/load-balancer/load-balancer-overview). Be sure to:
* Set forwarding rules to route TCP traffic from the load balancer's port **26257** to port **26257** on the nodes.
* Configure health checks to use HTTP port **8080** and path `/health?ready=1`. This health endpoint ensures that load balancers do not direct traffic to nodes that are live but not ready to receive requests.
1. Note the provisioned **IP Address** for the load balancer. You'll use this later to test load balancing and to connect your application to the cluster.
## Step 5. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 6. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 7. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 8. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 9. Run a sample workload
CockroachDB comes with a number of built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 10. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 11. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2. Install CockroachDB for Linux.
3. Run the `cockroach start` command, passing the new node's address as the `--advertise-addr` flag and pointing `--join` to the three existing nodes (also include `--locality` if you set it earlier).
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service):
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/master/_includes/v26.3/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
9. Repeat these steps for each additional node that you want in your cluster.
## Step 12. Use the database
Now that your deployment is working, you can:
1. Implement your data model.
2. Create users and grant them privileges.
3. Connect your application. Be sure to connect your application to the load balancer, not to a CockroachDB node.
4. Take backups of your data.
You may also want to adjust the way the cluster replicates data. For example, by default, a multi-node cluster replicates all data 3 times; you can change this replication factor or create additional rules for replicating individual databases and tables differently. For more information, see Replication Controls.
When running a cluster of 5 nodes or more, it's safest to increase the replication factor for important internal data to 5, even if you do not do so for user data. For the cluster as a whole to remain available, the ranges for this internal data must always retain a majority of their replicas.
## See also
* Production Checklist
* Manual Deployment
* Orchestrated Deployment
* Monitoring and Alerting
* Performance Tuning
* Local Deployment
# Deploy CockroachDB On-Premises
Source: https://docs.cockroachlabs.com/docs/v26.3/deploy-cockroachdb-on-premises
This tutorial shows you how to manually deploy a secure multi-node CockroachDB cluster on multiple machines, using [HAProxy](http://www.haproxy.org/) load balancers to distribute client traffic.
If you are only testing CockroachDB, or you are not concerned with protecting network communication with TLS encryption, you can use an insecure cluster instead. Select **Insecure** above for instructions.
To try CockroachDB Cloud instead of running CockroachDB yourself, refer to the Cloud Quickstart.
## Before you begin
### Requirements
* You must have CockroachDB installed locally. This is necessary for generating and managing your deployment's certificates.
* You must have [SSH access]() to each machine. This is necessary for distributing and starting CockroachDB binaries.
* Your network configuration must allow TCP communication on the following ports:
* `26257` for intra-cluster and client-cluster communication
* `8080` to expose your DB Console
* Carefully review the Production Checklist, including supported hardware and software, and the recommended Topology Patterns.
* Do not run multiple node processes on the same VM or machine. This defeats CockroachDB's replication and causes the system to be a single point of failure. Instead, start each node on a separate VM or machine.
* To start a node with multiple disks or SSDs, provide a separate `--store` flag for each disk when starting the `cockroach` process on the node. For more details about stores, see Start a Node.
If you start a node with multiple `--store` flags, it is not possible to scale back down to only using a single store on the node. Instead, you must decommission the node and start a new node with the updated `--store`.
* When starting each node, use the `--locality` flag to describe the node's location, for example, `--locality=region=west,zone=us-west-1`. The key-value pairs should be ordered from most to least inclusive, and the keys and order of key-value pairs must be the same on all nodes.
* When deploying in a single availability zone:
* To be able to tolerate the failure of any 1 node, use at least 3 nodes with the `default` 3-way replication factor. In this case, if 1 node fails, each range retains 2 of its 3 replicas, a majority.
* To be able to tolerate 2 simultaneous node failures, use at least 5 nodes and increase the `default` replication factor for user data to 5. The replication factor for important internal data is 5 by default, so no adjustments are needed for internal data. In this case, if 2 nodes fail at the same time, each range retains 3 of its 5 replicas, a majority.
* When deploying across multiple availability zones:
* To be able to tolerate the failure of 1 entire AZ in a region, use at least 3 AZs per region and set `--locality` on each node to spread data evenly across regions and AZs. In this case, if 1 AZ goes offline, the 2 remaining AZs retain a majority of replicas.
* To ensure that ranges are split evenly across nodes, use the same number of nodes in each AZ. This is to avoid overloading any nodes with excessive resource consumption.
* When deploying across multiple regions:
* To be able to tolerate the failure of 1 entire region, use at least 3 regions.
### Recommendations
* Decide how you want to access your DB Console:
| Access Level | Description |
| - | - |
| Partially open | Set a firewall rule to allow only specific IP addresses to communicate on port `8080`. |
| Completely open | Set a firewall rule to allow all IP addresses to communicate on port `8080`. |
| Completely closed | Set a firewall rule to disallow all communication on port `8080`. In this case, a machine with SSH access to a node could use an SSH tunnel to access the DB Console. |
## Step 1. Synchronize clocks
CockroachDB requires moderate levels of clock synchronization to preserve data consistency. For this reason, when a node detects that its clock is out of sync with at least half of the other nodes in the cluster by 80% of the maximum offset allowed (500ms by default), it spontaneously shuts down. This avoids the risk of consistency anomalies, but it's best to prevent clocks from drifting too far in the first place by running clock synchronization software on each node.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here, but other methods of clock synchronization are suitable as well.
1. SSH to the first machine.
2. Disable `timesyncd`, which tends to be active by default on some Linux distributions:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo timedatectl set-ntp no
```
Verify that `timesyncd` is off:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ timedatectl
```
Look for `Network time on: no` or `NTP enabled: no` in the output.
3. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
4. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
5. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
1. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
2. Repeat these steps for each machine where a CockroachDB node will run.
Compute Engine instances are preconfigured to use [NTP](http://www.ntp.org/), which should keep offsets in the single-digit milliseconds. However, Google can’t predict how external NTP services, such as `pool.ntp.org`, will handle the leap second. Therefore, you should:
* [Configure each GCE instance to use Google's internal NTP service](https://cloud.google.com/compute/docs/instances/configure-ntp#configure_ntp_for_your_instances).
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
Amazon provides the [Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), which uses a fleet of satellite-connected and atomic reference clocks in each AWS Region to deliver accurate current time readings. The service also smears the leap second.
* [Configure each AWS instance to use the internal Amazon Time Sync Service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html#configure-amazon-time-service).
* Per the above instructions, ensure that `etc/chrony.conf` on the instance contains the line `server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4` and that other `server` or `pool` lines are commented out.
* To verify that Amazon Time Sync Service is being used, run `chronyc sources -v` and check for a line containing `* 169.254.169.123`. The `*` denotes the preferred time server.
* If you plan to run a hybrid cluster across GCE and other cloud providers or environments, note that all of the nodes must be synced to the same time source, or to different sources that implement leap second smearing in the same way. See the Production Checklist for details.
[`ntpd`](http://doc.ntp.org/) should keep offsets in the single-digit milliseconds, so that software is featured here. However, to run `ntpd` properly on Azure VMs, it's necessary to first unbind the Time Synchronization device used by the Hyper-V technology running Azure VMs; this device aims to synchronize time between the VM and its host operating system but has been known to cause problems.
1. SSH to the first machine.
2. Find the ID of the Hyper-V Time Synchronization device:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ curl -O https://raw.githubusercontent.com/torvalds/linux/master/tools/hv/lsvmbus
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ python lsvmbus -vv | grep -w "Time Synchronization" -A 3
```
```
VMBUS ID 12: Class_ID = {9527e630-d0ae-497b-adce-e80ab0175caf} - [Time Synchronization]
Device_ID = {2dd1ce17-079e-403c-b352-a1921ee207ee}
Sysfs path: /sys/bus/vmbus/devices/2dd1ce17-079e-403c-b352-a1921ee207ee
Rel_ID=12, target_cpu=0
```
3. Unbind the device, using the `Device_ID` from the previous command's output:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ echo | sudo tee /sys/bus/vmbus/drivers/hv_utils/unbind
```
4. Install the `ntp` package:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo apt-get install ntp
```
5. Stop the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp stop
```
6. Sync the machine's clock with Google's NTP service:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpd -b time.google.com
```
To make this change permanent, in the `/etc/ntp.conf` file, remove or comment out any lines starting with `server` or `pool` and add the following lines:
```
server time1.google.com iburst
server time2.google.com iburst
server time3.google.com iburst
server time4.google.com iburst
```
Restart the NTP daemon:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo service ntp start
```
We recommend Google's NTP service because it handles ["smearing" the leap second](https://developers.google.com/time/smear). If you use a different NTP service that doesn't smear the leap second, be sure to configure client-side smearing in the same way on each machine. See the Production Checklist for details.
7. Verify that the machine is using a Google NTP server:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ sudo ntpq -p
```
The active NTP server will be marked with an asterisk.
8. Repeat these steps for each machine where a CockroachDB node will run.
## Step 2. Generate certificates
You can use `cockroach cert` commands or `openssl` commands to generate security certificates. This section features the `cockroach cert` commands.
Locally, you'll need to create the following certificates and keys:
* A certificate authority (CA) key pair (`ca.crt` and `ca.key`).
* A node key pair for each node, issued to its IP addresses and any common names the machine uses, as well as to the IP addresses and common names for machines running load balancers.
* A client key pair for the `root` user. You'll use this to run a sample workload against the cluster as well as some `cockroach` client commands from your local machine.
1. Install CockroachDB on your local machine, if you haven't already.
2. Create two directories:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir certs
```
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir my-safe-directory
```
* `certs`: You'll generate your CA certificate and all node and client certificates and keys in this directory and then upload some of the files to your nodes.
* `my-safe-directory`: You'll generate your CA key in this directory and then reference the key when generating node and client certificates. After that, you'll keep the key safe and secret; you will not upload it to your nodes.
3. Create the CA certificate and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-ca \
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
4. Create the certificate and key for the first node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
5. Upload the CA certificate and node certificate and key to the first node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ gcloud compute ssh \
--project \
--command "mkdir certs"
```
`gcloud compute ssh` associates your public SSH key with the GCP project and is only needed when connecting to the first node. See the [GCP docs](https://cloud.google.com/sdk/gcloud/reference/compute/ssh) for more details.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ scp certs/ca.crt \
certs/node.crt \
certs/node.key \
@@@@@
This is necessary because the certificates and keys for additional nodes will also be named `node.crt` and `node.key`. As an alternative to deleting these files, you can run the next `cockroach cert create-node` commands with the `--overwrite` flag.
1. Create the certificate and key for the second node, issued to all common names you might use to refer to the node as well as to the load balancer instances:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach cert create-node \
\
\
\
\
localhost \
127.0.0.1 \
\
\
\
--certs-dir=certs \
--ca-key=my-safe-directory/ca.key
```
2. Upload the CA certificate and node certificate and key to the second node:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ ssh @@@@@@`cockroach` client commands from your local machine. If you might also want to run `cockroach` client commands directly on a node (e.g., for local debugging), you'll need to copy the `root` user's certificate and key to that node as well.
On accessing the DB Console in a later step, your browser will consider the CockroachDB-created certificate invalid and you’ll need to click through a warning message to get to the UI. You can avoid this issue by using a certificate issued by a public CA.
## Step 3. Start nodes
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. Visit Releases and download the full binary of CockroachDB to the node.
2. On the node, follow the instructions to install CockroachDB.
3. Run the `cockroach start` command:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--certs-dir=certs \
--advertise-addr=
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
| `--cache`
`--max-sql-memory` | Increases the node's cache size to 25% of available system memory to improve read performance. The capacity for in-memory SQL processing defaults to 25% of system memory but can be raised, if necessary, to increase the number of simultaneous client connections allowed by the node as well as the node's capacity for in-memory processing of rows when using `ORDER BY`, `GROUP BY`, `DISTINCT`, joins, and window functions. For more details, see Cache and SQL Memory Size. |
| `--background` | Starts the node in the background so you gain control of the terminal to issue more commands. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=Start a Node.
Repeat these steps for each additional node that you want in your cluster.
For each initial node of your cluster, complete the following steps:
After completing these steps, nodes will not yet be live. They will complete the startup process and join together to form a cluster as soon as the cluster is initialized in the next step.
1. SSH to the machine where you want the node to run. Ensure you are logged in as the `root` user.
2. Install CockroachDB for Linux.
3. Create the Cockroach directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mkdir /var/lib/cockroach
```
4. Create a Unix user named `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ useradd cockroach
```
5. Move the `certs` directory to the `cockroach` directory.
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ mv certs /var/lib/cockroach/
```
6. Change the ownership of the `cockroach` directory to the user `cockroach`:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ chown -R cockroach /var/lib/cockroach
```
7. Download the [sample configuration template](https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v26.3/prod-deployment/securecockroachdb.service) and save the file in the `/etc/systemd/system/` directory:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
curl -o securecockroachdb.service https://raw.githubusercontent.com/cockroachlabs/docs/main/src/current/_includes/v23.2/prod-deployment/securecockroachdb.service
```
Alternatively, you can create the file yourself and copy the script into it:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
[Unit]
Description=Cockroach Database cluster node
Requires=network.target
[Service]
Type=notify
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start --certs-dir=certs --advertise-addr=\
This value must route to an IP address the node is listening on (with `--listen-addr` unspecified, the node listens on all IP addresses).
In some networking scenarios, you may need to use `--advertise-addr` and/or `--listen-addr` differently. For more details, see Networking. |
| `--join` | Identifies the address of 3-5 of the initial nodes of the cluster. These addresses should match the addresses that the target nodes are advertising. |
When deploying across multiple datacenters, or when there is otherwise high latency between nodes, it is recommended to set `--locality` as well. For more details, see Locality.
For other flags not explicitly set, the command uses default values. For example, the node stores data in `--store=cockroach-data` and binds DB Console HTTP requests to `--http-addr=localhost:8080`. To set these options manually, see Start a Node.
9. Start the CockroachDB cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl start securecockroachdb
```
10. Configure `systemd` to start CockroachDB automatically after a reboot:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
systemctl enable securecockroachdb
```
11. Repeat these steps for each additional node that you want in your cluster.
`systemd` handles node restarts in case of node failure. To stop a node without `systemd` restarting it, run `systemctl stop securecockroachdb`
## Step 4. Initialize the cluster
On your local machine, run the `cockroach init` command to complete the node startup process and have them join together as a cluster:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach init --certs-dir=certs --host=standard output, such as the CockroachDB version, the URL for the DB Console, and the SQL URL for clients.
## Step 5. Test the cluster
CockroachDB replicates and distributes data behind-the-scenes and uses a [Gossip protocol](https://wikipedia.org/wiki/Gossip_protocol) to enable each node to locate data across the cluster. Once a cluster is live, any node can be used as a SQL gateway.
When using a load balancer, you should issue commands directly to the load balancer, which then routes traffic to the nodes.
Use the built-in SQL client locally as follows:
1. On your local machine, launch the built-in SQL client, with the `--host` flag set to the address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach sql --certs-dir=certs --host= CREATE DATABASE securenodetest;
```
3. View the cluster's databases, which will include `securenodetest`:
```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
> SHOW DATABASES;
```
```
+--------------------+
| Database |
+--------------------+
| crdb_internal |
| information_schema |
| securenodetest |
| pg_catalog |
| system |
+--------------------+
(5 rows)
```
4. Use `\q` to exit the SQL shell.
## Step 6. Set up load balancing
Each CockroachDB node is an equally suitable SQL gateway to your cluster, but to ensure client performance and reliability, it's important to use load balancing:
* **Performance:** Load balancers spread client traffic across nodes. This prevents any one node from being overwhelmed by requests and improves overall cluster performance (queries per second).
* **Reliability:** Load balancers decouple client health from the health of a single CockroachDB node. In cases where a node fails, the load balancer redirects client traffic to available nodes.
[HAProxy](http://www.haproxy.org/) is one of the most popular open-source TCP load balancers, and CockroachDB includes a built-in command for generating a configuration file that is preset to work with your running cluster, so we feature that tool here.
1. On your local machine, run the `cockroach gen haproxy` command with the `--host` flag set to the address of any node and security flags pointing to the CA cert and the client cert and key:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
cockroach gen haproxy \
--certs-dir=certs \
--host=:26257 check port 8080
server cockroach2 :26257 check port 8080
server cockroach3 :26257 check port 8080
```
The file is preset with the minimal [configurations](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html) needed to work with your running cluster:
| Field | Description |
| - | - |
| `timeout connect`
`timeout client`
`timeout server` | Timeout values that should be suitable for most deployments. |
| `bind` | The port that HAProxy listens on. This is the port clients will connect to and thus needs to be allowed by your network configuration.
This tutorial assumes HAProxy is running on a separate machine from CockroachDB nodes. If you run HAProxy on the same machine as a node (not recommended), you'll need to change this port, as `26257` is likely already being used by the CockroachDB node. |
| `balance` | The balancing algorithm. This is set to `roundrobin` to ensure that connections get rotated amongst nodes (connection 1 on node 1, connection 2 on node 2, etc.). Check the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4-balance) for details about this and other balancing algorithms. |
| `option httpchk` | The HTTP endpoint that HAProxy uses to check node health. `/health?ready=1` ensures that HAProxy doesn't direct traffic to nodes that are live but not ready to receive requests. |
| `server` | For each included node, this field specifies the address the node advertises to other nodes in the cluster, i.e., the addressed pass in the `--advertise-addr` flag on node startup. Make sure hostnames are resolvable and IP addresses are routable from HAProxy. |
For full details on these and other configuration settings, see the [HAProxy Configuration Manual](http://cbonte.github.io/haproxy-dconv/1.7/configuration.html).
1. Upload the `haproxy.cfg` file to the machine where you want to run HAProxy:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
scp haproxy.cfg @built-in workloads for simulating client traffic. This step features CockroachDB's version of the [TPC-C](http://www.tpc.org/tpcc/) workload.
Be sure that you have configured your network to allow traffic from the application to the load balancer. In this case, you will run the sample workload on one of your machines. The traffic source should therefore be the **internal (private)** IP address of that machine.
1. SSH to the machine where you want to run the sample TPC-C workload.
This should be a machine that is not running a CockroachDB node, and it should already have a `certs` directory containing `ca.crt`, `client.root.crt`, and `client.root.key` files.
2. Install CockroachDB for Linux.
3. Use the `cockroach workload` command to load the initial schema and data, pointing it at the IP address of the load balancer:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload init tpcc \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
4. Use the `cockroach workload` command to run the workload for 10 minutes:
```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach workload run tpcc \
--duration=10m \
'postgresql://root@:26257/tpcc?sslmode=verify-full&sslrootcert=certs/ca.crt&sslcert=certs/client.root.crt&sslkey=certs/client.root.key'
```
You'll see per-operation statistics print to standard output every second:
```
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
1s 0 1443.4 1494.8 4.7 9.4 27.3 67.1 transfer
2s 0 1686.5 1590.9 4.7 8.1 15.2 28.3 transfer
3s 0 1735.7 1639.0 4.7 7.3 11.5 28.3 transfer
4s 0 1542.6 1614.9 5.0 8.9 12.1 21.0 transfer
5s 0 1695.9 1631.1 4.7 7.3 11.5 22.0 transfer
6s 0 1569.2 1620.8 5.0 8.4 11.5 15.7 transfer
7s 0 1614.6 1619.9 4.7 8.1 12.1 16.8 transfer
8s 0 1344.4 1585.6 5.8 10.0 15.2 31.5 transfer
9s 0 1351.9 1559.5 5.8 10.0 16.8 54.5 transfer
10s 0 1514.8 1555.0 5.2 8.1 12.1 16.8 transfer
...
```
After the specified duration (10 minutes in this case), the workload will stop and you'll see totals printed to standard output:
```
_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
600.0s 0 823902 1373.2 5.8 5.5 10.0 15.2 209.7
```
For more `tpcc` options, use `cockroach workload run tpcc --help`. For details about other workloads built into the `cockroach` binary, use `cockroach workload --help`.
1. To monitor the load generator's progress, open the DB Console by pointing a browser to the address in the `admin` field in the standard output of any node on startup.
Since the load generator is pointed at the load balancer, the connections will be evenly distributed across nodes. To verify this, click **Metrics** on the left, select the **SQL** dashboard, and then check the **SQL Connections** graph. You can use the **Graph** menu to filter the graph for specific nodes.
## Step 8. Monitor the cluster
Despite CockroachDB's various built-in safeguards against failure, it is critical to actively monitor the overall health and performance of a cluster running in production and to create alerting rules that promptly send notifications when there are events that require investigation or intervention.
For details about available monitoring options and the most important events and metrics to alert on, see Monitoring and Alerting.
## Step 9. Scale the cluster
You can start the nodes manually or automate the process using [systemd](https://www.freedesktop.org/wiki/Software/systemd/).
For each additional node you want to add to the cluster, complete the following steps:
1. SSH to the machine where you want the node to run.
2.