Deploy a Local Cluster in Kubernetes

1. Install the Helm client (version 3.0 or higher) and add the cockroachdb chart repository:
   $ helm repo add cockroachdb https://charts.cockroachdb.com/
   
   "cockroachdb" has been added to your repositories
   
2. Update your Helm chart repositories to ensure that you’re using the latest CockroachDB chart:
   $ helm repo update
   
3. The cluster configuration is set in the Helm chart’s values file.

1. Create a local YAML file (e.g., my-values.yaml ) to specify your custom values. These will be used to override the defaults in values.yaml.
2. To avoid running out of memory when CockroachDB is not the only pod on a Kubernetes node, you must set memory limits explicitly. This is because CockroachDB does not detect the amount of memory allocated to its pod when run in Kubernetes. We recommend setting conf.cache and conf.max-sql-memory each to 1/4 of the memory allocation specified in statefulset.resources.requests and statefulset.resources.limits.

conf:
  cache: "2Gi"
  max-sql-memory: "2Gi"

4. Install the CockroachDB Helm chart, specifying your custom values file. Provide a “release” name to identify and track this particular deployment of the chart, and override the default values with those in my-values.yaml.

$ helm install my-release --values {custom-values}.yaml cockroachdb/cockroachdb

5. Confirm that CockroachDB cluster initialization has completed successfully, with the pods for CockroachDB showing 1/1 under READY and the pod for initialization showing COMPLETED under STATUS:
   $ kubectl get pods
   
   NAME                                READY     STATUS      RESTARTS   AGE
   my-release-cockroachdb-0            1/1       Running     0          8m
   my-release-cockroachdb-1            1/1       Running     0          8m
   my-release-cockroachdb-2            1/1       Running     0          8m
   my-release-cockroachdb-init-hxzsc   0/1       Completed   0          1h
   
6. Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:
   $ kubectl get pv
   
   NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                      STORAGECLASS   REASON    AGE
   pvc-71019b3a-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-0   standard                 11m
   pvc-7108e172-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-1   standard                 11m
   pvc-710dcb66-fc67-11e8-a606-080027ba45e5   100Gi      RWO            Delete           Bound     default/datadir-my-release-cockroachdb-2   standard                 11m
   

​

Step 3. Use the built-in SQL client

1. Download the file:
   $ curl -OOOOOOOOO \
   https://raw.githubusercontent.com/cockroachdb/helm-charts/master/examples/client-secure.yaml
   
2. In the file, set the following values:
   • spec.serviceAccountName: my-release-cockroachdb
   • spec.image: cockroachdb/cockroach: {your CockroachDB version}
   • spec.volumes[0].project.sources[0].secret.name: my-release-cockroachdb-client-secret
3. Use the file to launch a pod and keep it running indefinitely:
   $ kubectl create -f client-secure.yaml
   
   pod "cockroachdb-client-secure" created
   
4. Get a shell into the pod and start the CockroachDB built-in SQL client:
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=./cockroach-certs \
   --host=my-release-cockroachdb-public
   
   # Welcome to the cockroach SQL interface.
   # All statements must be terminated by a semicolon.
   # To exit: CTRL + D.
   #
   # Client version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   # Server version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   
   # Cluster ID: 256a8705-e348-4e3a-ab12-e1aba96857e4
   #
   # Enter \? for a brief introduction.
   #
   root@my-release-cockroachdb-public:26257/defaultdb>
   

5. Run some basic CockroachDB SQL statements:
   > CREATE DATABASE bank;
   
   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   
   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   
   > SELECT * FROM bank.accounts;
   
     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   
6. Create a user with a password:
   > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
   
   You will need this username and password to access the DB Console later.
7. Exit the SQL shell and pod:
   > \q
   

​

Step 4. Access the DB Console

1. On secure clusters, certain pages of the DB Console can only be accessed by admin users. Get a shell into the pod and start the CockroachDB built-in SQL client: $ kubectl exec -it cockroachdb-client-secure \ —./cockroach sql \ —certs-dir=/cockroach-certs \ —host=my-release-cockroachdb-public
2. Assign roach to the admin role (you only need to do this once):
   > GRANT admin TO roach;
   
3. Exit the SQL shell and pod:
   > \q
   
4. In a new terminal window, port-forward from your local machine to the cockroachdb-public service:
   $ kubectl port-forward service/my-release-cockroachdb-public 8080
   
   Forwarding from 127.0.0.1:8080 -> 8080
   
   The port-forward command must be run on the same machine as the web browser in which you want to view the DB Console. If you have been running these commands from a cloud instance or other non-local shell, you will not be able to view the UI without configuring kubectl locally and running the above port-forward command on your local machine.
5. Go to https://localhost:8080 and log in with the username and password you created earlier. If you are using Google Chrome, and you are getting an error about not being able to reach localhost because its certificate has been revoked, go to chrome://flags/#allow-insecure-localhost, enable “Allow invalid certificates for resources loaded from localhost”, and then restart the browser. Enabling this Chrome feature degrades security for all sites running on localhost, not just CockroachDB’s DB Console, so be sure to enable the feature only temporarily.
6. In the UI, verify that the cluster is running as expected:
   • View the Node List to ensure that all nodes successfully joined the cluster.
   • Click the Databases tab on the left to verify that bank is listed.

​

Configure the cluster

1. Download and modify our StatefulSet configuration:
   $ curl -O https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/bring-your-own-certs/cockroachdb-statefulset.yaml
   
2. Update secretName with the name of the corresponding node secret. The secret names depend on your method for generating secrets. For example, if you follow the below steps using cockroach cert, use this secret name:
   secret:
     secretName: cockroachdb.node
   
3. The StatefulSet configuration deploys CockroachDB into the default namespace. To use a different namespace, search for kind: RoleBinding and change its subjects.namespace property to the name of the namespace. Otherwise, a failed to read secrets error occurs when you attempt to follow the steps in Initialize the cluster.

​

Create certificates

1. Create two directories:
   $ mkdir certs my-safe-directory
   

2. Create the CA certificate and key pair:
   $ cockroach cert create-ca \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
3. Create a client certificate and key pair for the root user:
   $ cockroach cert create-client \
   root \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
4. Upload the client certificate and key to the Kubernetes cluster as a secret:
   $ kubectl create secret \
   generic cockroachdb.client.root \
   --from-file=certs
   
   secret/cockroachdb.client.root created
   
5. Create the certificate and key pair for your CockroachDB nodes:
   $ cockroach cert create-node \
   localhost 127.0.0.1 \
   cockroachdb-public \
   cockroachdb-public.default \
   cockroachdb-public.default.svc.cluster.local \
   *.cockroachdb \
   *.cockroachdb.default \
   *.cockroachdb.default.svc.cluster.local \
   --certs-dir=certs \
   --ca-key=my-safe-directory/ca.key
   
6. Upload the node certificate and key to the Kubernetes cluster as a secret:
   $ kubectl create secret \
   generic cockroachdb.node \
   --from-file=certs
   
   secret/cockroachdb.node created
   
7. Check that the secrets were created on the cluster:
   $ kubectl get secrets
   
   NAME                      TYPE                                  DATA   AGE
   cockroachdb.client.root   Opaque                                3      41m
   cockroachdb.node          Opaque                                5      14s
   default-token-6qjdb       kubernetes.io/service-account-token   3      4m
   

​

Initialize the cluster

1. Use the config file you downloaded to create the StatefulSet that automatically creates 3 pods, each running a CockroachDB node:
   $ kubectl create -f cockroachdb-statefulset.yaml
   
   serviceaccount/cockroachdb created
   role.rbac.authorization.k8s.io/cockroachdb created
   rolebinding.rbac.authorization.k8s.io/cockroachdb created
   service/cockroachdb-public created
   service/cockroachdb created
   poddisruptionbudget.policy/cockroachdb-budget created
   statefulset.apps/cockroachdb created
   
2. Initialize the CockroachDB cluster:
   1. Confirm that three pods are Running successfully. Note that they will not be considered Ready until after the cluster has been initialized:
      $ kubectl get pods
      
      NAME            READY     STATUS    RESTARTS   AGE
      cockroachdb-0   0/1       Running   0          2m
      cockroachdb-1   0/1       Running   0          2m
      cockroachdb-2   0/1       Running   0          2m
      
   2. Confirm that the persistent volumes and corresponding claims were created successfully for all three pods:
      $ kubectl get pv
      
      NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS   REASON   AGE
      pvc-9e435563-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-0   standard                51m
      pvc-9e47d820-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-1   standard                51m
      pvc-9e4f57f0-fb2e-11e9-a65c-42010a8e0fca   100Gi      RWO            Delete           Bound    default/datadir-cockroachdb-2   standard                51m
      
   3. Run cockroach init on one of the pods to complete the node startup process and have them join together as a cluster:
      $ kubectl exec -it cockroachdb-0 \
      -- /cockroach/cockroach init \
      --certs-dir=/cockroach/cockroach-certs
      
      Cluster successfully initialized
      
   4. Confirm that cluster initialization has completed successfully. The job should be considered successful and the Kubernetes pods should soon be considered Ready:
      $ kubectl get pods
      
      NAME            READY     STATUS    RESTARTS   AGE
      cockroachdb-0   1/1       Running   0          3m
      cockroachdb-1   1/1       Running   0          3m
      cockroachdb-2   1/1       Running   0          3m
      

​

Step 3. Use the built-in SQL client

$ kubectl create \
-f https://www.cockroachlabs.com/docs/files/cockroach/cloud/kubernetes/bring-your-own-certs/client.yaml

pod/cockroachdb-client-secure created

1. Get a shell into the pod and start the CockroachDB built-in SQL client:
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   
   # Welcome to the cockroach SQL interface.
   # All statements must be terminated by a semicolon.
   # To exit: CTRL + D.
   #
   # Client version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   # Server version: CockroachDB CCL v19.1.0 (x86_64-unknown-linux-gnu, built 2019/04/29 18:36:40, go1.11.6)
   
   # Cluster ID: 256a8705-e348-4e3a-ab12-e1aba96857e4
   #
   # Enter \? for a brief introduction.
   #
   root@cockroachdb-public:26257/defaultdb>
   
   This pod will continue running indefinitely, so any time you need to reopen the built-in SQL client or run any other cockroach client commands (e.g., cockroach node), repeat step 2 using the appropriate cockroach command. If you’d prefer to delete the pod and recreate it when needed, run kubectl delete pod cockroachdb-client-secure.
2. Run some basic CockroachDB SQL statements:
   > CREATE DATABASE bank;
   
   > CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMAL);
   
   > INSERT INTO bank.accounts VALUES (1, 1000.50);
   
   > SELECT * FROM bank.accounts;
   
     id | balance
   +----+---------+
      1 | 1000.50
   (1 row)
   
3. Create a user with a password:
   > CREATE USER roach WITH PASSWORD 'Q7gc8rEdS';
   
   You will need this username and password to access the DB Console later.
4. Exit the SQL shell and pod:
   > \q
   

​

Step 4. Access the DB Console

1. On secure clusters, certain pages of the DB Console can only be accessed by admin users. Get a shell into the pod and start the CockroachDB built-in SQL client:
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   
2. Assign roach to the admin role (you only need to do this once):
   > GRANT admin TO roach;
   
3. Exit the SQL shell and pod:
   > \q
   
4. In a new terminal window, port-forward from your local machine to the cockroachdb-public service:
   $ kubectl port-forward service/cockroachdb-public 8080
   
   Forwarding from 127.0.0.1:8080 -> 8080
   
   The port-forward command must be run on the same machine as the web browser in which you want to view the DB Console. If you have been running these commands from a cloud instance or other non-local shell, you will not be able to view the UI without configuring kubectl locally and running the above port-forward command on your local machine.
5. Go to https://localhost:8080 and log in with the username and password you created earlier. If you are using Google Chrome, and you are getting an error about not being able to reach localhost because its certificate has been revoked, go to chrome://flags/#allow-insecure-localhost, enable “Allow invalid certificates for resources loaded from localhost”, and then restart the browser. Enabling this Chrome feature degrades security for all sites running on localhost, not just CockroachDB’s DB Console, so be sure to enable the feature only temporarily.
6. In the UI, verify that the cluster is running as expected:
   • View the Node List to ensure that all nodes successfully joined the cluster.
   • Click the Databases tab on the left to verify that bank is listed.