> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Transaction Retry Error Reference

When a [transaction](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) is unable to complete due to [contention](https://docs.cockroachlabs.com/docs/v24.2/performance-best-practices-overview#understanding-and-avoiding-transaction-contention) with another concurrent or recent transaction attempting to write to the same data, CockroachDB will [automatically attempt to retry the failed transaction](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) without involving the client (i.e., silently). If the automatic retry is not possible or fails, a *transaction retry error* is emitted to the client.

Transaction retry errors fall into two categories:

* *Serialization errors* indicate that a transaction failed because it could not be placed into a serializable ordering among all of the currently-executing transactions.

  * [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) transactions must address these errors with client-side intervention, where the client [initiates a restart of the transaction](#client-side-retry-handling), and [adjusts application logic and tunes queries](#minimize-transaction-retry-errors) for greater performance.
  * [`READ COMMITTED`](https://docs.cockroachlabs.com/docs/v24.2/read-committed) transactions do **not** return [serialization errors](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference) that require client-side handling.

* *Internal state errors* indicate that the cluster itself is experiencing an issue, such as being [overloaded](https://docs.cockroachlabs.com/docs/v24.2/ui-overload-dashboard), which prevents the transaction from completing. These errors generally require both cluster-side and client-side intervention, where an operator addresses an issue with the cluster before the client then [initiates a restart of the transaction](#client-side-retry-handling).

All transaction retry errors use the `SQLSTATE` error code `40001`, and emit error messages with the string [`restart transaction`](https://docs.cockroachlabs.com/docs/v24.2/common-errors#restart-transaction). Further, each error includes a [specific error code](#transaction-retry-error-reference) to assist with targeted troubleshooting.

When experiencing transaction retry errors, you should follow the guidance under [Actions to take](#actions-to-take), and then consult the reference for your [specific transaction retry error](#transaction-retry-error-reference) for guidance specific to the error message encountered.

## Overview

At the default `SERIALIZABLE` isolation level, CockroachDB always attempts to find a [serializable ordering](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) among all of the currently-executing transactions.

Whenever possible, CockroachDB will [auto-retry a transaction internally](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) without notifying the client. CockroachDB will only send a serialization error to the client when it cannot resolve the error automatically without client-side intervention.

[`READ COMMITTED`](https://docs.cockroachlabs.com/docs/v24.2/read-committed) transactions can transparently resolve serialization errors by [retrying individual statements](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-snapshots) rather than entire transactions.

## Actions to take

In most cases, the correct actions to take when encountering transaction retry errors are:

1. Under `SERIALIZABLE` isolation, update your application to support [client-side retry handling](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference#client-side-retry-handling) when transaction retry errors are encountered. Follow the guidance for the [specific error type](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference#transaction-retry-error-reference).

2. Take steps to [minimize transaction retry errors](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference#minimize-transaction-retry-errors) in the first place. This means reducing transaction contention overall, and increasing the likelihood that CockroachDB can [automatically retry](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) a failed transaction.

### Client-side retry handling

When running under [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) isolation, your application should include client-side retry handling when the statements are sent individually, such as:

```
> BEGIN;

> UPDATE products SET inventory = 0 WHERE sku = '8675309';

> INSERT INTO orders (customer, status) VALUES (1, 'new');

> COMMIT;
```

To indicate that a transaction must be retried, CockroachDB signals a serialization error with the `SQLSTATE` error code `40001` and an error message that begins with the string `"restart transaction"`.

To handle these types of errors, you have the following options:

* If your database library or framework provides a method for retryable transactions (it will often be documented as a tool for handling deadlocks), use it.
* If you're building an application in the following languages, Cockroach Labs has created adapters that include automatic retry handling:
  * **Go** developers using [GORM](https://github.com/jinzhu/gorm) or [pgx](https://github.com/jackc/pgx) can use the [`github.com/cockroachdb/cockroach-go/crdb`](https://github.com/cockroachdb/cockroach-go/tree/master/crdb) package. For an example, see [Build a Go App with CockroachDB](https://docs.cockroachlabs.com/docs/v24.2/build-a-go-app-with-cockroachdb).
  * **Python** developers using [SQLAlchemy](https://www.sqlalchemy.org) can use the [`sqlalchemy-cockroachdb` adapter](https://github.com/cockroachdb/sqlalchemy-cockroachdb). For an example, see [Build a Python App with CockroachDB and SQLAlchemy](https://docs.cockroachlabs.com/docs/v24.2/build-a-python-app-with-cockroachdb-sqlalchemy).
  * **Ruby (Active Record)** developers can use the [`activerecord-cockroachdb-adapter`](https://rubygems.org/gems/activerecord-cockroachdb-adapter). For an example, see [Build a Ruby App with CockroachDB and Active Record](https://docs.cockroachlabs.com/docs/v24.2/build-a-ruby-app-with-cockroachdb-activerecord).
* If you're building an application with another driver or data access framework that is [supported by CockroachDB](https://docs.cockroachlabs.com/docs/v24.2/third-party-database-tools), we recommend reusing the retry logic in our ["Simple CRUD" Example Apps](https://docs.cockroachlabs.com/docs/v24.2/example-apps). For example, **Java** developers accessing the database with [JDBC](https://jdbc.postgresql.org) can reuse the example code implementing retry logic shown in [Build a Java app with CockroachDB](https://docs.cockroachlabs.com/docs/v24.2/build-a-java-app-with-cockroachdb).
* If you're building an application with a language and framework for which we do not provide example retry logic, you might need to write your own retry logic. For an example, see the [Client-side retry handling example](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-example).
* **Advanced users, such as library authors**: See [Advanced Client-Side Transaction Retries](https://docs.cockroachlabs.com/docs/v24.2/advanced-client-side-transaction-retries).

#### Client-side retry handling example

For a conceptual example of application-defined retry logic, and testing that logic against your application's needs, see the [client-side retry handling example](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-example).

### Minimize transaction retry errors

In addition to the steps described in [Client-side retry handling](#client-side-retry-handling), which detail how to configure your application to restart a failed transaction, there are also a number of changes you can make to your application logic to reduce the number of transaction retry errors that reach the client application under `SERIALIZABLE` isolation.

Reduce failed transactions caused by [timestamp pushes](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache) or [read invalidation](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-refreshing):

* Limit the number of affected rows by following [optimizing queries](https://docs.cockroachlabs.com/docs/v24.2/apply-statement-performance-rules) (e.g., avoiding full scans, creating secondary indexes, etc.). Not only will transactions run faster, lock fewer rows, and hold locks for a shorter duration, but the chances of [read invalidation](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-refreshing) when the transaction's [timestamp is pushed](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache), due to a conflicting write, are decreased because of a smaller read set (i.e., a smaller number of rows read).

* Break down larger transactions (e.g., [bulk deletes](https://docs.cockroachlabs.com/docs/v24.2/bulk-delete-data)) into smaller ones to have transactions hold locks for a shorter duration. For example, use [common table expressions](https://docs.cockroachlabs.com/docs/v24.2/common-table-expressions) to group multiple clauses together in a single SQL statement. This will also decrease the likelihood of [pushed timestamps](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache). For instance, as the size of writes (number of rows written) decreases, the chances of the transaction's timestamp getting bumped by concurrent reads decreases.

* Use [`SELECT FOR UPDATE`](https://docs.cockroachlabs.com/docs/v24.2/select-for-update) to aggressively lock rows that will later be updated in the transaction. Updates must operate on the most recent version of a row, so a concurrent write to the row will cause a retry error ([`RETRY_WRITE_TOO_OLD`](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference#retry_write_too_old)). Locking early in the transaction forces concurrent writers to block until the transaction is finished, which prevents the retry error. Note that this locks the rows for the duration of the transaction; whether this is tenable will depend on your workload. For more information, see [When and why to use `SELECT FOR UPDATE` in CockroachDB](https://www.cockroachlabs.com/blog/when-and-why-to-use-select-for-update-in-cockroachdb/).

* Use historical reads ([`SELECT ... AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/v24.2/as-of-system-time)), preferably [bounded staleness reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads#when-to-use-bounded-staleness-reads) or [exact staleness with follower reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads#run-queries-that-use-exact-staleness-follower-reads) when possible to reduce conflicts with other writes. This reduces the likelihood of [`RETRY_SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference#retry_serializable) errors as fewer writes will happen at the historical timestamp. More specifically, writes' timestamps are less likely to be pushed by historical reads as they would [when the read has a higher priority level](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#transaction-conflicts). Note that if the `AS OF SYSTEM TIME` value is below the closed timestamp, the read cannot be invalidated.

* When replacing values in a row, use [`UPSERT`](https://docs.cockroachlabs.com/docs/v24.2/upsert) and specify values for all columns in the inserted rows. This will usually have the best performance under contention, compared to combinations of [`SELECT`](https://docs.cockroachlabs.com/docs/v24.2/select-clause), [`INSERT`](https://docs.cockroachlabs.com/docs/v24.2/insert), and [`UPDATE`](https://docs.cockroachlabs.com/docs/v24.2/update).

* If applicable to your workload, assign [column families](https://docs.cockroachlabs.com/docs/v24.2/column-families#default-behavior) and separate columns that are frequently read and written into separate columns. Transactions will operate on disjoint column families and reduce the likelihood of conflicts.

* As a last resort, consider adjusting the [closed timestamp interval](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#closed-timestamps) using the `kv.closed_timestamp.target_duration` [cluster setting](https://docs.cockroachlabs.com/docs/v24.2/cluster-settings) to reduce the likelihood of long-running write transactions having their [timestamps pushed](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache). This setting should be carefully adjusted if **no other mitigations are available** because there can be downstream implications (e.g., historical reads, change data capture feeds, statistics collection, handling zone configurations, etc.). For example, a transaction *A* is forced to refresh (i.e., change its timestamp) due to hitting the maximum [*closed timestamp*](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#closed-timestamps) interval (closed timestamps enable [Follower Reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads#how-stale-follower-reads-work) and [Change Data Capture (CDC)](https://docs.cockroachlabs.com/docs/v24.2/change-data-capture-overview)). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read.

> [!NOTE]
>
> If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time by which the data available in [Follower Reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads) and [CDC changefeeds](https://docs.cockroachlabs.com/docs/v24.2/change-data-capture-overview) lags behind the current state of the cluster. In other words, there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/).

Increase the chance that CockroachDB can [automatically retry](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) a failed transaction:

* [Send statements in transactions as a single batch](https://docs.cockroachlabs.com/docs/v24.2/transactions#batched-statements). Batching allows CockroachDB to [automatically retry](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) a transaction when [previous reads are invalidated](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-refreshing) at a [pushed timestamp](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache). When a multi-statement transaction is not batched, and takes more than a single round trip, CockroachDB cannot automatically retry the transaction. For an example showing how to break up large transactions in an application, see [Break up large transactions into smaller units of work](https://docs.cockroachlabs.com/docs/v24.2/build-a-python-app-with-cockroachdb-sqlalchemy#break-up-large-transactions-into-smaller-units-of-work).

- Limit the size of the result sets of your transactions to under 16KB, so that CockroachDB is more likely to [automatically retry](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries) when [previous reads are invalidated](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-refreshing) at a [pushed timestamp](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache). When a transaction returns a result set over 16KB, even if that transaction has been sent as a single batch, CockroachDB cannot automatically retry the transaction. You can change the results buffer size for all new sessions using the `sql.defaults.results_buffer.size` [cluster setting](https://docs.cockroachlabs.com/docs/v24.2/cluster-settings), or for a specific session using the `results_buffer_size` [connection parameter](https://docs.cockroachlabs.com/docs/v24.2/connection-parameters#additional-connection-parameters).

## Transaction retry error reference

Note that your application's retry logic does not need to distinguish between the different types of serialization errors. They are listed here for reference during [advanced troubleshooting](https://docs.cockroachlabs.com/docs/v24.2/performance-recipes#transaction-contention).

* [`RETRY_WRITE_TOO_OLD`](#retry_write_too_old)
* [`RETRY_SERIALIZABLE`](#retry_serializable)
* [`RETRY_ASYNC_WRITE_FAILURE`](#retry_async_write_failure)
* [`ReadWithinUncertaintyIntervalError`](#readwithinuncertaintyintervalerror)
* [`RETRY_COMMIT_DEADLINE_EXCEEDED`](#retry_commit_deadline_exceeded)
* [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found)
* [`ABORT_REASON_CLIENT_REJECT`](#abort_reason_client_reject)
* [`ABORT_REASON_PUSHER_ABORTED`](#abort_reason_pusher_aborted)
* [`ABORT_REASON_ABORT_SPAN`](#abort_reason_abort_span)
* [`ABORT_REASON_NEW_LEASE_PREVENTS_TXN`](#abort_reason_new_lease_prevents_txn)
* [`ABORT_REASON_TIMESTAMP_CACHE_REJECTED`](#abort_reason_timestamp_cache_rejected)
* [injected by `inject_retry_errors_enabled` session variable](#injected-by-inject_retry_errors_enabled-session-variable)

Each transaction retry error listed includes an example error as it would appear from the context of the client, a description of the circumstances that cause that error, and specific guidance for addressing the error.

### RETRY\_WRITE\_TOO\_OLD

```
TransactionRetryWithProtoRefreshError: ... RETRY_WRITE_TOO_OLD ...
```

**Error type:** Serialization error

**Description:**

The `RETRY_WRITE_TOO_OLD` error occurs when a transaction *A* tries to write to a row *R*, but another transaction *B* that was supposed to be serialized after *A* (i.e., had been assigned a higher timestamp), has already written to that row *R*, and has already committed. Under `SERIALIZABLE` isolation, this is a common error when you have too much contention in your workload.

**Action:**

Under [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) isolation:

1. Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:

   1. Send all of the statements in your transaction in a [single batch](https://docs.cockroachlabs.com/docs/v24.2/transactions#batched-statements).
   2. Use [`SELECT FOR UPDATE`](https://docs.cockroachlabs.com/docs/v24.2/select-for-update) to aggressively lock rows that will later be updated in the transaction.

Under [`READ COMMITTED`](https://docs.cockroachlabs.com/docs/v24.2/read-committed) isolation:

1. `RETRY_WRITE_TOO_OLD` errors are only returned in rare cases that can be avoided by adjusting the [result buffer size](#result-buffer-size).

### RETRY\_SERIALIZABLE

```
TransactionRetryWithProtoRefreshError: ... RETRY_SERIALIZABLE ...
```

The error message for `RETRY_SERIALIZABLE` contains additional information about the transaction conflict which led to the error, as shown below. This error message can also be viewed in the [DB Console](https://docs.cockroachlabs.com/docs/v24.2/ui-overview) by navigating to [**Insights Page** → **Workload Insights** → **Transaction Executions** and clicking on the transaction ID to see the **Failed Execution** insight](https://docs.cockroachlabs.com/docs/v24.2/ui-insights-page#failed-execution).

```
restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE  - failed preemptive refresh due to conflicting locks on /Table/106/1/918951292305080321/0 [reason=wait_policy] - conflicting txn: meta={id=1b2bf263 key=/Table/106/1/918951292305080321/0 iso=Serializable pri=0.00065863 epo=0 ts=1700512205.521833000,2 min=1700512148.761403000,0 seq=1}): "sql txn" meta={id=07d42834 key=/Table/106/1/918951292305211393/0 iso=Serializable pri=0.01253025 epo=0 ts=1700512229.378453000,2 min=1700512130.342117000,0 seq=2} lock=true stat=PENDING rts=1700512130.342117000,0 wto=false gul=1700512130.842117000,0
SQLSTATE: 40001
HINT: See: https://www.cockroachlabs.com/docs/v24.2/transaction-retry-error-reference.html#retry_serializable
```

**Error type:** Serialization error

**Description:**

> [!TIP]
>
> [`READ COMMITTED`](https://docs.cockroachlabs.com/docs/v24.2/read-committed) transactions do **not** produce `RETRY_SERIALIZABLE` errors.

At a high level, the `RETRY_SERIALIZABLE` error occurs when a transaction's timestamp is moved forward, but the transaction performed reads at the old timestamp that are no longer valid at its new timestamp. More specifically, the `RETRY_SERIALIZABLE` error occurs in the following three cases under [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) isolation:

1. When a transaction *A* has its timestamp moved forward (also known as *A* being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction *A* tried to write a key that transaction *B* had already read, and *B* was supposed to be serialized after *A* (i.e., *B* had a higher timestamp than *A*). CockroachDB will try to serialize *A* after *B* by changing *A*'s timestamp, but it cannot do that when another transaction has subsequently written to some of the keys that *A* has read and returned to the client. When that happens, the `RETRY_SERIALIZATION` error is signalled. For more information about how timestamp pushes work in our transaction model, see the [architecture docs on the transaction layer's timestamp cache](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#timestamp-cache).

2. When a [high-priority transaction](https://docs.cockroachlabs.com/docs/v24.2/transactions#transaction-priorities) *A* does a read that runs into a write intent from another lower-priority transaction *B*, and some other transaction *C* writes to a key that *B* has already read. Transaction *B* will get this error when it tries to commit, because *A* has already read some of the data touched by *B* and returned results to the client, and *C* has written data previously read by *B*.

3. When a transaction *A* is forced to refresh (i.e., change its timestamp) due to hitting the maximum [*closed timestamp*](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#closed-timestamps) interval (closed timestamps enable [Follower Reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads#how-stale-follower-reads-work) and [Change Data Capture (CDC)](https://docs.cockroachlabs.com/docs/v24.2/change-data-capture-overview)). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2.

*Failed preemptive refresh*

In the three preceding cases, CockroachDB will try to validate whether the read-set of the transaction that had its timestamp (`timestamp1`) pushed is still valid at the new timestamp (`timestamp3`) at commit time. This mechanism is called "performing a [read refresh](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#read-refreshing)". If the read-set is still valid, the transaction can commit. If it is not valid, the transaction will get a `RETRY_SERIALIZABLE - failed preemptive refresh` error. The refresh can fail for two reasons:

1. There is a committed value on a key that was read by the transaction at `timestamp2` (where `timestamp2` occurs between `timestamp1` and `timestamp3`). The error message will contain `due to encountered recently written committed value`. CockroachDB does not have any information about which conflicting transaction wrote to this key.
2. There is an intent on a key that was read by the transaction at `timestamp2` (where `timestamp2` occurs between `timestamp1` and `timestamp3`). The error message will contain `due to conflicting locks`. CockroachDB does have information about the conflicting transaction to which the intent belongs. The information about the [conflicting transaction](https://docs.cockroachlabs.com/docs/v24.2/ui-insights-page#serialization-conflict-due-to-transaction-contention) can be seen on the [DB Console Insights page](https://docs.cockroachlabs.com/docs/v24.2/ui-insights-page).

**Action:**

Under [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) isolation:

1. Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:
   1. Send all of the statements in your transaction in a [single batch](https://docs.cockroachlabs.com/docs/v24.2/transactions#batched-statements).
   2. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/v24.2/as-of-system-time).
   3. Use [`SELECT FOR UPDATE`](https://docs.cockroachlabs.com/docs/v24.2/select-for-update) to aggressively lock rows for the keys that were read and could not be refreshed.

### RETRY\_ASYNC\_WRITE\_FAILURE

```
TransactionRetryWithProtoRefreshError: ... RETRY_ASYNC_WRITE_FAILURE ...
```

**Error type:** Internal state error

**Description:**

The `RETRY_ASYNC_WRITE_FAILURE` error occurs when some kind of problem with your cluster's operation occurs at the moment of a previous write in the transaction, causing CockroachDB to fail to replicate one of the transaction's writes. This can happen if a [lease transfer](https://docs.cockroachlabs.com/docs/v24.2/architecture/replication-layer#epoch-based-leases-table-data) occurs while the transaction is executing, or less commonly if you have a [network partition](https://docs.cockroachlabs.com/docs/v24.2/cluster-setup-troubleshooting#network-partition) that cuts off access to some nodes in your cluster.

**Action:**

1. Retry the transaction as described in [client-side retry handling](#client-side-retry-handling). This is worth doing because the problem with the cluster is likely to be transient.
2. Investigate the problems with your cluster.  For cluster troubleshooting information, see [Troubleshoot Cluster Setup](https://docs.cockroachlabs.com/docs/v24.2/cluster-setup-troubleshooting).

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ReadWithinUncertaintyIntervalError

```
TransactionRetryWithProtoRefreshError: ReadWithinUncertaintyIntervalError:
        read at time 1591009232.376925064,0 encountered previous write with future timestamp 1591009232.493830170,0 within uncertainty interval `t <= 1591009232.587671686,0`;
        observed timestamps: [{1 1591009232.587671686,0} {5 1591009232.376925064,0}]
```

**Error type:** Serialization error

**Description:**

The `ReadWithinUncertaintyIntervalError` can occur when two transactions which start on different gateway nodes attempt to operate on the same data at close to the same time, and one of the operations is a write. The uncertainty comes from the fact that we cannot tell which one started first - the clocks on the two gateway nodes may not be perfectly in sync.

For example, if the clock on node *A* is ahead of the clock on node *B*, a transaction started on node *A* may be able to commit a write with a timestamp that is still in the "future" from the perspective of node *B*. A later transaction that starts on node *B* should be able to see the earlier write from node *A*, even if *B*'s clock has not caught up to *A*. The "read within uncertainty interval" occurs if we discover this situation in the middle of a transaction, when it is too late for the database to handle it automatically. When node *B*'s transaction retries, it will unambiguously occur after the transaction from node *A*.

> [!NOTE]
>
> This behavior is non-deterministic: it depends on which node is the [leaseholder](https://docs.cockroachlabs.com/docs/v24.2/architecture/life-of-a-distributed-transaction#leaseholder-node) of the underlying data range. It’s generally a sign of contention. Uncertainty errors are always possible with near-realtime reads under contention.

**Action:**

Under [`SERIALIZABLE`](https://docs.cockroachlabs.com/docs/v24.2/demo-serializable) isolation:

1. Be prepared to retry on uncertainty (and other) errors, as described in [client-side retry handling](#client-side-retry-handling).
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors). In particular, try to:
   1. Send all of the statements in your transaction in a [single batch](https://docs.cockroachlabs.com/docs/v24.2/transactions#batched-statements).
   2. Use historical reads with [`SELECT ... AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/v24.2/as-of-system-time).
3. If you [trust your clocks](https://docs.cockroachlabs.com/docs/v24.2/operational-faqs#what-happens-when-node-clocks-are-not-properly-synchronized), you can try lowering the [`--max-offset` option to `cockroach start`](https://docs.cockroachlabs.com/docs/v24.2/cockroach-start#flags), which provides an upper limit on how long a transaction can continue to restart due to uncertainty.

Under [`READ COMMITTED`](https://docs.cockroachlabs.com/docs/v24.2/read-committed) isolation:

1. `ReadWithinUncertaintyIntervalError` errors are only returned in rare cases that can be avoided by adjusting the [result buffer size](#result-buffer-size).

> [!NOTE]
>
> Uncertainty errors are a sign of transaction conflict. For more information about transaction conflicts, see [Transaction conflicts](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#transaction-conflicts).

### RETRY\_COMMIT\_DEADLINE\_EXCEEDED

```
TransactionRetryWithProtoRefreshError: TransactionRetryError: transaction deadline exceeded ...
```

**Error type:** Internal state error

**Description:**

The `RETRY_COMMIT_DEADLINE_EXCEEDED` error means that the transaction timed out due to being pushed by other concurrent transactions. This error is most likely to happen to long-running transactions. The conditions that trigger this error are very similar to the conditions that lead to a [`RETRY_SERIALIZABLE`](#retry_serializable) error, except that a transaction that hits this error got pushed for several minutes, but did not hit any of the conditions that trigger a `RETRY_SERIALIZABLE` error. In other words, the conditions that trigger this error are a subset of those that trigger `RETRY_SERIALIZABLE`, and that this transaction ran for too long (several minutes).

> [!NOTE]
>
> Read-only transactions do not get pushed, so they do not run into this error.

This error occurs in the cases described below.

1. When a transaction *A* has its timestamp moved forward (also known as *A* being "pushed") as CockroachDB attempts to find a serializable transaction ordering. Specifically, transaction *A* tried to write a key that transaction *B* had already read. *B* was supposed to be serialized after *A* (i.e., *B* had a higher timestamp than *A*). CockroachDB will try to serialize *A* after *B* by changing *A*'s timestamp.

2. When a [high-priority transaction](https://docs.cockroachlabs.com/docs/v24.2/transactions#transaction-priorities) *A* does a read that runs into a write intent from another lower-priority transaction *B*. Transaction *B* may get this error when it tries to commit, because *A* has already read some of the data touched by *B* and returned results to the client.

3. When a transaction *A* is forced to refresh (change its timestamp) due to hitting the maximum [*closed timestamp*](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer#closed-timestamps) interval (closed timestamps enable [Follower Reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads#how-stale-follower-reads-work) and [Change Data Capture (CDC)](https://docs.cockroachlabs.com/docs/v24.2/change-data-capture-overview)). This can happen when transaction *A* is a long-running transaction, and there is a write by another transaction to data that *A* has already read.

**Action:**

1. The `RETRY_COMMIT_DEADLINE_EXCEEDED` error is one case where the standard advice to add a retry loop to your application may not be advisable. A transaction that runs for long enough to get pushed beyond its deadline is quite likely to fail again on retry for the same reasons. Therefore, the best thing to do in this case is to shrink the running time of your transactions so they complete more quickly and do not hit the deadline.
2. If you encounter case 3 above, you can increase the `kv.closed_timestamp.target_duration` setting to a higher value. Unfortunately, there is no indication from this error code that a too-low closed timestamp setting is the issue. Therefore, you may need to rule out cases 1 and 2 (or experiment with increasing the closed timestamp interval, if that is possible for your application - see the note below).

If you increase the `kv.closed_timestamp.target_duration` setting, it means that you are increasing the amount of time by which the data available in [Follower Reads](https://docs.cockroachlabs.com/docs/v24.2/follower-reads) and [CDC changefeeds](https://docs.cockroachlabs.com/docs/v24.2/change-data-capture-overview) lags behind the current state of the cluster. In other words, there is a trade-off here: if you absolutely must execute long-running transactions that execute concurrently with other transactions that are writing to the same data, you may have to settle for longer delays on Follower Reads and/or CDC to avoid frequent serialization errors. The anomaly that would be exhibited if these transactions were not retried is called [write skew](https://www.cockroachlabs.com/blog/what-write-skew-looks-like/).

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_ABORTED\_RECORD\_FOUND

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORTED_RECORD_FOUND) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_ABORTED_RECORD_FOUND` error means that the client application is trying to use a transaction that has been aborted. This happens in one of the following cases:

* Write-write conflict: Another [high-priority transaction](https://docs.cockroachlabs.com/docs/v24.2/transactions#transaction-priorities) *B* encountered a write intent by our transaction *A*, and tried to push *A*'s timestamp.
* Cluster overload: *B* thinks that *A*'s transaction coordinator node is dead, because the coordinator node hasn't heartbeated the transaction record for a few seconds.
* Deadlock: Some transaction *B* is trying to acquire conflicting locks in reverse order from transaction *A*.

**Action:**

If you are encountering deadlocks:

* Avoid producing deadlocks in your application by making sure that transactions acquire locks in the same order.

If you are using only default [transaction priorities](https://docs.cockroachlabs.com/docs/v24.2/transactions#transaction-priorities):

* This error means your cluster has problems. You are likely overloading it. Investigate the source of the overload, and do something about it. For more information, see [Node liveness issues](https://docs.cockroachlabs.com/docs/v24.2/cluster-setup-troubleshooting#node-liveness-issues).

If you are using [high- or low-priority transactions](https://docs.cockroachlabs.com/docs/v24.2/transactions#transaction-priorities):

1. Retry the transaction as described in [client-side retry handling](#client-side-retry-handling)
2. Adjust your application logic as described in [minimize transaction retry errors](#minimize-transaction-retry-errors).

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_CLIENT\_REJECT

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_CLIENT_REJECT) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_CLIENT_REJECT` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_PUSHER\_ABORTED

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_PUSHER_ABORTED` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_ABORT\_SPAN

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_ABORT_SPAN) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_ABORT_SPAN` error is caused by the same conditions as the [`ABORT_REASON_ABORTED_RECORD_FOUND`](#abort_reason_aborted_record_found), and requires the same actions. The errors are fundamentally the same, except that they are discovered at different points in the process.

See [Minimize transaction retry errors](#minimize-transaction-retry-errors) for the full list of recommendations.

### ABORT\_REASON\_NEW\_LEASE\_PREVENTS\_TXN

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_NEW_LEASE_PREVENTS_TXN) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_NEW_LEASE_PREVENTS_TXN` error occurs because the timestamp cache will not allow transaction *A* to create a transaction record. A new lease wipes the timestamp cache, so this could mean the [leaseholder](https://docs.cockroachlabs.com/docs/v24.2/architecture/life-of-a-distributed-transaction#leaseholder-node) was moved and the duration of transaction *A* was unlucky enough to happen across a lease acquisition. In other words, leaseholders got shuffled out from underneath transaction *A* (due to no fault of the client application or schema design), and now it has to be retried.

**Action:**

Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).

### ABORT\_REASON\_TIMESTAMP\_CACHE\_REJECTED

```
TransactionRetryWithProtoRefreshError:TransactionAbortedError(ABORT_REASON_TIMESTAMP_CACHE_REJECTED) ...
```

**Error type:** Internal state error

**Description:**

The `ABORT_REASON_TIMESTAMP_CACHE_REJECTED` error occurs when the timestamp cache will not allow transaction *A* to create a transaction record. This can happen due to a [range merge](https://docs.cockroachlabs.com/docs/v24.2/architecture/distribution-layer#range-merges) happening in the background, or because the timestamp cache is an in-memory cache, and has outgrown its memory limit (about 64 MB).

**Action:**

Retry transaction *A* as described in [client-side retry handling](#client-side-retry-handling).

### injected by `inject_retry_errors_enabled` session variable

```
 TransactionRetryWithProtoRefreshError: injected by `inject_retry_errors_enabled` session variable
```

**Error type:** Internal state error

**Description:**

When the `inject_retry_errors_enabled` [session variable](https://docs.cockroachlabs.com/docs/v24.2/set-vars) is set to `true`, any statement (with the exception of [`SET` statements](https://docs.cockroachlabs.com/docs/v24.2/set-vars)) executed in the session inside of an explicit transaction will return this error.

For more details, see [Test transaction retry logic](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-example#test-transaction-retry-logic).

**Action:**

To turn off error injection, set the `inject_retry_errors_enabled` session variable to `false`.

## See also

* [Common Errors and Solutions](https://docs.cockroachlabs.com/docs/v24.2/common-errors)
* [Transactions](https://docs.cockroachlabs.com/docs/v24.2/transactions#automatic-retries)
* [Transaction Contention](https://docs.cockroachlabs.com/docs/v24.2/performance-best-practices-overview#transaction-contention)
* [Transaction Retry Error Example](https://docs.cockroachlabs.com/docs/v24.2/transaction-retry-error-example)
* [DB Console Transactions Page](https://docs.cockroachlabs.com/docs/v24.2/ui-transactions-page)
* [Architecture - Transaction Layer](https://docs.cockroachlabs.com/docs/v24.2/architecture/transaction-layer)