Technical Advisory 190483

Publication date: September 22, 2023

Description

Queries reading a table that uses at least one explicit column family in its definition and was from a backup using the following versions may return errors or incorrect results:

v22.2.7 to v22.2.13
v23.1.0 to v23.1.8

Restored tables that do not use explicit column families, or that were restored using versions prior to v22.2.7, v22.2.14 or later, or v23.1.9 or later, are unaffected. The process splits the key-value data into ranges. If the data being restored contains a table whose definition includes an explicit column family, the affected versions of RESTORE may incorrectly choose a split boundary. If those ranges are subsequently served by separate leaseholders, queries may fail to read expected data. These queries may then return errors or incorrect results. The incorrect results or errors can persist even after the cluster is upgraded to an unaffected version. They may require manual intervention by our support team to be permanently resolved. Observation of incorrect results or errors may be intermittent due to the movement of leaseholders or changes in query plans. Customers with tables that use at least one explicit column family and were restored using an affected version should run the following query:

SELECT start_pretty FROM crdb_internal.ranges_no_leases
WHERE table_id IN (SELECT id FROM system.descriptor
WHERE jsonb_array_length(crdb_internal.pb_to_json('desc', descriptor)->'table'->'families') > 1 AND id > 50) AND
start_pretty ~ '.*/[0-9]+/NULL';

If the result set is empty, no further action is required. If the result set is non-empty, contact our support team.

Statement

This is resolved by a fix in CockroachDB for v23.1 and v22.2. The PRs update the method used to select the boundaries at which ranges of data are split during RESTORE to avoid splitting between two column families of the same row. These fixes are available in CockroachDB maintenance releases v23.1.9 and v22.2.14.

Mitigation

Users of CockroachDB v23.1.0 through v23.1.8 and v22.2.7 through v22.2.13 are encouraged to upgrade to v23.1.10 or v22.2.14 before performing a RESTORE, especially if they are restoring tables which are defined with multiple explicit column families. Users who are operating a cluster that contains potentially affected tables should contact our support team to determine if they are affected.

Impact

Queries following the restoration of some types of backups, specifically those that include larger tables which are defined with multiple explicit column families, can produce errors or incorrect results. While the behavior in RESTORE which leads to this condition has since been fixed in v22.2.14 and v23.1.10, clusters which performed a potentially affected restore while running an affected version prior to upgrading may continue to be affected. Customers with such clusters should contact support to determine if they require remediation. Clusters which do not contain tables which utilize multiple column families and were restored on an affected version are unaffected. Questions about any technical alert can be directed to our support team.

​Description

​Statement

​Mitigation

​Impact

Description

Statement

Mitigation

Impact