Technical Advisory 44166

Publication date: February 12, 2020

Description

When running SHOW JOBS or viewing the Jobs page in the Admin UI, high memory usage can be incurred on a node to the point it could crash.

Statement

SHOW JOBS and the Jobs page in the Admin UI internally load all the job descriptions from the cluster in RAM before displaying them. Under reasonable production settings, a single backup job payload may exceed 5MB in size. Considering an hourly backup and default property for jobs.retention_time set to 336h, a single use of SHOW JOBS or a single user of the Jobs page in the Admin UI can incur ~1.7GB of memory utilization. This allocation is then multiplied by the number of concurrent accesses to the jobs table. Starting in CockroachDB v19.2.3, new jobs payloads are reduced in size. A later version will also avoid loading old job entries in memory when viewing recent jobs.

Mitigation

It is possible to reduce the number of job entries overall by setting the jobs.retention_time cluster setting to a value closer to 48h or 24h. For example:

SET CLUSTER SETTING jobs.retention_time='48:00:00'.

Additionally, if the nodes are observed to crash due to excessive memory usage, it may be necessary to truncate the job history. This can be achieved, for example, with:

DELETE from system.jobs
WHERE status = 'succeeded'
  AND created < (now() - '2 days'::interval);

Impact

All deployments running CockroachDB v19.2.0 to v19.2.2 are affected. Questions about any technical alert can be directed to our support team.

​Description

​Statement

​Mitigation

​Impact

Description

Statement

Mitigation

Impact