- Filter out specific rows and columns from changefeed messages to decrease the load on your downstream sink and support outbox workloads.
- Modify data before it emits to reduce the time and operational burden of filtering or transforming data downstream.
- Stabilize or customize the schema of your changefeed messages for increased compatibility with external systems.
Syntax
There are two possible components to CDC queries:- Projections select the columns that you want to emit data from.
- Predicates restrict the resulting column change data based on the filters you apply.
For a SQL diagram of the CDC query syntax, see the
CREATE CHANGEFEED page.
To emit different properties for a row, specify the following explicitly in CDC queries:
cdc_prev: A tuple-typed column that gives changefeeds access to the previous state of a row. For newly inserted rows in a table, thecdc_prevcolumn will emit asNULL. See the Emit the previous state of a row example for more detail.- CDC queries support system columns, for example:
-
crdb_internal_mvcc_timestamp: Records the timestamp of each row created in a table. If you do not have a timestamp column in the target table, you can accesscrdb_internal_mvcc_timestampin a changefeed. See the Determine the age of a row example.
When changes happen to a column that is part of a composite key, the changefeed will produce a delete message delete message and then an insert message.
Known limitations
- You can only apply CDC queries on a single table in each statement.
- Some stable functions, notably functions that return MVCC timestamps, are overridden to return the MVCC timestamp of the event, e.g.,
transaction_timestamporstatement_timestamp. Additionally, some time-based functions, such asnow()are not supported. We recommend using thetransaction_timestamp()function or thecrdb_internal_mvcc_timestampcrdb_internal_mvcc_timestampcolumn instead. - The following are not permitted in CDC queries:
- Volatile functions.
- Sub-select queries.
- Aggregate and window functions (i.e., functions operating over many rows).
deletechangefeed events will only contain the primary key. All other columns will emit asNULL. See Capture delete messages for detail on running a CDC query that emits the deleted values.ALTER CHANGEFEEDALTER CHANGEFEEDis not fully supported with changefeeds that use CDC queries. CDC queries. You can alter the options that a changefeed uses, but you cannot alter the changefeed target tables.- Creating a changefeed with CDC queries CDC queries on tables with more than one column family column family is not supported.
CDC query function support
The following table outlines functions that are useful with CDC queries:
You can also use the following functions in CDC queries:
- Functions marked as “Immutable” on the Functions and Operators page.
- Non-volatile user-defined functions. See the Queries and user-defined functions example.
- Functions that rely on session data. At the time of changefeed creation, information about the current session is saved. When a CDC query includes one of the functions that use session data, the query will evaluate the saved session data.
- The following “Stable” functions:
age()array_to_json()array_to_string()crdb_internal.cluster_id()date_part()date_trunc()extract()format()jsonb_build_array()jsonb_build_object()to_json()to_jsonb()row_to_json()overlaps()pg_collation_for()pg_typeof()quote_literal()quote_nullable()
Unsupported functions
You can not use the following functions with CDC queries:- Functions marked as “Volatile” on the Functions and Operators page.
- Functions listed in the Known limitations section on this page.
- Functions marked as “Stable” on the Functions and Operators page, except for those listed previously.
Examples
CDC queries allow you to customize your changefeed for particular scenarios. This section outlines several possible use cases for CDC queries. CDC queries useenvelope=bare message format by default. The bare message envelope places the output of the SELECT clause at the top level of the message instead of under an "after" key. When there is additional information that the changefeed is sending, such as updated or resolved timestamps, the messages will include a crdb field containing this information. Refer to the Changefeed Message Envelopes page for more detail.
Depending on how you are filtering or adapting the message envelope with a CDC query and which sink you’re emitting to, message output may vary from some of the example cases in this section.
Refer to CREATE CHANGEFEED for examples on using the foundational syntax to create a changefeed. For information on sinks, refer to the Changefeed Sinks page.
Filter columns
To only emit data from specific columns in a table, you can useSELECT {columns} to define the table’s columns.
As an example, using the users table from the movr database, you can create a changefeed that will emit messages including only the name and city column data:
Filter delete messages
To remove the delete messages from a changefeed stream, use theevent_op() function:
Capture delete messages
Delete changefeed messages will only contain the primary key value and all other columns will emit asNULL (see the Known limitations). To emit the deleted values, use the envelope=wrapped, format=json, and diff options:
before and after keys that contain the prior and current states of the row:
before value in the delete message, produced by the diff option, will include the entire row. That is, it will not include any projections from a CDC query.
Emit the previous state of a row
Changefeeds can access thecdc_prev hidden column on a table to emit the previous state of a row or column. cdc_prev is a tuple-typed column that contains the table’s columns.
To emit the previous state of a row, it is necessary to explicitly call cdc_prev:
cdc_prev tuple with the following syntax:
cdc_prev column will emit as NULL.
If you do not need to select specific columns in a table or filter rows from a changefeed, you can instead create a changefeed using the
diff option to emit a before field with each message. This field includes the value of the row before the update was applied.Reference TTL in a CDC query
In CockroachDB, table row deletes occur as a result of regular SQL transactions or through row-level TTL. When your changefeed emits delete event messages, you may need to distinguish between these two types of deletion. For example, only emitting messages for row-level TTL deletes from your changefeed. If you have TTL logic defined withttl_expiration_expression or ttl_expire_after, you can leverage CDC queries to determine whether or not a given row was expired at the time of the changefeed event, including a delete event.
Most users should use ttl_expiration_expression instead of ttl_expire_after for the following reasons:
- If you add
ttl_expire_afterto an existing table, it will cause a full table rewrite, which can affect performance. Specifically, it will result in a schema change that (1) creates a new hidden columncrdb_internal_expirationfor all rows, and (2) backfills the value of that new column tonow()+ttl_expire_after. - You cannot use
ttl_expire_afterwith an existingTIMESTAMPTZcolumn. - If you use
ttl_expiration_expression, you can use an existingTIMESTAMPTZcolumn called e.g.updated_at.
ttl_expiration_expression
In some cases, you may have custom expiration logic on rows in a table. You can also write a CDC query to emit rows that have deleted through row-level TTL using a custom TTL expression.
In the following example, the table uses the ttl_expiration_expression storage parameter to reference the expired_at column. To create a changefeed on this table to explicitly emit the previous state of the row for TTL deletions:
CREATE TABLE statement and further details on ttl_expiration_expression, refer to Using ttl_expiration_expression.
ttl_expire_after
When the table uses the ttl_expire_after storage parameter, you can emit rows that were deleted after expiring from the changefeed with syntax similar to:
- Accesses the
cdc_prevcolumn for the previous state of the row. - Searches for
deleteevents in that previous state. - Finds the TTL expiration timestamp of the deleted rows where it is earlier than the current statement timestamp.
CREATE TABLE statement and further details on ttl_expire_after, refer to Using ttl_expire_after.
This will only emit rows that were deleted after expiring. Furthermore, consider that a transactional SQL delete during the window between the row expiring and the TTL job running will also cause this message to emit from the changefeed.
Geofilter a changefeed
When you are working with aREGIONAL BY ROW table, you can filter the changefeed on the crdb_region column to create a region-specific changefeed:
REGIONAL BY ROW tables with changefeeds, see Changefeeds in Multi-Region Deployments.
Stabilize the changefeed message schema
As changefeed messages emit from the database, message formats can vary as tables experience schema changes. You can select columns with typecasting to prevent message fields from changing during a changefeed’s lifecycle:Shard changefeed messages
CDC queries allow you to emit changefeed messages from the same table to different endpoints. As a result, you can use queries to load balance messages across changefeed sinks without the need for an intermediate system. In this example, the query uses theride_id column’s UUID to shard the messages. The left() function filters the first character from the ride_id column and finds the specified initial characters. The example shards successfully by running a changefeed on the same table and dividing the 16 possible beginning UUID characters through to f.
Therefore, the first changefeed created:
View recent changes to a row
You can use CDC queries as a tool for debugging or investigating issues from the SQL shell. For example, you may need to identify what recently changed in a specific row. You can use thecursor option with the desired start time and a WHERE clause describing the row in question. Instead of sending to a sink, a “sinkless” changefeed will allow you to view the results in the SQL shell.
-
Find the start time. Use the
cluster_logical_timestamp()function to calculate the logical time. This will return the logical timestamp for an hour earlier than the statement run time: -
Run the changefeed without a sink and pass the start time to the
cursoroption: -
To find changes within a time period, use
cursorwith theend_timeoption:
Determine the age of a row
You can determine the age of a row by using thecrdb_internal_mvcc_timestamp system column and cdc_prev to access the row’s previous state:
Recover lost messages
In the event that an incident downstream has affected some rows, you may need a way to recover or evaluate the specific rows. Create a new changefeed that only watches for the affected row(s). Here, the example uses the row’s primary key:Customize changefeed messages
You can adapt your changefeed messages by filtering the columns, but it is also possible to build message fields with SQL expressions. In this example, the query adds asummary field to the changefeed message:
Create a scheduled changefeed to export filtered data
This example creates a nightly export of some filtered table data with a scheduled changefeed that will run just after midnight every night. The changefeed uses CDC queries to query the table and filter the data it will send to the sink:- If it runs into a failure,
on_execution_failure=retrywill ensure that the schedule retries the changefeed immediately. - If the previous scheduled changefeed is still running,
on_previous_running=startwill start a new changefeed at the defined cadence.
Queries and the outbox pattern
The transactional outbox pattern provides a way to publish events reliably through an outbox table before sending to the messaging system. CDC queries can help to streamline this process by eliminating the need for an outbox table in the database. If you also have a requirement to transform the data or remove delete messages from the changefeed payload, queries can achieve this. For example, you have three tables:users, accounts, and dogs. You need to send all changes to any of those tables to a single Kafka endpoint using a specific structure. Namely, a JSON object like the following:
- Selects the
event_timestampof the event and casts to anINT. - Sets the
typeof change using theevent_op()function. - Uses
jsonb_build_object()to construct the desired data field.
NULL in the cdc_prev output for an insert message, insert messages will be sent. Updates will not send, as long as the status was not previously NULL.
Queries and user-defined functions
You can create CDC queries that include user-defined functions. The followingCREATE FUNCTION statement builds the doubleRevenue() function at the database level:

