## Release announcements
* [Enhanced Cron Job Resilience With pg_cron in YugabyteDB](https://www.yugabyte.com/blog/enhanced-cron-job-resilience-in-yugabytedb/)
## v2.23.1.0 - November 12, 2024 {#v2.23.1.0}
**Build:** `2.23.1.0-b220`
### Downloads
**Docker:**
```sh
docker pull yugabytedb/yugabyte:2.23.1.0-b220
```
### Change log
View the detailed changelog
### Improvements
#### YSQL
* Changes the CatalogCacheMisses metric name to showcase separate cache misses for each catalog cache, assisting users in easily preloading necessary catalog tables. {{}}
* Enables in-place updates for non-key columns in indexes, reducing number of flushes. {{}}
* Introduces the `yb_make_next_ddl_statement_nonincrementing` configuration parameter to reduce catalog version increments during many consecutive DDLs, lowering memory and latency spikes. {{}}
* Updates advisory lock error message with a workaround hint instead of `not yet implemented.` {{}}
* Allows certain ALTER VIEW commands in the parser without requiring additional changes. {{}}
* Includes a modification to ysql_dump and Postgres that ensures all pg_class and pg_type OID preservation across a backup/restore when passing the include-yb-metadata flag. {{}}
* Streamlines sys table prefetching by automatically resetting catalog read time. {{}}
* Simplifies several ybc_pggate functions for better value return and removes unused functions. {{}}
* Revamps the restoration of old backups by only applying the check for `pg_class OID` if the new YSQL configuration parameter, `yb_ignore_heap_pg_class_oids`, is enabled. {{}}
* Unmodified columns on main table will continue to be written out when update optimizations are enabled, removing dependency on the feature controlled by `ysql_skip_row_lock_for_update`. {{}}
* Adds table-level Prometheus metrics for catcache misses at node-level, viewable at `:13000/prometheus-metrics` endpoint. {{}}
* Auto analyze service now tracks mutation counts from DDL changes and eliminates redundant analysis of indexes. {{}},{{}}
* Adds serialization/deserialization and equality checking mechanism for update optimization metadata. {{}}
* Allows adding a primary key using a unique index by performing a table rewrite. {{}}
* Enables the batch execution of explicit row lock requests in Pg, improving the handling and performance of such requests. {{}}
* Allows validation of foreign-key and check constraints with `ALTER TABLE ...VALIDATE CONSTRAINT`. {{}}
* Added yb_make_next_ddl_statement_nonincrementing to YbDbAdminVariables for yb_db_admin role. {{}}
* Allows preloading of foreign key lists in relcache, avoiding on-demand master fetches, controlled by `yb_enable_fkey_catcache`. {{}}
* Refines the PatchStatus function by storing relation id directly in the relations vector, avoiding the need to call the PgSession::LoadTable method. {{}}
#### YCQL
* Updates YCQL tables `system_schema.functions` and `system_schema.aggregates` to align with Cassandra, aiding Cortex migration. {{}}
#### DocDB
* Reduces risk of WAL over-garbage collection in xCluster and avoids unnecessary flushes by modifying the WAL retention policy. {{}}
* Enables reduction of duplicate code and custom flag filtering through `GetFlagInfos` relocation. {{}}
* Accelerates xCluster setup for tables by reducing setup time, storing all intermediate state in-memory, and limiting writes to `sys_catalog`. {{}}
* Allows adding new CatalogEntries without modifying the sys-catalog-tool through using helper functions. {{}}
* Enables reading stored vector index data from DocDB. {{}}
* Enables scanning of the `sequences_data` table for xCluster replication, aiding in the bootstrapping process by integrating current sequence information into the WAL from the source universe. {{}}
* Introduces `DEFINE_NEW_INSTALL_VALUE` macro to enable controlled rollout of new features in fresh clusters. {{}}
* Enables taking into account of changed data during a vector index data update. {{}}
* Adds utilities for HNSW vector index implementation, benchmarking, and file reading. {{}}
* Removes the `CallVisitor` template function to simplify the sys-catalog-tool code. {{}}
* Allows each sequence to have an update in the WAL for seamless xCluster replication. {{}}
* Introduces a command-line tool for building, validating, and benchmarking HNSW indexes with Usearch implementation. {{}}
* Offers support for different vector coordinate types including vector-related templates. {{}}
* Imports hnswlib header-only library for enhancements. {{}}
* Introduces a new gflag `max_disk_throughput_mbps` for automated control of write rejections when disk is full, replacing `reject_writes_min_disk_space_aggressive_check_mb`. {{}}
* Allows easier debugging with additional logging and updated log prefixes. {{}}
* Ensures that ddl_queue and replicated_ddls do not colocate on the same tablet in colocated databases. {{}}
* Integrates hnswlib into the vector indexing framework, adding wrapper and allowing different distance calculations. {{}}
* Eliminates the capability to rename xCluster replication groups, reducing potential confusion. {{}}
* Enables sequences replication feature in source universe by creating `sequences_data` table and incorporating an outgoing stream. {{}}
* Enables shared memory by default for executing read and write pg client queries to enhance performance. {{}}
* Streamlines automatic syncing of inline third-party dependencies and breaks down `thirdparty_tool` into separate modules. {{}}
* Enables automatic application of DDL changes in xCluster using `automatic_ddl_mode` proto field. {{}}
* Enables display of cluster_uuid in tserver /varz endpoint for better test and call_home tracking. {{}}
* Resets the `cdc_wal_retention_time_secs` value on table removal from xCluster replication and applies updates dynamically, eliminating need to drop and recreate replication. {{}}
* Ensures correct functioning of `search_path` by skipping its value's enquote during transaction setup. {{}}
* Enhances sequence generation consistency by making backend process stick to a logical connection when initializing a sequence variable with `nextval` function. {{}}
* Ensures DDL operations success with a sleep interval after database catalog version updates when Connection Manager is active. {{}}
* Updates the callhome URL to use HTTPS for enhanced security. {{}}
* Includes PID in yb_active_session_history for easy correlation with actual processes in YSQL, YCQL, and background activities. {{}}
* Allows reduction of thread stack size to avoid Linux hugepage backing. {{}}
* Allows explicit addition of gFlags to `gflag_allowlist.txt` for secure callhome data collection, plus `version_info` in tserver data. {{}}
* Adds `max_follower_heartbeat_delay` metric to alert users when the master follower heartbeat delay is too high. {{}}
* Removes the need to set `set_cdc_min_replicated_index` during xCluster bootstrap for improved log retention. {{}}
* Ensures the user interface accurately displays disk size, even when multiple data directories are used. {{}}
#### CDC
* Allows dynamic table addition alongside table removal from CDC streams without needing to disable or restart. {{}}
* Adds `DYNAMIC_TABLES_DISABLED` yb-admin option to disable dynamic table addition when creating new streams. {{}}
* Enables default non-eligible table cleanup in CDC streams, replaces the flag `enable_cleanup_of_non_eligible_tables_from_cdcsdk_stream` with `cdcsdk_enable_dynamic_table_addition_with_table_cleanup`, and removes flag protection for table identification. {{}}
* Syncs table removal from a CDC stream to a more performant async process, allowing both manual and automatic removal via the `RemoveUserTableFromCDCSDKStream` and `RemoveTablesFromCDCSDKStream` RPCs respectively. {{}}
* Disables `wal_level` checks for logical replication in YSQL as PG WAL is not used. {{}}
* Changes the default `wal_level` to `logical` to ensure compatibility with logical replication clients. {{}}
* Enables asynchronous removal of user tables from CDC streams by adjusting how background threads process and persist stream metadata. {{}}
* Enables tablet splitting on tables under CDCSDK stream by default using the GFLAG `enable_tablet_split_of_cdcsdk_streamed_tables`. {{}}
* Adds a tag for the slot name attribute in the CDC metrics. {{}}
#### yugabyted
* Handles OBJECT_NOT_FOUND errors during SELECT/UPDATE/DELETE after TRUNCATE when Connection Manager is enabled. {{}}
* Enables bitmap scans and removes size-based fetching in the pg parity feature by updating GFlags. {{}}
* Updates to the pg parity testcase now remove size-based fetching and include enabling bitmap scans. {{}}
* Changes the flag name from `enable_pg_parity_tech_preview` to `enable_pg_parity_early_access` in all branches. {{}}
* Stops sending gflags details in the callhome diagnostics to eliminate redundant data. {{}}
### Bug fixes
#### YSQL
* Removes unnecessary DCHECK for smoother rolling upgrades from release 2.20, enhancing `ysql_enable_db_catalog_version_mode` gflag functionality. {{}}
* Fixes an issue related to regular expression pushdown causing inconsistency and crashes by introducing thread local cache. {{}}
* Resets catalog read time after table prefetching to prevent outdated readings for on-demand loading from a master. {{}}
* Corrects IndexScan's indextlist to improve accuracy of variable references in subquery scenarios. {{}}
* Prevents individually moving a colocated table to a different tablespace, forcing users to move all tables collectively. {{}}
* Minimizes read restart errors during ANALYZE operation by using a fresh read time for each table, finding an optimal balance between read oldness and transactionality. {{}}
* Reduces the memory consumption of the `fk_reference_cache_` in large transactions, thereby ensuring minimal use of RAM. {{}}
* Prevents retry of schema mismatch errors in batched execution mode to maintain transaction atomicity. {{}}
* Revamped `Storage Read Requests` metric now better reflects true RPC count in `PrecastRequestSender` usage. {{}},{{}}
* Resolves `INSERT ON CONFLICT` bugs in temporary tables to ensure consistent outputs. {{}}
* Fixes a TupleDesc reference leak warning in nested INSERT ON CONFLICT. {{}}
* Restores unintentional grammar changes to YSQL ALTER SCHEMA command for bug fix. {{}}
* Fixes repeated calls in `DeleteTableInternal`, preventing `YbAdminSnapshotScheduleTest.SysCatalogRetention` failure on asan builds. {{}}
* Resolves potential memory corruption by mutating pushdown expressions outside the per tuple context. {{}}
* Eliminates the publishing of empty metrics from the `statementType` enum on the YSQL Metrics endpoint. {{}}
* Simplifies the process of identifying identical data by using storage equality, not semantic equality, during UPDATE queries, handling problems that arise when data types lack equality operators or when user-defined data types exhibit unusual semantic equality. {{}}
* Addresses a bug in YSQL that caused crashes while costing an index scan on a hash index with included columns using CBO. {{}}
* Adds more logs to aid in debugging a transaction status mismatch issue. {{}}
* Fixes the hanging issue when creating a colocated table by correcting the unconditional reading of the `relkind` column. {{}}
* Adds `log_dist` option to auto_explain docs for YSQL. {{}}
* Addresses the issue of invalid Attnum with Bitmap Scan +subplans +nested loops previously encountered in Index Scans. {{}}
* Adds table name to table-level metrics on `/metrics` endpoint for better identification. {{}}
* Fixes a bug that caused incorrect deducing of commit/abort status for transactions which do not change the table's schema. {{}}
* Adds HELP and TYPE metadata to YSQL Prometheus metrics endpoint for better metrics accessibility. {{}}
* Corrects an issue causing heap-buffer-overflow errors during ysqlconnmgr ASAN tests. {{}}
* Fixes a crash caused by using the `pg_hint_plan` with the `hint_table` enabled. {{}}
* Updates the copyright string from `YugaByteDB` to the correct version `YugabyteDB`, ensuring error-free linter runs. {{}}
* Allows for a modified calculation of net connections in YSQL connection manager with the warmup random mode. {{}}
* Prevents background parallel workers from executing check/assign hooks of YSQL configuration parameters, eliminating potential crashes. {{}}
* Modifies backup/restore process to skip column name checks for indexes, allowing for successful restoration even with renamed columns. {{}}
* Fixes crashes caused by using a RowComparisonExpression on a reordered primary key index. {{}}
* Improves cache re-invalidation for `ALTER TABLE` commands to avoid schema version mismatch errors within the same session. {{}}
* Streamlines ysql_dump process by eliminating faulty statements for colocated table unique constraint, enhancing database reliability. {{}}
* Introduces batching for the `INSERT ON CONFLICT` command to enhance performance by minimizing alternating read and write operations. {{}}
* Fixes the flaw in the current DDL atomicity workflow where only the first table's schema was compared, even if it doesn't change after a schema version increment. {{}}
* Alters encoding setup order to ensure `UTF8` use instead of `SQL_ASCII` and adds `pg_collation` to preloaded tables to prevent specific errors. {{}}
* Corrects a typo to prevent incorrect loading of pg_collation catalog caches, eliminating associated performance issues. {{}}
#### YCQL
* Eliminates unexpected error during shell type definition update across multiple sessions. {{}}
#### DocDB
* Prevents tablet bootstrap from getting stuck when replaying a truncate operation. {{}}
* Limits the response size of ListSnapshotSchedules RPC by excluding unnecessary per-object details to prevent overflow issues. {{}}
* Reduces the risk of deadlocks when making active read/write requests to a table right after a tablet-split. {{}}
* Simplifies handling of large RPC responses by catching errors earlier and appropriately using `narrow_cast`. {{}}
* Fixes errors from executing rpc callback twice due to lack of return after detecting disabled wait-queue or deadlock features. {{}}
* Switches write query duration from `CoarseMonoClock` to `MonoClock` for precise nanosecond granularity. {{}}
* Sets the `close_timestamp_micros` field for partially copied log segments to prevent time-based policy violation. {{}}
* Eliminates persistent logging issue after dropping a colocated table by removing unnecessary logs. {{}}
* Corrects leader balancing for geopartitioned tables to prevent early exits and ensure accurate global leader sorting. {{}}
* Adds `database_oid` to the `TSLocalLockManager` locking contract to distinguish objects from different databases. {{}}
* Adds debugging information for retryable requests rejected by followers despite acceptance by leaders. {{}}
* Corrects typo and includes minimum allowed value in the validation message for `xcluster_checkpoint_max_staleness_secs` flag. {{}}
* Prevents tablet partition collision during `ImportSnapshot` by not cloning `DELETED` or `REPLACED` tablets. {{}}
* Allows pg_cron to correctly handle SIGTERM by using `quickdie` instead of `die` or `pg_cron_sigterm`. {{}}
* Allows the retry of checkpoint if `leader not ready` or `service unavailable` errors occur, and prevents client retries on a failed checkpoint by storing an `InternalError` status. {{}}
* Restores `transaction_aware` field in `master_backup.proto` for compatibility with older DB versions. {{}}
* Enhances tablet limit flags by adding more details to the main flag and adjusting the help strings. {{}}
* Adds filtering for bootstrap intent iterators based on `min_replay_txn_start_ht` to prevent unnecessary SST files buildup during bootstrap. {{}}
* Fixes a compile error caused by a mismatch in the declaration of `protobuf_message_total_bytes_limit`. {{}}
* Reduces the potential for core dump errors during Point In Time Recovery (PITR) by utilizing shared_ptr in callbacks. {{}}
* Renames the `yql_endpoint_tserver_uuid` field to `top_level_node_id` in `/rpcz` to maintain consistency. {{}}
* Ensures correct and consistent updating of `grptablespace` column in `pg_yb_tablegroup` after `ALTER TABLE` command. {{}}
* Ensures the `CREATE TABLE ...TABLESPACE pg_default` command in colocated databases aligns with PostgreSQL semantics. {{}}
* Now treats YSQL configuration parameters in a case-insensitive manner, avoiding variable duplication and potential failures, aligning closer with Postgres behavior. {{}}
* Ensures node restarts successfully with `secure` enabled by introducing a timeout framework. {{}}
* Restores loading of old dumps that lack index PG class OIDs by enhancing the flag `yb_ignore_pg_class_oids`. {{}}
* Promotes cluster start-up when master_webserver_port is customized by considering user input during the `validate_and_set_configs` step. {{}}
* Allows the packed row size to exceed its limit during repacking, preventing tserver crash after `alter table add column` commands with a default value. {{}}
* The update stabilizes WriteBuffer, preventing potential crashes when constructed over non-zeroed memory. {{}}
* Adds a null check to prevent a system crash when cleaning up a recently applied transaction state after a flush operation. {{}}
* Resolves the shutdown issue with the shared exchange causing TServer to hang by properly managing concurrent threads and session destruction, and sets `pg_client_use_shared_memory` to false by default on Mac to enhance performance. {{}}
* Ensures "create table" requests for colocated tables don't fail due to mistaken tablet limit checks. {{}}
#### CDC
* Refactors CDCSDK to prevent data loss during workload with single shard transactions by ensuring WAL messages are committed before being replicated. {{}}
* Reduces disk IO overhead by optimizing the tablet-meta update process in CDCService. {{}}
* Introduces three new yb-admin commands : `disable_dynamic_table_addition_in_change_data_stream`, `remove_user_table_from_change_data_stream`, `validate_and_sync_cdc_state_table_entries_on_change_data_stream` for managing user tables in a CDC stream and addresses a race condition issue {{}},{{}},{{}}
* Obsoletes expired or unpolled tables from CDC stream and state table, reducing unnecessary resource retention and potential blocking of split tablet deletion. {{}}
* Filters out records related to index tables in colocated databases to prevent GetChanges call failures. {{}}
* Ensures old-tuples for updates are only sent when replica identity is FULL, matching PostgreSQL behavior. {{}}
* Enhances table removal from a CDC stream by fetching tablets from the local master copy instead of the entire cdc_state table, enhancing performance. {{}}
* Prevents crashes that occur due to concurrent GetChanges calls on the same producer tablet. {{}}
* Ensures entries for dropped tables are removed from the replica identity map, preventing replication errors. {{}}
#### yugabyted
* Offers corrected DB OID in the function `calculate_table_size`, now matching `YBCGetDatabaseOid(rel)` for improved reliability. {{}}
* Enables a node to successfully join using the `join` flag by implementing a retry mechanism. {{}}
## v2.23.0.0 - September 13, 2024 {#v2.23.0.0}
**Build:** `2.23.0.0-b710`
### Downloads
**Docker:**
```sh
docker pull yugabytedb/yugabyte:2.23.0.0-b710
```
### Highlights
We're excited to announce the [technical preview](/stable/releases/versioning/#tech-preview-tp) of four powerful new features in YugabyteDB aimed at simplifying operations, enhancing functionality, and improving performance.
**Instant database cloning** {{}}
[Instant database cloning](/stable/manage/backup-restore/instant-db-cloning/) allows you to quickly create independent copies of your database for data recovery, development, and testing. Clones can be created in seconds, and initially consume no additional disk space because they share the same data files as the original database. Clones can be created as of now, or as of any time in the recent past (within a configurable history retention period), so developers can experiment without impacting production performance. Clones also provide a safety net for rapid recovery from accidental data loss or corruption.
**pg_cron extension** {{}}
We're introducing support for the [pg_cron extension](/stable/additional-features/pg-extensions/extension-pgcron/), which brings a cron-based job scheduler directly into the database. With pg_cron, you can schedule YSQL commands using familiar cron syntax, including jobs on intervals as fine as seconds. While pg_cron operates on a single node known as the pg_cron leader, the queries it schedules leverage the full resources of the distributed cluster. In case of node failure, leadership is automatically transferred, ensuring continuous availability and smooth scheduling operations.
**Semi-automatic xCluster replication** {{}}
[Semi-automatic xCluster replication](/stable/deploy/multi-dc/async-replication/async-replication-transactional/) simplifies the management of YSQL transactional xCluster replication by operating at the database level instead of the table level. This reduces the need for IT admin involvement when tables are created or dropped, as xCluster management is only required for adding or removing entire databases from replication. DDL operations can be performed by authorized users or DBAs without needing elevated privileges. Semi-automatic xCluster supports YSQL transactional replication and is ideal for Disaster Recovery (DR) purposes.
**Improvement to backward scans** {{}}
We've improved the performance of backward scans by 10x. For example, descending scans on tables that are stored in ascending order are now much faster. Previously, backward scans were significantly slower than forward scans in YugabyteDB, causing the query optimizer to implement various workarounds, including adding a sort operator on top of the forward scan operator to keep the backward scan pattern comparable. Such optimizer tricks might have been ineffective in complex query plans. The improvements to backward scan performance now allows such queries to be faster out of the box. You enable the feature using the `use_fast_backward_scan` preview flag.
### Change log
View the detailed changelog
### Improvements
#### YSQL
* Enhances logging for DDL transaction conflicts and PG catalog version mismatches by including the DDL command tag and specific log details outside of the `log_ysql_catalog_versions` flag. {{}}
* Reduces per-backend memory consumption by reinstating TOAST compression for catalogue tables. {{}}
* Enables DDL atomicity feature by default by altering `ysql_yb_ddl_rollback_enabled`, `report_ysql_ddl_txn_status_to_master`, and `ysql_ddl_transaction_wait_for_ddl_verification` flags' defaults. {{}}
* Adds a new YSQL view for YCQL statement metrics, allowing it to be joined with YCQL wait events in the `yb_active_universe_history` table. {{}}
* Displays distinct prefix keys explicitly in the explain output, enhancing the clarity of indexing for users. {{}}
* Adds auto flag `ysql_yb_enable_ddl_atomicity_infra` to control DDL atomicity feature during the upgrade phase. {{}}
* Updates read time for each operation to simplify code and avoid applying used read time from obsolete operations. {{}}
* Allows YbInitPinnedCacheIfNeeded to only load the shared pinned cache, enhancing concurrent handling of DDLs in various databases. {{}}
* Rectifies a compilation error by eliminating duplicate declaration and unnecessary function triggered by merge issues. {{}}
* Avoids schema version mismatch errors during ALTER TABLE operations in cases where DDL atomicity is enabled. {{}}
* Adds new columns to localhost:13000/statements for more comprehensive database management, including user and database IDs along with varied block level statistics. {{}}
* Now logs global-impact DDL statements that increment all database catalog versions. {{}}
* Reorganizes extensions into three segregated directories for better access and ease of use. {{}}
* Resolves schema version mismatch errors that occur after an ALTER TABLE operation due to DDL transaction verification in non-debug builds. {{}}
* Introduces a new YSQL configuration parameter `yb_parallel_range_size` for better tuning of parallel range size. {{}}
* Removes the unused `keep_order` field from `YbctidGenerator` for cleaner results processing. {{}}
* Introduces a new YSQL configuration parameter `yb_enable_parallel_append` to disable the unannounced feature `parallel append`. {{}}
* Performs stylistic modifications and refactors in various YSQL scripts for better readability and performance. {{}}
* Adds support for creating vector indexes using a dummy ANN method `ybdummyann`, enabling preliminary vector-based searching in databases. {{}}
* Restricts the undesired usage of LWFunction by disallowing move copy. {{}}
* Simplifies the column binding logic for ybgin and lsm access methods and prepares for addition of user-defined index types. {{}}
* Enables the grammar for CREATE/DROP ACCESS METHOD for more flexible extension handling. {{}}
* Eradicates misleading log message during table creation with DDL atomicity enabled. {{}}
* Introduces a new enum PgYbrowidMode to deduplicate hidden internal column addition logic. {{}}
* Adds Save/Restore state functionality to `ConsistentReadPoint` using a new `Momento` class. {{}}
* Avoids renaming DocDb tables during legacy rewrite operations to prevent issues with backup/restore and improves handling of failed ADD/DROP primary key, ALTER TYPE operations. {{}}
* Stops python checks on all third-party extensions to avoid build failure. {{}}
* Simplifies and cleans up code in PgDml/PgSelect/PgSelectIndex classes, ensuring only necessary fields are used and removing redundant destructors, with no logic changes. {{}}
* Improves backward scans by updating the cost-based optimizer to consider backward scan enhancements, leading to significant execution-time improvements when `FLAGS_use_fast_backward_scan` is enabled. {{}}
* Introduces new role-related flags in `yb_backup.py` script to enhance backup and restore functionalities. {{}}
* Fixes various issues in the Batch Nested Loop Join code, particularly the new row array comparison, for clearer functioning and better documentation. {{}}
#### YCQL
* Now throws an error when using the unsupported GROUP BY clause in YCQL with autoflag `ycql_suppress_group_by_error` available for compatibility issues. {{}}
#### DocDB
* Adds an `ABORTED` state and an `abort_message` field to the `SysCloneStateInfoPB` object for better clone failure management. {{}}
* Offers a new stack trace tracking framework for improved visibility of disk I/O operations, configurable through the `track_stack_traces` flag. {{}}
* Resolves build failure caused by a problematic merge, offering better disk IO visibility by tracking IOs by stacktrace. {{}}
* Blocks writes based on the total number of bytes being flushed, not just when 2 memtables are flushing. {{}}
* Allows asynchronous DNS cache updating and resolution retry upon failure to reduce RPC call delays and prevent unexpected leadership changes. {{}},{{}}
* Introduces a new flag to toggle on or off recommended memory defaults for increased control over individual memory settings. {{}}
* Introduces utility to dump top heap stacks when memory limit is exceeded for enhanced debugging. {{}}
* Deprecates unused flags and old svc_num_workers flags for clearer user configurations. {{}}
* Allows local debug builds of YugabyteDB to restart from release builds to enhance debugging. {{}}
* Shifts xCluster-related functions from Catalocustomeranager to XClusterSourceManager for easier management. {{}}
* Speeds up backward scans by building rows from end to start, reducing unnecessary repositioning and Seek calls, enabled with `FLAGS_use_fast_backward_scan` flag. {{}}
* Boosts error messaging clarity when preview flags are not set in `allowed_preview_flags_csv`. {{}}
* Prevents GetChangesForXCluster from returning an invalid safe time in certain conditions. {{}}
* Broadens the rewrite_test_log.py script to incorporate more directory replacements such as home directory, YB_SRC_ROOT, LLVM toolchain directory, and third-party dependencies directory. {{}}
* Adjusts TServer memory percentage from 50% to 48% for (4,8]GiB boxes and sets new recommendations for boxes over 16 GiB. {{}}
* Incorporates the usearch and fp16 header-only libraries into the src directory, simplifying code import. {{}}
* Returns the original error messages from CreateTable and CreateTablegroup instead of an incorrect "Database not found" message. {{}}
* Enables replication of DDL schemas and users to maintain consistency between source and target. {{}}
* Updates `AreNodesSafeToTakeDown` to return earlier and deliver a readable error message when hitting a timeout, using a control flag, `are_nodes_safe_to_take_down_timeout_buffer_ms` with a default setting of 2 seconds. {{}}
* Reduces unnecessary alerts by removing "No active snapshot" warning from the logs. {{}}
* Ensures Data Definition Language (DDL) operations replicate exactly once by checking the `replicated_ddls` table prior to rerunning any DDL. {{}}
* Streamlines the creation of xCluster streams by unifying the scattered code into `XClusterClient::CreateXClusterStream`. {{}}
* Allows for faster failover in xCluster DR by skipping the cleanup process when `skip_producer_stream_deletion` is set on `DeleteUniverseReplicationRequestPB`. Safe for upgrades and rollbacks. {{}}
* Reduces extra reactor threads by reusing server messenger in AutoFlags. {{}}
* Reduces resource usage by enabling stateful service client to reuse the server's existing yb_client. {{}}
* Adds `SCHECK_PB_FIELDS_NOT_EMPTY` macro for validating non-empty fields, with individual checks on list elements. {{}}
* Splits up yb_xcluster_ddl_replication.c into additional util files for better project extensibility. {{}}
* Removes deprecated JSON output format in xCluster for clean and efficient function. {{}}
* Allows only single DDL query strings to prevent issues with DDL+DML mixes or multiple DDLs. {{}}
* Shifts certain RPC endpoint methods from the Catalocustomeranager to the new MasterClusterHandler class for easier management. {{}}
* Adds `read-time` option description in the `help` of `ysql_dump` allowing database dump at a specified past time. {{}}
* Renames and replaces `cdc_consumer_handler_thread_pool_size` with `xcluster_consumer_thread_pool_size` to reduce CPU and memory usage. {{}}
* Refreshes stack trace tracking UI endpoints to enable per-column sorting and optimize sorting script. {{}}
* Allows table locking by acquiring local server object locks for DDLs and DMLs, hosted by a local transaction participant, with session ID and tied to their lifetime. Includes performance upgrades to lock acquisition and conflict resolution. {{}}
* Simplifies replication setup by using table IDs instead of names, helping avoid issues caused by table renames or recreations. {{}}
* Increases speed of backward scans for flat document reader with support for packed row V2. {{}}
* Reuses the Tservers `yb::client` in `CDCService` to decrease 4 threads and a meta cache, addressing a `CDCService`/xCluster source issue. {{}}
* Allows unified usage of XClusterRemoteClient in XClsuterConsumer, centralizing client creation logic. {{}}
* Ensures replication health before succeeding `IsSetupUniverseReplicationDone`, improving error detection. {{}}
* Adds `external_hybrid_time` to log-dump output to detect xCluster target writes. {{}}
* Introduces two new `SOURCE_UNREACHABLE` and `SYSTEM_ERROR` enums to enable more detailed error reporting from the Poller. {{}}
* Allows requesting streams by producer table ids for xCluster DDL Replication to accurately match tables together. {{}}
* Enhances stack trace tracking endpoints usability by adding an access button, a reset tracking link, and timestamp details. {{}}
* Renames and shifts various members and functions from Catalocustomeranager to XClusterManager. {{}}
* Replaces the deprecated and unused namespace replication with DB Scoped replication for a more efficient solution. {{}}
* Makes pggate aware of fast backward scan capability for accurate cost identification in the Cost Based Optimizer. {{}}
* Added `emergency_repair_mode` flag and new yb-admin commands to handle corrupted CatalogEntity data without stopping `yb-master`. {{}}
* Relocates Setup, Bootstrap, Alter and Delete Target Replication functions for better organization. {{}}
* Changes column ID representation in debug builds to be compatible with release builds. {{}}
* Refactored the `PopulateTabletCheckPointInfo` function to improve its readability and maintainability. {{}}
* Removes display of InvalidFlags in the flags UI to clean up the user interface. {{}}
* Allows for storing and loading of vector indexes, ensuring effective management of these indexes. {{}}
* Integrates class `DocDBStatistics` with `ReadOperationData` for better statistics transmission to `IntentAwareIterator`. {{}}
* Introduces a new flag `max_disk_throughput_mbps` for automated control of write rejections when disk is full, replacing `reject_writes_min_disk_space_aggressive_check_mb`. {{}}
* Adds a flag `FLAGS_tablet_split_min_size_ratio` to control tablet splitting based on SST file sizes, ensuring better control over tablet size imbalance. {{}}
* Relocates heartbeat code for greater readability and Catalog Manager size reduction. {{}},{{}}
* Simplifies the clone state manager by moving persisted data to an in-memory structure. {{}}
* Relocates specific heartbeat code to `master_heartbeat_service.cc` for enhanced readability and easier tracking. {{}},{{}}
* Refactors heartbeat path code for easier navigation and reduction of Catalocustomeranager size, with no functional changes. {{}},{{}}
* Shifts tcmalloc profiling code to the Utils folder. {{}}
* Moves tablet splitting specific RPCs and functions from `catalog_manager.cc` to `tablet_split_manager.cc` for better code management. {{}},{{}}
* Introduces a new flag `enable_rwc_lock_debugging` to control slow lock debugging and fixes a bug in `rwc_lock.cc`. {{}}
* Changes CloneStateInfo object from scoped_refptr to std::shared_ptr for standardization. {{}}
* Allows setting only non-empty schema names in YBTableName. {{}}
* Transfers ownership of TabletSplitManager, CloneStateManager, and SnapshotCoordinator from Catalocustomeranager to Master for leaner dependency requirements. {{}},{{}}
#### CDC
* Preserves CDC stream even when all associated tables are dropped, tying its lifecycle to the database. {{}}
* Introduces three new yb-admin commands to remove a user table from a CDCSDK stream, descend dynamic table addition in a CDC stream, and validate CDC state for a particular stream, enhancing control over CDC streams. {{}},{{}}
* Prevents addition of tables with enum array column to the CDC stream to avoid crashes during consumption. {{}}
* Transforms the flag `yb_enable_cdc_consistent_snapshot_streams` from a preview into a default true auto flag. {{}}
* Enables dynamic table addition with Postgres replication consumption by setting retention barriers on new tables' tablets during creation. {{}}
* Allows modification of the publication refresh interval using the `cdcsdk_publication_list_refresh_interval_secs` flag. {{}}
* Adds a TServer flag, `ysql_yb_default_replica_identity`, for customizing default replica identity at table creation. {{}}
* Introduces `cdcsdk_enable_dynamic_table_addition` flag to manage dynamic table additions in replication slot consumption model. {{}}
* Introduces replication slot name for internal distinction between two consumption models in the code. {{}}
* Allows creating an old model stream via yb-admin in upgraded environments, ensuring only one stream type per database. {{}}
* Introduces replica identity in CDC to populate before image records, allowing table-level before image information fetching and retaining in stream metadata. {{}}
* Eliminates unnecessary NOTICE messages when setting yb_read_time from walsender, reducing message clutter. {{}}
* Enables transaction state to be cleared promptly after a table is deleted, preventing table deletion from getting stuck and resulting in faster functionality. {{}}
#### yugabyted
* Allows Connection Manager to handle error messages, preventing test failures in the YSQL layer. {{}}
* Ensures `RENAME DATABASE` query handles logical and physical connections correctly for consistent database behavior. {{}}
* Redefines tracking of role modifications using role OID in YSQL Connection Manager for accurate behavior during mid-session role renaming. {{}}
* Offers support for single-use YSQL configuration parameters in YSQL Connection Manager. {{}}
* Enables using role OID with `session_authorization` in YSQL Connection Manager for correct role alterations in a session. {{}}
* Allows sticky connections when setting certain YSQL configuration parameters not permitted in explicit transactions. {{}}
* Allows modification of YSQL configuration parameters on a running cluster by destroying the control connection. {{}}
* Adds a new `/pitr` endpoint and screen in the database page to list scheduled PITRs on yugabyted UI. {{}}
* Introduces `upgrade_finalize` command for smoother YugabyteDB version upgrades using yugabyted CLI and includes an `upgrade_ysql_timeout` flag. {{}}
* Directly enables `yb_enable_read_committed_isolation` and `ysql_enable_read_request_caching` on `yb-master` and `yb-tserver` processes. {{}}
* Delivers alerts on user interface when encountering node version mismatches in the cluster. {{}}
* Simplifies yugabyted by dropping Python2 support and transitioning the script to use Python3, replacing deprecated distutils package with shutil. {{}},{{}}
* Enables better handling of multi-valued flags in yugabyted without duplication, making the system more maintainable. {{}}
* Corrects the Sankey diagram for CPU usage by accurately calculating the total number of used/available cores. {{}}
* Made changes to string literals in `yugabyted` to avoid SyntaxWarning and added checks for exceptions during incorrect `advertise_address` input. {{}},{{}}
* Enables correct parsing of startup parameters with spaces in values when using the YSQL Connection Manager. {{}}
* Allows to specify multiple data directories using the new `additional_data_dir` configuration. {{}}
* Enables xCluster replication management between database clusters using new yugabyted commands. {{}}
* Ensures yugabyted UI metrics display properly with Kubernetes OSS operator deployed clusters. {{}}
* Elevates the reliability of the UUID retrieval process for tablet server nodes on the user interface. {{}}
* Ensures accurate CPU usage metrics by updating `prev_ticks_` at each metrics snapshotting iteration. {{}}
* Allows smooth node restart even if the `data_dir` parameter is missing in the user configuration file. {{}}
* Reduces `collect_logs` command failures by removing the yugabyted running check even when the yugabyted process is not running. {{}}
* Enhances `yugabyted configure_read_replica` commands with checks to gracefully handle failures when `data_placement_constraint` lacks `:`. {{}}
### Bug fixes
#### YSQL
* Fixes an error that occurs when decoding null values from a boolean column sorted as NULLS LAST in a secondary index. {{}}
* Fixes YSQL upgrade single connection mode error preventing new connection attempts before the old ones are released. {{}}
* Allows YB Admins to run pg_locks without requiring superuser status. {{}}
* Avoids failure when upgrading from version 2.14/2.16 to 2.20 by introducing a check to ensure pggate can handle RPC metrics sidecar before sending Scanned Rows count. {{}}
* Fixes memory leaks in pg_constraint/pg_attrdef local cache by adding a missing `hash_destroy` call in `YbCleanupTupleCache`. {{}}
* Resolves remaining memory leaks in CacheMemoryContext to stabilize cache memory after every catalog cache refresh. {{}}
* Allows YSQL DDL operations to wait for rollback/roll-forward operations to finish before proceeding. {{}}
* Allows more accurate modeling of base scan costs by taking into account the impact of storage index filters on secondary index. {{}}
* Resolves colocation option issues in table creation linked to table rewriting and partitioning. Enhances the `defGetBoolean` function to parse string values "0" and "1" as false and true respectively, and shifts a verification step to a earlier spot in the CREATE TABLE execution path, ensuring successful table partition creation. {{}},{{}}
* Renames the YSQL configuration parameter `ddl_rollback_enabled` to `yb_ddl_rollback_enabled` for specificity. {{}}
* Adds network latency cost to startup cost, yielding more accurate cost calculations in small tables. {{}}
* Disables bitmap scan by default to prevent unwarranted selection due to lower CBO costs. {{}}
* Reduces unnecessary log messages when `catalog_version_table_in_perdb_mode` is set to true. {{}}
* Corrects an issue where certain unbatchable filters weren't detected during indexpath formation when indexpath accepted batched values from multiple relations. Requires backports to 2.20 and 2.18. {{}}
* Exposes the YSQL configuration parameter `yb_enable_optimizer_statistics` as a flag `ysql_yb_enable_optimizer_statistics`. {{}}
* Corrects buffer overflow during placement validation in `ALTER TABLE SET TABLESPACE` operation. {{}}
* Allows for an enhanced readability and performance of yb_cost_index code, aiding in merging with the pg15 branch. {{}}
* The deadlock issue occurring when both a table and its index are deleted concurrently in yb-master has been resolved. {{}}
* Refines the YbGetOrdinaryColumnsNeedingPgRecheck condition to align with the ybIsTupMismatch implementation, ensuring Postgres rechecks index conditions when the "preliminary check" is skipped due to an invalid target key attnum. {{}}
* Corrects checks in YbIsScanCompatible to ensure the right-hand side (RHS) of all bound index conditions, not just inequalities, fits into the left-hand side (LHS) datatype. {{}}
* Prevents query layer retries for multi-statement queries to avoid redoing whole queries, ensuring idempotence. {{}}
* Fixes a bug that caused incorrect setting of global catalog version mode on TServer start. {{}}
* Fixes a bug in the index tuple width calculation for better YB base scans cost model. {{}}
* Ensures pushed down RowCompareExpressions correctly enforce non-null column references, rectifying previous behavior and enhancing data accuracy. {{}}
* Reduces the frequency of `schema version mismatch` errors during consecutive DDL operations by ensuring the up-to-date schema is fetched. {{}}
* Allows usage of `YsqlDdlRollbackEnabled` in pggate C++ code by correctly passing the result of `YbDdlRollbackEnabled`, reducing DDL atomicity g-flag issues in RF 1 clusters. {{}}
* Adds a new flag `ysql_min_new_version_ignored_count` to prevent a TServer crash caused by the downward shift in yb-master's catalog version, often surfacing post a PITR restore operation. {{}}
* The postgres process no longer crashes when running a "show all" command due to correct placement of the `yb_enable_ddl_atomicity_infra` description. {{}}
* Prevents failures in transaction restarts with UPDATE ...RETURNING queries in debug builds. {{}}
* Reverts updates from `Storage SQL` to `Remote SQL` and `Storage Filter` to `Remote Filter` for Foreign Scan. {{}}
* Adjusts inaccurate `ALTER TABLE` rewrite check for dropped rules to prevent unnecessary command failure. {{}}
* Fills in the "relation" column in `pg_locks` with the correct table OID after a table rewrite. {{}}
* Adjusts the value of YB_AT_REWRITE_ALTER_PRIMARY_KEY to prevent flag clashes and accommodate future upstream PG flags. {{}}
* Fixes the issue of PG crash when `yb_debug_log_catcache_events=1` is used before a database has been selected. {{}}
* Enables backward parallel scan capabilities, adjusting key bounds when conducting descending order scans. {{}}
* Prevents unnecessary CPU cycles and log flooding by not reading `pg_yb_catalog_version` when `enable_ysql=false`. {{}}
* Corrects the log message for successful column drop operations, ensuring accurate representation of DDL operations. {{}}
* Stops Batched Nest Loop (BNL) crashes by ensuring better indexing condition checks. {{}}
* Refines the logic to accurately push down join clauses to batched index scans without causing conflicts. {{}}
* Grants BNL hashtable its own expression context to prevent data overwrites during query execution. {{}}
* Re-enables rechecking for RowCompareExpressions to accurately handle NULL inputs in scan bound calculations. {{}}
* Resolves the `old-style-declaration` error in YbDdlRollbackEnabled by changing its definition to `static inline bool`. {{}}
* Prevents potential crashes by ensuring `yb_table_properties` pointer, in `load_relcache_init_file`, does not point to random, invalid memory. {{}}
* Makes `yb_get_range_split_clause` robust using PG TRY CATCH block, ensuring YB backup doesn't fail. {{}}
* Fixes memory leaks in ybcFetchNextHeapTuple by properly freeing the YBCStatus. {{}}
* Prevents core dumps by ensuring YSQL webserver destruction upon receiving a termination signal. {{}}
* Introduces new functions to enhance and consolidate the focus on tables stored in the system catalog. {{}}
* Allows `CreateNamespaceIfNotExists` function to retry on "already exists" error, preventing race conditions. {{}}
* Fixes the issue when a separately created and later attached partition does not properly inherit the parent's primary key using `ALTER TABLE ...ATTACH PARTITION`. {{}}
* Resolves potential database OID collision with `system_postgres` by excluding reserved OID 65535 in allocation. {{}}
* Allows skipping the relfilenode check on parent partition tables which do not get recreated during table rewrites. {{}}
* Now allows for correct backward prefix-based scanning by eliminating the problematic `kGroupEnd` marker that was leading to inaccurate seek results. {{}}
* Removes the unused function `Catalocustomeranager::WaitForDdlVerificationToFinish` for clarity. {{}}
* Ensures bitmap scans correctly recheck all results and avoid excluding rows, improving accuracy of outcomes. {{}}
* Prevents the `IN` expressions on single column from wrongly taking the tuple path, ensuring correct data processing. {{}}
* Fixes incorrect access to the scan plan's bind descriptor during tuple IN condition rechecks. {{}}
* Allows the creation of new shared relations during YSQL upgrade to have a global impact by incrementing the catalog version across every database. {{}}
* Allows resetting of stats collected by the ANALYZE command, including `reltuples`, `pg_statistic` rows, and `pg_statistic_ext` values. {{}}
* Corrects the YbGetOrdinaryColumnsNeedingPgRecheck function to return table column numbers instead of index numbers, preventing unnecessary data fetches and potential crashes or errors after dropping a column. {{}}
* Resolves a detected deadlock during ALTER TABLE operations, enhancing test stability. {{}}
* Reduces unexpected log messages by not invoking `YsqlDdlTxnCompleteCallback` if all table 'pb_txn_id's in the DDL transaction verifier state are already cleared, avoiding potential deadlock situations in DDL atomicity. {{}}
* Reduces prefix length in the index when using distinct index scan with included columns. {{}}
* Returns more accurate results when running EXPLAIN command by fixing relids of prefix keys under a subquery distinct index scan. {{}}
* Reduces sequence cache collision by incorporating both database and sequence OIDs as the entry key. {{}}
* Prevents a crash related to memory release associated with TupleTableSlots in SubPlans during a Values Scan. {{}}
* Allows faster data inserts into tables with identity columns. {{}}
* Enhances log output by adding missing newlines in the `yb_pclose_check` function and corrects memory allocation. {{}}
* Streamlines the "drop column" operation process, preventing hindrance even if the alter schema RPC is missed. {{}}
* Reduces ASAN/TSAN builds' pressure on t-server/master and avoids timeout issues by using less parallelism. {{}}
* Eliminates unnecessary waiting for concurrent transactions in the DEFERRABLE mode for READ ONLY serializable transactions. {{}}
* Fixes TServer crash when pushing down certain SAOP operations like `string_to_array`. {{}}
#### YCQL
* Allows the deletion of the Cassandra role in YCQLsh without it regenerating upon cluster restart, by adding a flag to mark if the role was previously created. {{}}
* Removes extra reads during the processing of `INSERT INTO ...RETURNS STATUS AS ROW` for CQL tables with a specific primary key, improving system load and efficiency. {{}}
* Now ensures simultaneous registration of new split tablet children, fully covering the keyspace during splits. {{}}
#### DocDB
* Fixes hidden split parent tablets wrongly appearing as leaderless in the master's leaderless tablet endpoint, ensuring accurate load balance status and preventing potential issues with Point-in-Time Recovery (PITR) operations. {{}}
* Ensures failed xCluster setup if the xCluster stream state update to `ACTIVE` does not occur. {{}}
* Fix ensures heartbeat processing doesn't blindly overwrite tablet replica state, avoiding potential data corruption. {{}}
* Fixes crash when parsing an invalid timestamp in LTO build by updating C++ dependencies and error handling. {{}}
* Removes unnecessary flush during snapshot deletion, preventing write blocks. {{}}
* Eliminates possible deadlock during setup replication by fixing the order in which locks are acquired. {{}}
* Fixes issue of scans not honoring timeouts, preventing indefinite reads and reducing CPU usage. {{}}
* Allows restoring a snapshot schedule from a time just before the oldest snapshot, improving usability. {{}}
* Removes assumption that every tablet server hosts tablets, preventing potential crashes. {{}}
* Resolves a heartbeat metrics issue ensuring full xCluster error information is sent to the new master even during a leader failover, and makes `tserver_heartbeat_metrics_interval_ms` runtime updatable. {{}}
* Adds validation to RPCs `DeleteSnapshot` and `RestoreSnapshot` to prevent deletion or use of snapshots involved in ongoing processes. {{}}
* Ensures `Create Table` operation fails if `Alter Replication` encounters an error, enhancing the reliability of replication setup. {{}}
* Converted the `ysql_skip_row_lock_for_update` to an auto-flag to resolve compatibility issues during upgrade, preventing incorrect DB record creations that can affect row visibility and integrity. {{}}
* Modifies memory consumption calculations for pending operations to ensure accurate rejection of new writes at bootstrap, preventing loading failures. {{}}
* Trims large error messages in AsyncRpc::Failed to prevent hitting memory limit and resulting unavailability. {{}}
* Excludes hidden tables from `generate snapshot` output to circumvent cloning failure from recreated tables. {{}}
* Prevents `unexpected leader` fatal errors by updating cached leader terms immediately after a leader change. {{}}
* Renames and updates the description of the flag `min_secustomerent_size_to_rollover_at_flush` for clarity. {{}}
* Changes the class of `enable_automatic_tablet_splitting` flag from `kLocalPersisted` (class 2) to `kExternal` (class 4) to eliminate setup issues with xCluster configurations. {{}}
* Switches from using scoped_refptr to std::shared_ptr for TabletInfo to handle cycles safely. {{}},{{}}
* Updates cotable IDs in flushed frontiers during a snapshot restore, preventing potential post-restore issues. {{}}
* Allows the persistent mapping of source-target schema versions when a new table is added to a colocated database, thus preventing replication from stalling after T-server restarts. {{}}
* Eliminates potential FATAL errors during reported tabletPB creation by ensuring retrieval of schema version is atomic. {{}}
* Ensures the correct order of destroying components, preventing possible concurrent calls on a WAL append callback. {{}}
* Adds a TSAN suppression to manage the apparent race condition in the function `boost::regex_match`. {{}}
* Fixes the compilation error for almalinux8 fastdebug gcc11 that was previously removed from the build matrix. {{}}
* Corrects a bug causing some tablet metrics to display incorrect `metric_type` attribute. {{}}
* Fixes a segmentation fault in yb-master by checking for a null pointer before dereferencing it, addressing an issue in the CDC run on `2.23.0.0-b37-arm`. {{ 
YugabyteDB
YugabyteDB Anywhere