From 18eaeb737dd595cf5d008e8c70e7b353b4331a74 Mon Sep 17 00:00:00 2001 From: Kirill Yukhin <kyukhin@tarantool.org> Date: Mon, 20 Nov 2023 11:14:19 +0300 Subject: [PATCH] Generate changelog for 3.0.0-beta1 NO_DOC=no code changes NO_TEST=no code changes NO_CHANGELOG=no code changes --- changelogs/3.0.0-beta1.md | 553 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 553 insertions(+) create mode 100644 changelogs/3.0.0-beta1.md diff --git a/changelogs/3.0.0-beta1.md b/changelogs/3.0.0-beta1.md new file mode 100644 index 0000000000..b432ccc1cf --- /dev/null +++ b/changelogs/3.0.0-beta1.md @@ -0,0 +1,553 @@ +Date: 2023-11-20 +Tag: 3.0.0-beta1 + +## Overview + +3.0.0-beta1 is the [beta][release_policy] version of the 3.0 release series. + +This release introduces 95 new features and resolves 51 bugs since +the 2.11 version. There can be bugs in less common areas. If you find any, +feel free to [report an issue][issues] on GitHub. + +Notable changes are: + +* New configuration capabilities (gh-8724) +* Fixes and improvements of the configuration process based on early adopters feedback. + +[release_policy]: https://www.tarantool.io/en/doc/latest/dev_guide/release_management/#release-policy +[issues]: https://github.com/tarantool/tarantool/issues + +## Compatibility + +Tarantool 3.x is backward compatible with Tarantool 1.10.x in the binary data +layout, client-server protocol, and replication protocol. + +Please [upgrade][upgrade] using the `box.schema.upgrade()` procedure to unlock +all the new features of the 2.x series. + +[upgrade]: https://www.tarantool.io/en/doc/latest/book/admin/upgrades/ + +## Functionality added or changed + +### Core + +- **[Breaking change]** Disabled DDL operations with an old system schema. + Now you have to run `box.schema.upgrade()` before you can execute any DDL + operations (gh-7149). +* Now `index_object.parts` contains the following methods, similar to the + `key_def` Lua module: `extract_key()`, `compare()`, `compare_with_key()`, + `merge()` (gh-7356). +* Introduced the `box.watch_once()` function to get the value currently + associated with a notification key on the local instance without subscribing + to future changes (gh-6493). +* Introduced the new `IPROTO_WATCH_ONCE` request to get the value currently + associated with a notification key on a remote instance without subscribing + to future changes (gh-6493). +* Introduced the `conn:watch_once()` net.box connection method to get the value + currently associated with a notification key on a remote instance without + subscribing to future changes. The new method is implemented using the + `IPROTO_WATCH_ONCE` request type (gh-6493). +* Introduced the default field values in the space format (gh-8157). +* Added support for functional variant of the default field values (gh-8609). +* `box.schema` can now be accessed before `box.cfg()`. +* Introduced new methods `tuple:info()` and `space:stat()` with the detailed + information on memory consumed by data (gh-6762). + +### Memtx + +* Added memtx read view statistics to `box.stat.memtx()` (gh-8501). + +### Vinyl + +* Introduced the `memory.tuple` statistic for `box.stat.vinyl()` that shows + the total size of memory occupied by all tuples allocated by the Vinyl engine + (gh-8485). + +### Replication + +* Added a new `bootstrap_leader` configuration option to specify the node from + which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` + to `'config'` and set `bootstrap_leader` value to either the URI or UUID of + the desired bootstrap leader. For example: + ```lua + box.cfg{ + bootstrap_strategy = 'config', + bootstrap_leader = 'localhost:3301', + replication = { + 'localhost:3301', + 'localhost:3302', + }, + listen = '3302', + } + ``` + (gh-7999). +* You may now control which node new replicas choose as a bootstrap leader + without touching node config. To do so, set `box.cfg.bootstrap_strategy` to + `'supervised'`, and the nodes will only bootstrap off the node on which you + called `box.ctl.make_bootstrap_leader()` last. + This works on an empty replica set bootstrap as well: start the admin console + before configuring the nodes. Then configure the nodes: + ```lua + box.cfg{ + bootstrap_strategy = 'supervised', + replication = ..., + listen = ..., + } + ``` + Finally, call `box.ctl.make_bootstrap_leader()` through the admin console + on the node you want to promote. All the nodes will bootstrap off that node + (gh-8509). +* A new option `box.cfg.cluster_name` allows assigning a human-readable name to + the entire cluster. It has to match in all instances and is displayed in + `box.info.cluster.name` (gh-5029). + +* A new option `box.cfg.replicaset_name` allows assigning a human-readable name + to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value + must be the same across all instances of one replicaset. The replicaset name + is displayed in `box.info.replicaset.name` (gh-5029). + +* A new option `box.cfg.instance_name` allows assigning a human-readable name to + the instance. It works the same as `box.cfg.instance_uuid`. Its value must be + unique in the replicaset. The instance name is displayed in `box.info.name`. + Names of other replicas in the same replicaset are visible in + `box.info.replication[id].name` (gh-5029). + +* Instance at rebootstrap can change its UUID while keeping its numeric ID if it + has the same non-empty instance name (gh-5029). +* Added the ability to set the `bootstrap_leader` configuration option to the + instance name of the desired bootstrap leader: + ```lua + box.cfg{ + bootstrap_strategy = 'config', + bootstrap_leader = 'leader-name', + replication = { + ... + }, + ... + } + ``` + (gh-7999, gh-8539). + +### LuaJIT + +* LuaJIT now can handle exceptions on traces (gh-7745). + +### Lua + +* **[Breaking change]** Added the new `varbinary` type to Lua. An object of + this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so + it can be used for storing binary blobs in the database. This also works the + other way round: data fields stored as `MP_BIN` are now decoded in Lua as + varbinary objects, not as plain strings, as they used to be. Since the latter + may cause compatibility issues, the new compat option `binary_data_decoding` + was introduced to revert the built-in decoder to the old behavior (gh-1629). +* Introduced the new Lua module `trigger`. This module enables managing and + calling triggers stored in trigger registry (gh-8656). +* Triggers from `space_object`, `box.session`, and `box.ctl` were moved to + the trigger registry (gh-8657). +* Introduced a new event `'tarantool.trigger.on_change'` in the trigger registry. + It is called when any event in the trigger registry is modified (gh-8664). +* Introduced helpers for encoding and decoding IPROTO packets in Lua (gh-8054). + +#### Msgpack + +* Improved error reporting for `msgpack.decode`. Now, an error raised by + `mgpack.decode` has a detailed error message and the offset in the input + data. If `msgpack.decode` failed to unpack a MsgPack extension, it also + includes the error cause pointing to the error in the extension data + (gh-7986). + +#### Netbox + +* Introduced the new `net.box` module function `from_fd` for creating a new + connection from a socket file descriptor number (gh-8984). + +#### Popen + +* Introduced new option `inherit_fds` for `popen.new`. The option takes + an array of file descriptor numbers that should be left open in the child + process (gh-8926). + +#### Socket + +* Introduced new socket functions `socket.socketpair`, `socket.from_fd`, and + `socket:detach` (gh-8927). + +#### Xlog + +* Introduced the `xlog.meta()` method for reading a meta block from an xlog + file. + +### SQL + +* Introduced the `SHOW CREATE TABLE` statement (gh-8098). +* **[Breaking change]** The `DROP CONSTRAINT` statement has been + improved (gh-9112). +* **[Breaking change]** Names in SQL are now case sensitive. To support backward + compatibility, a second lookup using a name normalized using the old rules is + added (gh-4467). +* `SHOW CREATE TABLE` no longer supports the DEFAULT clause (gh-8793). +* `SQL_EXPR` functions can now be set as a default value (gh-8793). +* A literal set as the default value can no longer have a `+` or `-` sign unless + the literal is numeric (gh-8793). +* **[Breaking change]** SQL now uses the `box` mechanism for default + values (gh-8793). + +### App + +* Added the new `--force-recovery` CLI option (gh-8876). + +### Box + +* Non-system spaces in a snapshot file are now sorted by their IDs. + As before, all non-system spaces are stored after system spaces (gh-7954). +* **[Breaking change]** Dropped support for `IPROTO_CALL16` from `net.box` + (i.e., the `call_16` option and the `call_16` method for `net.box` + connections) (gh-8636). +* Added support for square brackets in procedure resolution for Lua calls + (gh-8604). +* Added support for accepting IPROTO requests with a space or index name instead + of an identifier (gh-8146). +* Added a new `box.info` parameter `hostname` (gh-8605). +* Added the `box.cfg.memtx_sort_threads` parameter that specifies the number of + threads used to sort indexes keys on loading a memtx database. OpenMP is + not used to sort keys anymore (gh-3389). +* Changed the output of the `fselect` index method from a table of strings to + a multiline string and dropped the `print` and `use_nbsp` options. With the + `yaml_pretty_multiline` compat option enabled by default, multiline strings + now look good in the console, so there's no need to return a table of strings + to prettify the `fselect` output anymore. +* Allowed multiple updates to the same tuple field in a single call (gh-8226). +* Introduced `box.tuple.format` that enables format definition for tuples + created via `box.tuple.new` (standalone tuples) (gh-4693). +* **[Breaking change]** Disabled argument list syntax of `box.tuple.new` (this + was needed for gh-4693). It is possible to switch to the old behavior using + the compatibility option `box_tuple_new_vararg`. +* Sped up `index.select` and `index.pairs` with the `after` option by up to 30% + in a synthetic test by eliminating an extra buffer allocation. +* The function `box.session.push` is now deprecated. Consider using + `box.broadcast` instead (gh-8802). +* Introduced the new function `box.session.new` for creating a new IPROTO + session from a socket file descriptor number (gh-8801). +* Introduced the new `lua_eval` and `lua_call` object types for + `box.schema.user.grant`. Granting the `'execute'` privilege on `lua_eval` + allows the user to execute an arbitrary Lua expression with the + `IPROTO_EVAL` request. Granting the `'execute'` privilege on `lua_call` + allows the user to execute any global user-defined Lua function with + the `IPROTO_CALL` request (gh-8803). +* **[Breaking change]** Introduced the new `sql` object type for + `box.schema.user.grant`. Now only users with the `'execute'` privilege + granted on `sql` or `universe` can execute SQL expressions with the + `IPROTO_EXECUTE` or `IPROTO_PREPARE` requests. To revert to the old behavior + (no SQL access checks), use the `sql_priv` compat option (gh-8803). +* The maximum length of `box.cfg{}` string parameters is now 512 instead of 256. +* Added a new flag `tarantool.build.asan` that shows whether build + flag `ENABLE_ASAN` is set. +* Added the `leader_name` field to `box.info.election` (gh-8931). +* Now it is possible to specify the sort order for each part of an index + (gh-5529). +* Added support for sending tuple formats in IPROTO responses. Added a + `box_tuple_extension` backward compatibility option to disable sending + tuple formats in responses to IPROTO_CALL and IPROTO_EVAL requests (gh-8146). +* Added support for sending tuple formats in IPROTO_CALL and IPROTO_EVAL request + arguments. `box_tuple_extension` backward compatibility option can be used to + disable receiving tuple formats in IPROTO_CALL and IPROTO_EVAL request arguments + (gh-8633). + +### CLI + +* **[Behavior change]** Disabled starting the Lua REPL by default when running + Tarantool. Now, Tarantool yields the message that shows the command usage. + To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via + `stdin`, use dash (`-`) as the script name. For more information see a help + message by running `tarantool -h` (gh-8613). +* Introduced new command-line options `--name` and `--config` (gh-8613). + +### Compat + +* The following compatibility options' defaults were switched to new behavior: + * `yaml_pretty_multiline` + * `sql_seq_scan_default` + * `json_escape_forward_slash` + * `fiber_channel_close_mode` + * `fiber_slice_default` + * `box_cfg_replication_sync_timeout` + * `c_func_iproto_multireturn` + + More information on the new behavior can be found on the [Module compat](https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/) page. + +### Config + +* Initial version of the declarative server and cluster configuration + (gh-8724). +* All vinyl options are now supported (gh-8861). +* All feedback options are now supported (gh-8861). +* The `memtx.sort_threads` option is now supported (gh-8861). +* The `bootstrap_leader` option is now supported (gh-8861). +* All flight recorder options are now supported (gh-8861). +* All security hardening `box.cfg{}` options are now supported by + config (gh-8861). +* Added a low priority environment configuration source, which looks into + `TT_*_DEFAULT` variables. It is useful to declare defaults (gh-8862). +* It is now possible to access configuration from the application script using + the `config:get()` method (gh-8862). +* All metrics configuration options from `box.cfg{}` are now supported + in the YAML config (gh-8861). +* Added `--help-env-list` CLI option (gh-8862). +* Introduced a non-public API for extending the declarative config + functionality in Tarantool Community Edition (gh-8862). +* Improved the credentials applier: now it supports two-way synchronization + of roles and privileges for both users and roles (gh-8861). +* Removed the sha1 and sha256 hash entries from the config credentials + schema as not planned for development (gh-8967). +* Implemented a full password support in the `config.credentials` schema, + including a password setting, updating and removal for the `chap-sha1` + auth type (supported by both Tarantool Community Edition and Tarantool + Enterprise Edition) and the `pap-sha256` (just for Enterprise Edition + where it is available) (gh-8967). +* A group name from a topology defined by the cluster configuration doesn't + persisted anymore as `box.cfg.cluster_name` (gh-8862). +* The default directory and file paths are changed to `var/run/<...>`, + `var/log/<...>`, `var/lib/<...>` and so on (gh-8862). +* The directory that contains the configuration file is now added into the Lua + modules search paths (gh-8862). +* Most of vshard options are now added in the config (gh-9007). +* Added initial support of vshard (gh-9007). +* All audit options are now supported (gh-8861). +* Non-default privileges are now revoked from default users and roles + when they are removed from the config (gh-8967). +* Introduced the initial support for roles - programs that run when + a configuration is loaded or reloaded (gh-9078). +* On read-only instances, Tarantool now synchronizes credentials + in the background when switching to read-write mode instead of + skipping the synchronization (gh-8967). +* Now the permissions for space, function, and sequence can be granted + via the config. If the object has not been created yet, the grant is + postponed and executed after the creation of the object (gh-8967). +* Now the brand-new `lua_eval`, `lua_call`, and `sql` object types are + supported by credentials (gh-8967). +* Now a password hash (and salt) will be regenerated for users managed + in the configuration file if `security.auth_type` differs from a user's + `auth_type` (gh-8967). +* Introduced validation for replicaset_name/uuid and instance_name/uuid + mismatches before the recovery process when Tarantool is configured via + a YAML file or etcd. +* If Tarantool is configured via a YAML file or etcd, then instance and replica + set names are automatically set when possible (gh-8978). +* Added support for the `{{ replicaset_name }}` and `{{ group_name }}` + templates in addition to the existing `{{ instance_name }}` (gh-8862). + +### Console + +* Now multiline commands can be used by setting continuation marker + using the `\set continuation` command. + +### Debugger + +* Introduced readline support to tarantool debugger (gh-7738). + +### Fiber + +* Exported `fiber_set_name_n`, `fiber_name`, `fiber_id`, `fiber_csw` and + `fiber_find` into the public C API and usable via FFI as well. +* Make `fiber_set_joinable`, `fiber_set_ctx` and `fiber_get_ctx` + treat the NULL argument as the current fiber. + +### Space + +* Introduced space type: a new space definition field. +* Introduced the fully temporary space type. It is the same as data-temporary + but also has temporary metadata. Temporary spaces can now be created in + read-only mode, they disappear after server restart and don't exist on + replicas (gh-8323). + +### Test + +#### Fuzz + +* Added a grammar-based SQL fuzzer (gh-4826). + +### Build + +* Tarantool does not depend on libgomp anymore (gh-7689). +* Added building static `deb` and `rpm` packages. +* Dropped building distro-specific `deb` and `rpm` Tarantool packages. +* Dropped building Tarantool packages with GC32. Now only GC64 packages + are shipped. +* Exported all LuaJIT public functions from Tarantool binary (gh-3680). +* Updated libcurl to version 8.3.0. + +### Misc + +* Updated luarocks version to 3.9.2 (gh-6597). + +## Bugs fixed + +### Core + +* **[Breaking change]** The key `box.space._schema['cluster']` is renamed to + `'replicaset_uuid'`. That is not expected to be breaking because `_schema` is + an internal system space, but the key was visible in public and documented + (gh-5029). +* **[Breaking change]** The table `box.info.cluster` is renamed to + `box.info.replicaset`. The behaviour can be reverted using the `compat` option + `box_info_cluster_meaning` + (https://tarantool.io/compat/box_info_cluster_meaning) (gh-5029). +* Eliminated implicit conversion of unprintable utf-8 strings to binary blobs + when encoded in YAML. Now unprintable characters are encoded as escaped utf-8 + code points, for example, `\x80` or `\u200B` (gh-8756). +* Fixed decoding of escape sequences for single-byte character codes from YAML. + Before the fix, single-byte character codes between `0x80` and `0xff` would + be erroneously converted to two-byte UTF-8 code points, for example, `\x80` + would be decoded as `\uC280` (gh-8782). +* Now `index_object.parts.exclude_null` always contains `false` rather than + `null` when it is actually `false` (gh-8649). +* Fixed a crash that could happen when inserting an index definition with an + empty parts list directly into `box.space._index` (gh-8688). +* Fixed the memory leak on unpacking an invalid MsgPack error extension + (gh-8921). +* Fixed a use-after-free bug in fiber recycling code (gh-9020). +* Fixed a use-after-free bug in iproto server code (gh-9037). +* Fixed a heap-buffer-overflow bug in fiber creation code (gh-9026). +* An error is no longer raised on an attempt to print a dead fiber (gh-4265). +* Fixed a possible out-of-bound write in debug build on unpacking a MsgPack + error extension (gh-9098). +- Fixed a bug that caused a replication error after calling + `box.schema.downgrade` (gh-9049). +* Fixed a possible crash when unpacking an invalid MsgPack error extension + (gh-9136). +* Fixed a wrong assertion in index comparators when comparing decimals with + floats greater than `1e38`. The error was present only in the debug build. + Despite the failing assertion, the behavior after the assertion was correct + (gh-8472). +* Now the "Type Ctrl-C to exit Tarantool" message is displayed when a local + console is exited, but background fibers are running (gh-7017). +* Fixed a bug that could result in the incorrect `space:bsize()` when altering + a primary index concurrently with DML operations (gh-9247). +* Fixed a crash that happened while printing the stack trace on a stack + overflow bug (gh-9222). +* Fixed the behavior of `log.cfg{modules = ...}`. Now, instead of merging the + new log modules configuration with the old one, it completely overwrites the + current configuration, which is consistent with `box.cfg{log_modules = ...}` + (gh-7962). +* Fixed a bug in the network buffer that could result in a crash when there are + a lot of pending requests (gh-9218) +* Fixed a potential data corruption when the number of tuples in a hash index + is more than 2147483648 (gh-3594). +* Decremented the max space id (`box.schema.SPACE_MAX`). Now, the max space id + equals 2147483646. The limit was decremented because the old value is used as + an error indicator in the box C API. It's still possible to revert to the old + behavior with the compatibility module option `box_space_max` (gh-9118). +* The `tarantool -v` output and `tarantool.build.flags` string now include + build type-specific and LTO flags if any (gh-8022). +* Fixed a possible inconsistent state entering if fibers are joined incorrectly. + Now the `fiber_set_joinable` function panics if the fiber is dead or joined + already. The `fiber_join` and `fiber_join_timeout` functions now panic on a + double join if it is possible to detect it (gh-7562). + +### Replication + +* Fixed a bug that allowed to change replicaset UUID via `_schema` space. +* Fixed a bug when new instances could try to register via an anon instance + which previously failed to apply `box.cfg{replication_anon = false}`. +* Fixed a bug when `box.ctl.demote()` with `box.cfg{election_mode = 'off'}` + and an owned synchro queue could simply not do anything (gh-6860). +* Fixed a false-positive split-brain in a replica set on the first + promotion after an upgrade from versions before 2.10.1 (gh-8996). +* Fixed replicas writing corrupted xlogs when appending data to a local space + from an `on_replace` or `before_replace` trigger on a global replicated space. + Such xlogs were unrecoverable and caused other nodes to break replication with + the replica (gh-8746, gh-8958). + +### LuaJIT + +Backported patches from the vanilla LuaJIT trunk (gh-8825). The following issues +were fixed as part of this activity: + +* Fixed `BC_UCLO` insertion for returns. +* Fixed recording of `BC_VARG` with unused vararg values. +* Initialization instructions on trace are now emitted only for the first + member of a union. +* Prevent integer overflow while parsing long strings. +* Fixed various `^` operator and `math.pow()` function inconsistencies. +* Fixed parsing with predicting `next()` and `pairs()`. +* Fixed binary number literal parsing. Parsing of binary number with a zero + fractional part raises error too now. +* Fixed load forwarding optimization applied after table rehashing. +* Fixed recording of the `BC_TSETM`. +* Fixed the panic routine when `mprotect` fails to change flags for mcode area. +* Fixed frame for on-trace OOM handling. +* Fixed handling of instable types in TNEW/TDUP load forwarding. +* Handled table unsinking in the presence of `IRFL_TAB_NOMM`. +* Fixed a bug when an error could be raised on the non-currently executed + coroutine (gh-6323). +Backported patches from the vanilla LuaJIT trunk (gh-9145). The following issues +were fixed as part of this activity: + +* Fixed error handling after return from a child coroutine. + +### Lua + +#### Fio + +* The default permission mode for `fio.open()` was changed for newly + created files to 0666 (before umask) (gh-7981). + +### SQL + +* Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause + (ghs-119). +* Fixed a memory leak in a parser for a column's default rule (gh-9159). +* Fixed assertion in a debug build when a collation was added after an index + with more than one field (gh-9229). +* Fixed a bug that in some cases would not assign collations to an index created + during `CREATE TABLE` (gh-9229). +* A space no longer loses a default value after + `ALTER TABLE ADD COLUMN` (gh-8793). + +### Box + +- Fixed the ability to drop `_vinyl_deferred_delete` system space (gh-5279). +* Fixed a bug when it was not possible to access spaces named `_stream` and + `_stream_space_cache` and indexes named `_space` using the `net.box` stream + objects (gh-8598). +* **[Breaking change]** The `cord_slab_cache` symbol was removed from the public + API export (gh-7124). +* All DDL functions from the `box.schema` module are now wrapped into a + transaction to avoid database inconsistency on failed operations (gh-4348). +* Fixed crashes if `box.info.memory()`, `box.info.gc()`, `box.info.vinyl()`, + and `box.info.sql()` are called before `box.cfg{}` (gh-9173). +* **[Breaking change]** Added a `c_func_iproto_multireturn` option to the + `compat` module. The new behavior drops an additional array that wraps + multiple results returned via iproto (gh-4799). +* Fixed a bug that allows downgrading from a schema version more recent than + a Tarantool version (gh-9182). +* Fixed a bug when `on_shutdown` triggers weren't run if `os.exit()` was + called from `-e` command-line option (gh-9266). +* **[Breaking change]** `box.schema.user.grant()` now raises an error on + an attempt to grant the `execute` privilege on a space. Historically, + this action was allowed although it had no effect. It's still possible + to revert to the old behavior with the new compatibility option + `box_space_execute_priv` (gh-9277). +* Added a check that disables setting `box.cfg` and `log.cfg` options to + infinite numbers (NaN, Inf). Setting a `box.cfg` or `log.cfg` option to + an infinite number could result in a crash or invalid behavior (gh-4962). + +### Config + +* Fixed an error message if the cluster configuration was not provided or the + instance was not found in the cluster configuration during reload (gh-8862). +* Support parent directories creation for options that accept a directory or a + file (gh-8862). +* Create parent directories for `console.socket` and `log.file` (gh-8862). +* Create the `process.work_dir` directory (gh-8862). +* Consider all the paths as relative to `process.work_dir` when creating + necessary directories (gh-8862). +* Added the validation of instance, replica set, and group names in the + configuration, `--name` CLI option, and `TT_INSTANCE_NAME` environment + variable (gh-8862). + +### Http + +* Fixed a streaming connection stuck if etcd is stopped + unexpectedly (gh-9086). -- GitLab