diff --git a/changelogs/3.0.0-alpha1.md b/changelogs/3.0.0-alpha1.md new file mode 100644 index 0000000000000000000000000000000000000000..f6eec0e1a29ea59acbf6f67c37c7a281f3ce54d2 --- /dev/null +++ b/changelogs/3.0.0-alpha1.md @@ -0,0 +1,434 @@ +Date: 2023-07-05 +Tag: 3.0.0-alpha1 + +## Overview + +3.0.0-alpha1 is the [alpha][release_policy] version of the 3.0 release series. + +This release introduces 42 new features and resolves 70 bugs since +the 2.11 version. There can be bugs in less common areas. If you find any, +feel free to [report an issue][issues] on GitHub. + +Notable changes are: + +* New configuration capabilities (gh-8724) + +## Compatibility + +Tarantool 3.x is backward compatible with Tarantool 2.x in the binary data +layout, client-server protocol, and replication protocol. + +Please [upgrade][upgrade] using the `box.schema.upgrade()` procedure to unlock +all the new features of the 3.x series. + +[upgrade]: https://www.tarantool.io/en/doc/latest/book/admin/upgrades/ + +## Functionality added or changed + +### Core + +- **[Breaking change]** Disabled DDL operations with an old system schema. + Now you have to run `box.schema.upgrade()` before you can execute any DDL + operations (gh-7149). +* The `_schema.max_id` field in the `box.space._schema` function is now deprecated. + Now the `_space:max()` function is used instead of `_schema.max_id` (gh-5997). +* Added the new function `box.malloc.info()` for reporting memory usage of + Tarantool internal data structures allocated with `malloc()`. The function + is available only on Linux (gh-7311). +* Now `index_object.parts` contains the following methods, similar to the + `key_def` Lua module: `extract_key()`, `compare()`, `compare_with_key()`, + `merge()` (gh-7356). +* Introduced the `box.watch_once()` function to get the value currently + associated with a notification key on the local instance without subscribing + to future changes (gh-6493). +* Introduced the new `IPROTO_WATCH_ONCE` request to get the value currently + associated with a notification key on a remote instance without subscribing + to future changes (gh-6493). +* Introduced the `conn:watch_once()` net.box connection method to get the value + currently associated with a notification key on a remote instance without + subscribing to future changes. The new method is implemented using the + `IPROTO_WATCH_ONCE` request type (gh-6493). +* Introduced the default field values in the space format (gh-8157). + +### Memtx + +* Added memtx read view statistics to `box.stat.memtx()` (gh-8501). + +### Replication + +* Added a new `bootstrap_leader` configuration option to specify the node from + which a replica should bootstrap. To do this, set `box.cfg.bootstrap_strategy` + to `'config'` and set `bootstrap_leader` value to either the URI or UUID of + the desired bootstrap leader. For example: + ```lua + box.cfg{ + bootstrap_strategy = 'config', + bootstrap_leader = 'localhost:3301', + replication = { + 'localhost:3301', + 'localhost:3302', + }, + listen = '3302', + } + ``` + (gh-7999). +* You may now control which node new replicas choose as a bootstrap leader + without touching node config. To do so, set `box.cfg.bootstrap_strategy` to + `'supervised'`, and the nodes will only bootstrap off the node on which you + called `box.ctl.make_bootstrap_leader()` last. + This works on an empty replica set bootstrap as well: start the admin console + before configuring the nodes. Then configure the nodes: + ```lua + box.cfg{ + bootstrap_strategy = 'supervised', + replication = ..., + listen = ..., + } + ``` + Finally, call `box.ctl.make_bootstrap_leader()` through the admin console + on the node you want to promote. All the nodes will bootstrap off that node + (gh-8509). +* A new option `box.cfg.cluster_name` allows assigning a human-readable name to + the entire cluster. It has to match in all instances and is displayed in + `box.info.cluster.name` (gh-5029). + +* A new option `box.cfg.replicaset_name` allows assigning a human-readable name + to the replicaset. It works the same as `box.cfg.replicaset_uuid`. Its value + must be the same across all instances of one replicaset. The replicaset name + is displayed in `box.info.replicaset.name` (gh-5029). + +* A new option `box.cfg.instance_name` allows assigning a human-readable name to + the instance. It works the same as `box.cfg.instance_uuid`. Its value must be + unique in the replicaset. The instance name is displayed in `box.info.name`. + Names of other replicas in the same replicaset are visible in + `box.info.replication[id].name` (gh-5029). + +* Instance at rebootstrap can change its UUID while keeping its numeric ID if it + has the same non-empty instance name (gh-5029). +* Added the ability to set the `bootstrap_leader` configuration option to the + instance name of the desired bootstrap leader: + ```lua + box.cfg{ + bootstrap_strategy = 'config', + bootstrap_leader = 'leader-name', + replication = { + ... + }, + ... + } + ``` + (gh-7999, gh-8539). +* Implemented correct recovery of mixed transactions. To do this, set + `box.cfg.force_recovery` to `true`. If you need to revert to the old + behavior, don't set the `force_recovery` option (gh-7932). + +### LuaJIT + +* LuaJIT now can handle exceptions on traces (gh-7745). + +### Lua + +* Embedded the `tarantool/metrics` module for metrics aggregation and export (gh-7725). +* Metrics can now be configured using the `box.cfg()` function (gh-7725). +* **[Breaking change]** Added the new `varbinary` type to Lua. An object of + this type is similar to a plain string but encoded in MsgPack as `MP_BIN` so + it can be used for storing binary blobs in the database. This also works the + other way round: data fields stored as `MP_BIN` are now decoded in Lua as + varbinary objects, not as plain strings, as they used to be. Since the latter + may cause compatibility issues, the new compat option `binary_data_decoding` + was introduced to revert the built-in decoder to the old behavior (gh-1629). + +### SQL + +* Introduced the `SHOW CREATE TABLE` statement (gh-8098). + +### Box + +* Non-system spaces in a snapshot file are now sorted by their IDs. + As before, all non-system spaces are stored after system spaces (gh-7954). +* **[Breaking change]** Dropped support for `IPROTO_CALL16` from `net.box` + (i.e., the `call_16` option and the `call_16` method for `net.box` + connections) (gh-8636). +* Added support for square brackets in procedure resolution for Lua calls + (gh-8604). +* Added support for accepting IPROTO requests with a space or index name instead + of an identifier (gh-8146). +* Added a new `box.info` parameter `hostname` (gh-8605). +* Added the `box.cfg.memtx_sort_threads` parameter that specifies the number of + threads used to sort indexes keys on loading a memtx database. OpenMP is + not used to sort keys anymore (gh-3389). +* Changed the output of the `fselect` index method from a table of strings to + a multi-line string and dropped the `print` and `use_nbsp` options. With the + `yaml_pretty_multiline` compat option enabled by default, multi-line strings + now look good in the console, so there's no need to return a table of strings + to prettify the `fselect` output anymore. +* Allowed multiple updates to the same tuple field in a single call (gh-8226). +* Introduced `box.tuple.format` that enables format definition for tuples + created via `box.tuple.new` (standalone tuples) (gh-4693). +* **[Breaking change]** Disabled argument list syntax of `box.tuple.new` (this + was needed for gh-4693). It is possible to switch to the old behavior using + the compatibility option `box_tuple_new_vararg`. + +#### Cfg + +* Implemented a way to set a table as box.cfg{} options value via + environment variables (gh-8051). + +### Cli + +* **[Behavior change]** Disabled starting the Lua REPL by default when running + Tarantool. Now, Tarantool yields the message that shows the command usage. + To run the Lua REPL, just set the `-i` flag. To pass a Lua script contents via + `stdin`, use dash (`-`) as the script name. For more information see a help + message by running `tarantool -h` (gh-8613). +* Introduced new command-line options `--name` and `--config` (gh-8613). + +### Compat + +* The following compatibility options' defaults were switched to new behavior: + * `yaml_pretty_multiline` + * `sql_seq_scan_default` + * `json_escape_forward_slash` + * `fiber_channel_close_mode` + * `fiber_slice_default` + * `box_cfg_replication_sync_timeout` + + More information on the new behavior can be found on the [Module compat](https://www.tarantool.io/en/doc/latest/reference/reference_lua/compat/) page. + +### Console + +* Now multiline commands can be used by setting continuation marker + using the `\set continuation` command. + +### Test + +#### Fuzz + +* Now LuaJIT can be fuzzed using grammar-based fuzzer (gh-4823). +* Added a grammar-based SQL fuzzer (gh-4826). + +### Build + +* The `zstd` version was updated to pre-1.5.5 (gh-8391). +* Added `tt` weak dependency. The `tt` command line utility combines + `tarantoolctl` and `cartridge-cli` functionality into one tool. + The change is a part of pushing `tt` to the public (tarantool/tt#286). +* Tarantool does not depend on libgomp anymore (gh-7689). +* Hardening against memory corruption attacks is now enabled by default on + FreeBSD (gh-7536). +* Hardening against memory corruption attacks is now enabled by default on + AArch64 systems (gh-7536). +* Added building static `deb` and `rpm` packages. +* Dropped building distro-specific `deb` and `rpm` Tarantool packages. +* Dropped building Tarantool packages with GC32. Now only GC64 packages + are shipped. + +## Bugs fixed + +### Core + +* Fixed various bugs related to unsafe (i.e., from an unknown source) decoding + and validating of MsgPack extensions (ghs-73). +* Fixed a crash in the `net.box` client when a remote space had + a field type that the client version did not support (gh-4632). +* Fixed a bug when a space creation failed with a duplication error. + The issue occurred if the explicit and implicit space IDs were mixed. + Now the actual maximal space `id` is used to generate a new one (gh-8036). +* Relaxed the tuple format requirements on tuples passed as the page starting + position to `index:tuple_pos()` or to the `after` option of `index:select`. + Now, Tarantool validates only the key parts of the index being used and all + primary indexes (gh-8511). +* Fixed a crash that could happen when preparing a crash report on macOS + (gh-8445). +* Fixed an integer overflow issue in `net.box` (ghs-121). +* An `IPROTO_EVENT` packet now has the same sync number as the last + corresponding `IPROTO_WATCH` request (gh-8393). +* Fixed a bug because of which a dirty (not committed to WAL) DDL record could + be written to a snapshot and cause a recovery failure (gh-8530). +* **[Breaking change]** The key `box.space._schema['cluster']` is renamed to + `'replicaset_uuid'`. That is not expected to be breaking because `_schema` is + an internal system space, but the key was visible in public and documented + (gh-5029). +* **[Breaking change]** The table `box.info.cluster` is renamed to + `box.info.replicaset`. The behaviour can be reverted using the `compat` option + `box_info_cluster_meaning` + (https://tarantool.io/compat/box_info_cluster_meaning) (gh-5029). +* Fixed a bug when a tuple could be inserted even if it violates a `unique` + constraint of a functional index that has a nullable part (gh-8587). +* Fixed a bug when Tarantool failed to decode a request containing an unknown + IPROTO key. The bug resulted in broken connectivity between Tarantool 2.10 + and 2.11 (gh-8745). +* Fixed the node writing an empty `00000000000000000000.xlog` file regardless of + the actual vclock when interrupted during the initial `box.cfg()` call + (gh-8704). +* Fixed a bug causing the `ER_CURSOR_NO_TRANSACTION` failure for transactions + on synchronous spaces when the `on_commit/on_rollback` triggers are set + (gh-8505). +* Fixed a bug causing the effective session and user are not propagated to + `box.on_commit` and `box.on_rollback` trigger callbacks when transaction + is synchronous (gh-8742). +* Eliminated implicit conversion of unprintable utf-8 strings to binary blobs + when encoded in YAML. Now unprintable characters are encoded as escaped utf-8 + code points, for example, `\x80` or `\u200B` (gh-8756). +* Fixed a bug when MVCC sometimes lost gap record (gh-8326). +* Fixed a bug when MVCC rollback of prepared statement could break internal + invariants (gh-8648). +* Now MVCC engine automatically aborts a transaction if it reads changes + of a prepared transaction and this transaction is aborted (gh-8654). +* Disabled the backtrace collection feature on the AArch64 Linux platform + (gh-8572). +* Fixed a crash that could happen when Tarantool is started in the + [background mode](https://www.tarantool.io/en/doc/latest/reference/configuration/#confval-background) + (gh-6128). +* Fixed the collection of fiber backtraces on the M1/M2 macOS platform + (gh-8074). +* Fixed a bug that caused writing incorrect values into the `stream_id` field + of xlog headers (gh-8783). +* Fixed decoding of escape sequences for single-byte character codes from YAML. + Before the fix, single-byte character codes between `0x80` and `0xff` would + be erroneously converted to two-byte UTF-8 code points, for example, `\x80` + would be decoded as `\uC280` (gh-8782). + +### Memtx + +* Fixed the ability for the `memtx_memory` parameter to specify a memory size + less than required for the stable operation of the engine (gh-7389). +* Fixed a heap-use-after-free bug in the transaction manager, which could occur + when performing a DDL operation concurrently with a transaction on the same + space (gh-8781). + +### Vinyl + +* Vinyl space and index directories are now created on demand. + They are removed as soon as they become empty (gh-8441). + +### Replication + +* Fixed a bug related to `box.info.replication[...].upstream` being stuck in the "connecting" + state for several minutes after a replica DNS record change (gh-7294). +* Fixed an issue when the nodes synchronizing with + a hung leader reported the leader as alive. + This behavior led to the delay of the new elections (gh-7515). +* Fixed a bug that occurred on applier failure: a node could start an election + without having a quorum to do this (gh-8433). +* Fixed a bug that allowed to change replicaset UUID via `_schema` space. +* Fixed a bug when new instances could try to register via an anon instance + which previously failed to apply `box.cfg{replication_anon = false}`. +* Fixed a possible failure to promote the desired node by `box.ctl.promote()` on + a cluster with nodes configured with `election_mode = "candidate"` (gh-8497). +* Fixed nodes configured with `election_mode = 'candidate'` spuriously detecting + a split-vote when another candidate should win with exactly a quorum of votes + for it (gh-8698). +* Fixed a crash when using transactions with the linearizable isolation level during + a replica reconnect (gh-7991). +* Fixed a possible crash on bootstrap with `box.cfg.bootstrap_strategy = 'auto'` + when some of the bootstrapping nodes were stopped (gh-8757). + +### LuaJIT + +Backported patches from the vanilla LuaJIT trunk (gh-8069). The following issues +were fixed as part of this activity: + +* Fixed successful `math.min/math.max` call with no args (gh-6163). +* Fixed inconsistencies in `math.min/math.max` calls with a NaN arg (gh-6163). +* Fixed `pcall()` call without arguments on arm64. +* Fixed assembling of IR_{AHUV}LOAD specialized to boolean for aarch64. +* Fixed constant rematerialization on arm64. +* Fixed `emit_rma()` for the x64/GC64 mode for non-`mov` instructions. +* Limited Lua C library path with the default `PATH_MAX` value of 4096 bytes. +* The JIT engine was disabled by default on macOS platforms to improve + the user experience. If necessary, you can enable it with `jit.on` (gh-8252). +Backported patches from the vanilla LuaJIT trunk (gh-8516). The following issues +were fixed as part of this activity: + +* Fixed `IR_LREF` assembling for the GC64 mode on x86_64. +* Fixed canonicalization of +-0.0 keys for `IR_NEWREF`. +* Fixed result truncation for `bit.rol` on x86_64 platforms. +* Fixed saved bytecode encapsulated in ELF objects. +* Fixed `lua_yield()` invocation inside C hooks. +* Fixed memory chunk allocation beyond the memory limit. +* Fixed TNEW load forwarding with instable types. +* Fixed use-def analysis for `BC_VARG`, `BC_FUNCV`. + +### Lua + +* The `box.stat.memtx` function is now callable, it returns + all memtx statistics. The `box.stat.memtx.tx()` function + is now equivalent to the `box.stat.memtx().tx` function (gh-8448). +* Exported the missing `AUTH_TYPE` key to `box.iproto.key` (gh-8574). +* Stripped the unwanted `IPROTO_` prefix from the `POSITION`, `AFTER_POSITION`, + and `AFTER_TUPLE` keys exported to `box.iproto.key` (gh-8577). +* Fixed the xlog reader Lua module to show unknown row header fields. Before + this change the xlog reader silently skipped them. + +#### Fio + +* The default permission mode for `fio.open()` was changed for newly + created files to 0666 (before umask) (gh-7981). + +#### Http client + +* Fixed a bug where a response body cannot be decoded (gh-8363). +* Fixed the `Transfer-Encoding: chunked` setting being enabled even if + the `Content-Length` header exists for stream requests (gh-8744). + +### SQL + +* The `ANY` field type now supports collation in SQL (gh-8070). +* Now the `ARRAY`, `MAP`, and `INTERVAL` values cannot be used as + arguments in the `ORDER BY` clause (gh-6668). +* Fixed incorrect conversion of an integer greater than `INT64_MAX` or + less than `0` to a decimal number during SQL arithmetic operations (gh-8460). +* Fixed an assertion when selecting tuples with incomplete internal + format (gh-8418). +* Fixed integer overflow issues in built-in functions (ghs-119). +* Fixed a Use-After-Free vulnerability in the `WITH RECURSIVE` clause + (ghs-119). +* Fixed a possible assertion or segmentation fault when optimizing + `INSERT INTO ... SELECT FROM` (gh-8661). +* Fixed an integer overflow issue and added check for the `printf()` failure due + to too large size (ghs-122). + +### Box + +* Fixed a bug where `box.cfg.force_recovery` doesn't work when there are + no user spaces in a snapshot (gh-7974). +* Fixed a bug when large numbers were encoded incorrectly by `msgpackffi`. + It could lead to wrong select results with large number keys (gh-6119). +* Fixed a crash on an unknown option of the function (gh-8463). +* The `takes_raw_args` option is now handled correctly + during downgrading (gh-8457). +* Fixed several memory leaks on tuple encoding failures (gh-7939). +- Fixed the ability to drop `_vinyl_deferred_delete` system space (gh-5279). +* Fixed a bug (bad error message) in pagination related to the validation of the + `after` position option of the `:select` and `:pairs` methods of space and + index objects (gh-8716). +* Fixed a bug when it was not possible to access spaces named `_stream` and + `_stream_space_cache` and indexes named `_space` using the `net.box` stream + objects (gh-8598). +* Fixed the memory leaks caused by the multi-statement transaction errors in the + space index building and the space format checking operations (gh-8773). +* **[Breaking change]** The `cord_slab_cache` symbol was removed from the public + API export (gh-7124). +* Fixed the privilege check when using spaces with functional indexes and + constraints (gh-7873). + +### Console + +* Fixed `console.local_print()` failing on non-string arguments, which led to + some rare errors. For example, when connecting via tarantoolctl to cartridged + tarantool with incorrect credentials, a cdata error was passed through the + `local_print()`, which failed to interpret it (gh-8374). + +### Datetime + +* Fixed a bug with buffer overflow in `tnt_strptime` (gh-8502). +* Fixed an error in `datetime.set` when `timestamp` is passed along with `nsec`, +`usec`, or `msec` (gh-8583). +* Fixed errors when the string representation of a datetime object had +a negative nanosecond part (gh-8570). + +### Build + +* Enabled compiler optimizations for static build dependencies, which were + erroneously disabled in version 2.10.3 (gh-8606).