- Sep 17, 2021
-
-
Vladimir Davydov authored
This is a preparation for merging serializer_opts into luaL_serializer.
-
Vladimir Davydov authored
It's not needed now, because error marshaling is enabled automatically if the connector supports it (IPROTO_FEATURE_ERROR_EXTENSION is set in IPROTO_ID features). Closes #6428 @TarantoolBot document Title: Drop error_marshaling_enabled session setting box.session.setting.error_marshaling_enabled was used to enable encoding errors returned by CALL/EVAL in the extended format (as the MP_ERROR MsgPack extension). Now, the feature is enabled automatically if the connector supports it (sets IPROTO_FEATURE_ERROR_EXTENSION in IPROTO_ID features).
-
Vladimir Davydov authored
The new IPROTO protocol feature IPROTO_FEATURE_ERROR_EXTENSION enables encoding errors returned by CALL/EVAL commands as the MP_ERROR MsgPack extension. Note, the MP_ERROR extension can still be disabled globally by setting msgpack.cfg.encode_error_as_ext to false. If an IPROTO client doesn't set the feature bit, errors will be encoded as generic cdata objects (converted to strings by default). Needed for #6428 @TarantoolBot document Title: Document IPROTO_FEATURE_ERROR_EXTENSION A new feature bit for the IPROTO_ID command was added: ``` IPROTO_FEATURE_ERROR_EXTENSION = 2 ``` The protocol version was incremented - now it equals 2. If a network client sets this bit, errors returned by CALL/EVAL will be encoded as the MP_ERROR MsgPack extension (unless disabled globally by msgpack.cfg.encode_error_as_ext). If the bit is unset, errors will be encoded according to the serialization rules used for generic cdata objects (converted to strings by default). The built-in net.box connector sets this feature bit. The server sets this feature bit if it supports the MP_ERROR MsgPack extension so a net.box client can explicitly request the feature upon connecting to a server: ```lua net.box.connect(uri, {required_protocol_features = {'error_extension'}}) ```
-
Vladimir Davydov authored
This patch adds a new msgpack.cfg: encode_error_as_ext. Setting it makes msgpack and msgpackffi modules encode errors as the MP_ERROR msgpack extension. If the flag is unset, msgpack.encode behavior depends on encode_load_metatables, encode_use_tostring, and encode_invalid_as_nil options, see luaL_convertfield(), while msgpackffi.encode() will always encode errors as strings. The latter needs to be fixed, but it's out of the scope of this work and tracked separately, see #4499. The new option is enabled by default. Interaction with box.session.settings.error_marshaling_enabled: errors are encoded as the MP_ERROR msgpack extension when returned via IPROTO iff both error_marshaling_enabled and encode_error_as_ext are set. Closes #6433 @TarantoolBot document Title: Document msgpack.cfg.encode_error_as_ext The new option determines how error objects (see box.error.new) are encoded in the msgpack format: - If it's set, errors are encoded as the MP_ERROR msgpack extension. This is the default behavior. - If it's unset, the encoded format depends on other msgpack configuration options (encode_load_metatables, encode_use_tostring, encode_invalid_as_nil). With the otherwise default configuration, they are encoded as strings (see error.message). Functions affected by the default configuration (msgpack.cfg): - msgpack and msgpackffi modules - Storing errors in tuples and spaces (box.tuple.new) - Returning errors from IPROTO CALL/EVAL
-
Vladimir Davydov authored
error_to_mpstream and error_to_mpstream_noext have a lot in common. Factor it out into helper functions.
-
Vladimir Davydov authored
The function doesn't iterate over the stack of errors so it should have suffix '_one', like mp_encode_error_one.
-
Vladimir Davydov authored
It's customary for mp_encode_* functions to take a pointer to the buffer and return a pointer following the end of encoded data. mp_encode_error_one takes (char **) and advances it. Fix it to be consistent with other mp_encode_* functions.
-
Vladimir Davydov authored
There's no reason not to enable extended errors for CALL_16. Enable the feature and add a test. Needed for #6433
-
Vladimir Davydov authored
There's no way to encode an error yet so the test just hard-codes msgpack data. It will be fixed in the future, once we allow to encode errors with msgpack/msgpackffi. Needed for #6433
-
Vladimir Davydov authored
It's not used anymore.
-
Vladimir Davydov authored
Whether errors are encoded as a msgpack extension or not is determined by the serializer_opts::error_marshaling_enabled flag. Although an instance of serizlier_opts is passed to luamp_encode(), it doesn't propagate it to luamp_encode_extension_box(). The latter encodes an error as a msgpack extension if the error_marshaling_enabled flag is set in serializer_opts of the current session. This leads to a bug when luamp_encode() is called with error_marshaling_enabled unset while the current session has the flag set: 1. luaL_tofield() sets field->type to MP_EXT and field->ext_type to MP_UNKNOWN_EXTENSION, because the error_marshaling_enabled flag is unset: https://github.com/tarantool/tarantool/blob/b0431cf8f47e9d081f6a402bc18edb1d6ad49847/src/lua/serializer.c#L548 2. Basing on the ext_type, luamp_encode_r() skips the MP_ERROR swtich-case branch for the default branch and calls the luamp_encode_extension callback: https://github.com/tarantool/tarantool/blob/b0431cf8f47e9d081f6a402bc18edb1d6ad49847/src/lua/msgpack.c#L203 3. The callback implementation (luamp_encode_extension_box()) encodes the error, because the error_marshaling_enabled flag is set in the current session settings, and returns MP_EXT: https://github.com/tarantool/tarantool/blob/b0431cf8f47e9d081f6a402bc18edb1d6ad49847/src/box/lua/init.c#L420 4. luamp_encode_r() assumes that the callback didn't encode the extension, because it returned MP_EXT, and encodes it again as a string: https://github.com/tarantool/tarantool/blob/b0431cf8f47e9d081f6a402bc18edb1d6ad49847/src/lua/msgpack.c#L209 This results in a broken msgpack content. To fix this bug, let's do the following: - luaL_tofield() now sets ext_type to MP_ERROR unconditionally, irrespective of serializer_opts::error_marshaling_enabled. - luamp_encode_r() invokes the luamp_encode_extension callback for a MP_ERROR field only if error_marshaling_enabled is set. If the flag is unset, it proceeds with converting the field to string. - luamp_encode_extension_box() doesn't check serializer_opts anymore. It doesn't need to, because it's called iff error_marshaling_enabled is set. - YAML and JSON encoders are patched to handle the MP_ERROR field type by appending error::errmsg to the output (they use luaL_tofield() internally to determine the field type so they have to handle MP_ERROR). This basically disables error encoding as msgpack extension everywhere except returning an error from a Lua CALL/EVAL, in particular: - when creating a tuple with box.tuple.new(), - when inserting an error into a space, - when encoding an error with the msgpack module. This is okay, because the functionality has always been broken anyway. We will introduce a separate msgpack encoder option to enable encoding errors as MP_ERROR msgpack extension. Looking at the code links above, one is likely to wonder why error encoding was implemented via the encode extension callback in the first place. The lua/msgpack module knows about the MP_ERROR extension and even partially handles it so it'd be only natural to call the error encoder function directly, as we do with decimals and uuids. Unfortunately, we can't do it, because the error encoder is (surprise!) a part of the box library. I filed a ticket to move it to the core lib, see #6432. Closes #6431
-
Vladimir Davydov authored
Now net.box sends IPROTO_ID request on (re)connect to query features supported by the server and report its own features. The version and features reported by the server are checked against the new connection options: - required_protocol_version - min version (unsigned) - required_protocol_features - required features (array of strings) If the server version is older than specified or the server lacks certain features, the connection will fail. Features supported by the server are stored in the peer_protocol_version and peer_protocol_features fields of a connection. Closes #6253 @TarantoolBot document Title: Document required_protocol_version/features net.box options Two new options can now be passed to net.box.connect(): - required_protocol_version: min IPROTO protocol version that must be supported by the server. Type: unsigned integer. - required_protocol_features: array of IPROTO protocol features that must be supported by the server. Type: array of strings. If the server version is less than the specified or the server lacks certain features, the connection will fail with the corresponding error. Querying server features is implemented via the IPROTO_ID command. Currently, there are two features defined: streams and transactions. Irrespective of the options used, the actual version and features are reported via peer_protocol_version and peer_protocol_features fields of the connection. Example: ``` tarantool> require('net.box').connect(3301, { > required_protocol_version = 1, > required_protocol_features = {'transactions'}, > }) --- - peer_protocol_version: 1 peer_uuid: 7a8cfdbd-6bbc-4d10-99e5-cbbd06a2382f opts: required_protocol_version: 1 required_protocol_features: - transactions peer_protocol_features: transactions: true streams: true schema_version: 80 protocol: Binary state: active peer_version_id: 133632 port: '3301' ... ```
-
Vladimir Davydov authored
No functional changes, just get rid of some code duplication.
-
Vladimir Davydov authored
-
Vladimir Davydov authored
The new request can be used by a client to let the server know about supported IPROTO protocol features. The request body contains two fields: IPROTO protocol version and supported IPROTO protocol features. In reply to a IPROTO_ID request, the server sends its own protocol version and supported features. Currently, the actual protocol version is 1 and there are two features defined which are always set - streams and transactions. Part of #6253 @TarantoolBot document Title: Document IPROTO_ID request The new request code is 73. It takes a map in the body with the following keys: - IPROTO_VERSION (0x54) - protocol version (unsigned). - IPROTO_FEATURES (0x55) - array of protocol feature ids (unsigned). A client (connector) can send this request to let the server know about the protocol version and features it supports. The server may enable or disable certain functionality basing on the features supported by the client. In reply to the request, the server sends an IPROTO_OK response, in the body of which it reports its own protocol version and features (the format of the response body is the same as the request body). The request doesn't need authentication to pass. Currently, the actual protocol version is 1 and there are two features defined: - IPROTO_FEATURE_STREAMS - streams support (IPROTO_STREAM_ID header key), id = 0. - IPROTO_FEATURE_TRANSACTIONS - transactions support (IPROTO_BEGIN, IPROTO_COMMIT, IPROTO_ROLLBACK commands), id = 1.
-
Vladimir Davydov authored
- Add names for IPROTO_REPLICA_ANON, IPROTO_ID_FILTER, IPROTO_ERROR, and IPROTO_TERM. - Use "error_24" for IPROTO_ERROR_24, "error" for IPROTO_ERROR.
-
Vladimir Davydov authored
So that it can be used in a struct definition. Also, upper-case the name to conform to our coding conventions.
-
- Sep 16, 2021
-
-
Vladimir Davydov authored
The test is flaky, supposedly because of a bug in Lua JIT. Disabling Lua JIT makes the test pass. Workaround for #5081
-
- Sep 14, 2021
-
-
Alexander Turenko authored
I think that our current process is better. But if someone will want to understand why we stick with it, it is good to have explicit cons and pros.
-
Alexander Turenko authored
-
- Sep 10, 2021
-
-
Oleg Babin authored
This patch reworks approach to fiber management in Lua. Before this patch each action that should return fiber led to new userdata creation that was quite slow and made GC suffer. This patch introduces new field in struct fiber to store a reference to userdata that was created once for a fiber. It allows speedup operations as fiber.self() and fiber.id(). Simple benchmark shows that access to fiber storage is faster in two times, fiber.find() - 2-3 times and fiber.new/create functions don't have any changes.
-
Oleg Babin authored
Before this patch we used an assumption in lua fiber code that all valid lua refs are positive numbers. However there are special constants defined in lua header for such purpose LUA_NOREF and LUA_REFNIL [1]. This patch introduces special value in fiber module that is equal to LUA_REFNIL to be sure that this value will be correct in future static assert is added. Also this patch introduces several usages of this value - let's make fiber code more clearer and start to initialize ref values with LUA_NOREF value. Also it will be needed for future changes. [1] https://pgl.yoyo.org/luai/i/luaL_ref
-
Oleg Babin authored
For future changes we will need the fiber object, not only its id.
-
Oleg Babin authored
Ref is a lua reference to fiber storage. This patch just renames storage.lua.ref to storage.lua.storage_ref to underline that it's not a reference to fiber itself and needed only for fiber storage.
-
Andrey Saranchin authored
The problem is if we add a tuple to index and then fiber yield occurs (build-in-background), replace of this tuple will change it in building index, but it will not be changed back in case of rollback. Closes #5958
-
Andrey Saranchin authored
Currently, if we delete tuple while primary index is altering in background, a reference to removed tuple will be kept by primary index. Closes #6290 Part of #5958
-
- Sep 09, 2021
-
-
Mergen Imeev authored
This patch fixes the comparison between DECIMAL as a left value and DOUBLE greater than or equal to 1e38 or less than or equal to -1e38 as a right value. Any DOUBLE value greater than or equal to 1e38 is now greater than any DECIMAL value, and a DOUBLE value less than or equal to -1e38 is less than any DECIMAL value. This is because our decimal cannot contain more than 38 digits. Closes #6376
-
Serge Petrenko authored
Nullify all the output parameters passed to xrow_decode_subscribe(). This should help us get rid of cases when some of the parameters were left uninitialized when the SUBSCRIBE request didn't contain the corresponding fields. Also it's now possible to remove all the explicit parameter initializations for xrow_decode_* functions using xrow_decode_subscribe() internally. Do that as well. Follow-up #6034
-
EvgenyMekhanik authored
Fixed dropping incoming messages when connection is closed or SHUT_RDWR received and net_msg_max or readahead limit is reached. Now we don't close connection if an error occurred while writing response, but only set a special flag to stop further writing. Connection will be closed when we read 0 from socket and process all requests. Closes #6292
-
EvgenyMekhanik authored
Currently there are several statistic metrics for iproto, which can be obtained by using `box.stat.net`: `CONNECTIONS`, `REQUESTS`, `SENT` and `RECEIVED`. Add new metric `STREAMS` which contain statistics for iproto streams. Part of #6293 @TarantoolBot document Title: getting statistics for iproto streams was implemented Add new metric `STREAMS` to `box.stat.net`, which contain statistics for iproto streams. `STREAMS` contains same counters as `CONNECTIONS` metric in `box.stat.net`: current, rps and total. For example: STREAMS: current: 0 rps: 17980 totat: 112623
-
EvgenyMekhanik authored
Currently getting iproto statistics per thread is possible only in debug mode, using `ERRINJ_IPROTO_SINGLE_THREAD_STAT` (it can be used only in tests). This way have several disadvantages: first of all `errinj` is not a place to get statistics, secondly user may be interested in getting iproto statistics per thread. In this patch `box.stat.net.thread` was implemented, it can be used for getting same statistics as from `box.stat.net` but per thread. Part of #6293 @TarantoolBot document Title: getting network statistic per thread was implemented User has possibility to run several iproto threads, but at the moment he can get only general statistics, which combines statistics for all threads. Now `box.stat.net.thread` has been implemented to get the same statistics as when using `box.stat.net`, but for a thread. User can call `box.stat.net.thread()` as a function to get general network statistics per threads. For example for two iproto threads: --- - - SENT: total: 0 rps: 0 CONNECTIONS: current: 0 rps: 0 total: 0 REQUESTS: current: 0 rps: 0 total: 0 RECEIVED: total: 0 rps: 0 - SENT: total: 0 rps: 0 CONNECTIONS: current: 0 rps: 0 total: 0 REQUESTS: current: 0 rps: 0 total: 0 RECEIVED: total: 0 rps: 0 ... Also user can indexed it as a table by thread number. For example `box.stat.net.thread[1]` returns network statistics for first iproto thread: --- - SENT: total: 0 rps: 0 CONNECTIONS: current: 0 rps: 0 total: 0 REQUESTS: current: 0 rps: 0 total: 0 RECEIVED: total: 0 rps: 0 ...
-
- Sep 08, 2021
-
-
Vladimir Davydov authored
Make lbox_tuple_format_new (exported as box.internal.new_tuple_format to Lua) reuse tuple formats. It's safe, because formats exported to Lua are immutable. This fixes a net.box error when one creates/closes a lot of net.box connections: a net.box connection creates a few formats for a local version of the schema so if the garbage collector isn't invoked to clean up after it, Tarantool can run out of format ids. Closes #6217
-
Vladimir Davydov authored
An ephemeral space format doesn't have a dictionary so we ignore dictionaries in format hashing and comparison functions. However, formats exported to Lua (for net.box schema), which could also be reused, do define field names. Since we'd like to reuse those as well, we need to take field names into account while doing format hashing and comparison. Needed for #6217
-
Vladimir Davydov authored
Apart from ephemeral space formats, we can also reuse formats exported to Lua, e.g. for net.box schema. Let's rename the flag and update the comments to be relevant for all cases. Needed for #6217
-
Vladimir Davydov authored
This patch adds xmalloc, xcalloc, xrealloc, xstrdup, and xstrndup helper functions. Each of them calls the corresponding memory allocation function and panics if it fails. See the issue description for the full justification. Closes #3534
-
- Sep 02, 2021
-
-
Mergen Imeev authored
This patch fixes the problem with copying an empty string in mem_copy(). Previously, because the string length was 0, an error was thrown, but the diag was not set, which could lead to an error due to an empty diag or to a double free. Closes #6157 Closes #6399
-
- Sep 01, 2021
-
-
Nikita Pettik authored
It contains test case verifying that during parallel index build the first index to finish wins the race and aborts another TX(s). For the synchronization sake channels were used. However, in fact containing the same number of tuples in indexes we can't predict which index finishes first, so sometimes channel:get() executed inside fiber building index may hang. Hence, let's simply fill one of spaces with a bit more tuples and remove channel:get() from functions building indexes. Closes #6354
-
Nikita Pettik authored
In case we are updating "cluster" field in on_replace_dd_schema trigger we do not check verifying that value corresponding to "cluster" key is uuid. Meanwhile in contrast to other tuple_field_*() decoding functions tuple_field_uuid() EXPECTS that passed value is type of mp_string. So let's add extra string check to tuple_field_uuid(). Closes #6332
-
Nikita Pettik authored
On replace data-dictionary triggers do not expect that exceptions can be raised from their body. Unfortunately, on_replace_dd_schema still contains one invocation of diag_raise() which may lead to unpredictable results. So let's replace it with diag_set(). Part of #5332
-
EvgenyMekhanik authored
In previous commit closing connection without dropping requests was implemented. Now we can enable part of iproto transaction test which depends on it. Also test was reworked to make it more clear and all tests for 'vinyl' was removed because they are not related to streams.
-