lua: add varbinary type
Implementation notes: - The varbinary type is implemented as VLS cdata so we can't use the existing luaL_pushcdata and luaL_checkcdatas helpers for pushing an object of this type to Lua stack. Instead, we copied the implementation from the Lua JIT internals. - We already have the code handling `MP_BIN` fields in all built-in serializers. We just need to patch it to convert the data to/from a varbinary object instead of a plain string. - We updated the tuple.tostring method to set the NOWRAP base64 encoder flag when dumping binary blobs. The flag was apparently omitted by mistake because we mask all other new line characters while converting a tuple to a string. - The box/varbinary_type test was rewritten using the luatest framework with all the FFI code needed to insert binary data replaced with the new varbinary object. - We have to update quite a few SQL tests involving varbinary type because binary blobs are now returned as varbinary objects, not as plain strings, as they used to be. Closes #1629 @TarantoolBot document Title: Document the varbinary type The new module `varbinary` was introduced. The module implements the following functions: - `varbinary.new` - constructs a varbinary object from a plain string or cdata pointer and size (to be used with the `buffer` module). - `varbinary.is` - returns true if the argument is a varbinary object. ```Lua local bin = varbinary.new('data') assert(varbinary.is(bin)) assert(not varbinary.is('data')) ``` Like a plain string, a varbinary object stores arbitrary data. Unlike a plain string, it's encoded as a binary blob by the built-in encoders that support the binary type (MsgPack, YAML). (Actually, encoding binary blobs with the proper type is the main goal of the new type.) ``` tarantool> '\xFF\xFE' --- - "\xFF\xFE" ... tarantool> varbinary.new('\xFF\xFE') --- - !!binary //4= ... tarantool> msgpack.encode('\xFF\xFE') --- - "\xA2\xFF\xFE" ... tarantool> msgpack.encode(varbinary.new('\xFF\xFE')) --- - "\xC4\x02\xFF\xFE" ... ``` Note, the JSON format doesn't support the binary type so a varbinary object is still encoded as a plain string: ``` tarantool> json.encode('\xFF\xFE') --- - "\"\xFF\xFE\"" ... tarantool> json.encode(varbinary.new('\xFF\xFE')) --- - "\"\xFF\xFE\"" ... ``` The built-in decoders now decode binary data fields (fields with the 'binary' tag in YAML; the `MP_BIN` type in MsgPack) to a varbinary object by default: ``` tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE')) --- - true ... tarantool> varbinary.is(yaml.decode('!!binary //4=')) --- - true ... ``` This also implies that the data stored in the database under the 'varbinary' field type is now returned to Lua not as a plain string, but as a varbinary object. It's possible to revert to the old behavior by toggling the new compat option `binary_data_decoding` because this change may break backward compatibility: ``` tarantool> compat.binary_data_decoding = 'old' --- ... tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE')) --- - false ... tarantool> varbinary.is(yaml.decode('!!binary //4=')) --- - false ... ``` Please create a documentation page for the new compat option: https://tarantool.io/compat/binary_data_decoding A varbinary object implements the following meta-methods: - `__len` - returns the length of the binary data, in bytes. - `__tostring` - returns the data in a plain string. - `__eq` - returns true if the varbinary object contains the same data as another varbinary object or a string. ```Lua local bin = varbinary.new('foo') assert(#bin == 3) assert(tostring(bin) == 'foo') assert(bin == 'foo') assert(bin ~= 'bar') assert(bin == varbinary.new('foo')) assert(bin ~= varbinary.new('bar')) ``` There are no string manipulation methods, like `string.sub` or `string.match`. If you need to match a substring in a varbinary object, you have to convert it to a string first. For more details, see the [design document][1]. [1]: https://www.notion.so/tarantool/varbinary-in-Lua-a0ce453dcf5a46e3bc421bf80d4cc276
Showing
- changelogs/unreleased/gh-1629-varbinary.md 9 additions, 0 deletionschangelogs/unreleased/gh-1629-varbinary.md
- src/CMakeLists.txt 1 addition, 0 deletionssrc/CMakeLists.txt
- src/box/tuple_convert.c 2 additions, 2 deletionssrc/box/tuple_convert.c
- src/box/tuple_convert_BACKUP_246187.c 286 additions, 0 deletionssrc/box/tuple_convert_BACKUP_246187.c
- src/box/tuple_convert_BASE_246187.c 282 additions, 0 deletionssrc/box/tuple_convert_BASE_246187.c
- src/box/tuple_convert_LOCAL_246187.c 282 additions, 0 deletionssrc/box/tuple_convert_LOCAL_246187.c
- src/box/tuple_convert_REMOTE_246187.c 282 additions, 0 deletionssrc/box/tuple_convert_REMOTE_246187.c
- src/lua/compat.lua 16 additions, 0 deletionssrc/lua/compat.lua
- src/lua/init.c 2 additions, 0 deletionssrc/lua/init.c
- src/lua/msgpack.c 14 additions, 2 deletionssrc/lua/msgpack.c
- src/lua/msgpackffi.lua 29 additions, 3 deletionssrc/lua/msgpackffi.lua
- src/lua/serializer.c 10 additions, 1 deletionsrc/lua/serializer.c
- src/lua/utils.c 47 additions, 0 deletionssrc/lua/utils.c
- src/lua/utils.h 15 additions, 0 deletionssrc/lua/utils.h
- src/lua/varbinary.lua 65 additions, 0 deletionssrc/lua/varbinary.lua
- test/app-luatest/varbinary_test.lua 212 additions, 0 deletionstest/app-luatest/varbinary_test.lua
- test/app/uuid.result 2 additions, 2 deletionstest/app/uuid.result
- test/box-luatest/varbinary_test.lua 139 additions, 0 deletionstest/box-luatest/varbinary_test.lua
- test/box/varbinary_type.result 0 additions, 236 deletionstest/box/varbinary_type.result
- test/box/varbinary_type.test.lua 0 additions, 110 deletionstest/box/varbinary_type.test.lua
Loading
Please register or sign in to comment