Skip to content
Snippets Groups Projects
Commit 92730aae authored by Vladimir Davydov's avatar Vladimir Davydov Committed by dodokek
Browse files

lua: add varbinary type

Implementation notes:
 - The varbinary type is implemented as VLS cdata so we can't use
   the existing luaL_pushcdata and luaL_checkcdatas helpers for
   pushing an object of this type to Lua stack. Instead, we copied
   the implementation from the Lua JIT internals.
 - We already have the code handling `MP_BIN` fields in all built-in
   serializers. We just need to patch it to convert the data to/from
   a varbinary object instead of a plain string.
 - We updated the tuple.tostring method to set the NOWRAP base64
   encoder flag when dumping binary blobs. The flag was apparently
   omitted by mistake because we mask all other new line characters
   while converting a tuple to a string.
 - The box/varbinary_type test was rewritten using the luatest
   framework with all the FFI code needed to insert binary data
   replaced with the new varbinary object.
 - We have to update quite a few SQL tests involving varbinary type
   because binary blobs are now returned as varbinary objects, not
   as plain strings, as they used to be.

Closes #1629

@TarantoolBot document
Title: Document the varbinary type

The new module `varbinary` was introduced. The module implements the
following functions:
 - `varbinary.new` - constructs a varbinary object from a plain string
   or cdata pointer and size (to be used with the `buffer` module).
 - `varbinary.is` - returns true if the argument is a varbinary object.

```Lua
local bin = varbinary.new('data')
assert(varbinary.is(bin))
assert(not varbinary.is('data'))
```

Like a plain string, a varbinary object stores arbitrary data. Unlike
a plain string, it's encoded as a binary blob by the built-in encoders
that support the binary type (MsgPack, YAML). (Actually, encoding binary
blobs with the proper type is the main goal of the new type.)

```
tarantool> '\xFF\xFE'
---
- "\xFF\xFE"
...

tarantool> varbinary.new('\xFF\xFE')
---
- !!binary //4=
...

tarantool> msgpack.encode('\xFF\xFE')
---
- "\xA2\xFF\xFE"
...

tarantool> msgpack.encode(varbinary.new('\xFF\xFE'))
---
- "\xC4\x02\xFF\xFE"
...
```

Note, the JSON format doesn't support the binary type so a varbinary
object is still encoded as a plain string:

```
tarantool> json.encode('\xFF\xFE')
---
- "\"\xFF\xFE\""
...

tarantool> json.encode(varbinary.new('\xFF\xFE'))
---
- "\"\xFF\xFE\""
...
```

The built-in decoders now decode binary data fields (fields with the
'binary' tag in YAML; the `MP_BIN` type in MsgPack) to a varbinary
object by default:

```
tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE'))
---
- true
...

tarantool> varbinary.is(yaml.decode('!!binary //4='))
---
- true
...
```

This also implies that the data stored in the database under the
'varbinary' field type is now returned to Lua not as a plain string,
but as a varbinary object. It's possible to revert to the old behavior
by toggling the new compat option `binary_data_decoding` because this
change may break backward compatibility:

```
tarantool> compat.binary_data_decoding = 'old'
---
...

tarantool> varbinary.is(msgpack.decode('\xC4\x02\xFF\xFE'))
---
- false
...

tarantool> varbinary.is(yaml.decode('!!binary //4='))
---
- false
...
```

Please create a documentation page for the new compat option:
https://tarantool.io/compat/binary_data_decoding

A varbinary object implements the following meta-methods:
- `__len` - returns the length of the binary data, in bytes.
- `__tostring` - returns the data in a plain string.
- `__eq` - returns true if the varbinary object contains
  the same data as another varbinary object or a string.

```Lua
local bin = varbinary.new('foo')
assert(#bin == 3)
assert(tostring(bin) == 'foo')
assert(bin == 'foo')
assert(bin ~= 'bar')
assert(bin == varbinary.new('foo'))
assert(bin ~= varbinary.new('bar'))
```

There are no string manipulation methods, like `string.sub` or
`string.match`. If you need to match a substring in a varbinary
object, you have to convert it to a string first.

For more details, see the [design document][1].

[1]: https://www.notion.so/tarantool/varbinary-in-Lua-a0ce453dcf5a46e3bc421bf80d4cc276
parent b06e25ff
Loading
Showing
with 1695 additions and 356 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment