yaml: don't encode unprintable strings as binary blobs
Historically, we encode strings that contain invalid or non-printable utf-8 sequences in YAML as binary base64 blobs. We do that because of limitations/bugs of the YAML encoder, which refuses to encode invalid utf-8 strings. To work around this issue, we introduced the helper utf8_check_printable, which is basically a copy of yaml_check_utf8, and treat strings for which it fails as binary data (MP_BIN). This commit updates the YAML submodule to the version where all known issues with encoding invalid/unprintable utf-8 strings are fixed and removes special treatment of such strings (drops utf8_check_printable). Now unprintable or invalid utf-8 sequences are emitted as code points, e.g. '\xFF' or '\uFFFF'. This change is a pre-requisite for introducing the new varbinary type to Lua. Without it plain strings would be implicitly converted to varbinary after decoding/encoding them in YAML, which would be confusing. Closes #8756 NO_DOC=bug fix
Showing
- changelogs/unreleased/gh-8756-yaml-unprintable-utf8-encoding-fix.md 5 additions, 0 deletions.../unreleased/gh-8756-yaml-unprintable-utf8-encoding-fix.md
- src/box/lua/serialize_lua.c 1 addition, 3 deletionssrc/box/lua/serialize_lua.c
- src/box/tuple_convert.c 6 additions, 6 deletionssrc/box/tuple_convert.c
- src/lib/core/util.c 0 additions, 61 deletionssrc/lib/core/util.c
- src/trivia/util.h 0 additions, 9 deletionssrc/trivia/util.h
- test/app-tap/yaml.test.lua 7 additions, 5 deletionstest/app-tap/yaml.test.lua
- test/app/crypto.result 2 additions, 2 deletionstest/app/crypto.result
- test/app/crypto.test.lua 1 addition, 1 deletiontest/app/crypto.test.lua
- test/app/crypto_hmac.result 30 additions, 30 deletionstest/app/crypto_hmac.result
- test/app/crypto_hmac.test.lua 15 additions, 15 deletionstest/app/crypto_hmac.test.lua
- test/app/digest.result 32 additions, 32 deletionstest/app/digest.result
- test/app/digest.test.lua 17 additions, 16 deletionstest/app/digest.test.lua
- test/app/pack.result 2 additions, 2 deletionstest/app/pack.result
- test/app/uuid.result 4 additions, 4 deletionstest/app/uuid.result
- test/app/uuid.test.lua 2 additions, 2 deletionstest/app/uuid.test.lua
- test/box-luatest/gh_6934_nonprintable_error_message_test.lua 2 additions, 2 deletionstest/box-luatest/gh_6934_nonprintable_error_message_test.lua
- test/box/tuple.result 2 additions, 2 deletionstest/box/tuple.result
- test/box/varbinary_type.result 9 additions, 9 deletionstest/box/varbinary_type.result
- test/swim/swim.result 1 addition, 1 deletiontest/swim/swim.result
- test/vinyl/layout.result 8 additions, 4 deletionstest/vinyl/layout.result
Loading