console: check lua_yaml_encode error in lbox_console_format_yaml

This patch adds a missed check for `lua_yaml_encode` return value,
similar to the check in `console_dump_plain`. Now Lua error is raised
if YAML encoding failed for any reason (e.g. OOM or formatting error).

Before:
```
tarantool> box.error.new(box.error.ILLEGAL_PARAMS, '\x80')
~/test$ echo $?
0
~/test$
```

After:
```
tarantool> box.error.new(box.error.ILLEGAL_PARAMS, '\x80')
---
- error: 'console: an exception occurred when formatting the output:
    expected SCALAR, SEQUENCE-START, MAPPING-START, or ALIAS'
...

tarantool>
```

Part of #6781
Part of #6934

NO_DOC=bugfix
NO_TEST=see later commits
NO_CHANGELOG=see later commits