console: raise error if lua_encode failed

lua_encode can raise an exception (e.g. from luaL_checkfield) or return
`nil, err` result. Handle it in lbox_console_format_lua, similar to
lbox_console_format_yaml.

When fixed, it will expose another issue - nil value can not be
serialized:

```
tarantool> \set output lua
true;
tarantool> 1, nil, 2
1, {error = "console: exception while formatting the output:
    \"serializer: unexpected data (nd.field.size 0 nd.field.type 5)\""}, 2;
tarantool>
```

Fix this too.

Part of #6781
Part of #6934

NO_DOC=bugfix
NO_TEST=not a visible change
NO_CHANGELOG=not a visible change