box: cleanup on tuple encoding failure

Currently on tuple encoding failure we raise Lua error. In many placess
the error is not handled in Lua C code and we get misc leaks. Let's
instead pass error as return value.

Note that generally speaking encoding code can raise an error on OOM.
Which will lead to leak again. Hopefully application will be killed by
OOM killer instead. Other then that we expect no more errors in the
code. If code calls a user defined callback then pcall is used (see
lua_field_inspect_ucdata for example). So the turn from raising errors
to returning error code seems the right direction.

Closes #7939

NO_DOC=bugfix