lua: always try to serialize extensions using `luaL_convertfield`

Currently, we delegate the work of serializing extensions to serializers in
case of known extensions. In case of unknown extensions we try to convert
them from Lua using the serialization and string conversion metamethods.

However, there are cases where we would prefer to do the serialization in
Lua instead of delegating to serializers. Let's always try to serialize
extensions first using Lua via `luaL_convertfield`, and, if it fails, or
the conversion returns an extension again, fallback to the serializers.

Needed for #9105

NO_CHANGELOG=<refactoring>
NO_DOC=<refactoring>
NO_TEST=<refactoring>