lua/decimal: return pointer from luaT_isdecimal()

This way we can use just luaT_isdecimal() instead of two calls:
luaT_isdecimal() + luaT_checkdecimal() or luaT_isdecimal() +
luaL_checkcdata(). It is convenient and we already follow this way in
luaT_istuple().

The difference from luaT_checkdecimal() is that luaT_isdecimal() does
not raise a Lua exception. In may be undesirable and/or complicated to
handle in some contexts.

This is the preparation for exposing luaT_isdecimal() into the module
API.

Part of #7228

NO_DOC=refactoring, no user-visible changes
NO_TEST=will be tested in a next commit, after exposing to the
        module API
NO_CHANGELOG=refactoring, no user-visible changes