lua/serializer: encode Lua number -2^63 as integer

This patch fixes a bug that appeared after commit
3a13be1d ('lua: fix lua_tofield()
for 2**64 value') .

Due to this bug, -2^63 was serialized as double, although it
should be serialized as integer.

Closes #4672