box: forbid non-string types in key_def.new()

Currently if a non-string type is passed to luaT_key_def_set_part,
lua_tolstring returns null-pointer type_name, which is passed to
a string formatting function in diag_set.

Closes #5222

NO_DOC=bugfix

(cherry picked from commit 5215f3f3)