box: move index_opts::hint check from Lua to space_check_index_def

The checks in box.schema.index.create() and box.schema.index.alter()
were case sensitive, also it was possible to insert incorrect index
options directly into `box.space._index`. Fixed by adding checks
to memtx_space_check_index_def() and vinyl_space_check_index_def().

Closes #8937

NO_DOC=bugfix

(cherry picked from commit 4e25384b)