box: rework index_def_is_valid()

This patch renames index_def_is_valid() to index_def_check() and makes
it return 0 or -1 because that function sets a diag.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 5df76ade)