box: add iterator type checking

Return a more meaningful message about a malformed opts to
select, count and pairs methods of index object. Do not fail
with luajit error.

Also allow to pass specific iterator as a box.index.{ALL,...} directly.
For example, `s:pairs(nil, box.index.GT)`.

Fixes #6501

NO_DOC=bug fix

(cherry picked from commit 3a98e079)