net.box: add checking of options

Currently net.box's methods doesn't check types of passed options.
This can lead to Lua's internal errors, which are not self-explaining.

Let's add this functionality and raise errors with meaningful messages
in case of incorrect options.

Closes #6063
Closes #6530

NO_DOC=Not a visible change