box: allow vinyl_memory set to 0 in config

In 1.9 it was possible to have a vinylless configuration with
vinyl_memory=0, allow to do this in 1.10 by adjusting sanity
checks for vinyl_memory and memtx_memory. Now banning only
negative values.
memtx_memory check was changed for consistency, trying to
set memtx_memory to 0 fails anyways.
Also added a test to check that vinyl_memory can actually
be set to 0.

Closes: #3468