test: check hints corner cases

We must to enshure that hints don't broke tuple comparison.
As different field_types use different routines to calculate hint,
we must to ensure that the calculated values are compatible with
each other.

Also tested the corner cases for large double numbers, due to the
fact that there may be additional difficulties in hint construction.

Follow-up
    9fba29ab memtx: introduce tuple compare hint