decimal: fix index comparison with Inf, NaN

There was an assertion failure when inserting  a decimal into an index
which contained double Inf or NaN.

The reason for that was never checking decimal_from_*() return values,
and decimal_from_double() not being able to handle NaN or Inf, because
these values are not representable in decimal numbers.

Start handling decimal_from_<type> return values and fix decimal
comparison with Inf, NaN.

Closes #6377

NO_DOC=bugfix