sql: always treat NaN as NULL

In most cases, NaN was treated as NULL. But in case NaN was returned as
a result of a Lua or C user defined function, it was considered a
double. After this patch, NaN will also be considered NULL in the
specified cases.

Closes #6374
Closes #6572

NO_DOC=bugfix