lua: fix table.equals result when booleans compared

Before this patch comparison of two `false` values produced
incorrect result. That because nil and false are consieded in the
same way by if operator. This patch fixes an issue and introduces
corresponding tests.

Closes #6386

NO_DOC=bugfix

(cherry picked from commit 840a45c4)