lua: replace `api_check()` with `assert()`

`api_check()` is the LuaJIT internal assertion. To prevent inconsistency
during internal assertion changes (for example, during backporting), use
glibc's `assert()` instead.

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

(cherry picked from commit e78afb3b)