Our usual convention for C code is to return a negative value at failure
and set an entry to the diagnostics area.

When code uses this convention consistently, it is much easier to handle
failures when using it: you always know where to find an error type and
message and how to pass the error to a C or Lua caller.

See also the previous commit ('popen: add missed diag_set in
popen_signal/delete').

Part of #4031

Acked-by: Cyrill Gorcunov <gorcunov@gmail.com>