We need to duplicate a function for handling space upgrade in read view.
We can't just use func_new(func->def) to do this, because func_new sets
the given func_def to func->def, without copying. Usually, foo_new
duplicates the provided foo_def, e.g. see space_new. Let's make func_new
do the same.

Needed for https://github.com/tarantool/tarantool-ee/issues/163

NO_DOC=internal
NO_TEST=internal
NO_CHANGELOG=internal

box_process_call() uses func_call(), which not only calls the given
function, but also checks that the current user has the right to execute
it. As a result, we can't add auditing for only those function calls
that passed the access check (apparently, there's no reason to log
function calls that failed with an 'access denied' error - we have a
separate audit event for this).

To fix this, let's introduce func_call_no_access_check() helper, which
calls a function without checking access rights, and use it along with
existing func_access_check() in box_process_call(). func_call() is now
an inline function that calls func_access_check() and then on success
func_call_no_access_check().

It's probably wrong that func_call() checks access rights, because this
means that to use a space with a functional index/constraint, the user
needs not only read/write access to the space itself, but also execute
access to the function. I think we should check the right to execute
such function only once - on functional index/constraint creation, not
on every call, but I'm not going to change this now, because nobody's
complained so far, and a change like this needs a proper discussion
anyway.

NO_TEST=refactoring
NO_DOC=refactoring
NO_CHANGELOG=refactoring

There are cases when we need to be sure that a function is not
deleted and/or removed from func cache. For example constraints:
they must hold a pointer to struct func while it's very hard to
determine whether there'a constraint that points to given func.

Implement func pin/unpin for this purpose. You can pin a func to
declare that the func must not be deleted. To have to unpin it
when the func is not needed anymore.

NO_DOC=refactoring
NO_CHANGELOG=refactoring

Fill port with a corrupted data in debug (TRASH it) in order to
detect double port destruction early.

Add a comment for func_call function that describes what states
of ports are expected before and after func_call.

Fix port usage in lua using ffi - allocate a full port structure
instead of a (bit smaller) structure port_c. That makes any port
structure to have fixed determined size which in turn makes safe
to cast and use different port types.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

It never returns NULL anymore, because it uses xmalloc for memory
allocations.

Since we have low level module api now we can switch
the box.schema.func code to use it. In particular we
define schema_module structure as a wrapper over the
modules api -- it carries a pointer to general module
structure.

Because of modules reload functionality the schema modules
carry own cache of modules instances. Thus to make overall
code somehow close to modules api we do:

1) cache operations are renamed to cache_find/put/update/del;
2) C functions are switched to use module_func low level api;
3) because low level modules api carry own references we can
   take no explicit reference when calling a function.

Part-of #4642

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The modules subsystem hides some low-level operations under API.
In particular the modules subsystem is responsible for

 - modules lookup in Lua's "package.search" storage
 - modules caching to eliminate expensive load procedure
 - function symbol resolving

Because naming is intersecting with current module functions
sitting in box/func, lets rename the later to schema_module
prefix. We will use this prefix in next patches to point the
modules in box.schema.func are just a particular user of
the general modules engine.

Part-of #4642

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The only purpose of the module argument is to
notify the caller that the module doesn't exist.
Lets simplify the code and drop this argument.

Part-of #4642

Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

C functions are loaded from .so/.dylib dynamic libraries. A
library is loaded when any function from there is called first
time. And was supposed to be unloaded, when all its functions are
dropped from the schema (box.schema.func.drop()), and none of them
is still in a call. But the unloading part was broken.

In fact, box.schema.func.drop() never unloaded anything. Moreover,
when functions from the module were added again without a restart,
it led to a second mmap of the same module. And so on, the same
library could be loaded any number of times.

The problem was in a useless flag in struct module preventing its
unloading even when it is totally unused. It is dropped.

Closes #4648

Functions are stored in lists inside module objects. Module
objects are stored in a hash table, where key is a package name.
But the key was a pointer at one of module's function definition
object. Therefore, when that function was deleted, its freed
package name memory was still in the hash key, and could be
accessed, when another function was deleted.

Now module does not use memory of its functions, and keep a copy
of the package name.

The function func object used to provide a call method only for
C functions. In scope of this patch it reworked to be a uniform
function call frontend both for C and Lua functions.

Introduced classes func_c and func_lua, that provide own
constructors which produce implementation-specific object with
call and destroy methods.

Needed for #4182, #1260

Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.

To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
A new method get_msgpack returns a content of a port as a
msgpack data. The lifecycle of the returned value is
implementation-specific: it may either be returned directly from
the port, in which case the data will stay alive as long as the
port is alive, or it may be allocated on the fiber()->gc, in
which case the caller is responsible for cleaning up.

Needed for #4182, #1260

Tarantool used to assume that func_new call must not fail and
it used to build a new func object by given definition just on
func cache replace operation. We need to fix it to perform
user-dependent risky actions like Lua function assemble in
further patches.

The replace method is disallowed for _func space because it is
redundant and difficult to maintain in case of functions that
have pre-compiled runtime.

Needed for #4182, #1260

Closes #2946.

@TarantoolBot document
Title: fixed module reload
There was a bug in tarantool documentation:
https://tarantool.io/en/doc/1.7/book/box/
box_schema/#lua-function.box.schema.func.reload
Now it is allowed to reload all functions in loadable
module via one method. Legacy method including finction
name is forbidden.
box.schema.func.reload("utils")       -- ok since now
box.schema.func.reload("utils.func1") -- forbidden since now

Global reload is still unsupported because it seems
to be useless.
box.schema.func.reload()              -- invalid!

Needed for #2652

This patch adds ability to reload C procedures on the fly without
downtime. To achive that, Tarantool loads a new copy of shared
library and starts routing all new request to the new version.
The previous version remains active until all started calls
are finished. All shared libraries are loaded with RTLD_LOCAL,
therefore two or more copies can co-exist without any problems.
From now box loads all external modules via an unique symlink to
avoid caching inside dlopen().

If one of some module function is reloaded then all other functions
from this module will be reloaded.

Reviewed and heavily patched by Roman Tsisyk.

Closes #910

Mixing C-style error handling and C++ exceptions can lead to
serious problems.

In context of #910

Needed for #910

Need for #944

* Extract high-level IPROTO_CALL code to box.cc
* Move execute_c_call to func.cc
* Prepare src/lua/call.cc to rewrite in C

Incompatible changes:

* #300 logic was reverted. An attempt to call an unexisting Lua function
  without universal execute permissions now cause "access denied" message
  instead of "function does not exist".

  Since there is no cache for Lua procedures, existence of a function can be
  checked only by attempt to execute it. Caller **must** have a permission
  to execute function in order to execute functions. It's obvious, isn't it?

  Anyway, this patch doesn't affect user experience with using stored
  procedures. The two steps still must be performed to allow Lua calls
  using the binary protocol:
    1. A function must be defined in Lua
    2. A function must be declared in box.schema and caller must have
       execute permission for this object or for entire "universe".
  The order actually doesn't matter - the both steps must be done.

Introduce a layer of wrappers for a number of internal box functions,
dealing with accss to spaces and indexes.

These wrappers:
    * don't throw exceptions
    * have a common prefix box_
    * are exported in the server development headers

Rewrite Lua C bindings to use the public API described above.
Rewrite Lua FFI bindings to do the same.
Add test.

Implement basic support for stored functions in C/C++.

- split away struct func from struct func_def (function
object and function definition object).
- extend box.schema.func.create() to accept 'language' argument.
- move module API related cmake magic to its own file
- add test/lib to gitignore
- update to the new test-run
- add a test

Other changes:

 * Fix #708: catch exceptions in boxffi_iterator_next
 * Optimize index:get() to directly use index:findByKey() instead of
   box_process() + execute_select() + iterator->init() + iterator->next()

Add a new IPROTO request type - EVAL. EVAL compiles and executes arbitrary
Lua expression and returns raw result (without converting it to a list
of tuples).

Refactor statement transaction lifecycle. Avoid nested
statement transactions. Make sure there is no more than 1
statement transaction present in the system at the moment.

Resolve the bugs this revealed:
- in local hot standby we would parasite on the main or
script fiber session to process requests. When an assert
was added that there is no more than 1 transaction active
at a given moment in time, it turned out that transactions
running on behalf of local hot standby were running in
the scope of session of the run_script or sched fibers.

Create and use an own session there.

- refactor session on_connect/on_disconnect trigger
invocation: do not invoke these triggers for just any session,
only for real connects/disconnects (iproto, admin).

- split session creation/destruction and trigger invocation
Conclude that session storage implementation needs to be
done differently.

This patch add specialized versions of box.process for select, insert,
replace, update and delete requests. This optimization reduces the
number of Lua <-> C switches during request processing. New functions
create struct request directly without allocating temporary buffer for
entire request body.

The new server snapshot format is identical to format of
XLOG, i.e. it contains REPLACE statements with BOX_INSERT
flags for every tuple in a space.

This allows to streamline recovery, since rows from a snapshot
can be treated the same as rows from the write ahead log (XLOG).

This is an incompatible change:
 - tarantool 1.5 won't be able to read data of tarantool 1.6
 - tarantool 1.6, without extra effort, won't be able
 to read data of tarantool 1.5 (a conversion procedure is needed).

 This change as such doesn't break replication, but further changes
are in the pipeline which will inevitably finish this matter up as well.

Why this patch is necessary
---------------------------

To create system spaces dict-v5 branch employs on_replace triggers
mechanism, fired off by txn_replace(). Thus it's vital that all changes
go into spaces using txn_replace().

What else this patch does
-------------------------

- since now XLOG and SNAP have the same format, log_io.cc
code has become a bit simpler
- struct key_def is re-factored to simplify dynamic creation/deletion
of keys
- reference counting for tuple formats is added
- a number of error messages is improved to provide part no
in a multipart key.
- space cache is split away from space.cc into a separate
module, schema.[h,cc]

Instead of populating Lua box.space[] table once at
server start, update it with space data whenever it changes.
This is a pre-requisite patch for the new data dictionary,
which allows adding/dropping spaces on a running server.

Generalize session triggers to be usable for on_commit/on_rollback
events.

Fix style, add comments, expand the manual.