Every iteration over a secondary index tracks a point in the transaction
manager (due to lookup in the primary index). As a result, if the user
calls 'select' or 'pairs' over a huge data set, it will consume a lot of
memory due to this tracked points, even if the user doesn't uses
transactions.

To mitigate this, let's send all read only transactions to read view
immediately so that tracking is disabled completely during iteration.
Note, with this patch select() called outside a transaction doesn't
populate the cache any more, but it seems to be OK as caching large
select() requests results in cache thrashing.

Closes #2534

Currently, if tx is not NULL, tx->read_view is used, otherwise the
global read view is used. For the sake of #2534 (read view for read only
autocommit statements), we will need to pass an arbitrary read view to
this function. So let's add the corresponding argument. This also allows
us to drop env from the argument list.

While we are at it, let's also inline vy_index_full_by_stmt() as it is a
trivial wrapper around vy_index_get().

Needed for #2534

We don't account latency of point lookups, neither we emit a warning if
a point lookup took > too_long_threshold. To fix this, let's move the
too_long_threshold configuration parameter from vy_env to vy_index_env
and check it in both vy_read_iterator_next() and vy_point_lookup().
This will also allow us not to pass too_long_threshold (or vy_env for
that matter) all the way down to read iterator initialization.

The variable is only used for asserts, which
started to fail sporadically because of a missing initialization
since the last patch.

The point iterator is not actually an iterator: it doesn't have an
internal state and it acts as a function. Wrapping it into the iterator
protocol only complicates its usage. Let's turn it into a function.

This is not necessary, because if the index is primary, its key_def and
cmp_def are the same and so the read iterator will not compare extra
tuple parts as it is the case for secondary indexes.

tuple_extract_key() uses region to store the result. If the region is
not truncated, as it is currently the case, the memory consumption can
rocket sky high during a transaction execution. This is especially
critical in case of select() or pairs() over a secondary index.

To fix that, let's introduce vy_stmt_extract_key() wrapper, which would
store the result on malloc, and use it throughout the code. Note, this
doesn't add malloc() invocations, because malloc() has to be invoked by
vy_index_get() anyway - we just move it up.

A DNS issue can raise a SystemError, applier should handle this error

Fixed #3001

 - Use vy_tuple_compare() instead of vy_stmt_compare() in places where
   we know that both arguments are tuples (type != SELECT).

 - Use vy_tuple_compare() instead of tuple_compare() in the write
   iterator to assure that none of its arguments happens to be a key.

 - Use vy_stmt_compare() instead of vy_tuple_compare_with_key() in the
   read iterator when comparing the resulting statement to the search
   key, because the search key may be a tuple.

There's an optimization in vy_read_iterator_evaluate_src(): in case
the evaluated statement exactly matches the search key, we assume
that it precedes all statements that have been evaluated so far.
This assumption holds when we are looking for the next key, but it
results in a crash when we are applying UPSERTs:

  vy_read_iterator_evaluate_src: Assertion `vy_read_iterator_cmp_stmt(itr, src->stmt, itr->curr_stmt) < 0' failed

Actually, this optimization is rather dubious. It saves us one
comparison on the first iteration if the iterator type is LE/GE, the
search key is full, and there's an exact match. However, if there's
no exact match, it will add an extra comparison instead. One might
argue that we should optimize for exact match because of space.get(),
but for space.get(), the read iterator is not invoked at all - we use
point lookup instead. That being said, let's simply zap this
"optimization" altogether to fix this crash.

Closes #3003

A DNS issue can raise a SystemError, applier should handle this error

Fixed #3001

It is not used anywhere anymore. Move readahead configuration
to iproto.cc and zap ibuf.cc

The obuf part of applier->iobuf is unused so let's replace the iobuf
with ibuf. Couple of notes:

 - Do not bother about setting readahead for the ibuf (iobuf_readahead).
   We read rows one by one here anyway.

 - Do not free memory from the ibuf (iobuf_reset() does). The buffer may
   contain two rows at max, which is not a big deal.

Output buffers are now rotated independently of input buffers. The
rotation is done by the tx thread according to the following rules:

 - If both buffers are empty, choose any one.
 - If neither of buffers is empty, write to the current one.
 - If one of the buffers is empty while the other is not, choose
   the empty one (rotate).

The output buffer is modified (rotated, written, reset) exclusively by
the tx thread. The iproto thread just flushes its content to the socket.
To propagate the output buffer state (iproto flush position and tx write
position) between the two threads, we pass it in iproto_msg::wpos.

The patch was originally written by @kostja. I just rebased it, fixed
a couple of bugs, and added some comments.

Note, it updates the tarantool/small version to bring obuf_svp_reset().

Needed for #946

Rationale:
 - It's used in the only place.
 - I want to reuse it for key extraction into a tuple on malloc.

It's really annoying to pass vy_run_env along with vy_run or vy_slice
every time we want to read a run. Let's store a pointer to vy_run_env
immediately in vy_run. This is a widely accepted practice throughout
vinyl - we already do this in case of vy_index, vy_cache, and vy_mem.

vy_stmt_env doesn't really belong to vy_stmt infrastructure, actually it
isn't used there at all, only created and destroyed. Let's rename it to
vy_mem_env, move it to vy_mem.c, and replace vy_mem->allocator with a
pointer to vy_mem_env. This will allow us to account memory tree blocks
there.

Needed for #934

If applier state channel is closed by connect_all cleanup code then
applier_connect function could throw an unwanted exception on a write
to this channel.

Interrupt applier_pause() on fiber_cancel().

Applier orchestration uses fiber_cancel() to stop applier,
so applier_pause() should exit if the fiber is cancelled.

Add test.

Fixes #2991.

on_disconnect trigger works after session was closed,
but some fields were not destructed. E.g., descriptor was not
reinitialized.
Because of that, some code in on_disconnect trigger failed

Like fiber_cond_wait_timeout(), but waits for a deadline to pass.

Follow-up b4bf3fa0 ("applier: use fiber_cond instead of fiber_channel");

There was a real chance that httpd.py got stuck blocked on stdio.write
due to the buffer overflow. The very frequent heartbeat was set to flush
the buffer faster. This commit decreases heartbeat frequency but makes
stdio flush manually.

Lookup for http://mailru takes 10 sec in my environment.
This small change improves speed drastically.

Currently, to set the memory size in the low_quota.lua script, we use a
rather dirty trick: we create a symlink to low_quota.lua and code the
memory size in the name of the link (e.g. low_quota_2.lua). Instead we
can pass the memory size directly via the 'args' parameter so let's do
this.

While we are at it, let's also:

 - Fix a type in vinly/recovery_quota.test.lua:

     -- Disable dump and use all memory up to the limit.
     box.error.injection.set('ERRINJ_VY_RUN_WRITE', false)

   It should be 'true' obviously.

 - Remove useless switch to and from 'default' to restart a replica as
   the 'restart' command can be called from the replica script.

There's no check that range->begin can be NULL (for the leftmost range)
in vy_read_iterator_next_range(), which leads to a crash when trying to
compare range->begin to last_stmt. Add it.

  #0  0x5621e45fc1b1 in print_backtrace+9
  #1  0x5621e4507b9f in _ZL12sig_fatal_cbi+e2
  #2  0x7f819188c0c0 in __restore_rt+0
  #3  0x5621e456e34b in tuple_data+c
  #4  0x5621e456e814 in vy_tuple_compare_with_key+20
  #5  0x5621e4570cec in vy_read_iterator_next_range+139
  #6  0x5621e456fd87 in vy_read_iterator_next_key+275
  #7  0x5621e45710ad in vy_read_iterator_next+22c
  #8  0x5621e453db42 in vinyl_iterator_next+19a
  #9  0x5621e4513772 in iterator_next+cb
  #10 0x5621e45afe4a in box_select+32d
  #11 0x5621e45d4beb in _ZL11lbox_selectP9lua_State+187
  #12 0x5621e461c96b in lj_BC_FUNCC+34
  #13 0x5621e463f4e3 in lua_pcall+18e
  #14 0x5621e45e870c in luaT_call+29
  #15 0x5621e45e1ad7 in lua_fiber_run_f+c0
  #16 0x5621e4507914 in _ZL16fiber_cxx_invokePFiP13__va_list_tagES0_+1e
  #17 0x5621e45f9ba1 in fiber_loop+82
  #18 0x5621e479b31b in coro_init+4c

Fixes 5e414a73 ("vinyl: read iterator: do not reopen all sources when
range is changed")

Closes #2990

* JSON logging fails to encode functions/cdata/udata, it now uses tostring to
  encode some objects

closes gh-2899
closes gh-2900

If the master doesn't generate replication events, the lag on the slave
doesn't get updated. This is confusing, because due to this the lag may
stay high even if the replica is up-to-date with the master.

To fix this, this patch makes relay threads send slaves the current time
on the master in special "heartbeat" messages. A "heartbeat" message is
sent every time the relay thread is woken by timeout while waiting for
WAL events. Upon receiving a "heartbeat" message, a slave updates the
lag and continues waiting for more messages from the master.

Closes #2976

* Input history improvements.

Don't add to input history empty lines and two same lines together (to achieve behavior as in bash, python, etc.)

* Coding style fix.

* better error messages
* print trace

Set lua_atpanic with handler which prints line and file
where panic occured

But in some cases luajit doesn't call this function, though
program fails

Investigation shows that it happens inside unwind library

Apart from being an overkill, using a fiber_channel object for notifying
about applier state changes is actually unsafe, because fiber_channel
methods involve memory allocations and hence may fail due to OOM while
not methods using them expect this. For instance, applier_disconnect()
should never fail, but it actually may as it calls fiber_channel_put()
via applier_set_state().

To avoid unpredictable behavior caused by unhandled exceptions, let's
switch to plain and simple fiber_cond.

Report the LSN and the number of rows that caused the delay so that
the admin can find the problematic record in the xlog. Example:

  too long WAL write: 3 rows at LSN 65: 0.003 sec

Closes #2743

Currently, only user privileges are revoked. If an object has a role
privilege, an attempt to drop it will fail:

  > box.space.test:drop()
  ---
  - error: User '6' is not found
  ...

Fix it and add a test case.

Closes #2710

Prefer performance over correctness :(
iproto_write_error_blocking() may block entire server on a malformed
request. A malformed request can come only from an incorrect
client driver or an attacker. Remove the attack vector by
not attempting to respond to an incorrect request in blocking mode.

Minor code cleanup.

In a case of a fiber death, an active vinyl transaction in it is
not rolled back or commited. It just leaks. Fix it.

Closes #2983