A DNS issue can raise a SystemError, applier should handle this error

Fixed #3001

It is not used anywhere anymore. Move readahead configuration
to iproto.cc and zap ibuf.cc

The obuf part of applier->iobuf is unused so let's replace the iobuf
with ibuf. Couple of notes:

 - Do not bother about setting readahead for the ibuf (iobuf_readahead).
   We read rows one by one here anyway.

 - Do not free memory from the ibuf (iobuf_reset() does). The buffer may
   contain two rows at max, which is not a big deal.

Output buffers are now rotated independently of input buffers. The
rotation is done by the tx thread according to the following rules:

 - If both buffers are empty, choose any one.
 - If neither of buffers is empty, write to the current one.
 - If one of the buffers is empty while the other is not, choose
   the empty one (rotate).

The output buffer is modified (rotated, written, reset) exclusively by
the tx thread. The iproto thread just flushes its content to the socket.
To propagate the output buffer state (iproto flush position and tx write
position) between the two threads, we pass it in iproto_msg::wpos.

The patch was originally written by @kostja. I just rebased it, fixed
a couple of bugs, and added some comments.

Note, it updates the tarantool/small version to bring obuf_svp_reset().

Needed for #946

Rationale:
 - It's used in the only place.
 - I want to reuse it for key extraction into a tuple on malloc.

It's really annoying to pass vy_run_env along with vy_run or vy_slice
every time we want to read a run. Let's store a pointer to vy_run_env
immediately in vy_run. This is a widely accepted practice throughout
vinyl - we already do this in case of vy_index, vy_cache, and vy_mem.

vy_stmt_env doesn't really belong to vy_stmt infrastructure, actually it
isn't used there at all, only created and destroyed. Let's rename it to
vy_mem_env, move it to vy_mem.c, and replace vy_mem->allocator with a
pointer to vy_mem_env. This will allow us to account memory tree blocks
there.

Needed for #934

If applier state channel is closed by connect_all cleanup code then
applier_connect function could throw an unwanted exception on a write
to this channel.

Interrupt applier_pause() on fiber_cancel().

Applier orchestration uses fiber_cancel() to stop applier,
so applier_pause() should exit if the fiber is cancelled.

Add test.

Fixes #2991.

on_disconnect trigger works after session was closed,
but some fields were not destructed. E.g., descriptor was not
reinitialized.
Because of that, some code in on_disconnect trigger failed

Like fiber_cond_wait_timeout(), but waits for a deadline to pass.

Follow-up b4bf3fa0 ("applier: use fiber_cond instead of fiber_channel");

There was a real chance that httpd.py got stuck blocked on stdio.write
due to the buffer overflow. The very frequent heartbeat was set to flush
the buffer faster. This commit decreases heartbeat frequency but makes
stdio flush manually.

Lookup for http://mailru takes 10 sec in my environment.
This small change improves speed drastically.

Currently, to set the memory size in the low_quota.lua script, we use a
rather dirty trick: we create a symlink to low_quota.lua and code the
memory size in the name of the link (e.g. low_quota_2.lua). Instead we
can pass the memory size directly via the 'args' parameter so let's do
this.

While we are at it, let's also:

 - Fix a type in vinly/recovery_quota.test.lua:

     -- Disable dump and use all memory up to the limit.
     box.error.injection.set('ERRINJ_VY_RUN_WRITE', false)

   It should be 'true' obviously.

 - Remove useless switch to and from 'default' to restart a replica as
   the 'restart' command can be called from the replica script.

There's no check that range->begin can be NULL (for the leftmost range)
in vy_read_iterator_next_range(), which leads to a crash when trying to
compare range->begin to last_stmt. Add it.

  #0  0x5621e45fc1b1 in print_backtrace+9
  #1  0x5621e4507b9f in _ZL12sig_fatal_cbi+e2
  #2  0x7f819188c0c0 in __restore_rt+0
  #3  0x5621e456e34b in tuple_data+c
  #4  0x5621e456e814 in vy_tuple_compare_with_key+20
  #5  0x5621e4570cec in vy_read_iterator_next_range+139
  #6  0x5621e456fd87 in vy_read_iterator_next_key+275
  #7  0x5621e45710ad in vy_read_iterator_next+22c
  #8  0x5621e453db42 in vinyl_iterator_next+19a
  #9  0x5621e4513772 in iterator_next+cb
  #10 0x5621e45afe4a in box_select+32d
  #11 0x5621e45d4beb in _ZL11lbox_selectP9lua_State+187
  #12 0x5621e461c96b in lj_BC_FUNCC+34
  #13 0x5621e463f4e3 in lua_pcall+18e
  #14 0x5621e45e870c in luaT_call+29
  #15 0x5621e45e1ad7 in lua_fiber_run_f+c0
  #16 0x5621e4507914 in _ZL16fiber_cxx_invokePFiP13__va_list_tagES0_+1e
  #17 0x5621e45f9ba1 in fiber_loop+82
  #18 0x5621e479b31b in coro_init+4c

Fixes 5e414a73 ("vinyl: read iterator: do not reopen all sources when
range is changed")

Closes #2990

* JSON logging fails to encode functions/cdata/udata, it now uses tostring to
  encode some objects

closes gh-2899
closes gh-2900

If the master doesn't generate replication events, the lag on the slave
doesn't get updated. This is confusing, because due to this the lag may
stay high even if the replica is up-to-date with the master.

To fix this, this patch makes relay threads send slaves the current time
on the master in special "heartbeat" messages. A "heartbeat" message is
sent every time the relay thread is woken by timeout while waiting for
WAL events. Upon receiving a "heartbeat" message, a slave updates the
lag and continues waiting for more messages from the master.

Closes #2976

* Input history improvements.

Don't add to input history empty lines and two same lines together (to achieve behavior as in bash, python, etc.)

* Coding style fix.

* better error messages
* print trace

Set lua_atpanic with handler which prints line and file
where panic occured

But in some cases luajit doesn't call this function, though
program fails

Investigation shows that it happens inside unwind library

Apart from being an overkill, using a fiber_channel object for notifying
about applier state changes is actually unsafe, because fiber_channel
methods involve memory allocations and hence may fail due to OOM while
not methods using them expect this. For instance, applier_disconnect()
should never fail, but it actually may as it calls fiber_channel_put()
via applier_set_state().

To avoid unpredictable behavior caused by unhandled exceptions, let's
switch to plain and simple fiber_cond.

Report the LSN and the number of rows that caused the delay so that
the admin can find the problematic record in the xlog. Example:

  too long WAL write: 3 rows at LSN 65: 0.003 sec

Closes #2743

Currently, only user privileges are revoked. If an object has a role
privilege, an attempt to drop it will fail:

  > box.space.test:drop()
  ---
  - error: User '6' is not found
  ...

Fix it and add a test case.

Closes #2710

Prefer performance over correctness :(
iproto_write_error_blocking() may block entire server on a malformed
request. A malformed request can come only from an incorrect
client driver or an attacker. Remove the attack vector by
not attempting to respond to an incorrect request in blocking mode.

Minor code cleanup.

In a case of a fiber death, an active vinyl transaction in it is
not rolled back or commited. It just leaks. Fix it.

Closes #2983

Although a variable of this type is defined in appliers_connect_all(),
it doesn't seem to be used anywhere in this function. Looks like we
forgot to delete it after reworking the applier implementation. Delete
it now.

There is no reason to allocate it on a region. Struct request is not
stored longer than a single statement. In process_rw struct request is
used only to read its attributes. The request itself is not stored
anywhere in transaction between multiple statements.

Instead of try { xstream_write_xc(...); } catch (Exception *e) {...}
do if (xstream_write(...) != 0) { /* process error. */ }

It is more clear, than including half of the function into
try-catch.

* add peer uri to box.info.replication[x].upstream

Closes #2689

In case of a malformed packet or other error, do not
append to output buffer directly, but pump the message
through iproto thread.

This is a pre-requisite for a fix for gh-946.

Add new box configuration options:

  box.cfg.instance_uuid
  box.cfg.replicaset_uuid

Both options take a uuid as a string. If set, they force tarantool to
set the specified UUIDs for instance and/or replica set on bootstrap.
On recovery, they are used to check UUIDs stored in the snapshot against
the ones specified by the user - on mismatch recovery is aborted.

Closes #2967

Since commit ccd451eb ("Always touch snapshpot in checkpoint_daemon")
xlog/checkpoint_daemon leaves the checkpoint_interval set to 30 ms,
which can result in box.snapshot() failure in subsequence tests, as
the one shown below. Fix this.

  xlog/big_tx.test.lua                                    [ pass ]
  xlog/checkpoint_daemon.test.lua                         [ pass ]
  xlog/errinj.test.lua                                    [ disabled ]
  xlog/gh1433.test.lua                                    [ fail ]
  Test failed! Result content mismatch:
  --- xlog/gh1433.result  Fri Dec  1 17:09:56 2017
  +++ xlog/gh1433.reject  Fri Dec  1 17:15:49 2017
  @@ -17,5 +17,5 @@
   ...
   box.snapshot()
   ---
  -- ok
  +- error: Snapshot is already in progress
   ...

msgpack.ibuf_decode is deprecated.

 - Allow to pass a C buffer to msgpack.decode(). Syntax:

     buf = buffer.ibuf()
     ...
     obj, rpos = msgpack.decode(buf.rpos, buf:size())

 - Introduce a version of msgpack.decode() that doesn't check the
   supplied msgpack - msgpack.decode_unchecked(). It has the same
   signature as msgpack.decode() except if called on a C buffer it
   doesn't require the buffer size. It is supposed to supplant
   msgpack.ibuf_decode() over time.

 - Allow to store encoded objects in a user-supplied ibuf. Syntax:

     buf = buffer.ibuf()
     len = msgpack.encode(obj, buf)

   ('len' is the number of bytes stored in the buffer)

 - Add tests.

Closes #2755

If a space has secondary keys, an update operation generates a REPLACE
in the primary key and a DELETE + REPLACE in each secondary key. We
don't need a DELETE in the primary key, because a field indexed by the
primary key cannot be updated so a REPLACE is enough to update the tuple
stored in the index. On the contrary, a field indexed by a secondary key
can be updated so we need a DELETE to remove the old tuple from the
index.

As a result, if a field indexed by a secondary key gets updated often
(e.g. the user frequently calls space:update({x}, {{'+', 2, 1}}) on a
space with a secondary index over field #2), a lot of DELETE statements
will be generated. The DELETE statements won't be compacted until major
compaction so a range select over a secondary index may take long,
because it will have to iterate over all those useless DELETEs.

In fact, the REPLACE generated by an update operation can be safely
substituted with an INSERT in a secondary index. INSERT + DELETE are
annihilated on dump/compaction so that would solve the problem.
Unfortunately, we can't substitute REPLACE with INSERT immediately on
update, because statements are shared between primary and secondary
indexes in memory and we can't use an INSERT in a primary index in case
of update (see above). However, it is OK to turn REPLACE generated by an
update in a secondary key to an INSERT on dump/compaction. We just need
a way to identify such REPLACE statements somehow.

In contrast to normal REPLACEs, a REPLACE statement generated by an
update operation usually has a column mask. There's only one exception:
if an update operation updates all secondary keys, the column mask isn't
stored (vy_stmt_column_mask() returns UINT64_MAX). This is done for the
sake of memory usage minimization, but it doesn't seem to make much
sense: first, updates that touch all secondary indexes should be rare;
second, we save only 8 bytes per statement. Let's remove this
optimization and store column mask in REPLACE statements generated by
update operations unconditionally and use this information in the write
iterator to turn REPLACEs into INSERTs.

See #2875

The idea is simple: if the oldest statement for a given key among all
write iterator's sources is an INSERT, there is either no statements
for this key in older runs or the most recent statement is a DELETE;
in either case we can drop all leading DELETEs from this key's final
history and turn the first REPLACE if any into an INSERT.

Note, if the oldest statement is NOT an INSERT, but the first statement
in the output happens to be an INSERT, we must convert it to a REPLACE
otherwise we risk mistakenly optimizing new DELETEs for this key on the
next compaction.

Closes #2875

If the first statement for a particular key in a transaction write set
is INSERT, there is either no committed statements for this key or the
last committed statement is DELETE so we can

 - drop the final statement if it is a DELETE
 - turn the final statement into INSERT if it is a REPLACE

Needed for #2875

Use IPROTO_INSERT instead of IPROTO_REPLACE if the previous statement
for the same key is either absent or has type IPROTO_DELETE. This will
allow us to annihilate INSERT+DELETE pairs on compaction (currently, we
have to produce DELETE unless it's major compaction).

For now INSERT statements are produced only in the following cases:
 - space:insert() is called - see vy_insert();
 - space:upsert() is called, the space has secondary indexes, and
   the previous statement for the inserted key is either absent or
   has type DELETE - see vy_upsert();
 - space:replace() is called, the space has secondary indexes, and
   the previous statement for the inserted key is either absent or
   has type DELETE - see vy_replace_impl().

No special interpretation of INSERT statement has been added yet - they
are handled just like REPLACE statements. INSERT+DELETE annihilation
will be introduced by the following patches.

Needed for #2875

 - box/alter_limits.test.lua

   The test uses the following code to drop all indexes except primary:

     for k, v in pairs (s.index) do if v.id ~= 0 then v:drop() end end

   Since 's.index' is reinitialized by 'v:drop()', iteration over the
   table using 'pairs' may result in unpredictable behavior. Presumably,
   it is responsible for the following test failure:

     --- box/alter_limits.result     Fri Dec  1 08:30:16 2017
     +++ box/alter_limits.reject     Fri Dec  1 08:32:54 2017
     @@ -399,25 +399,22 @@
      -- unknown index type
      index = s:create_index('test', { type = 'nosuchtype' })
      ---
     -- error: Unsupported index type supplied for index 'test' in space 'test'
     +- error: 'Can''t create or modify index ''test'' in space ''test'': index id too big'

   Let's use a simple for-loop instead of 'pairs' here.

 - box/ddl.test.lua

   Right before the spurious failure shown below, the test executes the
   following line of code:

     box.begin() box.internal.collation.create('test2', 'ICU', 'ru_RU') box.commit()

   If the interpreter doesn't yield before proceeding to the next line,
   the transaction started at this line won't get aborted or committed,
   resulting in the test failure:

     --- box/ddl.result	Tue Oct 17 10:53:32 2017
     +++ box/ddl.reject	Fri Oct 20 11:16:17 2017
     @@ -295,10 +295,11 @@
      ...
      box.internal.collation.create('test', 'ICU', 'ru_RU')
      ---
     +- error: Space _collation does not support multi-statement transactions
      ...

   Fix this by explicitly aborting the transaction on the next line.

 - vinyl/ddl.test.lua

   The test checks that a vinyl transaction is aborted by a concurrent
   DDL using the following piece of code:

     ch = fiber.channel(1)
     s = box.schema.space.create('test', { engine = 'vinyl' })
     pk = s:create_index('primary', { parts = { 1, 'uint' } })
     sk = s:create_index('sec', { parts = { 2, 'uint' } })
     box.begin()
     s:replace({1, 2, 3})
     s:replace({4, 5, 6})
     s:replace({7, 8, 9})
     s:upsert({10, 11, 12}, {})
     _ = fiber.create(function () s:drop() ch:put(true) end)
     box.commit()
     ch:get()

   If the fiber performing table drop happens to yield before calling
   's:drop()', the transaction won't be aborted:

     --- vinyl/ddl.result	Wed Nov 29 06:14:09 2017
     +++ vinyl/ddl.reject	Wed Nov 29 06:19:09 2017
     @@ -650,7 +650,6 @@
      ...
      box.commit()
      ---
     -- error: Transaction has been aborted by conflict
      ...
      ch:get()
      ---

    Obviously, we should wait for the DDL fiber to complete before
    trying to commit the transaction so just swap 'ch:get()' and
    'box.commit()' here.