- Jun 08, 2018
-
-
Vladislav Shpilevoy authored
Box.session.push() allows to send a message to a client with no finishing a main request. Tarantool after this patch supports pushes over binary protocol. IProto message is encoded using a new header code - IPROTO_CHUNK. Push works as follows: a user calls box.session.push(message). The message is encoded into currently active obuf in TX thread, and then Kharon notifies IProto thread about new data. Originally Kharon is the ferryman of Hades who carries souls of the newly deceased across the rivers Styx and Acheron that divided the world of the living from the world of the dead. In Tarantool Kharon is a message and does the similar work. It notifies IProto thread about new data in an output buffer carrying pushed messages to IProto. Styx here is cpipe, and the boat is cbus message. One connection has single Kharon for all pushes. But Kharon can not be in two places at the time. So once he got away from TX to IProto, new messages can not send Kharon. They just set a special flag. When Kharon is back to TX and sees the flag is set, he immediately takes the road back to IProto. Herewith a user is not blocked to write to obuf when Kharon is busy. The user just updates obuf and set the flag if not set. There is no waiting for Kharon arrival back. Closes #2677
-
imarkov authored
Add lua bindings of PRIV_XXX constants. This patch helps to avoid using numerical constants of privilleges in schema.lua code. Relates #945
-
Vladimir Davydov authored
So as not to duplicate the same code over and over again.
-
Vladimir Davydov authored
An extra new line looks ugly in the log: 2018-06-06 15:22:22.682 [9807] main/101/interactive C> Tarantool 1.10.1-58-gd2272132 2018-06-06 15:22:22.682 [9807] main/101/interactive C> log level 5 2018-06-06 15:22:22.682 [9807] main/101/interactive I> mapping 268435456 bytes for memtx tuple arena... 2018-06-06 15:22:22.683 [9807] main/101/interactive I> mapping 134217728 bytes for vinyl tuple arena... 2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovery start 2018-06-06 15:22:22.692 [9807] main/101/interactive I> recovering from `./00000000000000000006.snap' 2018-06-06 15:22:22.721 [9807] main/106/applier/ I> remote master is 1.10.1 at 0.0.0.0:44441 2018-06-06 15:22:22.723 [9807] main/106/applier/ C> leaving orphan mode 2018-06-06 15:22:22.723 [9807] main/101/interactive C> replica set sync complete, quorum of 1 replicas formed 2018-06-06 15:22:22.723 [9807] main/101/interactive I> ready to accept requests
-
Vladimir Davydov authored
Neither recovery_new() nor recover_remaining_wals() need to modify it.
-
Vladimir Davydov authored
-
Ilya Markov authored
System views are used instead of direct reads of corresponding system spaces to explore all accessible objects such as spaces, functions, users and e.g. An operation with an inaccessible object produces a 'not found' error even if the object exists. In scope of #3250 Includes up fixes from Georgy
-
- Jun 07, 2018
-
-
Alexander Turenko authored
-
Alexander Turenko authored
* added --verbose to show output of successful TAP13 test (#73) * allow to call create_cluster(), drop_cluster() multiple times (#83) * support configurations (*.cfg files) in core = app tests * added return_listen_uri = <boolean> option for create_cluster() * save and print at fail tarantool log for core = app tests (#87)
-
Vladimir Davydov authored
This patch implements space_vtab::build_index callback for vinyl spaces. Now instead of returning an error in case the space is not empty, the callback will actually try to build a new index. The build procedure consists of four steps: 1. Prepare the LSM tree for building. This implies writing a special record to vylog, VY_LOG_PREPARE_LSM, and adding the new index to the vinyl scheduler so that it can be dumped during build. We need to log the new LSM tree so that we can keep track of run files created for it during build and remove them if build procedure fails. 2. Inserting tuples stored in the space into the new LSM tree. Since there may concurrent DML requests, we install a trigger to forward them to the new index. 3. Dumping the index to disk so that we don't have to rebuild it after recovery. 4. Committing the new LSM tree in vylog (VY_LOG_CREATE_LSM). Steps 1-3 are done from the space_vtab::build_index callback while step 4 is done after WAL write, from index_vtab::commit_create. While step 3 is being performed, new DML requests may be executed for the altered space. Those requests will be reflected in the new index thanks to the on_replace trigger, however they won't be recovered during WAL recovery as they will appear in WAL before the ALTER record that created the index. To recover them, we replay all statements stored in the primary key's memory level when replaying the ALTER record during WAL recovery. Closes #1653
-
Vladimir Davydov authored
Currently, we assume that no two runs of the same range intersect by LSN. This holds, because LSNs grow strictly monotonically, and no transaction may be split between two runs (as we pin each affected vy_mem until the transaction is complete). We ensure this with an assertion in vy_task_dump_complete. However, tuples inserted during index build will have arbitrary (not monotonically growing) LSNs. This is OK as for each particular key, two statements will still have different LSNs, but this may break the assertion in vy_task_dump_complete in case dump occurs while build is in progress. To avoid that, let's relax this limitation and assume that a dumped run may intersect by LSN with runs dumped before. Moreover, let's assume that it may have max LSN less than the max LSN stored on disk so that we should update vy_lsm::dump_lsn only if the dumped run has newer data. Needed for #1653
-
Konstantin Osipov authored
-
Vladimir Davydov authored
The fact that we may yield after we added a new slice created by dump, but before we removed the dumped in-memory index from the LSM tree complicates read iterator logic, as it has to detect such a case and filter out sources that contain duplicates. This logic relies on the fact that no two slices of the same range intersect by LSN. For the sake of ALTER we have to relax this limitation, as statements inserted during index build can have arbitrary (not monotonically growing) LSNs, so the no-LSN-intersection property won't be fulfilled for whole slices, only for individual keys. Since there shouldn't be more than 1000 ranges in the same LSM tree, yielding doesn't make much sense as iteration over the whole range tree should be pretty fast. Besides, dump isn't done frequently. That said, let's remove yielding altogether. Needed for #1653
-
Alexander Turenko authored
It is necessary for build on Ubuntu Bionic. Debian bugreport: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881481 Debhelper commit: https://github.com/Debian/debhelper/commit/740c628a1e571acded7e2aac5d6e7058e61da37f
-
lifemaker authored
-
Vladimir Davydov authored
We need to check unique constraint when building a new index. So let's factor out this helper function and pass space_name, index_name, and read view to it explicitly (because index_name_by_id isn't going to work for an index that is under construction and there's no tx when we are building a new index). Suggested by @Gerold103. Needed for #1653
-
Vladislav Shpilevoy authored
It is quite simple - just use stdout file descriptor as the destination for push messages. It is needed to make remote and local console be similar.
-
- Jun 06, 2018
-
-
Vladislav Shpilevoy authored
box.session.push allows to send some intermediate results in the scope of main request with no finalizing it. Messages can be sent over text and binary protocol. This patch allows to send text pushes. Text push is a YAML document tagged with '!push!' handle and 'tag:tarantool.io/push,2018' prefix. YAML tags is a standard way to define a type of the document. Console received push message just prints it to the stdout (or sends to a next console, if it is remote console too). Part of #2677
-
Konstantin Osipov authored
malloc() or getsockname() may reset the errno. Let's ensure we preserve errno when setting the diagnostics.
-
Vladimir Davydov authored
For the sake of recovery, we keep all vy_lsm_recovery_info objects in vy_recovery::index_id_hash, which maps space_id/index_id to the latest index incarnation. We build this hash while loading records from vylog, which is rather difficult for understanding, because an LSM tree may have a counterpart created by incomplete ALTER (see ->prepared) so we have to link and unlink those objects very carefully. Let's simplify the procedure by postponing hash building until we have loaded all records.
-
Vladimir Davydov authored
vy_mem_commit_stmt() expects statements to be committed in the order of increasing LSN. Although this condition holds now, it won't once we start using this function for building indexes. So let's remove this limitation. Needed for #1653
-
Konstantin Osipov authored
-
Vladimir Davydov authored
Since commit 1e1c1fdb ("vinyl: make read iterator always return newest tuple version") vinyl read iterator guarantees that any tuple it returns is the newest version in the iterator read view. However, if we don't bump mem version after assigning LSN to a mem statement, a read iterator using committed_read_view might not see it and return a stale tuple. Currently, there's no code that relies on this iterator feature, but we will need it for building new indexes. Without this patch, build (introduced later in the series) might return inconsistent results. Needed for #1653
-
Vladimir Davydov authored
Currently, we write new indexes to vylog only after successful WAL write (see vinyl_index_commit_create). This is incompatible with space ALTER - the problem is during ALTER vinyl may need to create new run files, which we need to track in order not to leave garbage if ALTER fails or tarantool exits before ALTER is complete. So this patch splits index creation in two stages, prepare and commit. The 'commit' stage is represented by existing VY_LOG_CREATE_LSM record, which is written from index_vtab::commit_create callback, just like before. For the 'prepare' stage we introduce a new record type, VY_LOG_REPARE_LSM, written from index_vtab::add_primary_key and index_vtab::build_index callbacks, i.e. before WAL write. For now, we don't write anything to prepared, but uncommitted indexes (this will be done later), but we do add prepared indexes to the scheduler so that they can be dumped and compacted. If ALTER fails, we drop prepared indexes in index_vtab::abort_create callback. Prepared but uncommitted indexes are ignored by backup and replication and cleaned up from vylog on restart. Note, we have to rework vinyl/errinj_vylog test in this patch, because index creation (and hence space truncation) commands now fail on vylog error, i.e. a situation when the same index is dropped and recreated multiple times in xlog without having corresponding records in vylog is now impossible. Also, space truncation is not linearizable for vinyl anymore as it may yield before WAL write, while trying to prepare an index in vylog. This is OK - we never promised it is. Remove the corresponding test case. Needed for #1653
-
- Jun 04, 2018
-
-
Vladimir Davydov authored
Vinyl worker threads can consume all disk bandwidth while performing dump or compaction, thus stalling DML requests, which also need some disk bandwidth for WAL. Memtx has a similar problem - it needs to write snapshot files. In case of memtx, we cope with this problem by limiting the write rate with box.cfg.snap_io_rate_limit option. Let's reuse this option for limiting vinyl dump/compaction rate. Closes #3220
-
Vladimir Davydov authored
fiber_sleep() works only if the current thread was created with cord_costart(). Since vinyl worker threads don't need fibers, they are created with cord_start() and hence can't use fiber_sleep(). So to be able to limit rate of vinyl dump/compaction, we have to use ev_sleep() instead of fiber_sleep() in xlog. This is fine by other xlog writers, because they don't use fibers either, neither they should as xlogs are written without coio. Needed for #3220
-
- Jun 02, 2018
-
-
Konstantin Osipov authored
-
Konstantin Osipov authored
-
Konstantin Osipov authored
-
Konstantin Osipov authored
-
Konstantin Osipov authored
-
Konstantin Osipov authored
-
Konstantin Osipov authored
Move SocketError to exception.[h,cc]
-
Konstantin Osipov authored
-
Konstantin Osipov authored
Fix a compiler warning with clang 6
-
- Jun 01, 2018
-
-
Vladimir Davydov authored
Even though we store last box.cfg.checkpoint_count checkpoints, we can only restore to the last one, because the backup API only allows to backup the last checkpoint. This patch adds an optional argument to box.backup.start() which specifies the index of the checkpoint to backup. If it is omitted or is 0, box.backup.start() will return files corresponding to the last checkpoint. If it is 1, it will back the previous checkpoint, and so on. Closes #3410
-
Vladimir Davydov authored
Follow-up #2634 Closes #3439
-
Konstantin Osipov authored
-
Vladimir Davydov authored
index.info() is about statistics so let's rename it to index.stat(). Since index.info() is not documented, we don't need to leave the old alias. Follow-up #3277
-
Vladimir Davydov authored
All global statistics are meant to be in box.stat module so let's move box.info.vinyl to box.stat.vinyl, but leave the old alias - we will remove it in 2.0. Patch all tests to use the new alias. Closes #3277
-