The reason of this refactoring is that we need Picodata specific
API rather then the API provided by tarantool module. For example,
table creation needs access to the clusterwide API for sharded and
global spaces.

Yaroslav Dynnikov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Don't update blake3 version, otherwise it results in

```
Updating constant_time_eq v0.2.5 -> v0.3.0

error: package `constant_time_eq v0.3.0` cannot be built because it
requires rustc 1.66.0 or newer, while the currently active rustc version
is 1.65.0
```

Egor Ivkov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This commit introduces new Lua function `pico.help()` along with
description of other public functions.

Current commit introduces distributed SQL in picodata (we use
'sbroad' library). Now it is possible to execute cluster-wide
SQL with 'pico.sql([[query]], {params})' lua method.

Current limitations:
1. DDL is still not implemented in picodata, so space sharding
   is mocked by '_pico_space' space (we'll remove it later).
2. We still have duplicating symbols of the Tarantool module in
   sbroad and in picodata rlibs.
3. We need to reuse piocdata's sharding functions and handle the
   case when different spaces in the query have different sharding
   functions.

Co-authored-by: Georgy Moshkin <gmoshkin@picodata.io>

Bumping `tarantool-module` fixes the `fiber::csw()` performance issue
that caused flaky tests in CI.

Also this patch increases the `test_sigsegv_child` timeout. The test is
flaky too, let's see if it helps.

Yaroslav Dynnikov authored 2 years ago and Yaroslav Dynnikov committed 2 years ago

Cargo used to complain:

```
error: the linking modifiers `+bundle` and `+whole-archive` are not
compatible with each other when generating rlibs
```

1. Remove build dependency on cmake crate. It doesn't fulfill all our
   needs, so we run cmake directly via `std::process::command`. It also
   makes `build.rs` easier to understand.
2. Arrange paths variable names.

lazy-static crate is deprecated and is only used in 2 places, where it
can be easily replaced with something simpler.

Georgy Moshkin authored 2 years ago and Georgy Moshkin committed 2 years ago

It's a small library whose benefits don't overweight the drawbacks.
Any additional dependency crate makes the build slower. But itertools in
particular is harmful because it hides some expensive computations
behind simple tempting to use functions such as `sorted_by`.

The code inside process_topology_request_async has become too complex
therefore TopologyRequest has lost its utility

This somehow broke the build for raft-proto, until I added `feature =
["bytes"]` for protobuf.

Georgy Moshkin authored 2 years ago and Georgy Moshkin committed 2 years ago

deactivation means that instance
- is demoted to learner (if it wasn't one already)
- is marked "inactive" so that it's ignored when determining
    the number of voters required for the cluster

It encapsulates the logics of a JoinRequest batch processing.

Topology module will be quite important in picodata. This first version
misses a lot of features, but a few commits later it's going to
implement quite a lot of logics.

When a new instance is joined - there's one complex thing: raft leader
has to decide where this new instance is going to be emplaced, i.e. what
replicaset should it join. There're many different parameters have an
influence - `repliction_factor`, `failure-domain`, and of course the
existing topology. So, this new `topology` module must make the decision.

This patch only refactors the current Picodata behavior, and doesn't
bring new features for its users. Instead, it opens the door to a future
development.

Also, this patch provides a unit-testing basis for the future features.

Remove prctl dependency that is not available on mac.

feat: raft peer discovery PoC

chore: prevent C functions from being optimized out

feat: improve peer discovery

fix: fix tests after making instance_id arg mandatory

Smart supevision with fork

Make it work

Under development

IPC messages to supervisor

One more little step: entrypoint enum

Arrange IPC from child to supervisor

Remove tarantool_main macro

Persist snapshot

Fix some fresh bugs

Implement postjoin

Discovery under refactoring

Enhance discovery

Working on discovery

Fix all discovery bugs known so far

Draft join algorithm

Fail applying snapshot

Joining a learner works

Cleanup snapshot generation

Reorganize traft code and call join automatically

Change peer.commit_index type from option to u64

Implement autopromotion to voter

Implement read_index

Take read_index before self-romotion

Cleanup excess logs

Cleanup logs and code

Deep refactoring in progress

Finish refactoring db schema

Embed entries applying inside traft node

Refactor raft node communication

Replace fiber channel with a mailbox, which is a `Vec<_>` + fiber cond.
It allows to batch raft commands in a more predictable way and makes
the code less error-prone.

Remove commented code

Simplify raft nodes interaction over net_box

Eliminate `traft::Message` struct because its internals aren't used.
Instead, serialize `raft::Message` using protobuf.

Batch ConnectionPool requests

1. Send messages in batches.
2. Allow changing connection uri.
3. Close unused connections after `inactivity_timeout`.

Enhance the raft node

1. Collect results from raft node
2. Fix initial bootstrap which used to fail due to fiber race.
3. Wrap raft storage operations in a transaction.

Bump tarantool module

Add documentation draft

Cleanup warnings

Try fixing tests

Fix test_storage_log

Fix test_traft_pool

Fix luatest single and couple

Start fixing threesome test

Implement concurrent join requests handling