To run the test you need xk6 installed
(https://github.com/tarantool/xk6-tarantool).

Georgy Moshkin authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This used to cause flakyness of test_bucket_rebalancing_respects_replication_factor
like this:

- governor proposes entry at index N with weight updates,
  and waits till it's applied

- commit index is not updated yet, but governor calls rpc::sharding to
  update the weights in the vshard subsystem on instances and passes the
  lagging commit index for syncing

- peers sync up to the lagging commit index and then reconfigure vshard
  to the outdated weights

- governor thinks everything is in order and waits for updates

- the test fails because buckets don't get rebalanced to the storages
  with weights set to 0

Now it is possible to make the second argument (query parameters)
optional. I.e. now we support 'pico.sql([[select * from t]])' as
well as 'pico.sql([[select * from t where a = ?]], {1})'.

Egor Ivkov authored 1 year ago and Егор Ивков committed 1 year ago

They should cover all the cases specified in the CaS Design Document

Current commit introduces distributed SQL in picodata (we use
'sbroad' library). Now it is possible to execute cluster-wide
SQL with 'pico.sql([[query]], {params})' lua method.

Current limitations:
1. DDL is still not implemented in picodata, so space sharding
   is mocked by '_pico_space' space (we'll remove it later).
2. We still have duplicating symbols of the Tarantool module in
   sbroad and in picodata rlibs.
3. We need to reuse piocdata's sharding functions and handle the
   case when different spaces in the query have different sharding
   functions.

Co-authored-by: Georgy Moshkin <gmoshkin@picodata.io>

Related issues:
- #237 Restrict cas predicate over primary index only.
Having secondary index in the cas predicate is problematic. At the moment we don't store enough information to check the predicate and storing this information would make the logic much complex
and was deemed not important enough for the user.

- #258 Dml semantics
Even though tarantool supports delete/update by secondary key, in the write-ahead log the operations are stored by primary key only. It was decided to also proceed in this direction. The operations will be represented as by primary key only and any secondary key update logic will probably be offloaded to clients/interfaces.

Dmitry Travyan authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This patch introduces `picodata connect` CLI command. It allows
connecting to interactive lua console of a picodata instance via iproto
The whole implementation comes down to invoking a lua script with
`require("console").start()`.

* _picodata_index keys now include space_id
* _picodata_space distribution type changed to "array"
* _picodata_space format type changed to "array"
* add Spaces::insert and Indexes::insert methods
* Ddl now has #[serde(tag = "kind", rename_all = "snake_case")]
* Dml now has #[serde(tag = "op_kind", rename_all = "snake_case")]
* Distribution now has #[serde(tag = "kind", rename_all = "snake_case")]

The main focus is on the snapshot recovery scenarios (bootstrap /
cathcup) interleaving with explicit CaS operations, rather than on
leadership changes and joins.

Consider the case when a client requests `index: X, term: T`, but
`term(X) != T`. Picodata used to check all entries in the range `[X+1,
last]`, but it didn't check the index `X` itself.

This patch provides this one extra check. Thereby it makes sure that
all entries in the range `[X, last]` belong to the same term `T ==
current_term`.

Refactor corresponding tests accordingly

Егор Ивков authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

- Add new variants to `traft::op::Op` for Ddl handling
- Provide corresponding Ddl operation structs
- Provide a builder for `DdlPrepare`

Georgy Moshkin authored 2 years ago and Yaroslav Dynnikov committed 1 year ago

This patch includes:

- Tarantool stored procedure `proc_cas`.
- Lua API `pico.cas`.
- Integration tests.

Now the function returns `first_index` instead of the number of entries
deleted - it has no practical purpose.

This patch adds a missing test for `pico.raft_compact_log` Lua API.
Also, it changes the API to be more alike other relevant Lua functions:
`raft_propose_*`, `raft_read_index`, `raft_timeout_now`.

Yaroslav Dynnikov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Adopted styleguide is "actual == expected", not vice versa.

Bumping `tarantool-module` fixes the `fiber::csw()` performance issue
that caused flaky tests in CI.

Also this patch increases the `test_sigsegv_child` timeout. The test is
flaky too, let's see if it helps.