Web UI feature is already enabled in CI `test` stage. This patch adds it
to the `pack` stage.

To do that, a path to a webui buldle can be passed explicitly, and it
does in CI which now builds it in a separate stage.

In default Rust features `webui` remains disabled by default to keep
minimal development environment truly minimal.

Anatoly Popov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

1. http-server:

    1. Get info about tiers from _pico_tier
    2. Get info about instance addresses from:
        - http: lua api
        - binary:  _pico_peer_addresses
    3. Switch from arrays to map where it makes sense
    4. Compose all the data to `cluster_state` object
    5. Get version information for lua api.

2. Tests:

    1. Add DeepDiff package to better diff for json
    2. Amend tests to new specification
    3. Get picodata version via lua api and use it in asserts

3. Chore:

    1. Add .envrc and .direnv to .gitingore (direnv specific files)
    2. Add .vscode to .gitignore.

Signed-off-by: Anatoly Popov <a.popov@picodata.io>

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

close: https://git.picodata.io/picodata/picodata/picodata/-/issues/449

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

After seeing failed checkout here
https://git.picodata.io/picodata/picodata/picodata/-/jobs/106805
I've found that gitlab supports specifying number of retries per job
stage. Use 3 attempts before giving up on fetching sources. See
https://docs.gitlab.com/ee/ci/runners/configure_runners.html#job-stages-attempts
for details

Having two different fixtures `binary_path` and
`path_to_binary_with_webui` results in weird failures in CI:

```
FAILED test/int/test_couple.py::test_exit_after_apply - conftest.TarantoolError:
    ('ER_NO_SUCH_PROC', "Procedure 'pico._inject_error' is not defined")
```

The root cause is that one `cargo build` overrides another with
different features enabled. To resolve the issue this patch
significantly refactors the test flow:

1. Web UI tests are skipped by default. One can enable it explicitly
   with `pytest --with-webui`

2. New fixture `cargo_build` behavior depends on environment:
   - In local dev environment it rebuilds the project with required features
   - In CI it skips rebuilding and runs tests against the existing build

3. The `path_to_binary_with_webui` fixture is removed.

See also:

- https://stackoverflow.com/a/55769818/1182787
- https://docs.pytest.org/en/7.4.x/how-to/writing_hook_functions.html
- https://docs.pytest.org/en/7.4.x/how-to/mark.html
- https://docs.pytest.org/en/7.4.x/example/simple.html#control-skipping-of-tests-according-to-command-line-option

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Before this patch we've had ADMIN_ID in schema.rs and ADMIN_USER_ID
in lib.rs. This patch removes ADMIN_USER_ID in favor of ADMIN_ID

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This patch adds initiator field to most of the events. Initiator is
added to members of Op enum which do not contain it already (e g owner
in CreateUser serves as initiator). To me it is reasonable to remove
as_user from CaS/ACL checks because it matches initiator. But I decided
not to do this just yet.

Additionally I've mande small cleanup to UserMetadata to avoid need for
intermediate step in decoding when we first fetch tuple and then decode
it into model struct.

Close https://git.picodata.io/picodata/picodata/picodata/-/issues/433

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Frequently when we want to execute commands from different users we need
to pass username and password separatly to each invokation. With this
patch this approach can be simplified and connection can be used
directlry in form of context manager.

Additionally this patch fixes handling of NOLOG env var. When it is
supplied logs are disabled which simplifies debugging of a test nbecause
with continuous output from multiple instances it is impossible to use
debugger efficiently.

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

At the later stage I discovered that sbroad applies the same validation
rules during parsing of grant/revoke statements and even its own
Privilege enum that is a ~copy of our PrivilegeType. Unfortunately
at the moment there is no way to share code between picodata and sbroad
efficiently (now only tarantool-module is shared and it is not suitable
for such kind of things) so it still makes sense to have this in
picodata because this is the point where all APIs converge to the single
point (CaS). In the future all other ways of validation should be
removed. Aside from sbroad similar kind of validation is performed
independently on lua API side.

Note that in prior commit 45ba7392 we've
removed all privileges from role super. This patch removes privileges
from admin that do not match the model: namely all privileges on
universe except session and usage. With this patch it is no longer possible
to grant or revoke such privileges.

Dmitry Rodionov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

This allows future constructors to enforce invariants.
Additionally this commit removes redundant set_schema_version on Op by
passing schema version to corresponding Op constructors.

Yaroslav Dynnikov authored 1 year ago and Yaroslav Dynnikov committed 1 year ago

Tested scenarios:

1. Alice is granted a privilege to create global/sharded table
2. Alice can create, read, and write it
3. Bob can't read it
4. Alice grants Bob a privilege to read it
5. Now Bob can read it, but he still can't write

This patch adds a proper support for auxiliary fields containing `Option<T>`.
As of now, KV pairs containing `Some(x)` will be printed as just `x`,
while the ones with `None` won't be printed at all.

Example:

```rust
crate::audit!(
    message: "demo audit event",
    title: "example",
    severity: Low,
    test: Some(0u32),
);
```

```json
{
  "id": "1.0.1",
  "message": "demo audit event",
  "severity": "low",
  "time": "2023-12-06T17:16:51.085+0300",
  "title": "example",
  "test": "0"
}
```

```rust
crate::audit!(
    message: "demo audit event",
    title: "example",
    severity: Low,
    test: None::<i32>,
);
```

```json
{
  "id": "1.0.1",
  "message": "demo audit event",
  "severity": "low",
  "time": "2023-12-06T17:16:51.085+0300",
  "title": "example"
}
```

This patch reintroduces instance-related events (e.g. `join_instance`)
which were accidentally disabled in a bugfix of the DML handler (see
commit 52ed9237 for more information) due to the lack of tests.

Furthermore, this patch adds the following events:

* expel_instance
* change_config, which tracks changes in `_pico_property`

This was the original intended change of the patch, but the apparent
breakage of the preexisting events called for action.

This change allows for creators of corresponding objects to consequently have all privileges on them.

Arseniy Volynets authored 1 year ago and Georgy Moshkin committed 1 year ago

- Test fails because of timeout on socket.
Current vshard multi storage call works
in 2 stages:
1. Ref needed storages
2. Execute stored procedure on the storages
When creating Ref on storage, vshard waits for all
buckets to become writable, if it happens that
rebalancer will start to move buckets, the query
execution will freeze until rebalancing is
done, which happens for this test sometimes.
- Comment out the flaky test until we switch
to new vshard multicall algorithm. Another
solution would be to increase timeout, but
there is no guarantee that new timeout is
big enough: I caught this error with 5s
timeout.

This could result in a broken instance if it terminated in between the
two storage writes. Now we always persist the commit index before
applying the persisted entries and do so in a transaction with unstable
entries or snapshot if they are present.