CONTRIBUTING.md

Contributing to Picodata

This document describes contributing to Picodata, primarily the ways you can build and test it.

Building Picodata from source

Required build tools

• Rust and Cargo 1.65 or newer
• Cmake 3.16 or newer
• gcc, g++
• libstc++-static
• (optional to build with Web UI) node v15+, yarn

Prerequisites for CentOS 8

Use the following commands to install the required build prerequisites. Note that you'll need recent Rust and Cargo versions installed using the recommended way from rustup.rs:

sudo dnf config-manager --set-enabled powertools
sudo dnf in -y gcc gcc-c++ make cmake git patch libstdc++-static
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

# Optional - to build with Web UI
sudo dnf module install nodejs:19
sudo corepack enable

Prerequisites for Ubuntu 22.04

Use the following command to install the required build prerequisites. Note that Ubuntu 22.04 provides recent Rust and Cargo versions, so it's preferable to install it via apt-get:

sudo apt-get install build-essential cargo git cmake -y

# Optional - to build with Web UI
curl -fsSL https://deb.nodesource.com/setup_19.x | sudo -E bash - &&\
sudo apt-get install -y nodejs
sudo corepack enable

Prerequisites for Alt Workstation p10

Use the following commands to install the required build prerequisites. Note that you'll need recent Rust and Cargo versions installed using the recommended way from rustup.rs:

su -
apt-get install gcc gcc-c++ cmake git patch libstdc++10-devel-static libgomp10-devel-static -y && exit
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Prerequisites for MacOs

Use the following commands to install the required build prerequisites. Note that you'll need recent Rust and Cargo versions installed using the recommended way from rustup.rs:

brew install git cmake make curl gcc msgpack
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

# Optional - to build with Web UI
brew install node yarn

Getting and building the source code

git clone https://git.picodata.io/picodata/picodata/picodata.git
cd picodata
git submodule update --init --recursive

Compile the project:

cargo build

This will build the debug version. If you want the release version, try this instead:

cargo build --release

If you want to enable Web UI for administration, build with these flags:

cargo build --features webui

When running picodata -http-listen should be supplied to serve Web UI.

The resulting binaries should appear under the target subdirectory.

Integration testing with pytest

The following refers to Ubuntu 20.04 LTS. The mileage with other distributions may vary.

Ubuntu

Installation

1. Install Python 3.10

   sudo add-apt-repository ppa:deadsnakes/ppa
   sudo apt install python3.10 python3.10-distutils
   curl -sSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
   python3.10 get-pip.py

2. Install pipenv:

   python3.10 -m pip install pipenv==2022.4.8

3. Install dependencies

   python3.10 -m pipenv install --deploy

Adding dependencies

python3.10 -m pipenv install <dependency-package-name>

Running

python3.10 -m pipenv run pytest
python3.10 -m pipenv run lint

or

python3.10 -m pipenv shell
# A new shell will be opened inside the pipenv environment
pytest
pipenv run lint

Running tests in parallel with pytest-xdist

python3.10 -m pipenv run pytest -n 20

---

macOS

Installation

The installation sequence requires Python 3.10 or newer, preferably installed via Homebrew.

First check if the required version is available:

brew info python

If it is, go and install it:

brew install python@3.11
brew link python
python3 --version

Check if the default python3 executable is available and provided by the Homebrew installation:

which python3

Note: Python versions <=3.9 such as the one provided by the older Xcode Developer Tools will not work.

If the python3 executable is provided by Homebrew, skip the following part and jump to pipenv installation. Otherwise, that needs to be fixed. Let's find out the local Homebrew installation details:

brew config

The HOMEBREW_PREFIX variable should point to the directory where brew installs packages. Let's create a symlink:

ln -s "$(brew config | sed -n "s/^HOMEBREW_PREFIX: //p" | tr -d "\n")/bin/python@3.10" /usr/local/bin/python3

Note: Make sure that /usr/local/bin is in your PATH.

Check python3 location and version. Now it should be provided by Homebrew:

which python3
python3 --version

Make sure the pip3 executable also points to the homebrew directory.

which pip3

If it doesn't, then similarly create a simlink for pip3:

ln -s "$(brew config | sed -n "s/^HOMEBREW_PREFIX: //p" | tr -d "\n")/bin/pip@3.10" /usr/local/bin/pip3

After that you can install pipenv:

pip3 install pipenv==2022.4.8

Set python path for pipenv:

pipenv --python /usr/local/bin/python3

Adding dependencies

pipenv install --deploy

Running

pipenv run pytest -n auto
pipenv run lint

Benchmarks and flamegraphs

Benchmarks

There is a simple benchmark based on pytest scenario. Quick run it with

make benchmark

The benchmark consists of two parts: Tarantool space operations (replaces) and Picodata raft writes to leader (NOPs). You will see both results.

The benchmark designed for quick evaluation of the current application performance.

Flamegraphs

It is possible to make flamegraphs while benchmarking a debug build.

• Install perf:

  apt install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`

• Install FlameGraph script:

  cd .. && git clone https://github.com/brendangregg/FlameGraph.git ; cd -

  Your folder structure will look like this:

  ..
  picodata (you are here)
  FlameGraph
  (other folders and files)

• Set kernel options for perf:

  sudo echo 0 > /proc/sys/kernel/perf_event_paranoid

  sudo echo 0 > /proc/sys/kernel/kptr_restrict

• Now run

  make flamegraph

  Flamegraphs will be placed in tmp/flamegraph dir.

SQL performance

To benchmark SQL, a custom K6 build (xk6) with Tarantool protocol support is used.

• Install golang and xk6.

• Run K6:

  make k6

  Performance summary can then be found in the test/manual/sql/ directory.