Alexey Protsenko authored 2 years ago and Dmitry Ivanov committed 3 months ago

Add to .gitlab.ci.yml test_linux, test_debian_docker_luacheck, coverage
from .travis.mk. Also sign package on build
Add checkpatch linter.
Add docker image build. Image copies original tarantool/tarantool from
Dockerhub

NO_DOC=ci change
NO_TEST=ci change
NO_CHANGELOG=ci change

Tarantool has hardcoded list of versions it can downgrade to. This list
should consist of all the released versions less than Tarantool version.
This workflow helps to make sure we update the list before release.

It is run on pushing release tag to the repo, checks the list and fails
if it misses some released version less than current. In this case we
are supposed to update downgrade list (with required downgrade code) and
update the release tag.

Closes #8319

NO_TEST=ci
NO_CHANGELOG=ci
NO_DOC=ci

(cherry picked from commit 6d856347)

Check check-entrypoint.sh comment for explanation of what entrypoint tag
is. The workflow fails if current branch does not have a most recent
entrypoint tag that it should have.

Part of #8319

NO_TEST=ci
NO_CHANGELOG=ci
NO_DOC=ci

(cherry picked from commit c06d0d14)

Some changes are not new features, rather developer tools updates
and improvements. There are also number of tweaks we can introduce
to improve testing and tests backporting across branches, which
also are not considered neither feature nor bug fix.

NO_DOC=changelog
NO_TEST=changelog
NO_CHANGELOG=changelog

(cherry picked from commit 4d6937ef)

We decided not to merge entries from Tarantool CE changelog to
Tarantool EE changelog. Instead we now add a link to CE release notes
to EE release notes. Update the script accordingly.

NO_DOC=tools
NO_TEST=tools
NO_CHANGELOG=tools

(cherry picked from commit 3dd1cffe)

One may encode a non-printable string under the MsgPack type MP_STR.
In fact, there's no way to encode binary data as MP_BIN from Lua so
users often do that, which breaks our gdb extension. Let's encode a
string in base64 if unicode() raises an error, like we do with MP_BIN.

While we are at it,
 - escape double quotes in printable strings;
 - fix MP_BIN encoding (b64encode takes and returns bytes).
 - don't use YAML !!binary prefix before base64 data - use bin: and str:
   instead.

Here's what it looks like:

```
(gdb) p *box_tuple_last
  <...>
  data = ["\"тест\\foo\"", str:/w==, bin:/w==]}
```

The tuple was created with the following command:

```
tarantool> msgpack = require('msgpack')
---
...

tarantool> box.tuple.new(
         >   msgpack.object_from_raw('\xae"тест\\foo"'), -- MP_STR unicode
         >   msgpack.object_from_raw('\xa1\xff'),        -- MP_STR binary
         >   msgpack.object_from_raw('\xc4\x01\xff'))    -- MP_BIN
---
- ['"тест\foo"', !!binary /w==, !!binary /w==]
...
```

Follow-up commit 08a171a4 ("gdb: support unicode in MP_STR type of
MsgPack").

NO_DOC=gdb extension
NO_TEST=gdb extension
NO_CHANGELOG=gdb extension

It doesn't belong to Tarantool sources. The tools directory looks like
the right place for it.

NO_DOC=code cleanup
NO_TEST=code cleanup
NO_CHANGELOG=code cleanup

By design after patching update.lua, introducing new data version,
a developer needs to update bootstrap.snap, which would have the
new data version.

Generation of bootstrap.snap is a bit tricky, there's a manual:
https://github.com/tarantool/tarantool/wiki/How-to-generate-new-bootstrap-snapshot

This commit presents a simple script that automates the process.
Just run it in tarantool build directory and check its output.
Check --help for script options.

Follow-up of tarantool/security#14

NO_DOC=internal tool
NO_TEST=internal tool
NO_CHANGELOG=internal tool

Regular expression now works on versions: alpha, beta, rc and so on.

NO_DOC=bugfix
NO_TEST=bugfix
NO_CHANGELOG=bugfix

Tabs were replaced with spaces to bypass checkpatch.

NO_DOC=bugfix
NO_TEST=bugfix
NO_CHANGELOG=bugfix

See https://github.com/tarantool/checkpatch

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

 1. There's a non-ASCII character in a comment (’). Replase it with a
    plain comma ('). Fixes commit
    19be79e6 ("tools: gen-release-notes:
    adapt for enterprise").
 2. In Python 2 subprocess.Popen() doesn't support the 'with' clause.
    Do not use the 'with' clause. Fixes commit
    5715f194 ("tools: gen-release-notes:
    factor out popen helper")

As a result CI fails. I wonder why it passed on the PR...

Follow-up #6855

NO_DOC=tools
NO_CHANGELOG=tools

We're going to reuse the existing tools/gen-release-notes script for
generation of Enterprise Edition (EE) changelogs. To do that, we need to

 1. Add 'enterprise' to the head of the sections list. The new section
    will be used by EE changelog entries. We want them to go before
    open-source changelog entries.
 2. Gather changelog entries from the 'changelog/unreleased' directory
    of the superproject directory. EE uses the open-source repository as
    its submodule.

NO_DOC=tools
NO_CHANGELOG=tools

So that we can gather changelog entries from more than one directory.

NO_DOC=tools
NO_CHANGELOG=tools

We will use it for running other commands.

NO_DOC=tools
NO_CHANGELOG=tools

Trivial refactoring that replaces source_dir and entries_dir_relative
changelog_entries_sorted() arguments with entries_dir. Needed to make
the following work easier.

NO_DOC=tools
NO_CHANGELOG=tools

This commit adds a new script, tools/check-commits. The script takes git
revisions in the same format as git-rev-list (actually, it feeds all its
arguments to git-rev-list) and runs some checks on each of the commits.

For example, to check the head commit, run:

  ./tools/check-commits -1 HEAD

To check the last five commits, run:

  ./tools/check-commits HEAD~5..HEAD

Currently, there are only two checks, but in future we may add more
checks, e.g. check diffs for trailing spaces:
 - The commit message contains a documentation request. Can be
   suppressed with NO_DOC=<reason> in the commit message.
 - A new changelog entry is added to changelog/unreleased by the commit.
   Can be suppressed with NO_CHANGELOG=<reason> in the commit message.

This commit also adds a new workflow triggered on pull request,
lint/commits. The workflow runs the tools/check-commits script on all
the commits added by the pull request. The workflow doesn't run on push,
because it's problematic to figure out what commits are new on a branch.
Besides, we don't want to run it on push to release branches, because
it's a pure dev workflow.

Example output:

Checking commit a33f3cc7 PASS
Checking commit 6f29f9d7 FAIL
SHA:     6f29f9d7
SUBJECT: iproto: introduce graceful shutdown protocol
  ERROR:
    Changelog not found in changelog/unreleased. If this commit
    doesn't require changelog, please add NO_CHANGELOG=<reason>
    to the commit message.
Checking commit fbc25aae FAIL
SHA:     fbc25aae
SUBJECT: Update small submodule
  ERROR:
    Missing documentation request ('@TarantoolBot document' not
    found in the commit message). If this commit doesn't need to
    be documented, please add NO_DOC=<reason> to the commit
    message.
  ERROR:
    Changelog not found in changelog/unreleased. If this commit
    doesn't require changelog, please add NO_CHANGELOG=<reason>
    to the commit message.

NO_DOC=ci
NO_CHANGELOG=ci

Now `## bugfix/luajit` and `## bugfix/LuaJIT` entries will form one
section.

Now one can write either `## bugfix/luajit` or `## bugfix/LuaJIT`. The
latter will NOT be transformed into `### Luajit` anymore. Both variants
now give `### LuaJIT` section inside `## Bugs fixed` section.

See `REDEFINITIONS` variable inside the script to understand how well
known headers (such as `## <...>/LuaJIT`) are prettified.

There is a problem with section grouping, when headers are written in
lower/title/mixed case. It'll be resolved in a next commit.

We run the script in CI as linter (see PR #6701), so it should handle
lack of any changelog entries gracefully.

Previously the script mistakely requires at least one feature and at
least one bugfix. However it is quite natural to have only bugfixes in a
bugfix release. Moreover, we added the script into CI as linter (see
PR #6701), so it should work even when we just start filling release
notes.

This patch introduces the use of RWS (repository web service) instead
of old update_repo.sh script for uploading packages to S3 repositories.
RWS fully replaces facilities of update_repo.sh + provides new features
like support of uploading ARM packages. Also, it is very simple to use,
just do `curl` to the RWS endpoint with some params and that's it.

Closes #6482

Sometimes to fix an issue we need to patch our submodule, update it
in tarantool and then patch tarantool using changes in submodule.

During development and review stages we have to create a branch (S)
in submodule and a branch (T) in tarantool that references head of
(S) branch.

When the fix is merged to mater it's very simple to make a mistake:
merge-and-push submodule's branch S and then tarantool's branch T.
It sounds obvious, but that's wrong: tarantool's master should
reference a commit from submodule's master, not a commit from
temporary developer's branch.

It's not hard to fix it but still a maintainer must always remember
that problem. In order to simplify their life it was decided to
create a script that is designed to check such thing before pushing
tarantool to origin/master.

Here is it, with simple usage:
./tools/check_push_master.sh && git push origin master

Make able to save packages in S3 buckets.

Closes: #6074

Make able to save packages in S3 buckets.

Closes: #5825

Make able to save packages in S3 buckets.

Closes: #5824

Script used to run Jepsen tests uses Terraform to prepare test environment.
When I made this script I had a wrong assumption about working directory
used by Terraform. Due to this sometimes job with Jepsen tests
failed on cleanup with message:

```
Error: Could not load plugin

Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider
"registry.terraform.io/terraform-providers/openstack" to obtain schema:
unknown provider "registry.terraform.io/terraform-providers/openstack"
```

Terraform documentation describes how Terraform uses working directories [1]
and there are at least to ways to resolve an issue. First one is using always
one directory before running terraform subcommands. Second one is using option
`-chdir` in all terraform commands [2].

1. https://www.terraform.io/docs/cli/init/index.html
2. https://www.terraform.io/docs/cli/commands/index.html#switching-working-directory-with-chdir

Closes #6089

This patch introduces two scripts to ease crash artefacts collecting and
loading for postmortem analysis:

* tarabrt.sh - the tool collecting a tarball with the crash artefacts
  the right way: the coredump with the binary, all loaded shared libs,
  Tarantool version (this is a separate exercise to get it from the
  binary built with -O2). Besides, the tarball has a unified layout, so
  it can be easily processed with the second script:
  - /coredump - core dump file on the root level
  - /binary - tarantool executable on the root level
  - /version - plain text file on the root level with
    `tarantool --version` output
  - /checklist - plain text file on the root level with the list of the
    collected entities
  - /etc/os-release - the plain text file containing operating system
    identification data
  - all shared libraries used by the crashed instance - their layout
    respects the one on the host machine, so they can be easily loaded
    with the following gdb command: set sysroot $(realpath .)

  The script can be easily used either manually or via
  kernel.core_pattern variable.

* gdb.sh - the auxiliary script originally written by @Totktonada, but
  needed to be adjusted to the crash artefacts layout every time. Since
  there is a unified layout, the original script is enhanced a bit to
  automatically load the coredump via gdb the right way.

Closes #5569

Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>
Reviewed-by: Sergey Bronnikov <sergeyb@tarantool.org>
Signed-off-by: Igor Munkin <imun@tarantool.org>

Enabling ubuntu-20.04 hosts for packaging workflows found that DEB
package Github Actions workflows do not need to install createrepo
tool. Also found that createrepo is not ready for ubuntu-20.04 as
described in (till ubuntu-21.04 where it is available as the new
version of this tool named as 'createrepo_c' as DEB package):

  3a7c2102 ('github-ci: ubuntu-20.04 misses createrepo package')

To fix it added 'createrepo_c' build and installation from sources
and changed in update_repo tool 'createrepo' tool to 'createrepo_c'.
This patch is needed to use these workflows on self-hosted runners
which run under ubuntu-20.04 by default for now.

Also checking the patch on ubuntu-20.04 hosts got the following issue:

  Regenerated DEB file: pool/xenial/main/t/tarantool/tarantool-common_2.8.0.185.g4c3e0eb-1_all.deb

  <botocore.awsrequest.AWSRequest object at 0x7f7998a4ca90>

  <botocore.awsrequest.AWSRequest object at 0x7f627d965070>
  make: *** [.gitlab.mk:131: deploy] Error 255
  Error: Process completed with exit code 2.

Found that there is already issue exists in Github Actions on it [1].
Provided solution to setup AWS region in environment helped to
workaround the issue [2].

Closes tarantool/tarantool-qa#110
Closes tarantool/tarantool-qa#111

[1]: https://github.com/aws/aws-cli/issues/5234
[2]: https://github.com/aws/aws-cli/issues/5234#issuecomment-635459464

Found that running package removement command from S3:

  ./tools/update_repo.sh -o=fedora -d=30 -b=s3://tarantool_repo/live/1.10 -r=tarantool-1.10.9.108

it searched to remove:

  Searching to remove: s3://tarantool_repo/live/1.10/fedora/30/SRPMS/Packages/tarantool-1.10.9.108-1.fedora30.src.rpm

while it had to be:

  Searching to remove: s3://tarantool_repo/live/1.10/fedora/30/SRPMS/Packages/tarantool-1.10.9.108-1.fc30.src.rpm

where 'fc30' had to be instead of 'fedora30'. This patch fixes it.

- install python3 on mac machines
- run test-run without specifying path to python interpreter
- get rid our own tarantool brew-tap that depends on Python 2

Part of #5652

Found issues and resolved:

- Added '$PROJECT_BINARY_DIR/build/' path creation before use.

- Set always ssh-agent restart before add new SSH key, because
  found that in some situations it may fail to add w/o it.

- Found mistake in TF_VAR_id variable setup, inverted ${CI_JOB_ID}
  variable check for it.

Also made improvements in script:

- Added all needed checks for all externally set mandatory variables.

- Added usage message with the list of mandatory and optional variables.

- Added comments to all script steps.

- Added step with checking nodes that they are reachable by SSH.

See https://github.com/tarantool/tarantool/discussions/5634

Added jobs for testing and deploying Fedora 33 packages.

Added Fedora 33 at update_repo tool to make able to save packages
in S3 buckets.

Closes #5502

Added package workflow to github actions. Set deployment of the
packages for the master branch and tags.

Closes #5638

It was added Fedora 32 gitlab-ci packaging job in commit:
  507c47f7a829581cc53ba3c4bd6a5191d088cdf ("gitlab-ci: add packaging for Fedora 32")

but also it had to be enabled in update_repo tool to make able to save
packages in S3 buckets.

Follows up #4966

Implemented ability to remove opensuse-leap OS packages.

Updated help message on remove option.

Added message which file to remove to be sure that the needed files
were searched to remove.

Found that Sources file destroys when module uploaded without sources.
Also found that it could happen for Packages file on modules uploading
without binaries. To fix it was added additional its downloading from
S3 if in modules it was not updated and routine was not used.