Due to inconsistency of Tarantool type casting while using strict
data types as "double" or "unsigned" it is needed
to use "number" data type in a whole bunch of cases.
However "number" may contain "decimal" that will be serialized into
string by JSON builtin module.

This commit adds "encode_decimal_as_number" parameter to json.cfg{}.
That forces to encode `decimal` as JSON number to force type
consistency in JSON output.
Use with catious - most of JSON parsers assume that number is restricted
to float64.

NO_DOC=we do not host doc

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Previously,

select "t1"."a" from (select "a" from "t") as "t1";

returned a result column name `t1` instead of `t1.a` because of
incorrect work of a dequoting function. The reason was that
previously sqlDequote() function finished its work when found the
first closing quote.

Old logic worked for simple selects where the column name doesn't
contain an explicit scan name ("a" -> a).
But for the sub-queries results sqlDequote() finished its work right
after the scan name ("t1"."a" -> t1). Now the function continues its
deqouting till it gets the null terminator at the end of the string.

Closes #7063

NO_DOC=don't change any public API, only a bug fix

Co-authored-by: Mergen Imeev <imeevma@gmail.com>

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 2 years ago

It is incorrect to create savepoints and truncate fiber->gc region
in the module_func_call(), as the module's function can also call
fiber_gc(). Actually, there is no sense in this action.

Previously we had a problem while calling a module function.
in module_func_call():

1. create a savepoint to fiber->gc
2. evaluate an arbitrary function. Let's assume it is a C function
    that creates and deletes prepared statements. When they don't
    fit into statement cache anymore (and we haven't configured or
    box to use transaction), sql_stmt_cache_insert() would call a
    fiber_gc(). As a result we can truncate the gc region to an
    invalid state (the amount of the used blocks would be less than
    we hade before the savepoint from step 1).
3. call fiber_gc() again (a savepoint points to an already freed
    position).

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Mergen Imeev authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Prior to this patch, some opcodes could use a pointer to struct space
that was set during parsing. However, the pointer to struct space is not
really something that defines spaces. A space can be identified by its
ID or name. In most cases, specifying space by pointer works fine, but
truncate() changes the pointer to space, resulting in a sigfault for
prepared statements using the above opcodes. To avoid this problem, a
new opcode has been introduced. This opcode uses the space ID to
determine the pointer to the struct space at runtime and stores it in
the MEM, which is later used in the mentioned opcodes.

Closes #7358

NO_DOC=bugfix

Mergen Imeev authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Opcode IteratorReopen is not used and should be dropped.

Part of #7358

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Actually there is no reason to throw an error and make a user
manually recreate prepared statement when it expires. A much more
user friendly way is to recreate it under hood when statement's
schema version differs from the box one.

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Problem description.

When we prepare a statement with parameters in the result columns
(for example box.prepare('select ?')) Tarantool has no information
about the type of the output column and set it to default boolean.
Then, on the execution phase, the type would be recalculated during
the parameter binding.

Tarantool expects that there is no way for parameter to appear in the
result tuple other than exactly be mentioned in the final projection.
But it is incorrect - we can easily propagate parameter from the inner
part of the join. For example

box.prepare([[select COLUMN_1 from t1 join (values (?)) as t2 on true]])

In this case column COLUMN_1 in the final projection is not a
parameter, but a "reference" to it and its type depends on the
parameter from the inner part of the join. But as Tarantool
recalculates only binded parameters in the result projection,
it doesn't change the default boolean metadata type of the COLUMN_1
and the query fails on comparison with the actual type of the tuple.

Solution.
As we don't want to patch Vdbe to make COLUMN_1 refer inner parameter,
it was decided to make a simple workaround: change the default
column type from BOOLEAN to ANY for parameters. It fixes the
comparison with the actual tuple type (we do not fail), but in some
cases get ANY column in the results where we would like to have
explicitly defined type. Also NULL parameters would also have ANY
type, though Tarantool prefers to have BOOLEAN in this case.

Closes https://github.com/tarantool/tarantool/issues/7283

NO_DOC=bug fix

Denis Smirnov authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Current test seems to be flaky. Previously it checked that we print
correctly in the logs the address of the replaced tuple during GC.
For some reason the test was searching for the first string with
`tuple_delete` in the log and expected it to be the one we are
looking for.

Current fix makes it search for exactly the deleted address we need,
not the first one appear in the log.

NO_DOC=test fix
NO_CHANGELOG=test fix

godzie44 authored 2 years ago and Дмитрий Кольцов committed 2 years ago

sql: add sql_execute_prepared_ext function, same as sql_execute_prepared but without `region` parameter
closes #2

NO_DOC=minor
NO_TEST=minor

godzie44 authored 2 years ago and Дмитрий Кольцов committed 2 years ago

- add box_tuple_data_offset function (return offset of the messagePack encoded data from the beginning of the tuple)
- add more export functions

closes #1

NO_DOC=build
NO_TEST=build

Alexey Protsenko authored 2 years ago and Дмитрий Кольцов committed 2 years ago

Add to .gitlab.ci.yml test_linux, test_debian_docker_luacheck, coverage
from .travis.mk. Also sign package on build
Add checkpatch linter.
Add docker image build. Image copies original tarantool/tarantool from
Dockerhub

NO_DOC=ci change
NO_TEST=ci change
NO_CHANGELOG=ci change

This reverts commit 33830978.

Follow-up #6477

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 6605de25)

This reverts commit 9d1f9f0e.

Follow-up #6477

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 7e1df16e)

Two things we need to do to fix build with OpenSSL 3.0:

1. Use EVP_MAC_* functions instead of HMAC_*
   https://www.openssl.org/docs/man3.0/man3/HMAC_CTX_new.html

2. Load the Legacy provider to enable legacy algorithms, such as MD4
   https://wiki.openssl.org/index.php/OpenSSL_3.0#Programming_in_OpenSSL_3.0

Closes #6477

NO_DOC=build fix
NO_TEST=build fix
NO_CHANGELOG=build fix

(cherry picked from commit e3bf73c8)

We redefine ssl_init and ssl_free in the EE build, because we need to do
some extra work there. Currently, it's fine to duplicate the bulk of the
OpenSSL library initialization code between EE and CE repositories, but
with the introduction of OpenSSL 3.0 it's going to become more
complicated so duplicating would look bad. Let's move the common code to
ssl_init_impl() and ssl_free_impl() helper functions.

Needed for #6477

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit f9739160)

ERR_error_string adds some extra information that depends on the OpenSSL
library version (code, module, method). This information says nothing to
the end user, and it results in different test results after updating to
OpenSSL 3.0. Let's use ERR_reason_error_string instead, which just
prints a human-readable error message.

Part of #6477

NO_DOC=minor change in error message
NO_CHANGELOG=minor change in error message

(cherry picked from commit 9cc130f0)

openssl_err_str is used for reporting OpenSSL errors. It calls
crypto_ERR_* functions using FFI. There's a typo in the code:
ffi.crypto_ERR_error_string is used instead of ffi.C.*.

We don't normally step on this, because OpenSSL doesn't return errors
in our configuration, but if it did for some reason (e.g. a cipher was
disabled in the library), we'd get a confusing error message.

NO_DOC=bug fix
NO_TEST=occur only on internal error
NO_CHANGELOG=occur only on internal error

(cherry picked from commit f72662c5)

Added 'tarantool/actions/cleanup' action to each job which uses
self-hosted runners.

The action cleans workspace directory of self-hosted runner after
previous run. The main reason to add this action is 'Need a single
revision' error [1] caused by a conflict of submodule versions,
the standard 'actions/checkout' action fails with this error. It's a
well-known problem and related issue [2] is still opened.

[1] https://github.com/tarantool/tarantool-qa/issues/145
[2] https://github.com/actions/checkout/issues/418

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

Closes tarantool/tarantool-qa#145

(cherry picked from commit 7ff87404)

To ensure that regular and GC64 jobs in packaging workflows don't
overwrite artifacts of each other, we need to use a different artifact
name per job.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 37759699)

In PR #7090 we forgot to update the path to gather failure artifacts for
many packaging workflows. Now it is fixed.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit db6bbb59)

Introduce helpers for each of our custom extension types.
The helpers will be used by mp_check() to validate extension contents
and make sure no malformed data is accepted by tarantool.

Closes #6857

NO_DOC=no user visible changes

(cherry picked from commit 5b47124a)

The inject remote method is used in testing quite a lot. For example,
when someone needs to pass arbitrary MsgPack to Tarantool.

The current way to use it is:
conn:_request(netbox._method.inject, opts, nil, nil, custom_msgpack)

This is quite long and ugly, so let's add a helper:
conn:_inject(custom_msgpack, opts)

Also, while we're at it, lets fix the following issue.
The _inject remote method is barely usable without knowing which sync
net.box will use for the next request.

On the one hand, the user has to encode some custom sync to the request
he's injecting.
On the other hand, net.box doesn't parse the custom sync and always uses
its own pre-generated one to wait for the request response.

So the user has to pick the correct sync value, which net.box uses
internally.

Let's make life simpler and introduce _next_sync method, which returns
the next sync, which will be used by net.box.

In-scope-of #6857
Closes #7177

NO_DOC=internal change
NO_CHANGELOG=internal change
NO_TEST=tested implicitly in next commit

(cherry picked from commit 5893d61d)

A Tarantool server is supposed to send a greeting message right after
accepting a new client so the first thing an applier does after
connecting to the master is reads the greeting. It does this without
timeouts. The problem is that if by mistake we connect to a wrong
instance, which doesn't send anything to clients, the applier will hang
forever (until the remote closes the socket), without logging any
errors.

This may happen even with a valid Tarantool instance - if SSL encryption
is enabled on the master, but not on the client, because the SSL
protocol assumes that the client initiates a connection by writing to
the socket first (before the server).

Let's add a timeout to the operation reading the greeting. The timeout
is set to replication_disconnect_timeout(), after which a connection is
broken if the master doesn't send heartbeats for that long. Note, we
don't add a timeout to other read/write operations issued to initiate a
replication connection, because if we received a greeting and it's
valid, then the master is likely to be fine.

Closes #7204

NO_DOC=bug

(cherry picked from commit c6485d8d)

The upgrade test worked incorrectly: the instances WAL directory contained
xlog with the latest schema version and the snapshot contained an older
schema version (2.7.1) than required (2.9.1), whereas there only need to
be a snapshot with 2.9.1 schema version — fix this descrepancy.

Also, schema upgrade is performed automatically: remove the redundant
`box.schema.upgrade` call.

NO_CHANGELOG=test fix
NO_DOC=test fix
NO_DOC=test fix

(cherry picked from commit 59ce25dd)

Commit b18dd47f ("Introduce backtrace=true option to fiber.info()")
introduced a way to skip backtraces in fiber.info() calls.
Commit 9da7e03e ("fiber: introduce fiber_o:info() and fiber_o:csw()")
introduced `options` function for fiber object however it ignored
passed options.
This patch fixed it. Currently fiber:info({backtrace = false})
returns info without backtrace.

Closes #7210

NO_DOC=bugfix

(cherry picked from commit 0eac13b9)

- Ubuntu Groovy Gorilla (20.10)
- Ubuntu Hirsute Hippo (21.04)

See for details: https://wiki.ubuntu.com/Releases

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit 2e067472)

- Fedora 30
- Fedora 31
- Fedora 32
- Fedora 33

See for details: https://docs.fedoraproject.org/en-US/releases/eol

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci

(cherry picked from commit eb891308)

FreeBSD Tarantool port has some problems if build uses cmake'
ExternalProject_Add which refers to the github repository.
Sumodule works better - so switching to using it.

NO_CHANGELOG=build
NO_DOC=build
NO_TEST=build

(cherry picked from commit 4eaff4e0)

Previous version of tzcode/CMakeLists.txt leaves untracked files under
source directory src/lib/tzcode/tz/. We have changed slightly an approach
used to checkout and build IANA github sources under build/ directory,
not in ${PROJECT_BINARY_DIR} (which may be pointing not to out-of-source
build directory, but rather to the repository root itself, if insource
build used).

Closes #7173

NO_CHANGELOG=infra
NO_DOC=infra
NO_TEST=infra

(cherry picked from commit 5f143288)

clang 13 includes a check for subtraction from NULL pointer which
is considered UB: historically, we had an alignment checking macro
which is affected by this. It seems like the intention of the
macro's author was to implicitly cast the pointer being checked to
uintptr_t without including stddef.h — replace this subtraction
with an explicit cast.

There is no way to set SQL_4_BYTE_ALIGNED_MALLOC, so the corresponding
part of the code was removed. Now there are only 8 byte alignment
assertions.

NO_CHANGELOG=UB fix
NO_DOC=UB fix
NO_TEST=UB fix

Co-authored-by: Georgiy Lebedev <curiousgeorgiy@gmail.com>
(cherry picked from commit 0289433a)

Copy-and-pasted from the corresponding of 2.10.0-rc1.

Generate changelog for 2.10.0 release.
Also, clean changelogs/unreleased folder.

NO_DOC=no code changes
NO_TEST=no code changes

This client error assumes two arguments: space's name and index's name.
Since from the context it's clear that primary index is affected,
let's drop index name from error message. It's quite handy for space
upgrade case, since in such situation we have lack of information
concerning space.

NO_CHANGELOG=<Minor change>
NO_DOC=<No functional changes>

(cherry picked from commit 6e76410a)

We are going to utilize primary key definition during space upgrade in
order to verify primary key invariant. So we need it to be stored in
space_upgrade metadata. Let's extend space_upgrade_new() signature and
pass there primary key definition. Also patch moves space_upgrade_new()
call a bit lower through code - now it's called after all indexes are
assigned (to simplify extraction of primary key definition).
Moreover, in order to print pretty error message during space upgrade
we should keep space in struct space_upgrade.

NO_DOC=ee
NO_TEST=ee
NO_CHANGELOG=ee

(cherry picked from commit 9255b607)

box.error() expects first argument to be numeric or table; passing
string to it is wrong. Accidentally in net.box module two places where
string is passed to box.error() as first argument: in :unprepare() and
:connect() methods. Let's fix them and pass error code ER_ILLEGAL_PARAMS.

NO_DOC=<bugfix>
NO_CHANGELOG=<Minor>

(cherry picked from commit a74c549a)

Fix wording, punctuation, and formatting.

NO_DOC=changelog
NO_TEST=changelog

(cherry picked from commit ac365ecf)

Previously SQL didn't validate for all cases, that the amount of the
source and destination columns during insertion is equal. The problem
was detected when we insert an incorrect amount of values into the
table. For example, the query

insert into t(a) select a, b from t

produced an instance crash. Fixed.

Closes #7132

NO_DOC=bug fix

Co-authored-by: Mergen Imeev <imeevma@gmail.com>
(cherry picked from commit dfa0ea2b)

We used to not implement comparison operators for interval
objects, thus any compare (even of equal) objects returned false:

	```
	tarantool> dt1 = datetime.new()
	---
	...

	tarantool> dt2 = datetime.now()
	---
	...

	tarantool> dt1 - dt1 == dt2 - dt2
	---
	- false
	...
	```

Now we implemented comparison for interval objects, where we compare
field by field starting from highest attribute (e.g. `year`) to
smallest attribute (e.g. `nsec`) and if any of comparison
returned non zero value then we return this result.

NO_DOC=bugfix
NO_CHANGELOG=bugfix

(cherry picked from commit 65a3c17f)

Use Olson/IANA tzdata for timezone handling in datetime string.

Closes of #6751

NO_TEST=doc

@TarantoolBot document
Title: Document timezone support

Timezones support
-----------------

Tarantool uses IANA tzdata aka Olson DB facilities for timezone
resolution at the moment of parsing of datetime literals or while
parsing `tz` attribute in constructor, or `:set{}` method call.

In addition to the `tzoffset` we provided before, we now define
`tzindex`, the unique index assigned by Tarantool to each known IANA
timezone.

```lua
date = require 'datetime'

tarantool> T = date.parse '2022-01-01T00:00 Europe/Moscow'

tarantool> T.tzindex
---
- 947
...

tarantool> T.tzoffset
---
- 180
...

tarantool> T.tz
---
- Europe/Moscow
...
```

Now `date.isdst` field (alone and as part of `:totable()` table) is
correctly calculated using `tzindex` and attributes of the selected
timezone in the Olson DB timezone.

```
tarantool> date.parse('2004-06-01T00:00 Europe/Moscow').isdst
---
- true
...

tarantool> date.parse('2004-12-01T00:00 Europe/Moscow'):totable()
---
- sec: 0
  min: 0
  yday: 336
  day: 1
  nsec: 0
  isdst: false
  wday: 4
  tzoffset: 180
  month: 12
  year: 2004
  hour: 0
...

```

All timezone names and abbreviations, known to the current Tarantool
version are available via `datetime.TZ` bidirectional array.

```lua
print(date.TZ['Europe/Moscow']) -- 947
print(date.TZ[947]) -- Europe/Moscow
```

Limitations
-----------

There were moments in past history, when local mean time in some
partcular zone used timezone offset not representable in whole
minutes, but rather in seconds, i.e. in Moscow before 1918 there
used to be offset +2 hours 31 minutes and 19 seconds. Please see
Olson dump for this period

```
$ ./src/lib/tzcode/install/usr/bin/zdump -c1880,1918 -i Europe/Moscow

TZ="Europe/Moscow"
-       -       +023017 MMT
1916-07-03      00:01:02        +023119 MMT
1917-07-02      00      +033119 MST     1
1917-12-27      23      +023119 MMT
```

Modern tzdata rules do not use such tiny fraction, and all timezones differ
to UTC in units measured in minutes, not seconds. Tarantool datetime uses
minutes internally as units for `tzoffset` so there is some loss of
precision if you try to operate with such ancient timestamps.

(cherry picked from commit 979a8fbd)

For performance reasons try to reuse previously
allocated structure across calls to tzalloc() if
zone appears to be the same.

NO_DOC=refactoring
NO_CHANGELOG=refactoring
NO_TEST=refactoring

(cherry picked from commit 6ba2cc36)