Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

This patch makes schema_object_type to be exported.

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

Dmitry Rodionov authored 1 year ago and Дмитрий Кольцов committed 1 year ago

Rename the members as well. Keep uint16_t in box_check_acess_space
as it is for other exported functions

NO_DOC=picodata internal patch
NO_CHANGELOG=picodata internal patch
NO_TEST=picodata internal patch

The trigger is invoked on IPROTO CALL/EVAL. The trigger callback is
passed a context with function name or eval expression and arguments
(MsgPack data). It will be used for auditing CALL/EVAL events in the EE
version.

NO_TEST=ee
NO_DOC=internal
NO_CHANGELOG=internal

box_process_call() uses func_call(), which not only calls the given
function, but also checks that the current user has the right to execute
it. As a result, we can't add auditing for only those function calls
that passed the access check (apparently, there's no reason to log
function calls that failed with an 'access denied' error - we have a
separate audit event for this).

To fix this, let's introduce func_call_no_access_check() helper, which
calls a function without checking access rights, and use it along with
existing func_access_check() in box_process_call(). func_call() is now
an inline function that calls func_access_check() and then on success
func_call_no_access_check().

It's probably wrong that func_call() checks access rights, because this
means that to use a space with a functional index/constraint, the user
needs not only read/write access to the space itself, but also execute
access to the function. I think we should check the right to execute
such function only once - on functional index/constraint creation, not
on every call, but I'm not going to change this now, because nobody's
complained so far, and a change like this needs a proper discussion
anyway.

NO_TEST=refactoring
NO_DOC=refactoring
NO_CHANGELOG=refactoring

Implement interactive transactions over iproto streams. Each stream
can start its own transaction, so they allows multiplexing several
transactions over one connection. If any request fails during the
transaction, it will not affect the other requests in the transaction.
If disconnect occurs when there is some active transaction in stream,
this transaction will be rollbacked, if it does not have time to commit
before this moment.

Part of #5860

@TarantoolBot document
Title: interactive transactions was implemented over iproto streams.
The main purpose of streams is transactions via iproto. Each stream
can start its own transaction, so they allows multiplexing several
transactions over one connection. There are multiple ways to begin,
commit and rollback transaction: using IPROTO_CALL and IPROTO_EVAL
with corresponding function (box.begin, box.commit and box.rollback),
IPROTO_EXECUTE with corresponding sql request ('TRANSACTION START',
'COMMIT', 'ROLLBACK') and IPROTO_BEGIN, IPROTO_COMMIT, IPROTO_ROLLBACK
accordingly. If disconnect occurs when there is some active transaction
in stream, this transaction will be rollbacked, if it does not have time
to commit before this moment. Add new command codes for begin, commit and
rollback transactions: `IPROTO_BEGIN 14`, `IPROTO_COMMIT 15` and
`IPROTO_ROLLBACK 16` accordingly.

The modules subsystem hides some low-level operations under API.
In particular the modules subsystem is responsible for

 - modules lookup in Lua's "package.search" storage
 - modules caching to eliminate expensive load procedure
 - function symbol resolving

Because naming is intersecting with current module functions
sitting in box/func, lets rename the later to schema_module
prefix. We will use this prefix in next patches to point the
modules in box.schema.func are just a particular user of
the general modules engine.

Part-of #4642

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

The only purpose of the module argument is to
notify the caller that the module doesn't exist.
Lets simplify the code and drop this argument.

Part-of #4642

Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

Make it possible to set temporary directory where
module will be copied before load.

@TarantoolBot document
Title: Justify module (re-)loading semantics

It is now possible to set directory where temporary
copies of modules to be loaded will be created.
It is done by setting $(TMPDIR) variable. It
will be "/tmp" if variable was not set.

Follow up #4945

API is different from box.session.push() - sync argument was
removed. It will disappear from Lua API as well, because it just
is not needed here. Session is omitted as well. Indeed, a user
can't push to a foreign session, and the current session can be
obtained inside box_session_push(). And anyway session is not in
the public C API.

Internally dump into iproto is done using obuf_dup(), just like
tuple_to_obuf() does. obuf_alloc() would be a bad call here,
because it wouldn't be able to split the pushed data into several
obuf chunks, and would cause obuf fragmentation.

Dump into plain text behaves just like a Lua push - it produces a
YAML formatted text or Lua text depending on output format. But to
turn MessagePack into YAML or Lua text an intermediate Lua
representation is used, because there are no a MessagePack -> YAML
and MessagePack -> Lua text translators yet.

Closes #4734

@TarantoolBot document
Title: box_session_push() C API

There is a new function in the public C API:
```C
    int
    box_session_push(const char *data, const char *data_end);
```

It takes raw MessagePack, and behaves just like Lua
`box.session.push()`.

box_process_call/eval() in the end check if there is an
active transaction. If there is, it is rolled back, and
an error is set.

But rollback is not needed anymore, because anyway in
the end of the request the fiber is stopped, and its
not finished transaction is rolled back. Just setting
of the error is enough.

Follow-up #4662

Closes #2200
Closes #4113
Closes #2233

@TarantoolBot document
Title: The box.internal.sql_function_create is forbidden
Legacy mechanism box.internal.sql_function_create to make some
Lua function available in SQL is forbidden now.

To make some function available in SQL you need to use
box.schema.func.create() mechanism: you need to specify
1) function language and language-specific options(e.g. you
   are able to define a persistent Lua function)
2) whether this function is_deterministic or not: deterministic
   functions allows to generate more efficient SQL VDBE bytecode
   so you better specify it when it is true
3) the function returns type: a Tarantool type string describes
   a type of value returned by function
4) param_list - a table of Tarantool's types strings desccribe
   function argument types
5) exports - a table of Tarantool's frontends where this function
   should be available ('LUA' by default). You need to specify
   {'LUA', 'SQL'} to make function available both in SQL requests
   and visible in box.func folder

Example:
-- Case1: C function
-- function1.so has int divide() symbol
box.schema.func.create("function1.divide", {language = 'C',
			returns = 'number',
			param_list = {'number', 'number'},
			is_deterministic = true,
			exports = {'LUA', 'SQL'}})
box.execute('SELECT "function1.divide"(6, 3)')
- metadata:
  - name: '"function1.divide"(6, 3)'
    type: number
  rows:
  - [2]
box.schema.func.drop("function1.divide")

-- Case2: Persistent Lua function
box.schema.func.create("SUMMARIZE", {language = 'LUA',
			returns = 'number',
			body = 'function (a, b) return a + b end',
			param_list = {'number', 'number'},
			is_deterministic = true,
			exports = {'LUA', 'SQL'}})
box.execute('SELECT summarize(1, 2)')
- metadata:
  - name: summarize(1, 2)
    type: number
  rows:
  - [3]
box.schema.func.drop("summarize")

Moreover there is a special predefined Lua function LUA that
allows to evaluate a custom Lua expressions in SQL.
You need to pass a string in form "return ...." to LUA function
that returns more than one value of any type.

Example:
box.execute('SELECT lua(\'return 1 + 1\')')
- metadata:
  - name: lua('return 1 + 1')
    type: any
  rows:
  - [2]
box.execute('SELECT lua(\'return box.cfg.memtx_memory\')')
- metadata:
  - name: lua('return box.cfg.memtx_memory')
    type: any
  rows:
  - [268435456]

This is a followup for 7691154a

In case of throwing client error because of inactive function
we did not destroy used port. It could possibly cause huge
memory leaks as could be seen with top or its analogues when
performing net.box test run in a loop.

Closes #4388

Refactoring: don't touch a fiber gc storage on a transaction rollback
explicitly. This relaxes dependencies between fiber and transaction
life cycles.

Prerequisites: #1254

The function func object used to provide a call method only for
C functions. In scope of this patch it reworked to be a uniform
function call frontend both for C and Lua functions.

Introduced classes func_c and func_lua, that provide own
constructors which produce implementation-specific object with
call and destroy methods.

Needed for #4182, #1260

Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.

To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
A new method get_msgpack returns a content of a port as a
msgpack data. The lifecycle of the returned value is
implementation-specific: it may either be returned directly from
the port, in which case the data will stay alive as long as the
port is alive, or it may be allocated on the fiber()->gc, in
which case the caller is responsible for cleaning up.

Needed for #4182, #1260

Before the patch Tarantool had a thread- and C-file- local array
of 4 static buffers, each 1028 bytes. It provided an API
tt_static_buf() allowing to return them one by one in a cycle.

Firstly, it consumed totally 200Kb of BSS memory in summary over
all C-files using these buffers. Obviously, it was a bug and was
not made intentionally. The buffers were supposed to be a one
process-global array.

Secondly, even if the bug above had been fixed somehow, sometimes
it would have been needed to obtain a bit bigger buffer. For
example, to store a UDP packet - ~1.5Kb.

This commit replaces these 4 buffers with small/ static allocator
which does basically the same, but in more granulated and
manoeuvrable way. This commit frees ~188Kb of BSS section.

A main motivation for this commit is a wish to use a single
global out-of-stack buffer to read UDP packets into it in the
SWIM library, and on the other hand do not pad out BSS section
with a new SWIM-special static buffer. Now SWIM uses stack for
this and in the incoming cryptography SWIM component it will need
more.

Closes #2946.

@TarantoolBot document
Title: fixed module reload
There was a bug in tarantool documentation:
https://tarantool.io/en/doc/1.7/book/box/
box_schema/#lua-function.box.schema.func.reload
Now it is allowed to reload all functions in loadable
module via one method. Legacy method including finction
name is forbidden.
box.schema.func.reload("utils")       -- ok since now
box.schema.func.reload("utils.func1") -- forbidden since now

Global reload is still unsupported because it seems
to be useless.
box.schema.func.reload()              -- invalid!

Allow define access privileges for all spaces, functions and sequences.
Read and write privileges are supported for spaces, execute privilege
for sequences. Privilege granting and revoking might be done through old api
without object identification:
  box.schema.user.grant("guest", "read", "space")

Prerequisite #945

* Add following behavior:
Owner of object can't utilize her own objects if she has not usage
access.
* Change access checks of space, sequence, function objects
Similar checks of other objects are performed in alter.cc.

Closes gh-3089

Currently, if a CALL/EVAL request leaves an open transaction at return,
we silently rollback it and print a warning to the log mentioning the
function name or eval expression to facilitate further debugging. After
issue #946 was fixed, we can't do that anymore, because request input,
which stores CALL/EVAL parameters, may be discarded before request
completion and hence be unavailable for logging. Without additional
information pointing at the culprit, the log message is pointless (see
issue #1100). We could copy the arguments, but that would slow down CALL
execution, which can't be justified solely by the need of verbose
logging. So let's stop being lenient and fail requests that do not close
transaction at return. This should encourage negligent users to finally
fix their code.

Follow-up #946

The iproto subsystem switches between two output buffers once in a while
in order to reclaim memory so passing a pointer to the output buffer
directly to box_process_call() or box_process_eval() is incorrect in
case the called function yields. To fix that, let's make these functions
return the CALL/EVAL result in a port object, which then can then be
encoded in msgpack with port_dump().

Needed for #946

So that it can be used not only for serializing a list of tuples, but
also for serializing a Lua stack that stores output of CALL/EVAL.

Needed for #946

We can do it for free now as all functions used by call.cc have already
been converted to C and there's nothing in call.cc that really needs any
C++ features.

Replace tnt_raise() with diag_set() and add a wrapper that raises
exception in case of error to be used in C++ code.

While we are at it, let's also move access_check_session_xc() to
the header file, because it's a trivial wrapper.

* Add box_on_access_denied API method
* Modify access error handlers in order to call the mentioned trigger
* Add new type of error - AccessDeniedError

Related #2911 "add audit log triggers"

Delete specifc access denied error code (ER_FUNCTION_ACCESS_DENIED,
ER_SPACE_ACCESS_DENIED, ER_FUNC_ACCESS_DENIED) and always
ER_ACCESS_DENIED code, which now contains object name and type

Pass operation type (create, drop, grant, revoke) to ER_ACCESS_DENIED.
Add a helper function schema_find_name() to schema.[h,cc].

In scope of gh-2911 "add triggers for audit log".

Heavily edited by @kostja

Add system privileges 'session' and 'usage'

* 'session' privilege lets user connect to database server
* 'usage' privilege lets user use his/her rights on database objects
* Both privileges are assigned to all users by default.

Implementation details:
* system privileges are special grant rights to 'universe'.
Therefore, they can be granted only by admin. Because of this fact,
during creation or deletion of user, we have to switch to 'admin' to
grant or revoke these rights.

Important changes:
* changed bootstrap.snap due to need to start admin with new privileges
* added auto upgrade script for 1.7.7

Fixes gh-2898.

With contributions by @kostja.

box.session.su() changes both user and effective user right now.
Changing only the session user seems to be rather difficult:
we need to keep the object allocated somewhere, and keeping
in mind request multiplexor in iproto, with which many requests
can share the same session, it can only be Lua stack.

While at it, change current_user() to effective_user() to
make it less ambiguous.

Introduce all the necessary ACL for ANSI SQL, as well as SESSION
and USAGE.

Change access storage type from uint8_t to a typedef.

Necessary for gh-2898.

CALL should check only EXECUTE access on universe instead of
READ, WRITE, EXECUTE.

Closes #3017

- Change signature of function access_check_func.
  Now it returns status instead of function.

Close #2816

This patch adds ability to reload C procedures on the fly without
downtime. To achive that, Tarantool loads a new copy of shared
library and starts routing all new request to the new version.
The previous version remains active until all started calls
are finished. All shared libraries are loaded with RTLD_LOCAL,
therefore two or more copies can co-exist without any problems.
From now box loads all external modules via an unique symlink to
avoid caching inside dlopen().

If one of some module function is reloaded then all other functions
from this module will be reloaded.

Reviewed and heavily patched by Roman Tsisyk.

Closes #910

Needed for #910

No semantic changes.

Follow up #2619 and #2507

* Add proper checks for premature end of request
* Convert to C

Follow up #2619

Create a special request type for call/eval requests.
Struct call_request is much lighter than struct request, and its
decoding is faster for call/eval request.
The new struct call_request allows call/eval without parameters.

In a future we are able to add into struct call_request more
members and merge it with box_function_ctx and lua_function_ctx.

Part of #2619

In order to create struct call_request it is neccessary to move
call/eval logic into separate call.c/.h files to avoid adding
call_request into more common box.h.

Part of #2619