Skip to content
Snippets Groups Projects
user avatar
Cyrill Gorcunov authored
Currently to run "C" function from some external module one
have to register it first in "_func" system space. This is
a problem if node is in read-only mode (replica).

Still people would like to have a way to run such functions
even in ro mode. For this sake we implement "box.lib" lua module.

Unlike `box.schema.func` interface the `box.lib` does not defer module
loading procedure until first call of a function. Instead a module
is loaded immediately and if some error happens (say shared
library is corrupted or not found) it pops up early.

The need of use stored C procedures implies that application is
running under serious loads most likely there is modular structure
present on Lua level (ie same shared library is loaded in different
sources) thus we cache the loaded library and reuse it on next
load attempts. To verify that cached library is up to day the
module_cache engine test for file attributes (device, inode, size,
modification time) on every load attempt.

Since both `box.schema.func` and `box.lib` are using caching to minimize
module loading procedure the pass-through caching scheme is
implemented:

 - box.lib relies on module_cache engine for caching;
 - box.schema.func does snoop into box.lib hash table when attempt
   to load a new module, if module is present in box.lib hash then
   it simply referenced from there and added into own hash table;
   in case if module is not present then it loaded from the scratch
   and put into both hashes;
 - the module_reload action in box.schema.func invalidates module_cache
   or fill it if entry is not present.

Closes #4642

Co-developed-by: default avatarVladislav Shpilevoy <v.shpilevoy@tarantool.org>
Signed-off-by: default avatarCyrill Gorcunov <gorcunov@gmail.com>

@TarantoolBot document
Title: box.lib module

Overview
========

`box.lib` module provides a way to create, delete and execute
`C` procedures from shared libraries. Unlike `box.schema.func`
methods the functions created with `box.lib` help are not persistent
and live purely in memory. Once a node get turned off they are
vanished. An initial purpose for them is to execute them on
nodes which are running in read-only mode.

Module functions
================

`box.lib.load(path) -> obj | error`
-----------------------------------

Loads a module from `path` and return an object instance
associate with the module, otherwise an error is thrown.

The `path` should not end up with shared library extension
(such as `.so`), only a file name shall be there.

Possible errors:

- IllegalParams: module path is either not supplied
  or not a string.
- SystemError: unable to open a module due to a system error.
- ClientError: a module does not exist.
- OutOfMemory: unable to allocate a module.

Example:

``` Lua
-- Without error handling
m = box.lib.load('path/to/library)

-- With error handling
m, err = pcall(box.lib.load, 'path/to/library')
if err ~= nil then
    print(err)
end
```

`module:unload() -> true | error`
---------------------------------

Unloads a module. Returns `true` on success, otherwise an error
is thrown. Once the module is unloaded one can't load new
functions from this module instance.

Possible errors:

- IllegalParams: a module is not supplied.
- IllegalParams: a module is already unloaded.

Example:

``` Lua
m = box.lib.load('path/to/library')
--
-- do something with module
--
m:unload()
```

If there are functions from this module referenced somewhere
in other places of Lua code they still can be executed because
the module continue sitting in memory until the last reference
to it is closed.

If the module become a target to the Lua's garbage collector
then unload is called implicitly.

`module:load(name) -> obj | error`
----------------------------------

Loads a new function with name `name` from the previously
loaded `module` and return a callable object instance
associated with the function. On failure an error is thrown.

Possible errors:
 - IllegalParams: function name is either not supplied
   or not a string.
 - IllegalParams: attempt to load a function but module
   has been unloaded already.
 - ClientError: no such function in the module.
 - OutOfMemory: unable to allocate a function.

Example:

``` Lua
-- Load a module if not been loaded yet.
m = box.lib.load('path/to/library')
-- Load a function with the `foo` name from the module `m`.
func = m:load('foo')
```

In case if there is no need for further loading of other
functions from the same module then the module might be
unloaded immediately.

``` Lua
m = box.lib.load('path/to/library')
func = m:load('foo')
m:unload()
```

`function:unload() -> true | error`
-----------------------------------

Unloads a function. Returns `true` on success, otherwise
an error is thrown.

Possible errors:
 - IllegalParams: function name is either not supplied
   or not a string.
 - IllegalParams: the function already unloaded.

Example:

``` Lua
m = box.lib.load('path/to/library')
func = m:load('foo')
--
-- do something with function and cleanup then
--
func:unload()
m:unload()
```

If the function become a target to the Lua's garbage collector
then unload is called implicitly.

Executing a loaded function
===========================

Once function is loaded it can be executed as an ordinary Lua call.
Lets consider the following example. We have a `C` function which
takes two numbers and returns their sum.

``` C
int
cfunc_sum(box_function_ctx_t *ctx, const char *args, const char *args_end)
{
	uint32_t arg_count = mp_decode_array(&args);
	if (arg_count != 2) {
		return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s",
				     "invalid argument count");
	}
	uint64_t a = mp_decode_uint(&args);
	uint64_t b = mp_decode_uint(&args);

	char res[16];
	char *end = mp_encode_uint(res, a + b);
	box_return_mp(ctx, res, end);
	return 0;
}
```

The name of the function is `cfunc_sum` and the function is built into
`cfunc.so` shared library.

First we should load it as

``` Lua
m = box.lib.load('cfunc')
cfunc_sum = m:load('cfunc_sum')
```

Once successfully loaded we can execute it. Lets call the
`cfunc_sum` with wrong number of arguments

``` Lua
cfunc_sum()
 | ---
 | - error: invalid argument count
```

We will see the `"invalid argument count"` message in output.
The error message has been set by the `box_error_set` in `C`
code above.

On success the sum of arguments will be printed out.

``` Lua
cfunc_sum(1, 2)
 | ---
 | - 3
```

The functions may return multiple results. For example a trivial
echo function which prints arguments passed in.

``` Lua
cfunc_echo(1,2,3)
 | ---
 | - 1
 | - 2
 | - 3
```

Module and function caches
==========================

Loading a module is relatively slow procedure because operating
system needs to read the library, resolve its symbols and etc.
Thus to speedup this procedure if the module is loaded for a first
time we put it into an internal cache. If module is sitting in
the cache already and new request to load comes in -- we simply
reuse a previous copy. In case if module is updated on a storage
device then on new load attempt we detect that file attributes
(such as device number, inode, size, modification time) get changed
and reload module from the scratch. Note that newly loaded module
does not intersect with previously loaded modules, the continue
operating with code previously read from cache.

Thus if there is a need to update a module then all module instances
should be unloaded (together with functions) and loaded again.

Similar caching technique applied to functions -- only first function
allocation cause symbol resolving, next ones are simply obtained from
a function cache.
f463b5fa
History

Tarantool

Build Status Build Status Code Coverage Telegram Slack Gitter Google Groups

https://tarantool.io/en/

Patch submissions and discussion of particular patches https://lists.tarantool.org/mailman/listinfo/tarantool-patches/

General development discussions https://lists.tarantool.org/mailman/listinfo/tarantool-discussions/

Tarantool is an in-memory database and application server.

Key features of the application server:

  • 100% compatible drop-in replacement for Lua 5.1, based on LuaJIT 2.1. Simply use #!/usr/bin/tarantool instead of #!/usr/bin/lua in your script.
  • full support for Lua modules and a rich set of own modules, including cooperative multitasking, non-blocking I/O, access to external databases, etc

Key features of the database:

  • ANSI SQL, including views, joins, referential and check constraints
  • MsgPack data format and MsgPack based client-server protocol
  • two data engines: 100% in-memory with optional persistence and an own implementation of LSM-tree, to use with large data sets
  • multiple index types: HASH, TREE, RTREE, BITSET
  • asynchronous master-master replication
  • authentication and access control
  • the database is just a C extension to the application server and can be turned off

Supported platforms are Linux/x86, FreeBSD/x86 and OpenBSD/x86, Mac OS X.

Tarantool is ideal for data-enriched components of scalable Web architecture: queue servers, caches, stateful Web applications.

To download and install Tarantool as a binary package for your OS, please visit https://tarantool.io/en/download/.

To build Tarantool from source, see detailed instructions in the Tarantool documentation at https://tarantool.io/en/doc/2.1/dev_guide/building_from_source/.

Please report bugs at https://github.com/tarantool/tarantool/issues We also warmly welcome your feedback in the discussion mailing list, tarantool@googlegroups.com.

Thank you for your interest in Tarantool!