Add merger for tuple streams (Lua part)
Fixes #3276. @TarantoolBot document Title: Merger for tuple streams The main concept of the merger is a source. It is an object that provides a stream of tuples. There are four types of sources: a tuple source, a table source, a buffer source and a merger itself. A tuple source just return one tuple. However this source (as well as a table and a buffer ones) supports fetching of a next data chunk, so the API allows to create it from a Lua iterator: `merger.new_tuple_source(gen, param, state)`. A `gen` function should return `state, tuple` on each call and then return `nil` when no more tuples available. Consider the example: ```lua box.cfg({}) box.schema.space.create('s') box.space.s:create_index('pk') box.space.s:insert({1}) box.space.s:insert({2}) box.space.s:insert({3}) s = merger.new_tuple_source(box.space.s:pairs()) s:select() --- - - [1] - [2] - [3] ... s = merger.new_tuple_source(box.space.s:pairs()) s:pairs():totable() --- - - [1] - [2] - [3] ... ``` As we see a source (it is common for all sources) has `:select()` and `:pairs()` methods. The first one has two options: `buffer` and `limit` with the same meaning as ones in net.box `:select()`. The `:pairs()` method (or `:ipairs()` alias) returns a luafun iterator (it is a Lua iterator, but also provides a set of handy methods to operate in functional style). The same API exists to create a table and a buffer source: `merger.new_table_source(gen, param, state)` and `merger.new_buffer_source(gen, param, state)`. A `gen` function should return a table or a buffer on each call. There are also helpers that are useful when all data are available at once: `merger.new_source_fromtable(tbl)` and `merger.new_source_frombuffer(buf)`. A merger is a special kind of a source, which is created from a key_def object and a set of sources. It performs a kind of the merge sort: chooses a source with a minimal / maximal tuple on each step, consumes a tuple from this source and repeats. The API to create a merger is the following: ```lua local key_def_lib = require('key_def') local merger = require('merger') local key_def = key_def_lib.new(<...>) local sources = {<...>} local merger_inst = merger.new(key_def, sources, { -- Ascending (false) or descending (true) order. -- Default is ascending. reverse = <boolean> or <nil>, }) ``` An instance of a merger has the same `:select()` and `:pairs()` methods as any other source. The `key_def_lib.new()` function takes a table of key parts as an argument in the same format as box.space.<...>.index.<...>.parts or conn.space.<...>.index.<...>.parts (where conn is a net.box connection): ``` local key_parts = { { fieldno = <number>, type = <string>, [ is_nullable = <boolean>, ] [ collation_id = <number>, ] [ collation = <string>, ] }, ... } local key_def = key_def_lib.new(key_parts) ``` A key_def can be cached across requests with the same ordering rules (typically requests to a same space).
Showing
- src/box/CMakeLists.txt 2 additions, 0 deletionssrc/box/CMakeLists.txt
- src/box/lua/init.c 6 additions, 1 deletionsrc/box/lua/init.c
- src/box/lua/merger.c 1143 additions, 0 deletionssrc/box/lua/merger.c
- src/box/lua/merger.h 47 additions, 0 deletionssrc/box/lua/merger.h
- src/box/lua/merger.lua 41 additions, 0 deletionssrc/box/lua/merger.lua
- test/box-tap/merger.test.lua 768 additions, 0 deletionstest/box-tap/merger.test.lua
Loading
Please register or sign in to comment