Skip to content
Snippets Groups Projects
Commit d4a4a74e authored by Andrey Saranchin's avatar Andrey Saranchin Committed by Vladimir Davydov
Browse files

misc: add changelog and doc request for pagination

Closes #7639

@TarantoolBot document
Title: Pagination

Introduce pagination to memtx and vinyl trees. It allows to start iteration
after last previously selected tuple. Since tuples can be huge in size,
there is a new entity called `position` - opaque object that represents
position of tuple in index. It is encoded to base64 format and can be
transferred and stored without any issues with non-printable characters.

Lua API of pagination:
- `select`
Select is provided with two new options: `fetch_pos` and `after`.
Option `after` can take tuple (or table, representing it) or
`position`. If it is passed, iteration will begin after tuple which
is described by the option. If there is no tuple described by `after` in
index, iteration starts with a tuple that would have been preceded by a
described one. Empty string ("") or `box.NULL` can be passed as a start
position. Here is an example:
```lua
last_tuple = box.NULL
while true do
    tuples = s:select(key, {limit=1000, after=last_tuple})
    if #tuples == 0 then
        break
    end
    last_tuple = tuples[#tuples]
    process_data(tuples)
end
```

The second option is `fetch_pos` - one can fetch a `position` of last
selected tuple, it will be returned as the second value. If no tuples
were fetched, `position` will be `nil`.
The snippet can be simplified by using `fetch_pos` option:
```lua
pos = ""
while true do
    tuples, pos = s:select(key, {limit=1000, after=pos, fetch_pos=true})
    if pos == nil then
        break
    end
    process_data(tuples)
end
```
However, the benefits of using position are not only lower memory
consumption and easier syntax - position is the only way to paginate
over multikey and functional indexes - an error will be thrown when you
use option `after` with tuple because tuple has not enough information
to describe its position in such indexes.

-`tuple_pos`
Index has a new method `index:tuple_pos(tuple)`, it returns `position`
of passed tuple (or a table, representing a tuple) in this index, even
if there is no such tuple. Passed tuple must match format of the space.
Does not work with multikey and functional indexes - an error will be
thrown.
```lua
pos = ""
while true do
    tuples = s:select(key, {limit=1000, after=pos})
    if #tuples == 0 then
        break
    end
    last_tuple = tuples[#tuples]
    pos = s.index.pk:tuple_pos(last_tuple)
    process_data(tuples)
end
```

-`pairs`
Pairs is provided only with `after` option with the same semantics.

```lua
for _, tuple in s:pairs(10, {after={10, 5}} do
    process_tuple(tuple)
end
```

-IPROTO
IPROTO is provided with new keys:

0x2e - IPROTO_AFTER_POSITION - start iteration after passed
`position`. It has type MP_STR.

0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple.
It has type MP_ARRAY.

0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple
in response. It has type MP_BOOL.

0x35 - IPROTO_POSITION - `position`, sent in response if
IPROTO_FETCH_POSITION is true. It has type MP_STR.

To start iteration from the beginning, one can send empty MP_STR as
IPROTO_AFTER_POSITION or send no `position` at all.

IPROTO_VERSION is bumped, new feature `pagination` is added.

-`net.box`
Net box select is provided with the same options `after` and
`fetch_pos`. It has the same behavior as in index select. The only
difference is format of returned values:
If option `buffer` is passed, the whole response is written to buffer,
only number of written bytes is returned.
If option `skip_header` is passed (if `skip_header` is passed, then
`buffer` is necessarily passed too), only IPROTO_DATA (without header)
is written in buffer, select returns number of bytes written as the
first value and `position` of last selected tuple as the second one.
If request is async and without buffer, then table with tuples is
returned if `fetch_pos` is false or nil (old behavior) and table with
table of tuples and `position` of last selected tuples is returned if
`fetch_pos` is true.
If no options, described above, are passed, then table with tuples is
returned as the first value and new `position` as the second one if
`fetch_pos` option is true.

Synchronous API:
```lua
pos = ""
while true do
    tuples, pos = conn.space.s:select(key, {limit=1000, after=pos,
        fetch_pos=true})
    if pos == nil then
        break
    end
    process_data(tuples)
end
```

Asynchronous API:
```lua
pos = ""
while true do
    ret = conn.space.s:select(key, {limit=1000, after=pos,
        fetch_pos=true, is_async=true})
    tuples = ret[1]
    pos = ret[2]
    if pos == nil then
        break
    end
    process_data(tuples)
end
```

NO_TEST=no changes
parent 5eee5d7e
No related branches found
No related tags found
No related merge requests found
## feature/core
* Introduced pagination support for memtx and vinyl tree indexes. Now, it's
possible to resume `pairs` and `select` from the position where the last
call stopped (gh-7639).
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment