From d4a4a74ee6264f85086de2d537b11dc69c1316fb Mon Sep 17 00:00:00 2001
From: Andrey Saranchin <Andrey22102001@gmail.com>
Date: Wed, 5 Oct 2022 17:26:05 +0300
Subject: [PATCH] misc: add changelog and doc request for pagination
Closes #7639
@TarantoolBot document
Title: Pagination
Introduce pagination to memtx and vinyl trees. It allows to start iteration
after last previously selected tuple. Since tuples can be huge in size,
there is a new entity called `position` - opaque object that represents
position of tuple in index. It is encoded to base64 format and can be
transferred and stored without any issues with non-printable characters.
Lua API of pagination:
- `select`
Select is provided with two new options: `fetch_pos` and `after`.
Option `after` can take tuple (or table, representing it) or
`position`. If it is passed, iteration will begin after tuple which
is described by the option. If there is no tuple described by `after` in
index, iteration starts with a tuple that would have been preceded by a
described one. Empty string ("") or `box.NULL` can be passed as a start
position. Here is an example:
```lua
last_tuple = box.NULL
while true do
tuples = s:select(key, {limit=1000, after=last_tuple})
if #tuples == 0 then
break
end
last_tuple = tuples[#tuples]
process_data(tuples)
end
```
The second option is `fetch_pos` - one can fetch a `position` of last
selected tuple, it will be returned as the second value. If no tuples
were fetched, `position` will be `nil`.
The snippet can be simplified by using `fetch_pos` option:
```lua
pos = ""
while true do
tuples, pos = s:select(key, {limit=1000, after=pos, fetch_pos=true})
if pos == nil then
break
end
process_data(tuples)
end
```
However, the benefits of using position are not only lower memory
consumption and easier syntax - position is the only way to paginate
over multikey and functional indexes - an error will be thrown when you
use option `after` with tuple because tuple has not enough information
to describe its position in such indexes.
-`tuple_pos`
Index has a new method `index:tuple_pos(tuple)`, it returns `position`
of passed tuple (or a table, representing a tuple) in this index, even
if there is no such tuple. Passed tuple must match format of the space.
Does not work with multikey and functional indexes - an error will be
thrown.
```lua
pos = ""
while true do
tuples = s:select(key, {limit=1000, after=pos})
if #tuples == 0 then
break
end
last_tuple = tuples[#tuples]
pos = s.index.pk:tuple_pos(last_tuple)
process_data(tuples)
end
```
-`pairs`
Pairs is provided only with `after` option with the same semantics.
```lua
for _, tuple in s:pairs(10, {after={10, 5}} do
process_tuple(tuple)
end
```
-IPROTO
IPROTO is provided with new keys:
0x2e - IPROTO_AFTER_POSITION - start iteration after passed
`position`. It has type MP_STR.
0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple.
It has type MP_ARRAY.
0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple
in response. It has type MP_BOOL.
0x35 - IPROTO_POSITION - `position`, sent in response if
IPROTO_FETCH_POSITION is true. It has type MP_STR.
To start iteration from the beginning, one can send empty MP_STR as
IPROTO_AFTER_POSITION or send no `position` at all.
IPROTO_VERSION is bumped, new feature `pagination` is added.
-`net.box`
Net box select is provided with the same options `after` and
`fetch_pos`. It has the same behavior as in index select. The only
difference is format of returned values:
If option `buffer` is passed, the whole response is written to buffer,
only number of written bytes is returned.
If option `skip_header` is passed (if `skip_header` is passed, then
`buffer` is necessarily passed too), only IPROTO_DATA (without header)
is written in buffer, select returns number of bytes written as the
first value and `position` of last selected tuple as the second one.
If request is async and without buffer, then table with tuples is
returned if `fetch_pos` is false or nil (old behavior) and table with
table of tuples and `position` of last selected tuples is returned if
`fetch_pos` is true.
If no options, described above, are passed, then table with tuples is
returned as the first value and new `position` as the second one if
`fetch_pos` option is true.
Synchronous API:
```lua
pos = ""
while true do
tuples, pos = conn.space.s:select(key, {limit=1000, after=pos,
fetch_pos=true})
if pos == nil then
break
end
process_data(tuples)
end
```
Asynchronous API:
```lua
pos = ""
while true do
ret = conn.space.s:select(key, {limit=1000, after=pos,
fetch_pos=true, is_async=true})
tuples = ret[1]
pos = ret[2]
if pos == nil then
break
end
process_data(tuples)
end
```
NO_TEST=no changes
---
changelogs/unreleased/pagination.md | 5 +++++
1 file changed, 5 insertions(+)
create mode 100644 changelogs/unreleased/pagination.md
diff --git a/changelogs/unreleased/pagination.md b/changelogs/unreleased/pagination.md
new file mode 100644
index 0000000000..0ace3798e2
--- /dev/null
+++ b/changelogs/unreleased/pagination.md
@@ -0,0 +1,5 @@
+## feature/core
+
+* Introduced pagination support for memtx and vinyl tree indexes. Now, it's
+ possible to resume `pairs` and `select` from the position where the last
+ call stopped (gh-7639).
--
GitLab