misc: add changelog and doc request for pagination
Closes #7639 @TarantoolBot document Title: Pagination Introduce pagination to memtx and vinyl trees. It allows to start iteration after last previously selected tuple. Since tuples can be huge in size, there is a new entity called `position` - opaque object that represents position of tuple in index. It is encoded to base64 format and can be transferred and stored without any issues with non-printable characters. Lua API of pagination: - `select` Select is provided with two new options: `fetch_pos` and `after`. Option `after` can take tuple (or table, representing it) or `position`. If it is passed, iteration will begin after tuple which is described by the option. If there is no tuple described by `after` in index, iteration starts with a tuple that would have been preceded by a described one. Empty string ("") or `box.NULL` can be passed as a start position. Here is an example: ```lua last_tuple = box.NULL while true do tuples = s:select(key, {limit=1000, after=last_tuple}) if #tuples == 0 then break end last_tuple = tuples[#tuples] process_data(tuples) end ``` The second option is `fetch_pos` - one can fetch a `position` of last selected tuple, it will be returned as the second value. If no tuples were fetched, `position` will be `nil`. The snippet can be simplified by using `fetch_pos` option: ```lua pos = "" while true do tuples, pos = s:select(key, {limit=1000, after=pos, fetch_pos=true}) if pos == nil then break end process_data(tuples) end ``` However, the benefits of using position are not only lower memory consumption and easier syntax - position is the only way to paginate over multikey and functional indexes - an error will be thrown when you use option `after` with tuple because tuple has not enough information to describe its position in such indexes. -`tuple_pos` Index has a new method `index:tuple_pos(tuple)`, it returns `position` of passed tuple (or a table, representing a tuple) in this index, even if there is no such tuple. Passed tuple must match format of the space. Does not work with multikey and functional indexes - an error will be thrown. ```lua pos = "" while true do tuples = s:select(key, {limit=1000, after=pos}) if #tuples == 0 then break end last_tuple = tuples[#tuples] pos = s.index.pk:tuple_pos(last_tuple) process_data(tuples) end ``` -`pairs` Pairs is provided only with `after` option with the same semantics. ```lua for _, tuple in s:pairs(10, {after={10, 5}} do process_tuple(tuple) end ``` -IPROTO IPROTO is provided with new keys: 0x2e - IPROTO_AFTER_POSITION - start iteration after passed `position`. It has type MP_STR. 0x2f - IPROTO_AFTER_TUPLE - start iteration after passed tuple. It has type MP_ARRAY. 0x1f - IPROTO_FETCH_POSITION - send position of last fetched tuple in response. It has type MP_BOOL. 0x35 - IPROTO_POSITION - `position`, sent in response if IPROTO_FETCH_POSITION is true. It has type MP_STR. To start iteration from the beginning, one can send empty MP_STR as IPROTO_AFTER_POSITION or send no `position` at all. IPROTO_VERSION is bumped, new feature `pagination` is added. -`net.box` Net box select is provided with the same options `after` and `fetch_pos`. It has the same behavior as in index select. The only difference is format of returned values: If option `buffer` is passed, the whole response is written to buffer, only number of written bytes is returned. If option `skip_header` is passed (if `skip_header` is passed, then `buffer` is necessarily passed too), only IPROTO_DATA (without header) is written in buffer, select returns number of bytes written as the first value and `position` of last selected tuple as the second one. If request is async and without buffer, then table with tuples is returned if `fetch_pos` is false or nil (old behavior) and table with table of tuples and `position` of last selected tuples is returned if `fetch_pos` is true. If no options, described above, are passed, then table with tuples is returned as the first value and new `position` as the second one if `fetch_pos` option is true. Synchronous API: ```lua pos = "" while true do tuples, pos = conn.space.s:select(key, {limit=1000, after=pos, fetch_pos=true}) if pos == nil then break end process_data(tuples) end ``` Asynchronous API: ```lua pos = "" while true do ret = conn.space.s:select(key, {limit=1000, after=pos, fetch_pos=true, is_async=true}) tuples = ret[1] pos = ret[2] if pos == nil then break end process_data(tuples) end ``` NO_TEST=no changes
changelogs/unreleased/pagination.md
0 → 100644
Please register or sign in to comment