flaky integration test when cluster is replicated

integration test test_change_cache_by_config_replica can fail because of replication delay. we need to be sure that space is created on storage-replica before query execution to avoid flaky result. see error below

Failed tests:
-------------

1) configuration.sql_cache.test_change_cache_by_config_replica
...d-cartridge/test_app/test/integration/sql_cache_test.lua:39: expected: nil, actual: Sbroad Error: Lua error (dispatch IR): LuaError(ExecutionError("Sbroad Error: Failed to execute the query: LuaError(ExecutionError(\"Space 'space_for_breake_cache' does not exist\"))"))
stack traceback:
	...d-cartridge/test_app/test/integration/sql_cache_test.lua:39: in function 'configuration.sql_cache.test_change_cache_by_config_replica'
	...
	[C]: in function 'xpcall'

to get this error run script:

#!/bin/bash
iter=0
while true
do
    ((iter=iter+1))
    echo $iter
    make test_integration
    if [ $? -ne 0 ]; then
        break
    fi
    sleep 3
done

added Tests label

This error occurs because Tarantool uses async replication by default, and it is possible to try selecting from a replica that does not have the space yet. There are two ways to fix the flaky test:

wait replication
Return the cluster when replication ends. Replace the following code:

#sbroad-cartridge/test_app/test/helper/cluster_async_replication.lua

helper.start_test_cluster = function (cfg)
    fio.rmtree(helper.datadir)
    fio.mktree(helper.datadir)

    helper.cluster = cartridge_helpers.Cluster:new(...)

    helper.cluster:start()
    helper.cluster:upload_config(cfg)
end

with this

#sbroad-cartridge/test_app/test/helper/cluster_async_replication.lua

helper.start_test_cluster = function (cfg)
    fio.rmtree(helper.datadir)
    fio.mktree(helper.datadir)

    helper.cluster = cartridge_helpers.Cluster:new(...)

    helper.cluster:start()
    
    local fiber = require("fiber")
    local storage11 = helper.cluster:server("storage-1-1").net_box
    local storage12 = helper.cluster:server("storage-1-2").net_box
    local start_vclock_11 = storage11:eval("return box.info.vclock")
    local start_vclock_12 = storage12:eval("return box.info.vclock")

    helper.cluster:upload_config(cfg)
    local vclock_11 = storage11:eval("return box.info.vclock")
    local vclock_12 = storage12:eval("return box.info.vclock")

    local WAITING_TIMEOUT = 60
    local fiber_sleep = 0.01
    local wait_start = fiber.clock()

    while vclock_11[1] - start_vclock_11[1] ~= vclock_12[1] - start_vclock_12[1] do
        fiber.sleep(fiber_sleep)
        vclock_11 = storage11:eval("return box.info.vclock")
        vclock_12 = storage12:eval("return box.info.vclock")

        local current_time = fiber.clock()
        if current_time > wait_start + WAITING_TIMEOUT then
            t.fail("timeout exceed waiting replication")
        end
    end
    t.ssert_equals(vclock_11[1] - start_vclock_11[1], vclock_12[1] - start_vclock_12[1])
end

that fix does not solve all posssible problem. for example, you can fail if execute insert and then select. for handling thise cases you need to have function handler.wait_replication that will be wait vclock synchronization

That fix does not solve all possible problems. For example, you may fail if you execute an insert and then select. To handle these cases, you need to have a function handler.wait_replication that will wait for vclock synchronization.

use sync replication
Alternatively, you can use sync replication. Notice that Tarantool supports sync replication for individual spaces. So, I think we will need to iterate over all spaces from the config and enable the is_sync flag. Also, there are specific instance settings. See the docs

assigned to @laa

mentioned in commit cf4bf90e

mentioned in merge request !254 (merged)

mentioned in commit b67a8100

closed with commit b67a8100

changed milestone to %2023, Q1