Expected 3000 buckets in the cluster, got 2929
Denis Smirnov, [08/07/2024 18:03] У меня под нагрузкой происходит явное забитие сетевого буфера, после которого у вшарда теряется 71 или 72 бакета (ровно по количеству бакетов на каком-то узле пикодаты). Есть ли у тебя идеи, что бы это могло быть?
bucket_loss.txt
июл 08 17:52:17 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:17.799
[1831689] main/131/to:default-6005 network.rs:272 W> error when sending
message to peer: deadline expired, raft_id: 36
июл 08 17:52:18 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:18.199 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:18 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:18.530 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:18 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:18.610 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.027 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.153 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.300 [1831689] main/409/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.463 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.913 [1831689] main/411/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:19 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:19.977 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:20 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:20.350 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:20 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:20.513 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:20 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:20.887 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:21 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:21.068 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:21 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:21.356 [1831689] main/417/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:21 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:21.528 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:21 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:21.934 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:22 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:22.145 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:22 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:22.404 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:22 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:22.826 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:23 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:23.498 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:23 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:23.498 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:23 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:23.904 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:24 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:24.444 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:24 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:24.871 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:24 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:24.872 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:24 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:24.947 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:25 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:25.062 [1831689] main/431/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:25 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:25.634 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:25 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:25.656 [1831689] main/433/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(1)
июл 08 17:52:26 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:26.128 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:26 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:26.567 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:26 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:26.967 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:27 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:27.133 [1831689] main/439/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:27 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:27.722 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:27 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:27.789 [1831689] main/441/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:28 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:28.158 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:28 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:28.576 [1831689] main/443/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:28 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:28.593 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:28 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:28.996 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:29 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:29.115 [1831689] main/447/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:29 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:29.464 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:29 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:29.602 [1831689] main/449/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:29 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:29.960 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:30 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:30.213 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:30 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:30.642 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:31 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:31.138 [1831689] main/453/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:31 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:31.334 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:31 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:31.718 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:31 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:31.723 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:31 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:31.888 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:32 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:32.139 [1831689] main/457/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:32 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:32.441 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:32 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:32.608 [1831689] main/459/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:32 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:32.972 [1831689] main/131/to:default-6005 network.rs:272 W> error when sending message to peer: deadline expired, raft_id: 36
июл 08 17:52:33 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:33.245 [1831689] main/461/iproto-in/big3-vtb-4.picodata.int:13306 mod.rs:454 W> received unwaited message for SyncIndex(0)
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.828 [1831689] main/467/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.829 [1831689] main/463/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.829 [1831689] main/366/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.830 [1831689] main/400/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.830 [1831689] main/404/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:43 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:43.830 [1831689] main/399/pico.sql vshard.rs:134 E> LuaError(ExecutionError("vshard/replicaset.lua:1261: attempt to index field 'conn' (a nil value)")): dql_with_single_plan
июл 08 17:52:44 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:44.350 [1831689] main/468/pico.sql vshard.rs:134 E> LuaError(ExecutionError("sbroad/sbroad-core/src/helper.lua:84: Expected 3000 buckets in the cluster, got 2929")): dql_with_single_plan
июл 08 17:52:44 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:44.351 [1831689] main/403/pico.sql vshard.rs:134 E> LuaError(ExecutionError("sbroad/sbroad-core/src/helper.lua:84: Expected 3000 buckets in the cluster, got 2929")): dql_with_single_plan
июл 08 17:52:44 big1-vtb-1 picodata[1831689]: 2024-07-08 17:52:44.351 [1831689] main/399/pico.sql vshard.rs:134 E> LuaError(ExecutionError("sbroad/sbroad-core/src/helper.lua:84: Expected 3000 buckets in the cluster, got 2929")): dql_with_single_plan