wal: introduce limits on simultaneous writes
Since the introduction of asynchronous commit, which doesn't wait for a WAL write to succeed, it's quite easy to clog WAL with huge amounts write requests. For now, it's only possible from an applier, since it's the only user of async commit at the moment. This happens when replica is syncing with master and reads new transactions at a pace higher than it can write them to WAL (see docbot request for detailed explanation). To ameliorate such behavior, we need to introduce some limit on not-yet-finished WAL write requests. This is what this commit is trying to do. A new counter is added to wal writer: queue_size (in bytes) together with a corresponding configuration setting: `wal_queue_max_size`. The counter is increased on every new submitted request, and decreased once the tx thread receives a confirmation that a specific request was written. Actually, the limit is added to an abstract journal queue, but currently works only for wal writer, since it's the only possible journal when applier is working. Once size reaches its maximum value, applier is blocked until some of the write requests are finished. The size limit isn't strict, i.e. if there's at least one free byte, the whole write request fits and no blocking is involved. The feature is ready for `box.commit{is_async=true}`. Once it's implemented, it should check whether the queue is full and let the user decide what to do next. Either wait or roll the tx back. Closes #5536 @TarantoolBot document Title: new configuration option: 'wal_queue_max_size' `wal_queue_max_size` puts a limit on the amount of concurrent write requests submitted to WAL. `wal_queue_max_size` is measured in number of bytes to be written (0 means unlimited, which was the default behaviour before). The option only affects replica behaviour at the moment, and defaults to 16 megabytes. The option limits the pace at which replica reads new transactions from master. Here's when the option comes in handy: Before this option was introduced such a situation could be possible: there are 2 servers, a master and a replica, and the replica is down for some period of time. While the replica is down, master serves requests at a reasonable pace, possibly close to its WAL throughput limit. Once the replica reconnects, it has to receive all the data master has piled up and there's no limit in speed at which master sends the data to replica, and, without the option, there was no limit in speed at which replica submitted corresponding write requests to WAL. This lead to a situation when replica's WAL was never in time to serve the requests and the amount of pending requests was constantly growing. There was no limit for memory WAL write requests take, and this clogging of WAL write queue could even lead to replica using up all the available memory. Now, when `wal_queue_max_size` is set, appliers will stop reading new transactions once the limit is reached. This will let WAL process all the requests that have piled up and free all the excess memory.
Showing
- changelogs/unreleased/wal-queue-limit.md 8 additions, 0 deletionschangelogs/unreleased/wal-queue-limit.md
- src/box/applier.cc 2 additions, 2 deletionssrc/box/applier.cc
- src/box/box.cc 28 additions, 2 deletionssrc/box/box.cc
- src/box/box.h 1 addition, 0 deletionssrc/box/box.h
- src/box/journal.c 41 additions, 0 deletionssrc/box/journal.c
- src/box/journal.h 84 additions, 1 deletionsrc/box/journal.h
- src/box/lua/cfg.cc 9 additions, 0 deletionssrc/box/lua/cfg.cc
- src/box/lua/load_cfg.lua 3 additions, 0 deletionssrc/box/lua/load_cfg.lua
- src/box/txn.c 2 additions, 2 deletionssrc/box/txn.c
- src/box/txn.h 2 additions, 1 deletionsrc/box/txn.h
- src/box/txn_limbo.h 1 addition, 1 deletionsrc/box/txn_limbo.h
- src/box/wal.c 8 additions, 0 deletionssrc/box/wal.c
- src/box/wal.h 7 additions, 0 deletionssrc/box/wal.h
- test/app-tap/init_script.result 1 addition, 0 deletionstest/app-tap/init_script.result
- test/box-tap/cfg.test.lua 2 additions, 1 deletiontest/box-tap/cfg.test.lua
- test/box/admin.result 2 additions, 0 deletionstest/box/admin.result
- test/box/cfg.result 4 additions, 0 deletionstest/box/cfg.result
- test/replication/gh-5536-wal-limit.result 133 additions, 0 deletionstest/replication/gh-5536-wal-limit.result
- test/replication/gh-5536-wal-limit.test.lua 59 additions, 0 deletionstest/replication/gh-5536-wal-limit.test.lua
- test/replication/suite.cfg 1 addition, 0 deletionstest/replication/suite.cfg
Loading
Please register or sign in to comment