vinyl: use separate thread pools for dump and compaction tasks
Using the same thread pool for both dump and compaction tasks makes estimation of dump bandwidth unstable. For instance, if we have four worker threads, then the observed dump bandwidth may vary from X if there's high compaction demand and all worker threads tend to be busy with compaction tasks to 4 * X if there's no compaction demand. As a result, we can overestimate the dump bandwidth and trigger dump when it's too late, which will result in hitting the limit before dump is complete and hence stalling write transactions, which is unacceptable. To avoid that, let's separate thread pools used for dump and compaction tasks. Since LSM tree based design typically implies high levels of write amplification, let's allocate 1/4th of all threads for dump tasks and use the rest exclusively for compaction.
Loading
Please register or sign in to comment