Draft: feat: respond to picodata heartbeats over iproto in iproto thread
Summary
This MR adds necessary plumbing to parse & respond to picodata-specific heartbeat messages fully in the iproto thread, without hitting the TX thread. A iproto_picodata_hb_handler hook is added to let picodata define its own logic for generating the heartbeat response message based on its raft state.
To allow responding to the heartbeat messages, a static Instead, writing to the socket is done directly in the message handler, producing limitations 3 & 4.pri_obuf buffer is added, which, if not empty, will be sent out before any responses buffered from the TX thread.
Still a WIP, because I am not sure whether it has everything needed to stuff picodata raft heartbeats into the message or not.
Limitations
-
This approach will not allow us to create new iproto connections while the TX thread is busy, since that still requires a roundtrip to the TX thread. This only allows sending (and getting responses to) heartbeats over existing connections.
-
Requests other than heartbeats can fill up the per-connection request queue (which is sized according to
iproto_net_msg_maxpicodata config parameter). After this happens, no responses to pings will be sent out. -
The heartbeat response is dropped if the connection is busy writing a response (waiting for the socket to become available for writing). This is done to prevent heartbeat responses being inserted into the middle of a response. This might be an issue for the picodata client, which usually expects a response to all of its messages.
-
If a write of the heartbeat response could not be completed in one go without blocking, the connection will be closed. This is unlikely to happen, unless the being actively written to. This, however, is itself unlikely due to limitation 3.
Meta
Necessary for picodata#989 Docs follow-up: not necessary