memtx: move memtx_tx logic to memtx_tx.c

Hide structures and functions that are not required for API.

No logical changes.

Part of #8122

NO_DOC=refactoring
NO_TEST=refactoring
NO_CHANGELOG=refactoring

(cherry picked from commit 4cc04ca9)