vinyl: fix use after free on key alloc error in iterator constructor

Error labes got mixed up. Fix the order and add a test.

Fixes commit 3f026339 ("vinyl: implement iterator pagination").

Closes #8372

NO_DOC=bug fix
NO_CHANGELOG=fix for unreleased feature

(cherry picked from commit 074e4eb8)