ci: fix RPM package builds on aarch64 runners

We're using LXD containers as aarch64 runners. For some reason, OOM
killer just kills the compilation process while package building when
`make -j $(nproc)`. The issue happens only with builds where LTO is
enabled. It's found, that `-j6` works fine. The bigger value causes
problems.

NO_DOC=ci
NO_TEST=ci
NO_CHANGELOG=ci