A little update for this thread. My pack manager seems to be working now, in a sane, reliable and repeatable way. I am now focusing on making the distro a standalone environment.
I want to be able to bootstrap my distro from within it. The real missing part here is gcc. That bitch is a hell to get right.
I've dropped my "in-house" cross-compiler for something more professional and reliable: crosstools-ng. It provides a toolchain that is more solid than my hacky solution, but also less flexible (my toolchain could ONLY compile static binaries, and didn't make use of any directory having a /usr prefix).
So I'll need to investigate it a bit further. Still, I've been able to use it with most of my ports, so it's a nice step forward.

If anyone is interrested in giving it a try, I can help!