I have never used a ports system, but does it do any of that automatically? It can't know that I'm done using the dep. So I need to manually run clean-up correct?

I'm not opposed to a clean-up process, just trying to figure out a clean way to have minimal updates. So far I've settled on a container, much like a chroot build system. I have my runtime OS that then only does updates on required packages, nothing superfluous. My build container then keeps all the deps installed permanently, and they get updated when I run system updates. I'm not overly concerned with the build container becoming bloated.

I think I'm just over-complicating it... As I tend to do.