I don't see why/how using pipes or sockets to ask a separate process to draw these window decorations would be any better than calling the API of a framework library(*), which is what most GUI applications currently do.

I.e., the API is already the interface enforcing the kind of separation of concerns you're looking for. For the process at the other end of the pipe/socket to work, you're going to have to send it pretty much the same data you're passing to the API, only now you'd be passing it via a more cumbersome mechanism.

(* Doesn't have to be a callback-driven framework, of course. A widget library or an immediate mode UI library like imgui or imtui is an interesting solution to some situations.)