I personally think the current environment is very organic.

While the most common userland tools can be bloated for some usages, I don't think there's any claim about their reliability or general usefulness. There are alternatives for every scenario, from rugged systems to lightweight embedded ones. And having the system in the open give the ability to configure it to fit personal needs.

You mention extra flags and features, lack of consistency between them, and duplicate behavior, in my opinion that's more of a programmer's moral perception issue than an actual one.

From my perspective, the problem that rises in popular softwares is that they are gradually partnering with others, creating and adding too much coupling. Coupling is insidious because changing any of the lowest layer will ripple through the whole stack. The more you add coupling and the more the system looks like one rigid piece. My guess is that this is what we see in the partnership of the major DE developers, the freedesktop, etc..
Another one of my guess is that the team working on those softwares are trading technical debt for quick wins of being able to use bleeding edge features. That's the environment we live in today, fast iterations.