There's a patch for dwm that require exactly this. It tries to mimic plan9's swallowing feature. When you an application opens a window, the calling terminal is "replaced" by the new app. For that, it must search for a calling terminal whenever a new window is popping.

Just like vain said, it traverses the process tree (using /proc/$PID/stat, where the 4th value is the parent PID), and for each PID that maps internally to a window (you must keep track of the PID per window), check whether the application class match a predefined one ("St", by default). They also improved it to work on OpenBSD (using libkvm).

See the full patch here: https://dwm.suckless.org/patches/swallow...de9b0.diff

This means that you must rely on X11 and set manually what is and is not a terminal.

At some point, I think it's easier to do that try to "guess" what is a terminal, because many processes can open a pseudo TTY for various reasons (ssh-agent for example). Even you shell ! So that means you'd stop at the shell rather than at the actual terminal.

Edit:

After thinking about it more, I suppose you can somehow guess if a PID is a terminal emulator or not. You'd have to make a few assumptions for what defines a terminal emulator:

• it has the $DISPLAY environment variable set
  
• it is controlling a pseudo TTY (has /dev/ptmx open)
  
• it can be associated with a graphical window
  

If all 3 conditions are met, it could be safe to assume you find a terminal emulator.