You worst moment on a Unix system - Psychology, Philosophy, and Licenses

Users browsing this thread: 8 Guest(s)
movq
Long time nixers
It’s always data loss, isn’t it? Probably because it’s so painful. I only lost a couple of screenshots once, because I ran `rm *` in `$HOME`. No `-r`, though, so it only picked up those few stray JPGs.

One of the worst moments in the recent past (a few years ago, I think) was probably running `iptables -nvL | less` on all of our client-facing OpenVPN servers at the same time (something like clusterssh).

That command looks totally harmless. What I didn’t know was that `iptables` grabs a file lock (`/run/xtables.lock`), queries rules from the kernel, prints them, and then releases that lock. In that order. If you have a lot of rules, then writing this output to the pipe will eventually block because the pipe buffer is full. Meaning, the lock is still being held, basically until `less` has read until EOF from the pipe.

Now, when a new clients connects, we had OpenVPN create a new set of iptables rules for this client. And this is where the disaster happened: OpenVPN accepts the connection, then runs `iptables -w -A …`, and then waits for this to finish.

In other words, me running `iptables -nvL | less` grabbed the file lock and the next client that connected tried to grab that lock as well, but it blocked due to `-w` and thus it blocked the entire OpenVPN process. Boom, no more traffic for a couple of hundred VPN clients. Including myself.

(We have since introduced a wrapper script `iptables-show` which basically does `iptables -nvL >foo && less foo`.)

(13-11-2020, 03:24 PM)zge Wrote: lost my bookmark collection I had started when I was 4 years old.
Was that a figure of speech or did you really start this young?


Messages In This Thread
You worst moment on a Unix system - by venam - 23-05-2016, 01:40 AM
RE: You worst moment on a Unix system - by jkl - 23-05-2016, 11:00 AM
RE: You worst moment on a Unix system - by cjm - 23-05-2016, 11:26 AM
RE: You worst moment on a Unix system - by z3bra - 23-05-2016, 12:19 PM
RE: You worst moment on a Unix system - by rocx - 23-05-2016, 12:32 PM
RE: You worst moment on a Unix system - by xero - 23-05-2016, 03:46 PM
RE: You worst moment on a Unix system - by Dworin - 24-05-2016, 01:44 AM
RE: You worst moment on a Unix system - by z3bra - 24-05-2016, 03:37 AM
RE: You worst moment on a Unix system - by pkal - 13-11-2020, 03:24 PM
RE: You worst moment on a Unix system - by movq - 13-11-2020, 05:31 PM
RE: You worst moment on a Unix system - by pkal - 13-11-2020, 05:43 PM
RE: You worst moment on a Unix system - by s0kx - 24-11-2020, 12:21 PM
RE: You worst moment on a Unix system - by venam - 24-11-2020, 12:36 PM
RE: You worst moment on a Unix system - by freem - 24-11-2020, 01:43 PM
RE: You worst moment on a Unix system - by venam - 24-11-2020, 01:57 PM
RE: You worst moment on a Unix system - by freem - 24-11-2020, 05:07 PM
RE: You worst moment on a Unix system - by jkl - 24-11-2020, 05:13 PM
RE: You worst moment on a Unix system - by Dworin - 24-11-2020, 10:35 PM
RE: You worst moment on a Unix system - by pfr - 25-11-2020, 02:51 AM