Users browsing this thread: 4 Guest(s)
z3bra
Grey Hair Nixers
Dear Unix diary,

Today I discovered a neat trick involving /proc
Some user complained AGAIN that their development server was full. Indeed, they have a component whose logs are set in "DEBUG" mode, which makes it fill the available space rather quickly (and they don't want to do anything about it, because they're lazy asses!).
So as usual, I log onto the machine, perform the traditionnal "df -h" semi-automatically, and then navigate to the log directory that I know is guilty. But this time, I was surprised:

Code:
# cd /opt/app-data/guilty/var/log
# du -sh *opt/app-data/guilty/var/log
5.1M    APP_20150720.log
72K     APP_20150721.log
72K     APP_20150722.log
72K     APP_20150723.log
237M    APP_20150724.log
0       APP_20150725.log
0       APP_20150726.log
17K     APP_20150727.log

Ah, so there might be another guilty app! which one is it?

Code:
# du -sh /opt/app-data/*
500M    app01
123M    app02
17M     app03
400M    guilty

# du -sh /
7.6G    /

WTF? Then I got enlightenned... Someone deleted a huge ass file, while it was still openned!
Dammit, I already told them not to play heroes on the box, and call me instead...
After some research and the help of our good IRC members (props to SOMEONE[1] for finding the link).

This stackoverflow post advise to use "lsof" to find the openned. I already though about it, but lsof was not installed on the box, and the disk being full didn't allow me to install it. So I had to go with another option.
Hopefully, another answer on the page suggested an alternative, which I tried immediately:

Code:
# find /proc/*/fd -ls | grep  '(deleted)'
35225653    0 lrwx------   1 appuser users          64 Jul 27 18:56 /proc/3568/fd/1 -> /dev/pts/0\ (deleted)
35225657    0 lrwx------   1 appuser users          64 Jul 27 18:56 /proc/3632/fd/2 -> /dev/pts/0\ (deleted)

I found them! I told them that restarting their component would free the space, but being too busy "coding", they couldn't afford a reboot. So I just temporarily emptied the file descriptors by hand ("> /proc/3568/fd/1" , and let them go with it.
/proc/3568/fd/1
Code:
# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       73G   6.7G 66.3G 9%  /

Hooray! Unix will always impress me!

[1] ymbx doesn't deserves the honor of being cited. Die alone man.


Messages In This Thread
UNIX Diary - by z3bra - 18-08-2014, 11:03 AM
RE: UNIX Diary - by venam - 18-08-2014, 11:37 AM
RE: UNIX Diary - by cjm - 18-08-2014, 12:01 PM
RE: UNIX Diary - by venam - 18-08-2014, 12:18 PM
RE: UNIX Diary - by cjm - 18-08-2014, 02:11 PM
RE: UNIX Diary - by venam - 19-08-2014, 12:50 PM
RE: UNIX Diary - by z3bra - 19-08-2014, 01:18 PM
RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 02:15 PM
RE: UNIX Diary - by vompatti - 19-08-2014, 04:20 PM
RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 07:53 PM
RE: UNIX Diary - by venam - 20-08-2014, 12:54 PM
RE: UNIX Diary - by vompatti - 20-08-2014, 06:33 PM
RE: UNIX Diary - by z3bra - 21-08-2014, 04:53 AM
RE: UNIX Diary - by DotDev - 21-08-2014, 07:50 AM
RE: UNIX Diary - by shtols - 21-08-2014, 12:55 PM
RE: UNIX Diary - by venam - 21-08-2014, 01:22 PM
RE: UNIX Diary - by venam - 23-08-2014, 06:55 AM
RE: UNIX Diary - by z3bra - 23-08-2014, 07:27 AM
RE: UNIX Diary - by venam - 24-08-2014, 04:39 AM
RE: UNIX Diary - by xero - 24-08-2014, 11:24 PM
RE: UNIX Diary - by pizzaroll1 - 25-08-2014, 12:37 PM
RE: UNIX Diary - by z3bra - 26-08-2014, 03:38 AM
RE: UNIX Diary - by cjm - 27-08-2014, 12:48 AM
RE: UNIX Diary - by xero - 28-08-2014, 11:11 AM
RE: UNIX Diary - by venam - 30-08-2014, 01:47 AM
RE: UNIX Diary - by kirby - 30-08-2014, 09:56 AM
RE: UNIX Diary - by venam - 30-08-2014, 11:22 AM
RE: UNIX Diary - by venam - 09-09-2014, 12:21 PM
RE: UNIX Diary - by venam - 04-11-2014, 12:27 AM
RE: UNIX Diary - by berk - 10-09-2014, 09:12 AM
RE: UNIX Diary - by xero - 10-09-2014, 10:53 AM
RE: UNIX Diary - by venam - 10-09-2014, 11:51 AM
RE: UNIX Diary - by xero - 11-09-2014, 11:18 AM
RE: UNIX Diary - by z3bra - 12-09-2014, 08:08 AM
RE: UNIX Diary - by xero - 17-09-2014, 10:51 AM
RE: UNIX Diary - by pizzaroll1 - 17-09-2014, 01:07 PM
RE: UNIX Diary - by hades - 17-09-2014, 07:47 PM
RE: UNIX Diary - by venam - 18-09-2014, 12:37 AM
RE: UNIX Diary - by xero - 18-09-2014, 10:51 AM
RE: UNIX Diary - by venam - 18-09-2014, 12:18 PM
RE: UNIX Diary - by earsplit - 19-09-2014, 02:23 PM
RE: UNIX Diary - by vompatti - 20-09-2014, 11:52 AM
RE: UNIX Diary - by pizzaroll1 - 20-09-2014, 08:00 PM
RE: UNIX Diary - by srp - 21-09-2014, 02:09 AM
RE: UNIX Diary - by kirby - 21-09-2014, 03:25 PM
RE: UNIX Diary - by venam - 30-09-2014, 01:00 PM
RE: UNIX Diary - by xero - 08-10-2014, 10:31 AM
RE: UNIX Diary - by cjm - 09-10-2014, 04:14 PM
RE: UNIX Diary - by kirby - 09-10-2014, 08:15 PM
RE: UNIX Diary - by JerrySpringerIsMyDad - 24-10-2014, 10:03 PM
RE: UNIX Diary - by xero - 28-10-2014, 10:38 AM
RE: UNIX Diary - by venam - 28-10-2014, 01:18 PM
RE: UNIX Diary - by neeasade - 09-01-2015, 10:44 AM
RE: UNIX Diary - by z3bra - 09-01-2015, 02:04 PM
RE: UNIX Diary - by z3bra - 19-02-2015, 07:05 AM
RE: UNIX Diary - by venam - 19-02-2015, 04:55 PM
RE: UNIX Diary - by xero - 20-02-2015, 11:49 AM
RE: UNIX Diary - by z3bra - 28-04-2015, 04:13 PM
RE: UNIX Diary - by bsdkeith - 29-04-2015, 05:01 AM
RE: UNIX Diary - by z3bra - 29-04-2015, 10:29 AM
RE: UNIX Diary - by xero - 29-04-2015, 03:09 PM
RE: UNIX Diary - by greduan - 02-05-2015, 09:58 AM
RE: UNIX Diary - by cjm - 07-05-2015, 11:14 AM
RE: UNIX Diary - by z3bra - 07-05-2015, 12:20 PM
RE: UNIX Diary - by neeasade - 24-07-2015, 11:10 PM
RE: UNIX Diary - by z3bra - 25-07-2015, 08:44 AM
RE: UNIX Diary - by Houseoftea - 25-07-2015, 11:32 PM
RE: UNIX Diary - by movq - 26-07-2015, 03:52 AM
RE: UNIX Diary - by z3bra - 27-07-2015, 02:02 PM
RE: UNIX Diary - by z3bra - 24-08-2015, 07:07 PM
RE: UNIX Diary - by xero - 25-08-2015, 10:38 AM
RE: UNIX Diary - by rocx - 30-08-2015, 11:40 PM
RE: UNIX Diary - by z3bra - 31-08-2015, 04:40 AM
RE: UNIX Diary - by venam - 31-08-2015, 07:30 AM
RE: UNIX Diary - by pranomostro - 03-09-2015, 05:34 PM
RE: UNIX Diary - by kirby - 03-09-2015, 06:57 PM
RE: UNIX Diary - by xero - 10-09-2015, 11:30 AM
RE: UNIX Diary - by venam - 14-09-2015, 03:45 AM
RE: UNIX Diary - by darthlukan - 14-09-2015, 02:53 PM
RE: UNIX Diary - by dkeg - 15-09-2015, 07:49 AM
RE: UNIX Diary - by darthlukan - 05-10-2015, 03:03 PM
RE: UNIX Diary - by pranomostro - 05-10-2015, 05:00 PM
RE: UNIX Diary - by apk - 06-10-2015, 02:41 PM
RE: UNIX Diary - by Mafia - 06-10-2015, 09:33 PM
RE: UNIX Diary - by kirby - 12-10-2015, 03:04 PM
RE: UNIX Diary - by pranomostro - 13-10-2015, 08:37 AM
RE: UNIX Diary - by venam - 13-10-2015, 09:38 AM
RE: UNIX Diary - by xero - 13-10-2015, 10:55 AM
RE: UNIX Diary - by pranomostro - 13-10-2015, 11:18 AM
RE: UNIX Diary - by venam - 15-10-2015, 01:34 PM
RE: UNIX Diary - by darthlukan - 15-10-2015, 04:44 PM
RE: UNIX Diary - by pranomostro - 15-10-2015, 06:05 PM
RE: UNIX Diary - by xero - 15-10-2015, 06:58 PM
RE: UNIX Diary - by darthlukan - 16-10-2015, 02:29 AM
RE: UNIX Diary - by pranomostro - 16-10-2015, 10:09 AM
RE: UNIX Diary - by darthlukan - 19-10-2015, 02:10 PM
RE: UNIX Diary - by venam - 20-10-2015, 02:05 AM
RE: UNIX Diary - by darthlukan - 20-10-2015, 02:28 AM
RE: UNIX Diary - by thetornainbow - 20-10-2015, 08:52 AM
RE: UNIX Diary - by Houseoftea - 20-10-2015, 09:11 AM
RE: UNIX Diary - by z3bra - 20-10-2015, 12:18 PM
RE: UNIX Diary - by darthlukan - 21-10-2015, 02:31 AM
RE: UNIX Diary - by z3bra - 30-11-2015, 07:08 PM
RE: UNIX Diary - by apk - 30-11-2015, 08:49 PM
RE: UNIX Diary - by z3bra - 01-12-2015, 08:24 AM
RE: UNIX Diary - by apk - 01-12-2015, 11:43 AM
RE: UNIX Diary - by arcetera - 02-12-2015, 08:57 PM
RE: UNIX Diary - by venam - 03-12-2015, 02:20 AM
RE: UNIX Diary - by pranomostro - 03-12-2015, 11:32 AM
RE: UNIX Diary - by dcat - 07-12-2015, 06:16 PM
RE: UNIX Diary - by venam - 11-12-2015, 07:30 AM
RE: UNIX Diary - by xero - 11-12-2015, 03:04 PM
RE: UNIX Diary - by pranomostro - 13-12-2015, 08:31 AM
RE: UNIX Diary - by venam - 13-12-2015, 03:22 PM
RE: UNIX Diary - by strang3quark - 13-12-2015, 03:57 PM
RE: UNIX Diary - by rocx - 13-12-2015, 04:37 PM
RE: UNIX Diary - by Wildefyr - 13-12-2015, 07:29 PM
RE: UNIX Diary - by venam - 14-12-2015, 01:37 AM
RE: UNIX Diary - by movq - 14-12-2015, 12:44 PM
RE: UNIX Diary - by venam - 14-12-2015, 12:56 PM
RE: UNIX Diary - by pranomostro - 14-12-2015, 01:24 PM
RE: UNIX Diary - by movq - 19-12-2015, 03:26 PM
RE: UNIX Diary - by venam - 20-12-2015, 05:02 AM
RE: UNIX Diary - by movq - 20-12-2015, 08:13 AM
RE: UNIX Diary - by pranomostro - 20-12-2015, 10:21 AM
RE: UNIX Diary - by z3bra - 20-12-2015, 03:46 PM
RE: UNIX Diary - by movq - 22-12-2015, 03:27 PM
RE: UNIX Diary - by XcelQ - 24-01-2016, 01:32 PM
RE: UNIX Diary - by pranomostro - 26-01-2016, 02:17 PM
RE: UNIX Diary - by XcelQ - 28-01-2016, 03:17 AM
RE: UNIX Diary - by rain1 - 19-04-2016, 08:28 AM
RE: UNIX Diary - by josuah - 22-04-2016, 05:45 PM
RE: UNIX Diary - by movq - 07-05-2016, 01:34 PM
RE: UNIX Diary - by rocx - 16-05-2016, 02:21 AM
RE: UNIX Diary - by venam - 10-06-2016, 01:11 AM
RE: UNIX Diary - by venam - 14-09-2016, 01:12 AM
RE: UNIX Diary - by xero - 14-09-2016, 11:36 AM
RE: UNIX Diary - by jkl - 14-09-2016, 11:58 AM
RE: UNIX Diary - by venam - 31-01-2017, 09:27 AM
RE: UNIX Diary - by rocx - 18-02-2017, 02:17 AM
RE: UNIX Diary - by fre d die - 15-03-2017, 06:13 PM
RE: UNIX Diary - by rocx - 15-03-2017, 07:07 PM
RE: UNIX Diary - by robotchaos - 07-04-2017, 02:13 AM
RE: UNIX Diary - by evbo - 08-04-2017, 03:00 AM
RE: UNIX Diary - by jkl - 08-04-2017, 08:30 AM
RE: UNIX Diary - by venam - 09-04-2017, 04:56 AM
RE: UNIX Diary - by robotchaos - 10-04-2017, 03:56 PM
RE: UNIX Diary - by jkl - 10-04-2017, 04:03 PM
RE: UNIX Diary - by robotchaos - 10-04-2017, 04:22 PM
RE: UNIX Diary - by jkl - 10-04-2017, 04:25 PM
RE: UNIX Diary - by josuah - 11-04-2017, 09:19 AM
RE: UNIX Diary - by jkl - 11-04-2017, 09:39 AM
RE: UNIX Diary - by evbo - 18-04-2017, 02:20 PM
RE: UNIX Diary - by robotchaos - 18-04-2017, 03:43 PM
RE: UNIX Diary - by nas - 12-06-2017, 09:33 AM
RE: UNIX Diary - by Tmplt - 12-07-2017, 02:00 AM
RE: UNIX Diary - by xero - 13-07-2017, 03:12 PM
RE: UNIX Diary - by venam - 20-10-2017, 03:12 AM
RE: UNIX Diary - by Houseoftea - 04-01-2018, 08:40 PM
RE: UNIX Diary - by rocx - 05-01-2018, 02:04 AM
RE: UNIX Diary - by Tmplt - 21-01-2018, 10:02 PM
RE: UNIX Diary - by jkl - 22-01-2018, 08:14 AM
RE: UNIX Diary - by Tmplt - 22-01-2018, 01:02 PM
RE: UNIX Diary - by mrtn - 22-01-2018, 04:47 PM
RE: UNIX Diary - by jkl - 22-01-2018, 04:53 PM
RE: UNIX Diary - by jkl - 31-03-2018, 09:22 PM
RE: UNIX Diary - by jkl - 15-05-2018, 12:17 PM
RE: UNIX Diary - by z3bra - 20-09-2018, 07:07 PM
RE: UNIX Diary - by xero - 25-10-2018, 01:27 PM
RE: UNIX Diary - by venam - 26-10-2018, 01:08 AM
RE: UNIX Diary - by Steph - 28-10-2018, 05:23 PM
RE: UNIX Diary - by rocx - 04-11-2018, 01:48 PM
RE: UNIX Diary - by jkl - 05-11-2018, 10:18 AM
RE: UNIX Diary - by pranomostro - 06-11-2018, 09:06 AM
RE: UNIX Diary - by anthk - 10-01-2019, 12:20 PM
RE: UNIX Diary - by pranomostro - 10-01-2019, 04:38 PM
RE: UNIX Diary - by gaak - 14-01-2019, 09:47 PM
RE: UNIX Diary - by josuah - 10-12-2019, 08:39 PM
RE: UNIX Diary - by z3bra - 15-05-2020, 12:19 PM
RE: UNIX Diary - by wolf - 18-06-2020, 04:52 PM
RE: UNIX Diary - by fre d die - 08-07-2020, 04:47 PM
RE: UNIX Diary - by seninha - 08-07-2020, 09:34 PM
RE: UNIX Diary - by movq - 28-09-2020, 03:19 PM
RE: UNIX Diary - by z3bra - 29-09-2020, 11:00 AM
RE: UNIX Diary - by jkl - 30-09-2020, 03:52 AM
RE: UNIX Diary - by movq - 28-05-2021, 03:18 PM
RE: UNIX Diary - by movq - 27-09-2021, 11:56 AM
RE: UNIX Diary - by venam - 17-12-2021, 03:49 PM
RE: UNIX Diary - by z3bra - 20-12-2021, 08:34 AM