UNIX Diary - Psychology, Philosophy, and Licenses
Users browsing this thread: 2 Guest(s)
|
|||
Dear Unix diary,
today I've been a bad sysadmin. It just happened. I host my own git repository, and earlier this evening I was working on my crux port tree, when I decided to commit and push my work. But this time, something went wrong and git didn't let me push any reference. Amongst all the messages returned by git, I saw this one: Code: remote: fatal: write error: No space left on device Fucking shit. I instantly imagine what's happening: my /var partition wasn't correctly sized upon creation. This is where I host my website, gopherhole, git repo, pictures, videos, ... Every 'production' service. And after serving me well for several years, it's now full. Hopefully, I had setup all my partitions on top of LVM, and let like 200GiB available, just in case things go wrong. And they did. So here am I, staring at my red prompt, typing a few commands: Code: root ~# df -h Ok, so it's not the first time this happens, remember? You already grew your /home partition, and it went good! Just do the same with /var! It works without a reboot! What was those commands again? Code: root ~# lvextend -L +20G vg0/var Phew... I'm safe now! So what the hell was going on? I decided to investigate a bit further, to see what I should watch next time. That's how I realised that I did a HUGE mistake... Code: root ~# cd /var/ My backup system doesn't check wether it saves to a mountpoint or not. Shit. For a whole week, all my backups where created in my /var partition instead of a backup USB drive meant for this purpose. And it filled it up pretty quickly. My backup system send me a mail after each backup, explaining me how it went. The fact it's saving to a mountpoint or not is written in it. I just stopped checking. Silly me. I realise that this issue could have been easily solved by mounting my backup disk elsewhere, then moving the files, and remounting where it should be. But I didn't. Instead, I grew a partition that didn't need to be (the backups filled 48GiB out of 50Gib allocated to /var), and this partition can't be shrinked anymore, as it's an XFS filesystem. So today I learnt two things, the hard way: 1. Don't do anything until you know what's going on 2. Configure systems checks and READ THEM I hope you'll learn from my mistakes. For now I think I'll just print this over my desktop, as a reminder: Code: root ~# df -h /var/ |
|||
Messages In This Thread |
UNIX Diary - by z3bra - 18-08-2014, 11:03 AM
RE: UNIX Diary - by venam - 18-08-2014, 11:37 AM
RE: UNIX Diary - by cjm - 18-08-2014, 12:01 PM
RE: UNIX Diary - by venam - 18-08-2014, 12:18 PM
RE: UNIX Diary - by cjm - 18-08-2014, 02:11 PM
RE: UNIX Diary - by venam - 19-08-2014, 12:50 PM
RE: UNIX Diary - by z3bra - 19-08-2014, 01:18 PM
RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 02:15 PM
RE: UNIX Diary - by vompatti - 19-08-2014, 04:20 PM
RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 07:53 PM
RE: UNIX Diary - by venam - 20-08-2014, 12:54 PM
RE: UNIX Diary - by vompatti - 20-08-2014, 06:33 PM
RE: UNIX Diary - by z3bra - 21-08-2014, 04:53 AM
RE: UNIX Diary - by DotDev - 21-08-2014, 07:50 AM
RE: UNIX Diary - by shtols - 21-08-2014, 12:55 PM
RE: UNIX Diary - by venam - 21-08-2014, 01:22 PM
RE: UNIX Diary - by venam - 23-08-2014, 06:55 AM
RE: UNIX Diary - by z3bra - 23-08-2014, 07:27 AM
RE: UNIX Diary - by venam - 24-08-2014, 04:39 AM
RE: UNIX Diary - by xero - 24-08-2014, 11:24 PM
RE: UNIX Diary - by pizzaroll1 - 25-08-2014, 12:37 PM
RE: UNIX Diary - by z3bra - 26-08-2014, 03:38 AM
RE: UNIX Diary - by cjm - 27-08-2014, 12:48 AM
RE: UNIX Diary - by xero - 28-08-2014, 11:11 AM
RE: UNIX Diary - by venam - 30-08-2014, 01:47 AM
RE: UNIX Diary - by kirby - 30-08-2014, 09:56 AM
RE: UNIX Diary - by venam - 30-08-2014, 11:22 AM
RE: UNIX Diary - by venam - 09-09-2014, 12:21 PM
RE: UNIX Diary - by venam - 04-11-2014, 12:27 AM
RE: UNIX Diary - by berk - 10-09-2014, 09:12 AM
RE: UNIX Diary - by xero - 10-09-2014, 10:53 AM
RE: UNIX Diary - by venam - 10-09-2014, 11:51 AM
RE: UNIX Diary - by xero - 11-09-2014, 11:18 AM
RE: UNIX Diary - by z3bra - 12-09-2014, 08:08 AM
RE: UNIX Diary - by xero - 17-09-2014, 10:51 AM
RE: UNIX Diary - by pizzaroll1 - 17-09-2014, 01:07 PM
RE: UNIX Diary - by hades - 17-09-2014, 07:47 PM
RE: UNIX Diary - by venam - 18-09-2014, 12:37 AM
RE: UNIX Diary - by xero - 18-09-2014, 10:51 AM
RE: UNIX Diary - by venam - 18-09-2014, 12:18 PM
RE: UNIX Diary - by earsplit - 19-09-2014, 02:23 PM
RE: UNIX Diary - by vompatti - 20-09-2014, 11:52 AM
RE: UNIX Diary - by pizzaroll1 - 20-09-2014, 08:00 PM
RE: UNIX Diary - by srp - 21-09-2014, 02:09 AM
RE: UNIX Diary - by kirby - 21-09-2014, 03:25 PM
RE: UNIX Diary - by venam - 30-09-2014, 01:00 PM
RE: UNIX Diary - by xero - 08-10-2014, 10:31 AM
RE: UNIX Diary - by cjm - 09-10-2014, 04:14 PM
RE: UNIX Diary - by kirby - 09-10-2014, 08:15 PM
RE: UNIX Diary - by JerrySpringerIsMyDad - 24-10-2014, 10:03 PM
RE: UNIX Diary - by xero - 28-10-2014, 10:38 AM
RE: UNIX Diary - by venam - 28-10-2014, 01:18 PM
RE: UNIX Diary - by neeasade - 09-01-2015, 10:44 AM
RE: UNIX Diary - by z3bra - 09-01-2015, 02:04 PM
RE: UNIX Diary - by z3bra - 19-02-2015, 07:05 AM
RE: UNIX Diary - by venam - 19-02-2015, 04:55 PM
RE: UNIX Diary - by xero - 20-02-2015, 11:49 AM
RE: UNIX Diary - by z3bra - 28-04-2015, 04:13 PM
RE: UNIX Diary - by bsdkeith - 29-04-2015, 05:01 AM
RE: UNIX Diary - by z3bra - 29-04-2015, 10:29 AM
RE: UNIX Diary - by xero - 29-04-2015, 03:09 PM
RE: UNIX Diary - by greduan - 02-05-2015, 09:58 AM
RE: UNIX Diary - by cjm - 07-05-2015, 11:14 AM
RE: UNIX Diary - by z3bra - 07-05-2015, 12:20 PM
RE: UNIX Diary - by neeasade - 24-07-2015, 11:10 PM
RE: UNIX Diary - by z3bra - 25-07-2015, 08:44 AM
RE: UNIX Diary - by Houseoftea - 25-07-2015, 11:32 PM
RE: UNIX Diary - by movq - 26-07-2015, 03:52 AM
RE: UNIX Diary - by z3bra - 27-07-2015, 02:02 PM
RE: UNIX Diary - by z3bra - 24-08-2015, 07:07 PM
RE: UNIX Diary - by xero - 25-08-2015, 10:38 AM
RE: UNIX Diary - by rocx - 30-08-2015, 11:40 PM
RE: UNIX Diary - by z3bra - 31-08-2015, 04:40 AM
RE: UNIX Diary - by venam - 31-08-2015, 07:30 AM
RE: UNIX Diary - by pranomostro - 03-09-2015, 05:34 PM
RE: UNIX Diary - by kirby - 03-09-2015, 06:57 PM
RE: UNIX Diary - by xero - 10-09-2015, 11:30 AM
RE: UNIX Diary - by venam - 14-09-2015, 03:45 AM
RE: UNIX Diary - by darthlukan - 14-09-2015, 02:53 PM
RE: UNIX Diary - by dkeg - 15-09-2015, 07:49 AM
RE: UNIX Diary - by darthlukan - 05-10-2015, 03:03 PM
RE: UNIX Diary - by pranomostro - 05-10-2015, 05:00 PM
RE: UNIX Diary - by apk - 06-10-2015, 02:41 PM
RE: UNIX Diary - by Mafia - 06-10-2015, 09:33 PM
RE: UNIX Diary - by kirby - 12-10-2015, 03:04 PM
RE: UNIX Diary - by pranomostro - 13-10-2015, 08:37 AM
RE: UNIX Diary - by venam - 13-10-2015, 09:38 AM
RE: UNIX Diary - by xero - 13-10-2015, 10:55 AM
RE: UNIX Diary - by pranomostro - 13-10-2015, 11:18 AM
RE: UNIX Diary - by venam - 15-10-2015, 01:34 PM
RE: UNIX Diary - by darthlukan - 15-10-2015, 04:44 PM
RE: UNIX Diary - by pranomostro - 15-10-2015, 06:05 PM
RE: UNIX Diary - by xero - 15-10-2015, 06:58 PM
RE: UNIX Diary - by darthlukan - 16-10-2015, 02:29 AM
RE: UNIX Diary - by pranomostro - 16-10-2015, 10:09 AM
RE: UNIX Diary - by darthlukan - 19-10-2015, 02:10 PM
RE: UNIX Diary - by venam - 20-10-2015, 02:05 AM
RE: UNIX Diary - by darthlukan - 20-10-2015, 02:28 AM
RE: UNIX Diary - by thetornainbow - 20-10-2015, 08:52 AM
RE: UNIX Diary - by Houseoftea - 20-10-2015, 09:11 AM
RE: UNIX Diary - by z3bra - 20-10-2015, 12:18 PM
RE: UNIX Diary - by darthlukan - 21-10-2015, 02:31 AM
RE: UNIX Diary - by z3bra - 30-11-2015, 07:08 PM
RE: UNIX Diary - by apk - 30-11-2015, 08:49 PM
RE: UNIX Diary - by z3bra - 01-12-2015, 08:24 AM
RE: UNIX Diary - by apk - 01-12-2015, 11:43 AM
RE: UNIX Diary - by arcetera - 02-12-2015, 08:57 PM
RE: UNIX Diary - by venam - 03-12-2015, 02:20 AM
RE: UNIX Diary - by pranomostro - 03-12-2015, 11:32 AM
RE: UNIX Diary - by dcat - 07-12-2015, 06:16 PM
RE: UNIX Diary - by venam - 11-12-2015, 07:30 AM
RE: UNIX Diary - by xero - 11-12-2015, 03:04 PM
RE: UNIX Diary - by pranomostro - 13-12-2015, 08:31 AM
RE: UNIX Diary - by venam - 13-12-2015, 03:22 PM
RE: UNIX Diary - by strang3quark - 13-12-2015, 03:57 PM
RE: UNIX Diary - by rocx - 13-12-2015, 04:37 PM
RE: UNIX Diary - by Wildefyr - 13-12-2015, 07:29 PM
RE: UNIX Diary - by venam - 14-12-2015, 01:37 AM
RE: UNIX Diary - by movq - 14-12-2015, 12:44 PM
RE: UNIX Diary - by venam - 14-12-2015, 12:56 PM
RE: UNIX Diary - by pranomostro - 14-12-2015, 01:24 PM
RE: UNIX Diary - by movq - 19-12-2015, 03:26 PM
RE: UNIX Diary - by venam - 20-12-2015, 05:02 AM
RE: UNIX Diary - by movq - 20-12-2015, 08:13 AM
RE: UNIX Diary - by pranomostro - 20-12-2015, 10:21 AM
RE: UNIX Diary - by z3bra - 20-12-2015, 03:46 PM
RE: UNIX Diary - by movq - 22-12-2015, 03:27 PM
RE: UNIX Diary - by XcelQ - 24-01-2016, 01:32 PM
RE: UNIX Diary - by pranomostro - 26-01-2016, 02:17 PM
RE: UNIX Diary - by XcelQ - 28-01-2016, 03:17 AM
RE: UNIX Diary - by rain1 - 19-04-2016, 08:28 AM
RE: UNIX Diary - by josuah - 22-04-2016, 05:45 PM
RE: UNIX Diary - by movq - 07-05-2016, 01:34 PM
RE: UNIX Diary - by rocx - 16-05-2016, 02:21 AM
RE: UNIX Diary - by venam - 10-06-2016, 01:11 AM
RE: UNIX Diary - by venam - 14-09-2016, 01:12 AM
RE: UNIX Diary - by xero - 14-09-2016, 11:36 AM
RE: UNIX Diary - by jkl - 14-09-2016, 11:58 AM
RE: UNIX Diary - by venam - 31-01-2017, 09:27 AM
RE: UNIX Diary - by rocx - 18-02-2017, 02:17 AM
RE: UNIX Diary - by fre d die - 15-03-2017, 06:13 PM
RE: UNIX Diary - by rocx - 15-03-2017, 07:07 PM
RE: UNIX Diary - by robotchaos - 07-04-2017, 02:13 AM
RE: UNIX Diary - by evbo - 08-04-2017, 03:00 AM
RE: UNIX Diary - by jkl - 08-04-2017, 08:30 AM
RE: UNIX Diary - by venam - 09-04-2017, 04:56 AM
RE: UNIX Diary - by robotchaos - 10-04-2017, 03:56 PM
RE: UNIX Diary - by jkl - 10-04-2017, 04:03 PM
RE: UNIX Diary - by robotchaos - 10-04-2017, 04:22 PM
RE: UNIX Diary - by jkl - 10-04-2017, 04:25 PM
RE: UNIX Diary - by josuah - 11-04-2017, 09:19 AM
RE: UNIX Diary - by jkl - 11-04-2017, 09:39 AM
RE: UNIX Diary - by evbo - 18-04-2017, 02:20 PM
RE: UNIX Diary - by robotchaos - 18-04-2017, 03:43 PM
RE: UNIX Diary - by nas - 12-06-2017, 09:33 AM
RE: UNIX Diary - by Tmplt - 12-07-2017, 02:00 AM
RE: UNIX Diary - by xero - 13-07-2017, 03:12 PM
RE: UNIX Diary - by venam - 20-10-2017, 03:12 AM
RE: UNIX Diary - by Houseoftea - 04-01-2018, 08:40 PM
RE: UNIX Diary - by rocx - 05-01-2018, 02:04 AM
RE: UNIX Diary - by Tmplt - 21-01-2018, 10:02 PM
RE: UNIX Diary - by jkl - 22-01-2018, 08:14 AM
RE: UNIX Diary - by Tmplt - 22-01-2018, 01:02 PM
RE: UNIX Diary - by mrtn - 22-01-2018, 04:47 PM
RE: UNIX Diary - by jkl - 22-01-2018, 04:53 PM
RE: UNIX Diary - by jkl - 31-03-2018, 09:22 PM
RE: UNIX Diary - by jkl - 15-05-2018, 12:17 PM
RE: UNIX Diary - by z3bra - 20-09-2018, 07:07 PM
RE: UNIX Diary - by xero - 25-10-2018, 01:27 PM
RE: UNIX Diary - by venam - 26-10-2018, 01:08 AM
RE: UNIX Diary - by Steph - 28-10-2018, 05:23 PM
RE: UNIX Diary - by rocx - 04-11-2018, 01:48 PM
RE: UNIX Diary - by jkl - 05-11-2018, 10:18 AM
RE: UNIX Diary - by pranomostro - 06-11-2018, 09:06 AM
RE: UNIX Diary - by anthk - 10-01-2019, 12:20 PM
RE: UNIX Diary - by pranomostro - 10-01-2019, 04:38 PM
RE: UNIX Diary - by gaak - 14-01-2019, 09:47 PM
RE: UNIX Diary - by josuah - 10-12-2019, 08:39 PM
RE: UNIX Diary - by z3bra - 15-05-2020, 12:19 PM
RE: UNIX Diary - by wolf - 18-06-2020, 04:52 PM
RE: UNIX Diary - by fre d die - 08-07-2020, 04:47 PM
RE: UNIX Diary - by seninha - 08-07-2020, 09:34 PM
RE: UNIX Diary - by movq - 28-09-2020, 03:19 PM
RE: UNIX Diary - by z3bra - 29-09-2020, 11:00 AM
RE: UNIX Diary - by jkl - 30-09-2020, 03:52 AM
RE: UNIX Diary - by movq - 28-05-2021, 03:18 PM
RE: UNIX Diary - by movq - 27-09-2021, 11:56 AM
RE: UNIX Diary - by venam - 17-12-2021, 03:49 PM
RE: UNIX Diary - by z3bra - 20-12-2021, 08:34 AM
|