UNIX Diary

UNIX Diary - Psychology, Philosophy, and Licenses

Users browsing this thread: 6 Guest(s)

	z3bra Offline \| 28-04-2015, 04:13 PM \| #58

Dear Unix diary,

today I've been a bad sysadmin.
It just happened. I host my own git repository, and earlier this evening I was working on my crux port tree, when I decided to commit and push my work. But this time, something went wrong and git didn't let me push any reference. Amongst all the messages returned by git, I saw this one:

Code:
remote: fatal: write error: No space left on device

Fucking shit. I instantly imagine what's happening: my /var partition wasn't correctly sized upon creation. This is where I host my website, gopherhole, git repo, pictures, videos, ... Every 'production' service. And after serving me well for several years, it's now full.

Hopefully, I had setup all my partitions on top of LVM, and let like 200GiB available, just in case things go wrong. And they did.

So here am I, staring at my red prompt, typing a few commands:

Code:
root ~# df -h

Filesystem                Size      Used Available Use% Mounted on

mdev                      1.0M         0      1.0M   0% /dev

shm                     499.4M         0    499.4M   0% /dev/shm

/dev/dm-1                 4.0G    797.9M      3.2G  20% /

tmpfs                    99.9M    208.0K     99.7M   0% /run

cgroup_root              10.0M         0     10.0M   0% /sys/fs/cgroup

/dev/sda1                96.8M     14.5M     77.3M  16% /boot

/dev/mapper/vg0-var      50.0G     50.0G     20.0K 100% /var

/dev/mapper/vg0-home    100.0G     12.9G     85.2G  13% /home

/dev/mapper/vg0-data    600.0G    346.7G    252.1G  58% /data

tmpfs                   499.4M         0    499.4M   0% /tmp

tmpfs                   499.4M     32.4M    467.0M   6% /home/z3bra/tmp

/dev/mapper/vg0-data    600.0G    346.7G    252.1G  58% /var/lib/mpd/music

root ~# mount | grep /var

/dev/mapper/vg0-var on /var type xfs (rw,relatime,attr2,inode64,noquota)

root ~# lvs

  LV   VG   Attr       LSize

  data vg0  -wi-ao---- 600.00g

  home vg0  -wi-ao---- 100.00g

  root vg0  -wi-ao----   4.00g

  swap vg0  -wi-ao----   1.00g

  var  vg0  -wi-ao----  50.00g

root ~# vgs

  VG   #PV #LV #SN Attr   VSize   VFree

  vg0    1   5   0 wz--n- 931.41g 176.41g

Ok, so it's not the first time this happens, remember? You already grew your /home partition, and it went good! Just do the same with /var! It works without a reboot!

What was those commands again?

Code:
root ~# lvextend -L +20G vg0/var

  Extending logical volume var to 70.00 GiB

  63e74d07f000-63e74d2c1000 r-xp 00000000 fd:01 8430401                    /lib/libdevmapper.so.1.02: mlock failed: Out of memory

  63e74d2c6000-63e74d4cb000 r-xp 00000000 fd:01 8430404                    /lib/libdevmapper-event.so.1.02: mlock failed: Out of memory

  Logical volume var successfully resized

  Internal error: Reserved memory (9064448) not enough: used 9084928. Increase activation/reserved_memory?

root ~# lvs

  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  data vg0  -wi-ao---- 600.00g

  home vg0  -wi-ao---- 100.00g

  root vg0  -wi-ao----   4.00g

  swap vg0  -wi-ao----   1.00g

  var  vg0  -wi-ao----  70.00g

root ~# xfs_growfs -d /var

meta-data=/dev/mapper/vg0-var    isize=256    agcount=4, agsize=3276800 blks

         =                       sectsz=4096  attr=2, projid32bit=1

         =                       crc=0

data     =                       bsize=4096   blocks=13107200, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

log      =internal               bsize=4096   blocks=6400, version=2

         =                       sectsz=4096  sunit=1 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

data blocks changed from 13107200 to 18350080

root ~# df -h

Filesystem                Size      Used Available Use% Mounted on

mdev                      1.0M         0      1.0M   0% /dev

shm                     499.4M         0    499.4M   0% /dev/shm

/dev/dm-1                 4.0G    797.9M      3.2G  20% /

tmpfs                    99.9M    208.0K     99.7M   0% /run

cgroup_root              10.0M         0     10.0M   0% /sys/fs/cgroup

/dev/sda1                96.8M     14.5M     77.3M  16% /boot

/dev/mapper/vg0-var      70.0G     50.0G     20.0G  71% /var

/dev/mapper/vg0-home    100.0G     12.9G     85.2G  13% /home

/dev/mapper/vg0-data    600.0G    346.7G    252.1G  58% /data

tmpfs                   499.4M         0    499.4M   0% /tmp

tmpfs                   499.4M     32.4M    467.0M   6% /home/z3bra/tmp

/dev/mapper/vg0-data    600.0G    346.7G    252.1G  58% /var/lib/mpd/music

Phew... I'm safe now! So what the hell was going on? I decided to investigate a bit further, to see what I should watch next time.
That's how I realised that I did a HUGE mistake...

Code:
root ~# cd /var/

root var# du -sh *

48.5G   backup

156.7M  cache

0       db

0       empty

228.8M  git

5.7M    gopher

4.5G    lib

0       local

0       lock

7.9M    log

0       mail

0       run

40.0K   spool

0       tmp

1.1G    www

root var# cd backup/

root backup# du -sh *

12.0K   bin

20.0K   etc

48.5G   out

20.0K   usr

84.0K   var

root backup# mountpoint out

out is not a mountpoint

root backup# cd out/

root out# ll

total 50841516

drwxr-sr-x    2 backup   users       4.0K Apr 28 02:11 ./

drwxr-sr-x    8 backup   users       4.0K Feb  2 20:24 ../

-rw-r--r--    1 backup   users       5.3G Apr 25 07:43 data

-rw-r--r--    1 backup   users          0 Apr 25 07:43 data.0.BAK

-rw-r--r--    1 backup   users      12.0G Apr 26 04:37 homedir

-rw-r--r--    1 backup   users      12.0G Apr 22 04:43 homedir.0.BAK

-rw-r--r--    1 backup   users      12.0G Apr 25 05:00 homedir.1.BAK

-rw-r--r--    1 backup   users      44.0K Apr 26 04:42 homedir.2.BAK

-rw-r--r--    1 backup   users       1.2G Apr 28 02:11 production

-rw-r--r--    1 backup   users       1.2G Apr 21 02:10 production.0.BAK

-rw-r--r--    1 backup   users       1.2G Apr 22 02:11 production.1.BAK

-rw-r--r--    1 backup   users       1.2G Apr 23 02:11 production.2.BAK

-rw-r--r--    1 backup   users       1.2G Apr 24 02:11 production.3.BAK

-rw-r--r--    1 backup   users       1.2G Apr 25 02:12 production.4.BAK

-rw-r--r--    1 backup   users          0 Apr 26 02:11 production.5.BAK

-rw-r--r--    1 backup   users       5.3M Apr 27 02:12 production.6.BAK

-rw-r--r--    1 backup   users          0 Apr 28 02:11 production.7.BAK

My backup system doesn't check wether it saves to a mountpoint or not. Shit.
For a whole week, all my backups where created in my /var partition instead of a backup USB drive meant for this purpose. And it filled it up pretty quickly.

My backup system send me a mail after each backup, explaining me how it went. The fact it's saving to a mountpoint or not is written in it. I just stopped checking. Silly me.

I realise that this issue could have been easily solved by mounting my backup disk elsewhere, then moving the files, and remounting where it should be. But I didn't. Instead, I grew a partition that didn't need to be (the backups filled 48GiB out of 50Gib allocated to /var), and this partition can't be shrinked anymore, as it's an XFS filesystem.

So today I learnt two things, the hard way:

1. Don't do anything until you know what's going on
2. Configure systems checks and READ THEM

I hope you'll learn from my mistakes. For now I think I'll just print this over my desktop, as a reminder:

Code:
root ~# df -h /var/

Filesystem                Size      Used Available Use% Mounted on

/dev/mapper/vg0-var      70.0G      1.5G     68.5G   2% /var

Messages In This Thread

UNIX Diary - by z3bra - 18-08-2014, 11:03 AM

RE: UNIX Diary - by venam - 18-08-2014, 11:37 AM

RE: UNIX Diary - by cjm - 18-08-2014, 12:01 PM

RE: UNIX Diary - by venam - 18-08-2014, 12:18 PM

RE: UNIX Diary - by cjm - 18-08-2014, 02:11 PM

RE: UNIX Diary - by venam - 19-08-2014, 12:50 PM

RE: UNIX Diary - by z3bra - 19-08-2014, 01:18 PM

RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 02:15 PM

RE: UNIX Diary - by vompatti - 19-08-2014, 04:20 PM

RE: UNIX Diary - by pizzaroll1 - 19-08-2014, 07:53 PM

RE: UNIX Diary - by venam - 20-08-2014, 12:54 PM

RE: UNIX Diary - by vompatti - 20-08-2014, 06:33 PM

RE: UNIX Diary - by z3bra - 21-08-2014, 04:53 AM

RE: UNIX Diary - by DotDev - 21-08-2014, 07:50 AM

RE: UNIX Diary - by shtols - 21-08-2014, 12:55 PM

RE: UNIX Diary - by venam - 21-08-2014, 01:22 PM

RE: UNIX Diary - by venam - 23-08-2014, 06:55 AM

RE: UNIX Diary - by z3bra - 23-08-2014, 07:27 AM

RE: UNIX Diary - by venam - 24-08-2014, 04:39 AM

RE: UNIX Diary - by xero - 24-08-2014, 11:24 PM

RE: UNIX Diary - by pizzaroll1 - 25-08-2014, 12:37 PM

RE: UNIX Diary - by z3bra - 26-08-2014, 03:38 AM

RE: UNIX Diary - by cjm - 27-08-2014, 12:48 AM

RE: UNIX Diary - by xero - 28-08-2014, 11:11 AM

RE: UNIX Diary - by venam - 30-08-2014, 01:47 AM

RE: UNIX Diary - by kirby - 30-08-2014, 09:56 AM

RE: UNIX Diary - by venam - 30-08-2014, 11:22 AM

RE: UNIX Diary - by venam - 09-09-2014, 12:21 PM

RE: UNIX Diary - by venam - 04-11-2014, 12:27 AM

RE: UNIX Diary - by berk - 10-09-2014, 09:12 AM

RE: UNIX Diary - by xero - 10-09-2014, 10:53 AM

RE: UNIX Diary - by venam - 10-09-2014, 11:51 AM

RE: UNIX Diary - by xero - 11-09-2014, 11:18 AM

RE: UNIX Diary - by z3bra - 12-09-2014, 08:08 AM

RE: UNIX Diary - by xero - 17-09-2014, 10:51 AM

RE: UNIX Diary - by pizzaroll1 - 17-09-2014, 01:07 PM

RE: UNIX Diary - by hades - 17-09-2014, 07:47 PM

RE: UNIX Diary - by venam - 18-09-2014, 12:37 AM

RE: UNIX Diary - by xero - 18-09-2014, 10:51 AM

RE: UNIX Diary - by venam - 18-09-2014, 12:18 PM

RE: UNIX Diary - by earsplit - 19-09-2014, 02:23 PM

RE: UNIX Diary - by vompatti - 20-09-2014, 11:52 AM

RE: UNIX Diary - by pizzaroll1 - 20-09-2014, 08:00 PM

RE: UNIX Diary - by srp - 21-09-2014, 02:09 AM

RE: UNIX Diary - by kirby - 21-09-2014, 03:25 PM

RE: UNIX Diary - by venam - 30-09-2014, 01:00 PM

RE: UNIX Diary - by xero - 08-10-2014, 10:31 AM

RE: UNIX Diary - by cjm - 09-10-2014, 04:14 PM

RE: UNIX Diary - by kirby - 09-10-2014, 08:15 PM

RE: UNIX Diary - by JerrySpringerIsMyDad - 24-10-2014, 10:03 PM

RE: UNIX Diary - by xero - 28-10-2014, 10:38 AM

RE: UNIX Diary - by venam - 28-10-2014, 01:18 PM

RE: UNIX Diary - by neeasade - 09-01-2015, 10:44 AM

RE: UNIX Diary - by z3bra - 09-01-2015, 02:04 PM

RE: UNIX Diary - by z3bra - 19-02-2015, 07:05 AM

RE: UNIX Diary - by venam - 19-02-2015, 04:55 PM

RE: UNIX Diary - by xero - 20-02-2015, 11:49 AM

RE: UNIX Diary - by z3bra - 28-04-2015, 04:13 PM

RE: UNIX Diary - by bsdkeith - 29-04-2015, 05:01 AM

RE: UNIX Diary - by z3bra - 29-04-2015, 10:29 AM

RE: UNIX Diary - by xero - 29-04-2015, 03:09 PM

RE: UNIX Diary - by greduan - 02-05-2015, 09:58 AM

RE: UNIX Diary - by cjm - 07-05-2015, 11:14 AM

RE: UNIX Diary - by z3bra - 07-05-2015, 12:20 PM

RE: UNIX Diary - by neeasade - 24-07-2015, 11:10 PM

RE: UNIX Diary - by z3bra - 25-07-2015, 08:44 AM

RE: UNIX Diary - by Houseoftea - 25-07-2015, 11:32 PM

RE: UNIX Diary - by movq - 26-07-2015, 03:52 AM

RE: UNIX Diary - by z3bra - 27-07-2015, 02:02 PM

RE: UNIX Diary - by z3bra - 24-08-2015, 07:07 PM

RE: UNIX Diary - by xero - 25-08-2015, 10:38 AM

RE: UNIX Diary - by rocx - 30-08-2015, 11:40 PM

RE: UNIX Diary - by z3bra - 31-08-2015, 04:40 AM

RE: UNIX Diary - by venam - 31-08-2015, 07:30 AM

RE: UNIX Diary - by pranomostro - 03-09-2015, 05:34 PM

RE: UNIX Diary - by kirby - 03-09-2015, 06:57 PM

RE: UNIX Diary - by xero - 10-09-2015, 11:30 AM

RE: UNIX Diary - by venam - 14-09-2015, 03:45 AM

RE: UNIX Diary - by darthlukan - 14-09-2015, 02:53 PM

RE: UNIX Diary - by dkeg - 15-09-2015, 07:49 AM

RE: UNIX Diary - by darthlukan - 05-10-2015, 03:03 PM

RE: UNIX Diary - by pranomostro - 05-10-2015, 05:00 PM

RE: UNIX Diary - by apk - 06-10-2015, 02:41 PM

RE: UNIX Diary - by Mafia - 06-10-2015, 09:33 PM

RE: UNIX Diary - by kirby - 12-10-2015, 03:04 PM

RE: UNIX Diary - by pranomostro - 13-10-2015, 08:37 AM

RE: UNIX Diary - by venam - 13-10-2015, 09:38 AM

RE: UNIX Diary - by xero - 13-10-2015, 10:55 AM

RE: UNIX Diary - by pranomostro - 13-10-2015, 11:18 AM

RE: UNIX Diary - by venam - 15-10-2015, 01:34 PM

RE: UNIX Diary - by darthlukan - 15-10-2015, 04:44 PM

RE: UNIX Diary - by pranomostro - 15-10-2015, 06:05 PM

RE: UNIX Diary - by xero - 15-10-2015, 06:58 PM

RE: UNIX Diary - by darthlukan - 16-10-2015, 02:29 AM

RE: UNIX Diary - by pranomostro - 16-10-2015, 10:09 AM

RE: UNIX Diary - by darthlukan - 19-10-2015, 02:10 PM

RE: UNIX Diary - by venam - 20-10-2015, 02:05 AM

RE: UNIX Diary - by darthlukan - 20-10-2015, 02:28 AM

RE: UNIX Diary - by thetornainbow - 20-10-2015, 08:52 AM

RE: UNIX Diary - by Houseoftea - 20-10-2015, 09:11 AM

RE: UNIX Diary - by z3bra - 20-10-2015, 12:18 PM

RE: UNIX Diary - by darthlukan - 21-10-2015, 02:31 AM

RE: UNIX Diary - by z3bra - 30-11-2015, 07:08 PM

RE: UNIX Diary - by apk - 30-11-2015, 08:49 PM

RE: UNIX Diary - by z3bra - 01-12-2015, 08:24 AM

RE: UNIX Diary - by apk - 01-12-2015, 11:43 AM

RE: UNIX Diary - by arcetera - 02-12-2015, 08:57 PM

RE: UNIX Diary - by venam - 03-12-2015, 02:20 AM

RE: UNIX Diary - by pranomostro - 03-12-2015, 11:32 AM

RE: UNIX Diary - by dcat - 07-12-2015, 06:16 PM

RE: UNIX Diary - by venam - 11-12-2015, 07:30 AM

RE: UNIX Diary - by xero - 11-12-2015, 03:04 PM

RE: UNIX Diary - by pranomostro - 13-12-2015, 08:31 AM

RE: UNIX Diary - by venam - 13-12-2015, 03:22 PM

RE: UNIX Diary - by strang3quark - 13-12-2015, 03:57 PM

RE: UNIX Diary - by rocx - 13-12-2015, 04:37 PM

RE: UNIX Diary - by Wildefyr - 13-12-2015, 07:29 PM

RE: UNIX Diary - by venam - 14-12-2015, 01:37 AM

RE: UNIX Diary - by movq - 14-12-2015, 12:44 PM

RE: UNIX Diary - by venam - 14-12-2015, 12:56 PM

RE: UNIX Diary - by pranomostro - 14-12-2015, 01:24 PM

RE: UNIX Diary - by movq - 19-12-2015, 03:26 PM

RE: UNIX Diary - by venam - 20-12-2015, 05:02 AM

RE: UNIX Diary - by movq - 20-12-2015, 08:13 AM

RE: UNIX Diary - by pranomostro - 20-12-2015, 10:21 AM

RE: UNIX Diary - by z3bra - 20-12-2015, 03:46 PM

RE: UNIX Diary - by movq - 22-12-2015, 03:27 PM

RE: UNIX Diary - by XcelQ - 24-01-2016, 01:32 PM

RE: UNIX Diary - by pranomostro - 26-01-2016, 02:17 PM

RE: UNIX Diary - by XcelQ - 28-01-2016, 03:17 AM

RE: UNIX Diary - by rain1 - 19-04-2016, 08:28 AM

RE: UNIX Diary - by josuah - 22-04-2016, 05:45 PM

RE: UNIX Diary - by movq - 07-05-2016, 01:34 PM

RE: UNIX Diary - by rocx - 16-05-2016, 02:21 AM

RE: UNIX Diary - by venam - 10-06-2016, 01:11 AM

RE: UNIX Diary - by venam - 14-09-2016, 01:12 AM

RE: UNIX Diary - by xero - 14-09-2016, 11:36 AM

RE: UNIX Diary - by jkl - 14-09-2016, 11:58 AM

RE: UNIX Diary - by venam - 31-01-2017, 09:27 AM

RE: UNIX Diary - by rocx - 18-02-2017, 02:17 AM

RE: UNIX Diary - by fre d die - 15-03-2017, 06:13 PM

RE: UNIX Diary - by rocx - 15-03-2017, 07:07 PM

RE: UNIX Diary - by robotchaos - 07-04-2017, 02:13 AM

RE: UNIX Diary - by evbo - 08-04-2017, 03:00 AM

RE: UNIX Diary - by jkl - 08-04-2017, 08:30 AM

RE: UNIX Diary - by venam - 09-04-2017, 04:56 AM

RE: UNIX Diary - by robotchaos - 10-04-2017, 03:56 PM

RE: UNIX Diary - by jkl - 10-04-2017, 04:03 PM

RE: UNIX Diary - by robotchaos - 10-04-2017, 04:22 PM

RE: UNIX Diary - by jkl - 10-04-2017, 04:25 PM

RE: UNIX Diary - by josuah - 11-04-2017, 09:19 AM

RE: UNIX Diary - by jkl - 11-04-2017, 09:39 AM

RE: UNIX Diary - by evbo - 18-04-2017, 02:20 PM

RE: UNIX Diary - by robotchaos - 18-04-2017, 03:43 PM

RE: UNIX Diary - by nas - 12-06-2017, 09:33 AM

RE: UNIX Diary - by Tmplt - 12-07-2017, 02:00 AM

RE: UNIX Diary - by xero - 13-07-2017, 03:12 PM

RE: UNIX Diary - by venam - 20-10-2017, 03:12 AM

RE: UNIX Diary - by Houseoftea - 04-01-2018, 08:40 PM

RE: UNIX Diary - by rocx - 05-01-2018, 02:04 AM

RE: UNIX Diary - by Tmplt - 21-01-2018, 10:02 PM

RE: UNIX Diary - by jkl - 22-01-2018, 08:14 AM

RE: UNIX Diary - by Tmplt - 22-01-2018, 01:02 PM

RE: UNIX Diary - by mrtn - 22-01-2018, 04:47 PM

RE: UNIX Diary - by jkl - 22-01-2018, 04:53 PM

RE: UNIX Diary - by jkl - 31-03-2018, 09:22 PM

RE: UNIX Diary - by jkl - 15-05-2018, 12:17 PM

RE: UNIX Diary - by z3bra - 20-09-2018, 07:07 PM

RE: UNIX Diary - by xero - 25-10-2018, 01:27 PM

RE: UNIX Diary - by venam - 26-10-2018, 01:08 AM

RE: UNIX Diary - by Steph - 28-10-2018, 05:23 PM

RE: UNIX Diary - by rocx - 04-11-2018, 01:48 PM

RE: UNIX Diary - by jkl - 05-11-2018, 10:18 AM

RE: UNIX Diary - by pranomostro - 06-11-2018, 09:06 AM

RE: UNIX Diary - by anthk - 10-01-2019, 12:20 PM

RE: UNIX Diary - by pranomostro - 10-01-2019, 04:38 PM

RE: UNIX Diary - by gaak - 14-01-2019, 09:47 PM

RE: UNIX Diary - by josuah - 10-12-2019, 08:39 PM

RE: UNIX Diary - by z3bra - 15-05-2020, 12:19 PM

RE: UNIX Diary - by wolf - 18-06-2020, 04:52 PM

RE: UNIX Diary - by fre d die - 08-07-2020, 04:47 PM

RE: UNIX Diary - by seninha - 08-07-2020, 09:34 PM

RE: UNIX Diary - by movq - 28-09-2020, 03:19 PM

RE: UNIX Diary - by z3bra - 29-09-2020, 11:00 AM

RE: UNIX Diary - by jkl - 30-09-2020, 03:52 AM

RE: UNIX Diary - by movq - 28-05-2021, 03:18 PM

RE: UNIX Diary - by movq - 27-09-2021, 11:56 AM

RE: UNIX Diary - by venam - 17-12-2021, 03:49 PM

RE: UNIX Diary - by z3bra - 20-12-2021, 08:34 AM

View a Printable Version