What do you use as a system monitor - Servers Administration, Networking, & Virtualization

Users browsing this thread: 4 Guest(s)
venam venam Offline | 10-02-2022, 01:22 PM | #1
Administrators
Hello nixers,
Here's a simple thread: What do you use as a system monitor and do you have recommendations.

I normally go for htop and top but I'm looking to broaden my horizons.

Be sure to mention the OS it runs on (if only a particular one), and provide a link.
neeasade neeasade Offline | 10-02-2022, 10:08 PM | #2
Grey Hair Nixers
I use htop. as far as "finding out when shit is whack", I monitor--

- temp and battery in a systemd user service: https://github.com/neeasade/dotfiles/blo...nsor_alert

- disk space via a section on my panel that pops up: https://github.com/neeasade/dotfiles/blo...isc#L4-L20

I guess it might also be nice to be alerted when there's high IO or cpu usage or something (over some time period/non-bursty)
movq movq Offline | 12-02-2022, 04:51 AM | #3
Long time nixers
htop here, too. It can usually answer all of my questions.

One of the things it can't do is show network usage. For my Linux desktop/laptop, I just have a widget in my status bar (along with CPU usage and such). It's surprisingly rare that I need to show network usage on a server in real time (we have long-term statistics using influx/telegraf). When I do, I usually went for iftop. I just found bmon which has a slightly nicer UI and doesn't need root.


(10-02-2022, 10:08 PM)neeasade Wrote: temp

That's interesting. Why do you need an alert when CPU temp goes up? Cooling issues?
neeasade neeasade Offline | 12-02-2022, 05:02 PM | #4
Grey Hair Nixers
> Why do you need an alert when CPU temp goes up? Cooling issues?

It was for an old thin thinkpad I had that probably needed thermal paste re-applied -- Positioning and a cooler could matter a lot, and so I needed something to nudge me to keep that in mind

edit: I was also QUITE mobile at the time for context. coffee shops, conference rooms, a friends living room, different every day.
seninha seninha Offline | 12-02-2022, 09:36 PM | #5
Long time nixers
On my OpenBSD desktop, I configured apmd to log every minute at /var/log. I also configured sensorsd.conf to set a high temperature value that, when passed, a message should be logged.
Then, I run an awk script at X login that reads apmd log (from tail -f) and writes a notification to xnotify's fifo when a interesting line is read.
It currently only greps for lines reporting low battery and high temperature (see the image below, at top right).

For my dock/bar I use WindowMaker-style dockapps, in particular wmbubble, which compiles some CPU and memory info as a duck swimming on a pool of bubbling water. When a mouse is over it, wmbubble shows the load average graph.

[Image: mon.png]
venam venam Offline | 13-02-2022, 05:49 AM | #6
Administrators
On that note, I also forgot I have a script for battery status that pops up at certain intervals.
It's running as a systemd user timer (~/.config/systemd/user/timers.target.wants).
The timer unit
Code:
[Unit]
Description=Timer to run battery-notify

[Timer]
OnActiveSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target
and service unit:
Code:
[Unit]
Description=battery-notify

[Service]
ExecStart=%h/docu/bin/check-batt

[Install]
WantedBy=default.target

And the actual script that gets called check-batt, which is very similar to what neeasade is doing but with threshold values:
Code:
#!/usr/bin/env bash

BAT_NUM='1'
STATUS_FILE=/tmp/check-bat.tmp
BATTPERC=$(cat /sys/class/power_supply/BAT${BAT_NUM}/capacity)
BATTSTATUS=$(cat /sys/class/power_supply/BAT${BAT_NUM}/status)
LAST_STATUS=''
LAST_NOTIF_LEVEL=0
NOTIF_LEVEL=0

if [ -e $STATUS_FILE ] ; then
    . $STATUS_FILE
fi

save_status() {
    LAST_STATUS=$BATTSTATUS
    echo "LAST_STATUS=${BATTSTATUS}" > $STATUS_FILE
    echo "LAST_NOTIF_LEVEL=${NOTIF_LEVEL}" >> $STATUS_FILE
}

notify() {
    level=$1
    msg=$2
    DISPLAY=:0 /usr/bin/notify-send -t 3500 -u "$level" "POWER" "$msg"
}

# Battery status changed
if [[ "$BATTSTATUS" != "$LAST_STATUS" ]] ; then
    notify "normal" "battery is now $BATTSTATUS, Capacity: $BATTPERC"
    NOTIF_LEVEL=$LAST_NOTIF_LEVEL
    save_status
    exit
fi

if [ "$BATTSTATUS" = Discharging ] && [ "$BATTPERC" -le 10 ]; then
    NOTIF_LEVEL=10
    notify "critical" "battery is less than 10% - Connect charger, going to sleep in 10s"
    sleep 10
    BATTSTATUS=$(cat /sys/class/power_supply/BAT${BAT_NUM}/status)
    if [ "$BATTSTATUS" = Discharging ]; then
        notify "critical" "going to suspend now"
        sleep 3
        logger "Critical battery threshold, suspending"
        save_status
        systemctl suspend
        exit
    else
        notify "normal" "battery is now charging, cancelling suspend"
        save_status
        exit
    fi
fi

if [[ $BATTPERC -le 20 ]] ; then
    NOTIF_LEVEL=20
    notify "critical" "battery $BATTSTATUS is less than 20%"
    save_status
elif [[ $BATTPERC -le 30 ]] ; then
    NOTIF_LEVEL=30
    if [ "$NOTIF_LEVEL" != "$LAST_NOTIF_LEVEL" ]; then
        notify "normal" "battery $BATTSTATUS is at 30%"
        save_status
    fi
elif [[ $BATTPERC -le 50 ]] ; then
    NOTIF_LEVEL=50
    if [ "$NOTIF_LEVEL" != "$LAST_NOTIF_LEVEL" ] ; then
        notify "low" "battery $BATTSTATUS is at 50%"
        save_status
    fi
elif [[ $BATTPERC -le 60 ]] ; then
    NOTIF_LEVEL=60
    if [ "$NOTIF_LEVEL" != "$LAST_NOTIF_LEVEL" ] ; then
        notify "normal" "battery $BATTSTATUS is at 60%"
        save_status
    fi
fi
pfr pfr Offline | 14-02-2022, 10:31 PM | #7
Nixers
(12-02-2022, 09:36 PM)seninha Wrote: On my OpenBSD desktop

Man, I love your desktop. Everytime I see it I'm intrigued. I wish you would just fork a distro or release a pre-configured live image. It's so polished and consistant. Harmonious even.

Anyway, I keep saying I'm going to use shod once I have a desktop setup (with a mouse). Eventually!

Back to the topic at hand, I quite like nmon as a system monitor.
VMS VMS Offline | 15-02-2022, 01:14 PM | #8
Members
(13-02-2022, 05:49 AM)venam Wrote: On that note, I also forgot I have a script for battery status that pops up at certain intervals.

I do have one too on NetBSD, and the concept is somehow similar:

Code:
#!/bin/sh
    CHARGE=$(/usr/sbin/envstat | grep charge: | sed 's/.*(\(.*\))/\1/')
    BATT=$(echo $CHARGE | awk '{print $1}')

if envstat | grep -Eiq 'charging.*true'; then
  STATUS=charging
else
  STATUS=discharging
fi
notify-send "$(echo -e "battery = $BATT \nstatus  = $STATUS")"

It relies on envsys(4) (Environmental Systems framework), which was written by the creator of Void's XBPS and can print a lot of useful info from monitor devices and sensors in quite a fancy and human-readable fashion.

Usually I bind this to something like Mod4 + b in CTWM.
Similarly, I bind Mod4 + v to:

Code:
#!/bin/sh
VOL=$(mixerctl -n outputs.master | cut -d , -f 1 | \
     awk '{print $1/254*100}' | cut -d . -f 1)
notify-send "volume: $VOL %"

In my ~/.profile I have this little function to monitor power-management related events. I usually have a sticky term running it in foreground.

Code:
pwrd()
        {
             ( tail -f -n0 /var/log/messages & ) | sed -n -e 's/^.*sensor_battery: //p'
        }

As a pseudo graphical resource monitor, I really love systat; the interface might look a bit ugly by modern standards, as it relies on the native curses library, which is more limited compared to ncurses, but it does its job and does it well. systat can display various statistics at the same time including cpu load, virtual memory, processes, swap usage, disk i/o, open sockets, established connections, network traffic...for example `systat iostat` looks like that:

[Image: systat.png]

To quickly display the number of total used buffers in megabytes, I also use this script; useful on server and public access BSD systems like SDF or GREX.

Code:
#!/bin/sh
vmstat -s | awk '
/ bytes per page$/ { bpp = $1 }
/ cached file pages$/ { cfp = $1 }
/ cached executable pages$/ { cep = $1 }
END { print((cfp + cep) * bpp / 1024 / 1024); }'

As for graphical tools, I really like a tool ported from old versions of SPARC32 Solaris and called xmeter. I was unable to find a description to link here, but, from the man page:

Quote: Xmeter displays a periodically updating histogram of the system
statistics gathered by rstat(3) for the specified hosts. Meters can be
displayed in a vertical, horizontal or rectangular arrangement. As
statistics range between 4 user defineable levels (OK, WARN, ERROR or
FATAL), the background, foreground, highlight, border and internal
border colors, and the background bitmap of each meter can be changed.

rstat is a daemon capable of returning performance statistics obtained from the kernel. These can be read using the rup(1) command. rstat is a Sun RPC protocol, and I suppose that's why I was unable to find a port of xmeter for Linux.
pyratebeard pyratebeard Offline | 18-02-2022, 09:20 AM | #9
Long time nixers
I use the same battery script as venam, and I also use htop(1).

Another tool that I quite like is glances.
jkl jkl Offline | 21-02-2022, 11:31 AM | #10
Long time nixers
I’m perfectly fine with the built-in tools. I noticed that the additional overhead is just not worth it.

--
<mort> choosing a terrible license just to be spiteful towards others is possibly the most tux0r thing I've ever seen
prx* prx* Offline | 24-02-2022, 06:02 PM | #11
Members
Hi,
On OpenBSD, I use this little script to output in tmux status or dwm's bar various info : mpd current song, battery remaining, audio volume, date. There are also functions for disk usage, cpu and ram that I actually don't use (top/vmstat/systat are good for this)

[Image: oKVt.png]

Code:
fs_perc()
{
    df -h $1 | awk 'NR==2 {print $5}'
}

get_vol()
{
    sndioctl output.level | awk -F = '{printf "%d\n",$2*100}'
}

ram() {
    top | awk '/Memory/ { print $3 }'
}

loadavg()
{
    #uptime | awk -F'[a-z]:' '{print $2}' | tr -d ' '
    sysctl -n vm.loadavg
}

bat()
{
    apm | awk 'NR==1 && !/absent/ {printf "%s", $4}'
    #test $(apm -a) -eq 0 && printf "⚡%s%%" $(apm -l)
}

mpdstatus()
{
mpc | awk '
(NR == 1) {
    status = $0
    if (length(status) > 42) {
        status = substr($0, 1, 42) "..."
    }
}
(NR == 2) && /playing/ {
    isplaying = 1
}
END { if (isplaying) { printf "[%s]", status } }
'
}

temp()
{
    sysctl hw.sensors | awk -F"=" '{print $2; exit}'
}

printf "%s / ⚡ %s / 🔈 %s / %s" \
     "$(mpdstatus)"\
     "$(bat)" \
     "$(get_vol)%" \
     "$(date "+📅 %Y-%m-%d / 🕓 %H:%M")"
yakumo.izuru yakumo.izuru Offline | 17-04-2022, 11:04 PM | #12
Members
I'm usually using htop and relying on conky for the rest, which is mainly CPU load, RAM usage, assigned IP address, battery charge[*], uptime[*] and date/time[*]
  • Also handled by tmux via apm(8), get_uptime.sh (a script I wrote), and the default date/time setting
Code:
_                _            ,                                
(_|   |_/o       | |          /|   |                      o    
  |   |      __  | |  _ _|_    |___|  _   ,_    _  _  _     _|_
  |   |  |  /  \_|/  |/  |     |   |\|/  /  |  / |/ |/ |  |  |  
   \_/   |_/\__/ |__/|__/|_/   |   |/|__/   |_/  |  |  |_/|_/|_/
~Izuru Yakumo
maksim maksim Offline | 29-09-2022, 09:28 AM | #13
Members
I use Btop, it is an awesome TUI utility with a good UI. I also made an Everforest theme for it, which is already inside of Btop.
gaak gaak Offline | 31-10-2022, 04:21 AM | #14
Members
(12-02-2022, 09:36 PM)seninha Wrote: On my OpenBSD desktop, I configured apmd to log every minute at /var/log. I also configured sensorsd.conf to set a high temperature value that, when passed, a message should be logged.

This for everything. When the current state is nominal, nothing should be displayed.

Getting everyone to agree on a message bus is not easy. eg. battery low/high goes to X11 display, while the daily security report goes to email.

Code:
#!/usr/bin/awk -f

BEGIN { max_disk = 0.950 ; min_ifree = 2000; max_load = 1.0; }

/^$/ { is_disk = 0; is_network = 0; }
/^disks:/ { is_disk = 1; next; }
/^network:/ { is_network = 1; next; }
# remove date-time as that breaks duplicate output checking
/^OpenBSD/ {
    for ( i = 4 ; i <= NF; i++ ) {
        $i = "";
    }
}
/load averages:/ {
    if ( $NF > max_load ) { print; }
    next;
}

! is_network && ! is_disk { print; }

is_network {
  name = $1; colls = int($NF); ofail = int($(NF-1)); ifail = int($(NF-3));
  if ( colls > 0 || ofail > 0 || ifail > 0 ) {
    printf("net %s input errs %s output errs %s collisions %s\n", name, ifail, ofail, colls);
  }
}

is_disk {
  name = $NF; total = int($2); used = int($3); ifree = int($7);
  if ( name == "on" ) { next; }
  if ( used / total > max_disk ) {
    printf("disk %s at %.2f used\n", name, used / total);
  }
  if ( ifree < min_ifree ) {
    printf("disk %s ifree at %d\n", name, ifree);
  }
}


View a Printable Version