Ok, it seems like I did not explain it right.

Well, it goes like this:
The program is set up, and, after the work is done, it is shut down.
If an error occurs during these two periods, it just exits without attempting
to clean things up.

When an error occurs after the setup, the shutdown routine is called and the
program then exits.

@strangequark:
Well, if i can't create a directory in /tmp and start a process, it is the users
fault. And the consequences are not that bad-some files in /tmp.

@rocx:
When it fails, it shouts.
I don't like programs that try to repair themselves.
»Windows is searching for a solution, please wait…«
I understand this rule as »if there is a 100% chance the problem can be solved elegantly
by itself, do it, otherwise shout while dying.«
This is what I tried to do.

This keeps the code fairly simple, but does things right.

Dear UNIX diary,

today, I exploited memory overcommit and copy-on-write.

I have a program that reads data and stores it in a circular buffer. That is, when the buffer is "full", new data is simply stored at the beginning of the buffer, thus overwriting old data. The program just keeps the data in ordinary memory, i.e. it does not save it on a hard drive or something else.

This program can be fed with arbitrary data, for example raw audio data. This means you keep recording and recording, but you will always only have ~50MB (or the last ten minutes) of audio data available.

So far, so good.

What I want to do next is send my program a signal and it shall save the buffer as an OGG/Vorbis file. Meanwhile, I still want to keep recording to the very same buffer -- without any interruption. Of course, as new data arrives, old data will be overwritten, so I must make a copy of my buffer prior to encoding it. Creating that copy could be a problem if the original buffer is, say, 80% of my total RAM.

Enter: Kernel magic.

Okay, I have a process with some allocated memory. I can solve my problem by simply doing a fork(). The child process will be a clone of the parent -- including its memory. The child can then encode the audio data as OGG/Vorbis while the parent continues to record new data.

What's awesome about this is that the kernel does not really create a copy of the memory[1]. Both processes refer to the same memory pages. Only when one process writes to a "shared" page (the parent does this when recording new audio data), a real copy will be created. Thus, only that particular data in my circular buffer which arrives while I'm encoding needs to be copied. This only amounts to a fraction of the total buffer size, which allows me to do the whole operation even with very large buffers.

And, I, myself, don't have to implement any of this. The kernel takes care of everything.


[1]: At least, Linux behaves this way by default. Last time I checked, FreeBSD and OpenBSD did the same thing.

First time I hear of a circular buffer of limited size, if that's the name you give this thing.
I've used queues, stacks, circular queues, all sorts of trees, and a lot of different data structures but I can't think of why a circular buffer is more useful than stack for buffering.

Why the circularity?

In essence, my "circular buffer" is just this:


Dead simple.

Starting from "i", I can go back to read the previous "fill" bytes (you'll have to wrap around, of course). (The real code reads larger chunks, not single bytes.)

The goal was to record audio data, continuously, for several hours. But: We are always only interested in the last 10 minutes. Data older than that can be discarded.

I have to admit that this was pretty much the first thing that came to my mind. I didn't really look for alternatives. Am I missing something here? :)

I like that concept.

Isn't the same process used in dash cams? It only records the last X minutes, and new data override the oldest data. It would be kinda tricky to replay it once it's stopped, you'd have to know the buffer size, its memory address and the position where it stopped writing.

(20-12-2015, 03:46 PM)z3bra Wrote: Isn't the same process used in dash cams? It only records the last X minutes, and new data override the oldest data.

I believe so. Flight recorders do the same thing. All these things just sit there, wait for some kind of event, and then need to provide data from the past.

I wonder why I never tried this as a child: http://www.instructables.com/id/Audio-Cassette-Loop/

Okay unix diary so i want to create a lot of stuff but i have no idea how to go about it or what language to use.

Web browser engine that suckless, mta that suckless for something like quark from suckless. a cms, and my own kernel and unix tools from scartch hopefully in golang, maybe a golang mta and web server and kernel.

I have no idea how to start any of these project or go about it i don't even know any language well enough to start a project like the ones I listed. so Diary what should I do?

@XcelQ:
First, write your plans down. I promise you that everything will take
much longer than expected, so it is good to write notess and specifications.

Second, use which tools suit you best. Don't fall into the hipster trap.
If you want to start s huge project (your project is huge), you can't afford learning a new language.
For the stuff you want to write, I would recommend to start learning golang.

Third: read some books about the stuff you want to do. Start with the easy stuff,
try to learn things in sequence, not in paralell.

(26-01-2016, 02:17 PM)pranomostro Wrote: @XcelQ:
First, write your plans down. I promise you that everything will take
much longer than expected, so it is good to write notess and specifications.

Second, use which tools suit you best. Don't fall into the hipster trap.
If you want to start s huge project (your project is huge), you can't afford learning a new language.
For the stuff you want to write, I would recommend to start learning golang.

Third: read some books about the stuff you want to do. Start with the easy stuff,
try to learn things in sequence, not in paralell.

Thanks for the comment, i'm going to start learning a language and stick with it and try easy stuff first even though that doesn't interest me. But i should try to learn something.

Dear UNIX diary,

Yesterday I installed OpenBSD for the first time. I set it up with dwm+dmenu+st - just starting out so I have a lot of docs to read to get my footing.

Today I tried to fix a bug in the release version of st since my terminals were crashing & vanishing, sent away a patch and it had already been fixed in git *doh*. It was good practice anyway. Hopefully I will be able to make good contributions in future.

TL;DR: Non-free hardware drivers, dumb internet service providers... What a mess!


Dear UNIX diary,

Both of my laptops have half installed OSes (debian) and do not boot.

I can't install anything on my third laptop as it has a 32 bit UEFI and a 64 bit processor, and is now broken anymay.

I have one USB key with debian netinstall, but to get internet with the cable, I need to connect via a web page, and there is no browser included in the ISO (not even text mode nor cURL).

Tethering from Android does not works with debian (it would have worked as I con open the connection page from the phone before tethering).

Using the wifi does not work, as all of my devices wireless controller use proprietary drivers that are obvously not included in any of these installation images.

I could try to use a raspberry pi, but I have no adapter from HDMI to VGA (still the standard in France, even if it is old), and don't even have a VGA cable. My other Raspberry have a screen board connected with GPIO, but the provided image needs configuration before the console to be displayed on the screen.

Last solution: using my android phone as a bootable media to install a distro, but then I will have to downgrade android from 5.x.x to 4.x.x because all versions with fancy material design can not get this "drive droid" app to work. Fortunately, I can do it without a desktop, as I have a recovery mode that can install .zip images.

While I will have installed ArchLinux, which works, I will have to be fast, as the systemd journal somethig takes up to 98% of my CPU and make the laptop crash a few time after I booted. Then I cannot download a full iso, and have to look for a netinstall or a minimal lightweight.

So after trying:
- Debian (no browser),
- Alpine (no browser),
- Void (not booting),
- Arch (systemd journal eating all CPU),
- Crux (no browser),
- OpenBSD (no browser),
- NixOS (not installing due to configuration failure),
- Windows^K^K^K^K^K^K^K Ubuntu (I'm allergic)
- SliTaz (freezes during install),
- Devuan (no browser),
- ArchBang without systemd (does not fully boot);
- Gentoo (no browser);
I will then try a few other to see what boots and hos a browser.

Epilogue: Now with an OpenRC-based on both of my machine (alpine linux and obarun, arch-derived), I reallize that I learned about the choices distro-maker makes, and multiple different ways to install a distro. Moreover, now that I know that the story ends well, I find it rather fun.

There is always a way to linux.

Dear UNIX diary,

a few months ago, I started having shutdown issues. My system would sometimes hang for a while before finally powering down. While trying to debug the problem, I found out about systemd's debug shell. The nice thing about it is that it's "always" available, even when the system is just about to power off.

I've been using the following script to shut down my system ever since:


Once started, you can access a root shell on TTY 9. (That's why it shouldn't run all the time.)

I really like the idea of always having a rescue shell available during shutdown. This has been bothering me for years. When the system is shutting down, you're usually completely "helpless". If something hangs/freezes, you can only hope that your init system is able to handle the situation -- if it can't, you're screwed and you risk data loss.

With a rescue shell like that you have a much greater chance of fixing things.

Dear UNIX Diary,

For a little project of our's, my homeskillet Adam and I decided to use the httpd that comes with OpenBSD on our box instead of Nginx. Upon transition, it didn't appear that anything served. Constant twiddling later, I pull it up in elinks to check on the status of the project while still logged into our box. For some reason, it worked in elinks. Same goes to the success on my end in eww. But yet the site would not load in Firefox. Most peculiar.

Turns out aforementioned homeskillet nabbed some certificates a la Let's Encrypt and the TLS shindig was throwing us for a loop. Protip for OpenBSD httpd users who want to use TLS: Make sure you configure it properly in /etc/httpd.conf, because the tls segment is a braced segment.


Tomorrow, I'll be tackling some more complicated parts of the project with the httpd. User directories, for example.

Dear Unix Diary,

You made me suffer a lot yesterday... Actually Arch did!

Don't blame adam, blame Arch and its machiavellism!

I've been keeping my packages up to date but I've been doing that every 4-5 days and my box had an uptime of 24 days.

Suddenly, 2 days ago, when I went through the update process I noticed that I had to download 900MB and replace more than 500 pkgs, WHAT IS HAPPENING HERE?

Ok, No prob, let's update those.

I remembered a few days ago I was explaining the difference between the TTY and the X environment to my SO and when I switched to TTY2 and startx it wasn't starting.
At the time I didn't give it too much attention, I even forgot about it.

I let the stuffs download and update during the night and when I wake up the next morning, reboot, and X doesn't start.

Quick analysis => It seems to come from urxvt, I'm starting my X session over it.
Quick fix => Let's start the session over xterm and see what else is broken.

...

• The touchpad
  
• Urxvt
  
• Postgresql
  
• PHP
  
• Phppgadmin
  
• Ruby
  
• Some Python modules
  
• Perl
  

Calm down fellow, nothing to worry about.

Let's start with the touchpad.

"You've done that multiple times, you can do it again"
"Why the hek should I have to redo it!"

Ok, let's check the X11 files.

50-synaptics.conf is missing but it's in a .pacsave file, insane.
No need anymore to check the Arch wiki about touchpad

That's one.

Urxvt is crying about the wrong link to the perl library.
Damn perl updates breaking everything.

Let's try compiling it from the AUR:

(EE)

Dammit!

What could be wrong, let's remove some support by using the --disable flag.
No result.

Why not reinstall with pacman from the main repo.

No freaking way... It's working again, it even kept my configuration, that's great!

That's two.

I have scripts on my machine that uses my postgresql to store logs.
They didn't start correctly at boot that's when I noticed postgres couldn't start.

Ok, systemd magic tells me to check the logs in journactl -ex or journalct -u postgresql

I'm just paraphrasing here:

Yeah, that's verbose.

Where are my text logs when I need them.

Let's check /var/log/error.log, nope, /var/log/postTAB nope nope neither, /var/log/messages.log, really.


Damn, couldn't you do that during the update.

I ran the script in the wiki and my DB is up again.

That's three.

Let's see what's up with php.

They switched to php7 apparently.

And the support for phar and openssl are built in, that explains why I had a warning.

Let's just remove those extension from php.ini.

That's four.

What about phppgadmin.

It's shouting about php incompatibility or drop of support for the constructor that are named like the class.

Wow, I didn't know that was even possible, how old is that code?

Quote:xero: that was php4 style, naming the class and the constructor the same. it was ok in php5 and now deprecated in 7

Seems like it was fixed in a GH pull here but not in the main repo.

I'll fix it manually 'cause I kick asses!

I went through all the classes files and patched them.

That's five.

I use ruby for jekyll, nothing else.

Jekyll doesn't start, yep, get over it!

I tried to install the gem again, didn't find much about "re-installing" a gem.


No luck, it kept saying I was missing some module.

I kept trying to remove the gem and reinstalling them for a while, as normal user and as root, I thought it would make a difference, and then gave up.

I then completely removed my ~/.gem directory and tried again.

Ok, it's installing in ~/.gem/ruby/2.3.0 not ~/.gem/ruby/2.2.0 and my ruby path in my .zshrc points to the 2.2.0.

This is working... Eh nope.

Pygments isn't there => gem install pygments => No such module => gem install rb-pygments => not working.

Damn, can I use something else, let's update my _config.yml to use "rouge" instead, whatever that is.

Sweet, jekyll is up again!

But weirdly it's not parsing well my H tags.
Maybe it's adhering to the "newest" markdown standards, I know "standards" and markdown don't go along.

It seems to work when I add a space after the "#" but not when the word is directly next to it.

Sed comes to the rescue (even though my skill are rusty):

The rest...
I'll fix on the fly, they are just modules.

Cheers my Unix friends!

Dear Unix diary,
DVDs are annoying.

Yesterday we wanted to watch a movie on a dvd my brother landed me.
I first tried on my parabola box and it wasn't recognized by the dvd player, I also tried on my girlfriend's ubuntu box and it wasn't recognized either.
Later, my brother tried it on his own ubuntu machine and it worked fine, the only difference was the media player he was using: VLC.

So I installed VLC on my Debian laptop and tried the DVD, it worked fined!
Great, so now let's watch it on the TV via HDMI.
Unfortunately the HDMI wasn't recognized on this laptop... No idea why.

At this point I started searching for a way to play the DVD from mplayer or mpv.
I found about the `dvdnav://` and other options. One in particular was the one allowing to play from an ISO file.

I used dd to create an iso of the DVD and then transfer it to my parabola machine that had issues reading the DVD earlier on.

Fine, now the parabola box is connected to HDMI and it can play DVDs with menus.

Unfortunately, 55min into the film and the film suddenly stops.
I first thought the ISO might have been corrupted but as it turned out the DVD itself was scratched and also couldn't play.

(14-09-2016, 01:12 AM)venam Wrote: Dear Unix diary,
DVDs are annoying.

yup. when i learn about a new movie i want to see, i grab a torrent and stream it to my kodi powered htpc. analogue disks are so passé ;D

Torrents are still a thing? I thought everyone has finally understood how much greater Kad2 is.
No web browser required. No web server (for .torrent files or Magnet links) required either.

Dear Unix diary,
The last few weeks I've been compiling links for the newsletter, reading pages after pages everyday.
I've also been trying to convey and join ideas in novel way for the podcast in the hope they'll be more understandable.

But even so, I still feel like I don't know you at all.

And so I'm on my quest to go again through books about shell scripting, server administration, C programming, and if I can reach the end I'll start with driver development.

Dear UNIX Diary,

Never thought I'd say this, but sometimes the ability to use lookaheads in regular expressions would be nice. If there's one field that Emacs could use work in, it would be the regexps. Never thought I'd even use regexp features this advanced until some markup I wrote needed refactoring. Simplified my markup to the point it didn't need body or head tags. While I was at it, I even removed quotes from single-token attribute properties. Big mistake. It's good habit to quote them anyway.

Where do lookaheads come in? An example tag:

El gasp. No quotes. To add them, all I'd need to do is surround what comes after an equals sign with a quote. Easy? Not really. In regular expression jargon, a "lookahead" is setting part of a pattern to look for without actually including that pattern as part of the match. If I had lookaheads, it'd be as simple as something along the lines of:

That one might not work, but it's more or less what I know right now. Like I said, Emacs doesn't support this feature. Writing around that limitation, I had to match for an equal sign, capture a nonwhitespace token, then replace it with an equal sign (because lack of lookaheads) and the captured group surrounded by quotes. Evil Ex command story short when selecting a region:

Quite a mouthful, ain't it? I suppose I'll see what I can do to make this easier.

Yeah, a patch exists or I could have used Perl as a foreign substitution agent, but that's not really helping at the moment. The best step of prevention is by doing things right the first time.

Dear Unix Diary,
Yesterday i installed pass. I decided it was about time i changed all my passwords to something more secure, and after a bit of research i decided to use pass as my password manager. After changing all the usual passwords to randomly generated ones i decided i should also change my root and user pass. However what i failed to think of was that i wouldnt be able to open pass without logging in...

This lead to me needing to boot froma LiveCD(usb) to chroot into my machine and change the usr pass to log in...

Dear UNIX Diary,

Speaking of pass, I finally got email working. Never had any luck using OfflineIMAP. Trying isync grabbed me the wonders. Where does password-store come into play? Exactly what I was dreading with OfflineIMAP: keeping an encrypted password. It's an easy run control setting. Just punch in your PGP passphrase and let fly.

Dear UNIX Diary,

I am going to embark on another epic adventure, installing FreeBSD on my laptop. I've seen mention of it all over the place, especially here. Much like I did with Linux years and years ago, I just adopted it even though I had no idea what I was doing. So day 0, replacing the ssd in my laptop to the new home of my FreeBSD install.

Dear UNIX Diary,

By pure coincidence, I also decided to give FreeBSD a fair shake. I've tried every major (and some minor) Linux distro under the sun, but never a BSD. I'm going to try and make it my daily driver for two weeks and see how it goes, looking forward to stumbling through a new system again.

Next step: an actual Unix. ;->

Dea Unix diary,

I'm still on the quest of discovery to your underlying principles.
The past two weeks I've been studying and researching on the topic of terminals, a huge subject that has a heavy baggage of history.
I'll unpack a summary of it in next week podcast episode, unfortunately I wasn't able to finish it in the past weeks.

The newsletter and podcast projects have taken a big amount of my time but they have been so fruitful, along the way I've learned quite a bunch and I hope I've been able to share some too. The community has been very supportive with feedbacks, corrections, and new insights, I'm grateful for that.

In the next few days I'll try to add the transcript for the most recent episodes that don't have any (The older ones may need to be recorded again).

My first ideas of the podcast and newsletter were for them to be community projects. However the podcast didn't turn out that way. You can still hop in and record your own episode or join or propose subject if you feel like it. As for the newsletter I was hoping we could rotate the person that compiles it together but that also didn't turn out that way.

I've also created a new forums theme, which I hope will make the website more accessible to the outside community. The theme is both light and includes a night mode. It is now the default theme, but that doesn't mean everyone is limited to it, there's still the freedom of choosing whichever theme pleases you.

I relauched the webcasts thread: https://nixers.net/showthread.php?tid=1924 I'm truly looking forward to new posts.

Other than that, I started the eudyptula challenge and it's super fun.

damn it jkl, always making me feel behind the curve. so what's an actual Unix to you? direct and simple please

BSD had started as a distribution of Unix. Technically, it is "Unix-based" (unlike GNU/Linux). However, as the free BSDs had been rewritten completely due to a certain AT&T problem, they don't share much code with "the Unix" anymore. It is debatable if they still are actual Unices.

IBM's AIX and Oracle's Solaris are real Unices though.

right on, thanks for the clarification. so you don't have any direct problems with these adhering to the Unix philosophy, just not actual unix code-based. am i correct?

Dear Unix Diary,

I agree with evbo. looking forward to stumbling around on a new system. and boy am i. X will not run in framebuffer mode. tried everything as described in https://forums.freebsd.org/threads/59727/ still no X... continuing to troubleshoot.

(10-04-2017, 04:22 PM)robotchaos Wrote: you don't have any direct problems with these adhering to the Unix philosophy

I use OpenBSD and FreeBSD quite daily, so I don't have problems with them. I just don't feel well with the mixing of terms. Unix's reputation has largely suffered from non-Unices.