Your dream project - Programming On Unix

Users browsing this thread: 3 Guest(s)
z3bra
Grey Hair Nixers
To make this community stronger, I had an idea. Let's all share our "dream projects" (Unix related), and help each others accomplish them!
By dream project, I mean a project you feel you would love to get involved in, but feel like you're not up to the task, or lack resources.

I'll start with mine: I want to setup a decentralized online backup solution with other people, but I don't have anyone to peer with, and don't know which tool to use for this task.
BANGARANG, MOTHERFUCKER
jkl
Long time nixers
Quote: a decentralized online backup solution with other people

So, basically, Resilio Sync (formerly Bittorrent Sync)?

Mine's still a machine to slap people over the internet. My main problem is not how to control the GPIO stuff (that looks pretty easy), but I suck at hardware robotics.

edit: Ha!
z3bra
Grey Hair Nixers
(25-07-2017, 08:38 AM)jkl Wrote:
Quote: a decentralized online backup solution with other people

So, basically, Revilo Sync (formerly Bittorrent Sync)?

Resilio Sync you mean? I use that at work already, and I considered this solution yeah. My main problem with it is that they only provide pre-compiled binaries, and there all users have the same access to the data. I'd prefer a solution with builtin encryption support, so that even if my data is spread amongst 10 servers, I'm still the only one able to access it (given that I don't loose the key of course).
I like the idea of using bittorrent underneath though, as it's pretty efficient network-wise, and fault-tolerant as well. I considered syncthing as well, but many users reported loosing data because of it, which makes it a pretty bad backup tool!

I started cooking something on my own for this purpose (see synk). For now it's just a multihost rsync, but I plan on having it support directory recursion, and non-POSIX systems as well. I would love to implement it over bittorrent, but couldn't find a simple library to use the protocol in C, so for now I keep calling ssh + rsync (and it works rather well for now).
With this solution, encryption needs to be done outside the tool, for example:

Code:
# tar cj /etc /home | openssl enc -e -aes256 -pass file:/etc/secret > /var/backup/$(hostname)-$(date +%Y%m%d).tbz.enc
# synk /var/backup/*.enc -h jkl.nixers.net -h dcat.nixers.net -h xero.nixers.net -h venam.nixers.net

That's no rocket science as you can see, but still require external tools.

(25-07-2017, 08:38 AM)jkl Wrote: Mine's still a machine to slap people over the internet. My main problem is not how to control the GPIO stuff (that looks pretty easy), but I suck at hardware robotics.

Now that's a good idea. I'd had fist bump as well, to keep entertaining the suspense when the arm moves ;)
To go slap only, a servomotor and a rubber band would be enough I guess. Then to slap, you perform a full rotation with the rubber band.

Now you need to create and RFC for that protocol
jkl
Long time nixers
(25-07-2017, 09:14 AM)z3bra Wrote: Resilio Sync you mean?

Ah, sorry. I tried it for a very short time (with the old name) and I lost track of it for years. Just pretend I never wrote "Revilo"!

(25-07-2017, 09:14 AM)z3bra Wrote: My main problem with it is that they only provide pre-compiled binaries, and there all users have the same access to the data. I'd prefer a solution with builtin encryption support, so that even if my data is spread amongst 10 servers, I'm still the only one able to access it (given that I don't loose the key of course).

I see. I'm not sure if it makes sense to have a decentralized protocol for sharing files you don't want anyone to see. Why would anyone want to participate in your backups then?
Maybe a combination of Syncthing and GnuPG encryption would work for you. But I surely don't want to torpedo your idea for the sake of it.

(25-07-2017, 09:14 AM)z3bra Wrote: Now that's a good idea. I'd had fist bump as well, to keep entertaining the suspense when the arm moves ;)

Ha, awesome: Hit them and then fist-bump them for successfully having been hit.

(25-07-2017, 09:14 AM)z3bra Wrote: To go slap only, a servomotor and a rubber band would be enough I guess.

I found out that a rubber band would be a horrible idea as it's not predictable enough. The slapping speed needs to be regulated well or else it could leave serious damage on the opponent which is not what I'd want. (Also, my hardware skills are limited. I wouldn't even know a servo motor if I saw one, so I don't really consider myself to be capable of building something with one.)

(25-07-2017, 09:14 AM)z3bra Wrote: Now you need to create and RFC for that protocol

The protocol itself is ready but undocumented. I'd probably have to read about how to create an RFC first.
z3bra
Grey Hair Nixers
As I said, syncthing seems to loose data over time, which would totally defeat the point of using it for backups. But something similar would definitely rock!
Having your own backups encrypted and saved in different places means that nobody would be able to steal your data, even though you have it stored over different random places over the internet.

A servomotor is the kind of motor you find in these radio commanded cars for kids. Typical electric motor, nothing special about it. You put a switch controlled by the raspi, and switch it on to slap.
r4ndom
Members
(25-07-2017, 08:01 AM)z3bra Wrote: I'll start with mine: I want to setup a decentralized online backup solution with other people, but I don't have anyone to peer with, and don't know which tool to use for this task.

I'm pretty sure this is an easy thing to do with blockchains. I'm currently studying the hyperledger fabric framework, which is a blockchain framework able to easily set up a custom smart contract within (the logic behind the blockchain). It is in addition a permissioned blockchain, so only permitted peers are able to use it. This way you can spread this backup tool between other nixer user's and you can build your own blockchain network with it. The only problem would be the scalability. Since blockchains do always grow in size it would be a terribly big database in a short time if someone decides to backup his movies/music/pictures with it.
z3bra
Grey Hair Nixers
(25-07-2017, 04:06 PM)r4ndom Wrote:
(25-07-2017, 08:01 AM)z3bra Wrote: I'll start with mine: I want to setup a decentralized online backup solution with other people, but I don't have anyone to peer with, and don't know which tool to use for this task.

I'm pretty sure this is an easy thing to do with blockchains. I'm currently studying the hyperledger fabric framework, which is a blockchain framework able to easily set up a custom smart contract within (the logic behind the blockchain). It is in addition a permissioned blockchain, so only permitted peers are able to use it. This way you can spread this backup tool between other nixer user's and you can build your own blockchain network with it. The only problem would be the scalability. Since blockchains do always grow in size it would be a terribly big database in a short time if someone decides to backup his movies/music/pictures with it.

that is an interesting idea indeed, thanks for sharing! I'm more interested in an all purpose backup tool though (my compressed data is about 600Gib), so it would probably not scale well
darthlukan
Members
I've been reading through Window Manager docs and trying to write one myself for a while. Every time I get some headway, I scrap it and start over, never satisfied with my garbage C skills. I want to write it in C because I'm weak in that language (I could have probably already finished it in Python or even Go, but then I wouldn't be learning what I want to learn or building what I want to build). It's so bad, that I haven't even put it up in a Git repo because the history would just be a ton of scrapping commits and deletions.
z3bra
Grey Hair Nixers
(26-07-2017, 02:43 PM)darthlukan Wrote: I've been reading through Window Manager docs and trying to write one myself for a while. Every time I get some headway, I scrap it and start over, never satisfied with my garbage C skills. I want to write it in C because I'm weak in that language (I could have probably already finished it in Python or even Go, but then I wouldn't be learning what I want to learn or building what I want to build). It's so bad, that I haven't even put it up in a Git repo because the history would just be a ton of scrapping commits and deletions.

Writing a WM is a big project, and as such, you first need to organize and plan what your code will be. For such a project, you'll need to be consistent in style, and stick with it until you have something working. Don't change your mind mid-way, or you'll loose a huge amount of time.

Regarding the actual code, don't hesitate to look into other projects, particularly swm, 2bwm and wmutils (we wrote wmutils so that the code is easy to understand and reuse).
There is also libwm which you can use to focus on features rather than dealing with windows themselves.
Tmplt
Long time nixers
Lately I've been wanting a torrent client with the same model of mpd. I currently use rtorrent, but I find the configuration confusing, and having to run it in a detached screen/tmux session feels needless. And I'm not even going to mention the endless trouble I've had when trying to set rutorrent up. Unfortunately I'm quite busy with both uni and other projects, but one day... (it goes without saying that I'd use libgbt).

EDIT: I want this so I would be able to do everything I want through a CLI, but this might also be possible with rtorrent and its XMLRPC interface. I'll have to investigate this further.

EDIT2: I'd aptly name it gbtd, methinks.
grah
Members
(17-10-2017, 03:12 PM)Tmplt Wrote: Lately I've been wanting a torrent client with the same model of mpd. I currently use rtorrent, but I find the configuration confusing, and having to run it in a detached screen/tmux session feels needless. And I'm not even going to mention the endless trouble I've had when trying to set rutorrent up. Unfortunately I'm quite busy with both uni and other projects, but one day... (it goes without saying that I'd use libgbt).

EDIT: I want this so I would be able to do everything I want through a CLI, but this might also be possible with rtorrent and its XMLRPC interface. I'll have to investigate this further.

EDIT2: I'd aptly name it gbtd, methinks.

Have you ever used btpd? I'd say it's very similar to mpd and you can do everything through a CLI.
z3bra
Grey Hair Nixers
btpd is nice! it cannot handke magnet links though, and doesn't do DHT peering iirc.
There were many torrents I couldn't download because of that.

For anyone interested, I started writing a torrent lib (2 actually, one project died):
* libgbt (cc -lgbt ;), now dead project)
* libeech (cc -leech)

none of them is finished, but I might get there someday!
grah
Members
(27-02-2019, 11:45 AM)z3bra Wrote: btpd is nice! it cannot handke magnet links though, and doesn't do DHT peering iirc.
There were many torrents I couldn't download because of that.

Yeah it's a shame it doesn't have those features, I'm surprised more people didn't build onto it because it's a solid core.

(27-02-2019, 11:45 AM)z3bra Wrote: For anyone interested, I started writing a torrent lib (2 actually, one project died):
* libgbt (cc -lgbt ;), now dead project)
* libeech (cc -leech)

none of them is finished, but I might get there someday!

Wow that's impressive! It may not be finished but you sure have put a lot of time and effort into it.
z3bra
Grey Hair Nixers
wolf
Members
I wouldn't call it a "dream" project, but a study roadmap.

I probably have mentioned to some of you that I'm trying to write a wiki system with C. The content ( markdown, images, et cetera ) could be published by some control version system or API ( or both ). A simple but effective ACL on viewing would be nice plus some search system. I also would love to avoid any RDBMS.
I was trying to use inotify(7) to provide automatic build of new documents, but probably I'll get some headaches with recursive directory monitoring and software portability ( especially when using with UNIX ).

I guess it can be done with a toolchain linke BHCS+Manticore, maybe. But my goal is mainly to learn some coding.
Doom
Members
I don't have a "dream roadmap" but am trying to decide on a studying roadmap, as wolf posted:

I'm interested in writing command-line utilities and interfaces, I'm stuck between doing it all in C or Go.

I've picked up C over the last few months. It has helped me improve the way I think about projects. But, there is a lot of things I've come to find inconvenient, particularly with memory management and working with strings.

I switched to Go and thought it was easier. I was productive and finished the utility scripts I'd planned to do in C in an afternoon. Doing it C, however, would've taken me longer.

However, I'm still drawn to C. I use a lot of JavaScript in my day job, and I see the similarities, even writing my JavaScript in C-like fashion. I like how close C is to the metal and how small the executables are in the final product.

tl;dr: I'm stuck between C and Go and would appreciate some guidance.
venam
Administrators
(13-05-2019, 01:21 PM)Doom Wrote: tl;dr: I'm stuck between C and Go and would appreciate some guidance.

In my opinion, choosing between languages before diving into a project usually comes from the experience you got with them before. What you got stuck on, what worked fined, what were the strong points, etc..

If you don't have a clue and haven't done any of those before then try both.
Doom
Members
(14-05-2019, 12:30 AM)venam Wrote:
(13-05-2019, 01:21 PM)Doom Wrote: tl;dr: I'm stuck between C and Go and would appreciate some guidance.

In my opinion, choosing between languages before diving into a project usually comes from the experience you got with them before. What you got stuck on, what worked fined, what were the strong points, etc..

If you don't have a clue and haven't done any of those before then try both.

Thanks, venam.

I've actually worked on two projects in both languages. Using Go, I finished the projects relatively quickly, with the ability to write tests on the fly.

Using C, the projects took longer but, for some reason, I felt a bigger sense of accomplishment. Maybe because I had to worry about more details and fixing bugs required more thinking.
evbo
Members
(14-05-2019, 01:56 PM)Doom Wrote: Using Go, I finished the projects relatively quickly, with the ability to write tests on the fly. ... Using C, the projects took longer but, for some reason, I felt a bigger sense of accomplishment. Maybe because I had to worry about more details and fixing bugs required more thinking.

Ah, the age old question of development speed vs program speed :)

With experience you should be able to determine what language and libraries you'll need before you start. Many times a modern managed language with a good standard library is the right choice for your project. I'm partial to Go and C# but that's just me.

I guess the reverse question you can ask yourself is whether Go's disadvantages (runtime, garbage collection, etc) are going to hinder your project in a way that C wouldn't.
Doom
Members
(15-05-2019, 01:55 PM)evbo Wrote: I guess the reverse question you can ask yourself is whether Go's disadvantages (runtime, garbage collection, etc) are going to hinder your project in a way that C wouldn't.

Interesting. The only thing that is hindering when working with Go is the requirement to not waste a variable. While that's great, it becomes a pain when the project grows. I don't have that problem in C.

***

In terms of libraries and such, Go blows C away, with its ease of installing and importing modules when I need them.