Building code, applying recipes?

Building code, applying recipes? - Printable Version
+- nixers (https://nixers.net)
+-- Forum: Development & Graphics (https://nixers.net/Forum-Development-Graphics)
+--- Forum: Programming On Unix (https://nixers.net/Forum-Programming-On-Unix)
+--- Thread: Building code, applying recipes? (/Thread-Building-code-applying-recipes)

Building code, applying recipes? - freem - 13-08-2020

Hello.
This will be my 1st post here, so hopefully it won't be out of topic or contain other issues.

I am certain most if not all of you already knows what I'll say in the first lines, but I think it is better to (try to) define the context.

When writing programs, especially in compiled languages, one often have to apply sets of rules on sets of files, eventually producing new files, on which again sets of rules might be applied.
The probably most know example in this regard is the generation of an executable binary from C code: usually, ".c" "text" files are compiled to ".o" object files, which are linked together in some binary, but one could also mention the gettext translation system, where some code source is searched for patterns, those patterns being extracted to generate a catalog (a ".pot" file), that translators copy/paste for the language they want to support, and start to change (writing a ".po" file). When the translation is done, it is compiled into a final ".mo" file which will be used by the application.
So, no, it's not limited to binary programs (and there are other uses, too, for example code generators like flex, yacc, coco/R...).

This have to be done by 2 kinds of people: those who creates the program (I will call them "devs"), and those who package or installs it (I will call them "users").
Two different usages, and different needs, too.

The "users" just wan't to have the final binary, without being annoyed with too many steps. They most often use complete "builds", maybe even just delete the source code when they have the binary, since they no longer need the code.
Fast build time is, here, merely a convenience, while being able to *not* have to tinker is an important feature.

The "devs" on the other hands, needs to repeatedly rebuild the code, in order to hunt for bugs, to try fixes, or to implement new features.
Build times here are critical, and usually being able to tinker easily with the build system is also quite important.
Since the need for speed is important, projects are usually not rebuilt from scratch, instead, tools try to only rebuild the files that were changed or depends on the changed files.

Now that there is some context, let name some players in the area, the ones I had to work with (or sometimes, against):

home-made build scripts. Ok enough for very small projects and code snippets, only mentioned because it's possible... I do that often tbh, for my snippets and other junk code.
make (Makefiles): the old one. 1 name, multiple implementations, each one is different. Have implicit rules that may match your needs (but often won't while still getting in the way when you ask it to be verbose, and also depends on implementation). The common feature set is unable to easily handle multiple outputs of 1 rule. It is sometimes written by hand, but most often people use generators to work around those problems (and others). "Devs" can't easily ask it to put intermediate files into specific directories, which results in polluting the source code ones. Same to have multiple configurations (debug build, release build, static analysis, etc). Diagnostic is poor, notably because of if you ask for it, it will go through all the implicit rules, bloating the log.
GNU autotools... another old one, previously the de-facto standard for C and C++ on unix-like systems I believe, generates Makefiles. What to say... it's easy for "users" (as long as things works flawlessly, ofc), but each time I tried to contribute to a project using that thing, if I had to do changes the build system, I gave up. It's complex, bloated, and have a real bad reputation on windows, which is often a target. I'm not sure if it's possible to handle multiple configurations easily, I *really* try to avoid messing with projects that use that... I regularly got missing commands in the middle of the build, and good luck to find which one, since the diagnostic is really poor, being a combination of make and shell tools.
CMake. Again, a generator. One of it's strengths is that it is easy to grasps the basics when hacking someone else's code. Other strengths are, with no particular order: *not* limited to generate Makefiles, good diagnostics, damn easy to have multiple build configurations. Still, it's users tend to try to reinvent pkg-config on unix-like systems, which *SUCKS*. It is often considered the current de-facto standard for C and C++. Oh, forgot that. Even if it's better than autotools, the syntax is still ugly, lot of syntactic sugar.
scons... I have not really tried it, just used it as a user. Claims to be a make replacement, built in python, configuration is, also, in python, which is why I don't even wan't to try it until I'm paid for that: I don't like languages that forces a particular codestyle for one, and second, while other tools use (or favor a lot) declarative, non-turing-complete syntax, which is relatively easy to hack, this one encourages adding complexity to what should be simple. I quote them: "Configuration files are Python scripts--use the power of a real programming language to solve build problems.". Stupid. Oh, and it tries to do the whole job, too. It replaces both Make *and* the autotools/cmake/whatever. Meh.
meson: another one I don't have tried, have interesting selling points, despite being written in a language I don't like (but if I don't have to work with it, it's fine). Honestly, I think it may be a good alternative to CMake, except that it seems to only be able to generate build configs for ninja.
ninja. This one is make replacement, the one I am slowly but certainly starting to love. Why? Because unlike make, it have a very clear and concise documentation. It have no default rules that will bite you. It does one thing, but does it correctly. Unlike make, it supports 2 types of outputs (which can all consist of multiple files) and 3 types of dependencies which include a feature to automatically discover the includes a file depends on (well, actually, it understands the compiler's output). Oh. And it has *versionning* too. It's approach still have some "problems", of course: you must specify files one by one (unlike make which have globbing) and you must write all rules by yourself. Now, this is intended, as they say: "You should generate your ninja files using another program.". That's probably true for big projects, really. But for projects where there are only a handful of files, it's actually pretty nice to write those. TBH I even think it would be possible to write a build.ninja file that would extract the list of files to compile from git (or another (D)VCS) to generate target-specific build configurations.

And you, what do you use, and what do you think about the current state?

RE: Building code, applying recipes? - venam - 14-08-2020

Recipe making is a hard topic. You got to find a way in your tool to be able to specify easily and represent cleanly the tree of dependencies between the steps.

The tool I choose really depends on what I want to achieve. You've mainly listed tools that are used to automate building software, but recipe making isn't limited to software, you can have a recipe to place files in your blog, or to help you rearrange a directory, or to run a series of test units. Jenkins could be a solution to this in a CI environment.

When it comes to building software, you normally have to add more complexity, some language twist to it, some ways to check if libraries are present on the system, to find out on which system you are running, to download the libraries and packages, and more.

Personally, I tend to use Make for C based languages, and otherwise I rely on shell or perl scripts. That is because most often, I don't have to face this problem because the language I rely on already has a way to handle building itself into a package via a file that describes the dependency and environment. Otherwise, if I ever need to set a test environment I'll rely on docker or vagrant or write a small install script which would consist of moving files to certain places.

RE: Building code, applying recipes? - z3bra - 14-08-2020

I mostly (only?) write C, for which I will always write a makefile. It's almost always the same (see safe/makefile as an example), and uses a "config.mk" file to change the build system, with well established macros, so the "users" can customize it according to what they're used to (LDFLAGS, DESTDIR, PREFIX, …).
I write strictly POSIX makefiles, which isn't too hard, and IMO only lacks a way to auto discover source files (for which shell assignement, "SRC != find . -name *.c", works with both GNU and BSD make).

Edit:You can take a look at this link for a good way to write portable makefiles (gopher).

What I like about make is the convenience of its implicit rules. So convenient, that you might not even need a makefile at all !

Code:
$ ls -l

total 4

-rw-r--r-- 1 z3bra users 121 Aug 14 09:27 sqrt.c

$ cat sqrt.c

#include <stdlib.h>

#include <math.h>

int

main(int argc, char *argv[])

{

        return argc > 1 ? sqrt(atoi(argv[1])) : -1;

}

$ make LDFLAGS=-lm sqrt

cc   -lm  sqrt.c   -o sqrt

$ ./sqrt 16; echo $?

4

Thanks to implicit rules, and builtin macros, make knows how to build target "sqrt" without me having to specify a makefile. This is pretty handy, and I use it very often !

Now make is my "marry one" choice. The (IMO) saner alternative that only lacks a wider adoption to rule the world, is plan9's mk. It is make, but simpler. There are not macros (with weird expansion times), no implicit rules (easier to debug), and better shell integration (eg, auto discover source files).

I usually provide an mkfile along with my programs, so I can be the "change I want to see in the world" :)
It can even reuse the makefile's config.mk (if it's strictly POSIX ofc). Here is for example the mkfile for the same project I shared above: safe/mkfile. It ressemble the makefile quite a lot !

Coming from plan9, mk integrates better with the system than make. When make defines "Macros", mk will export them as variables in the environment when building the recipes. This means that mk will NOT expand the variables before passing them to the shell, and it will pass the recipes "as-is", without modifying them. This is boring, predictable, awesome to use. No need to learn a new "language" atop of your shell, and pre-process macros in your head to debug recipes.

I definitely prefer mk to make.

Another option to consider is djb's redo, through apenwarr's implementation.
It's more similar to ninja, and looks appealing, but I must say that it seems to clutter the project with lots of files. I never tried that though, so I don't want to elaborate on the subject. Looks interresting :)

RE: Building code, applying recipes? - jkl - 14-08-2020

I usually use CMake for C/C++ and native tools for other languages. Adoption is the key and CMake works just fine, even on Windows.

RE: Building code, applying recipes? - freem - 17-08-2020

(14-08-2020, 03:20 AM)venam Wrote: Recipe making is a hard topic. You got to find a way in your tool to be able to specify easily and represent cleanly the tree of dependencies between the steps.

The tool I choose really depends on what I want to achieve. You've mainly listed tools that are used to automate building software, but recipe making isn't limited to software

Because it's what I know the most, never had time to dig a lot into others... although I did spent a bit of time on Rex (rexify.org), I must admit I never thought about it like a tool too apply recipes.
Thanks for that thought.

(14-08-2020, 03:20 AM)venam Wrote: Jenkins could be a solution to this in a CI environment.

CI tools... I wonder, how are they different from cmake, make, younameit?

(14-08-2020, 03:20 AM)venam Wrote: Otherwise, if I ever need to set a test environment I'll rely on docker or vagrant or write a small install script which would consist of moving files to certain places.

I know I should learn docker, but I also know I would tend to prefer LXD. And I'm still annoyed by the fact none of those tools are portable AFAIK, since they explicitly rely on a specific kernel's features.
I remember how much I had to relearn when I moved from windows to Debian, and I currently do not even know if I will still use a linux-based system in few years.
Worse, I have the feeling that those tools are (ab)used to justify the lack of documentation about what the "packaged" software really does, how it works, and what it needs as dependencies.
The tools are not in fault, of course, but the spirit I perceive around them annoys me. Maybe I'm wrong. Hopefully I am.

(14-08-2020, 04:48 AM)z3bra Wrote: I mostly (only?) write C, for which I will always write a makefile. It's almost always the same (see safe/makefile as an example), and uses a "config.mk" file to change the build system, with well established macros, so the "users" can customize it according to what they're used to (LDFLAGS, DESTDIR, PREFIX, …).
I write strictly POSIX makefiles, which isn't too hard

I could have stayed with Makefiles too, but could never find a clear documentation. Also, I spent time those last weeks into a compiler-compiler named coco/R, which takes 1 .atg file as input, and generates 4 files as outputs.
I spent hours fighting the tool, reading doc, searching the web about how I could do that, in vain. And I already had written some Makefiles (for a baremetal C project, which imply I had to find info from a mess of code and rare docs, found a nice blog on the way though).
Then I tried ninja. Got the thing understood and under control in few hours, despite 1st time working with it.
But, thanks for the link, I'll take a closer look at it.

Quote:Edit:You can take a look at this link for a good way to write portable makefiles (gopher).

I will, with this proxy :)

Quote:Now make is my "marry one" choice. The (IMO) saner alternative that only lacks a wider adoption to rule the world, is plan9's mk. It is make, but simpler. There are not macros (with weird expansion times), no implicit rules (easier to debug), and better shell integration (eg, auto discover source files).

Everything coming from p9 always seems attractive to me!
If someone knows about an install party for plan9 in normandy, France, don't hesitate and give me the info :D

Quote:Coming from plan9, mk integrates better with the system than make.

What a builder would have me quickly sold to is, DVCS integration. Or at least the possibility to (call "git ls-files \*.c" for example).
Make probably can, actually, since it can summons shell scripts, but I think (I need to confirm this by reading the doc you gave though) it actually lacks some features, that ninja have, but ninja also lacks features... because they intend it to be a tool for tools, which does make sense. At least they do avoid the feature creep, and I think it could be possible to generate ninja files with ninja, to solve the problem. Duh.

Quote:When make defines "Macros", mk will export them as variables in the environment when building the recipes. This means that mk will NOT expand the variables before passing them to the shell, and it will pass the recipes "as-is", without modifying them. This is boring, predictable, awesome to use. No need to learn a new "language" atop of your shell, and pre-process macros in your head to debug recipes.

This does look a little like ninja, tbh. But with probably more features, still. I'll have to take a read on that, too!

Quote:Another option to consider is djb's redo, through apenwarr's implementation.
It's more similar to ninja, and looks appealing, but I must say that it seems to clutter the project with lots of files. I never tried that though, so I don't want to elaborate on the subject. Looks interresting :)

Indeed, it does. Thanks for sharing.

(14-08-2020, 04:58 PM)jkl Wrote: I usually use CMake for C/C++ and native tools for other languages. Adoption is the key and CMake works just fine, even on Windows.

I disagree that cmake works fine. I had tons of issue with it even for normal projects, with pretty well known libraries.
It integrates with *NO* operating system. For windows users, it's the norm so they don't don't notice. For linux devs though, why does not it uses pkg-config by default (instead of needing a 'if unix do somoething'? Would be much more reliable than the builtin system.
Also, it uses absolute paths, which have 3 direct consequences:
1) your login name is written in the binaries you distribute with C and C++, especially if you use __LINE__ and __FILE__ macros (and you should, makes logs a lot more useful)
2) it prevents people to move their folders without recursive removal of the damn build folder
3) prevents reproducible build unlike the cmake developper (the one who wrote the script, minds you) inserts hacks
Also, it's so fucking complex to use than I would never dare to use cmake for baremetal or with non-conventional stuff like compiler-compilers like coco/R. And yes, those are things I do need to do.

Indeed, it's widely adopted. It's also a lot easier to use than autotools. But it still requires way too much work to have an acceptable script to produce binaries, which is imo a proof that no, it does not works fine.
If adoption was really the key, we would all be still using autotools, actually. Or make. Not cmake. Cmake rose because adoption is *not* key, solving problems is the key, and cmake solved more that what was there for many situations, imo. But not all situations.

[edit]
I hope I was not harsh on that last bit... not the intent, but I do remember the time I've lost fighting against that thing a bit too much.
[/edit]

RE: Building code, applying recipes? - jkl - 18-08-2020

Probably my projects weren’t complex enough yet.
I might look into meson/ninja again...

RE: Building code, applying recipes? - z3bra - 23-08-2020

Does anyone have a real experience with redo ?

It looks very interesting, but always felt fairly complex to manage for me.

RE: Building code, applying recipes? - eadwardus - 25-08-2020

I use make ("Makefile" and "config.mk") as it works for me and it's simple when compared to the more usual choices (like autohell). The only alternative (from the ones that i know) that i would consider to use is plan9's mk. There may be interesting tools like ninja(samurai is nice) or redo, but i prefer to have a single static file.

RE: Building code, applying recipes? - venam - 24-07-2021

I've been recently trying to learn meson. While the documentation is scarce, in my opinion, I'm still impressed by how clean the build scripts can be. I don't have much experience with other build systems for C-family languages but so far I like it.

Let's revive this thread as we had a really interesting related discussion on IRC the other day.