24

I've recently been using some build tools for a Nodejs project at work when I realized that most languages' main build tool/system use a different language than the underlying programming language itself.

For example, make does not use C or C++ to write scripts and ant (nor Maven) doesn't use Java as their language for scripting.

Newer languages like Ruby do use the same language for build tools like rake, which makes sense to me. But why hasn't this always been the case? What't the advantage to having a build tool that uses a different language from the underlying language?

joshin4colours
  • 3,678
  • 1
  • 24
  • 37
  • 14
    Maybe the general-purpose language isn't a good enough fit for the specialist usage of the tool? For example, I wouldn't want to write make files in C! Sometimes a DSL is better. – Andres F. Sep 21 '15 at 18:16
  • 3
    Have you considered the joy of trying to writing a build tool/script in C? or Java? –  Sep 21 '15 at 18:17
  • @AndresF. good point. On the other hand, if I know C well, but want to use a build tool, now I have to learn a second language. Why not just use one? – joshin4colours Sep 21 '15 at 18:21
  • @joshin4colours Consider writing a program that finds and uses the correct `${CC}` for the environment. Do you have vendor cc? or gcc? or clang? or clang pretending to be gcc? Make sure that you don't recompile that .o file if the dependencies are older than it. Invoke lex (flex or bison) with a specified file, get the lex.yy.c file and compile it... Sure, if you're just doing `system("cc foo.c");` that's one thing. If you are doing a complex build with system level dependencies and different dependencies based on the libraries available on the system that's quite another. –  Sep 21 '15 at 18:29
  • 13
    Look at it from another angle: Why not use make to write application software? – whatsisname Sep 21 '15 at 18:31
  • 4
    [This article](http://www.defmacro.org/ramblings/lisp.html) is primarily about what the author believes makes Lisp great, but has some relevant information on why Ant uses XML instead of Java. – 8bittree Sep 21 '15 at 18:34
  • 2
    @MichaelT: `Have you considered the joy of trying to writing a build tool/script in C? or Java?` Have you considered the horrendous mess that you have to twist XML into when a non-trivial build contains requirements best expressed with imperative logic constructs? – Mason Wheeler Sep 21 '15 at 18:52
  • 6
    If you wanted to write a program in C that automates building C programs, wouldn't you just end up writing `make` again (which is implemented in C anyway)? – Brandin Sep 21 '15 at 18:59
  • 1
    @MasonWheeler Or you use make (don't *need* to use ant for builds) - though modern Java without maven-esque dependency downloads is Not Fun (tm). Could always use Gradle (and Groovy) if you want a scripting language based build script. –  Sep 21 '15 at 19:01
  • @Brandin exactly, but that's my question: why wasn't make written in C in the first place? (for example) – joshin4colours Sep 21 '15 at 19:27
  • 6
    @joshin4colours make *is* written in C. The language that make uses, however, is a declarative language that invokes the shell as needed. –  Sep 21 '15 at 19:38
  • 3
    There's this old saying that a lathe is the only tool in a workshop that can build itself, nowadays it's the 3D printer that can print itself (to some degree). Not many tools have that ability, but even if they don't, they are still great tools nonetheless. The question is what's the advantage if a language is not well suited to build itself? Does that compromise its other purposes? – null Sep 21 '15 at 20:15
  • What happens when the project has multiple Things? C library AND java? – Eris Sep 22 '15 at 00:39
  • 1
    Because if you write your build system in C, then you would want to write a build system, that builds your build system and then you would want to write... – Alexander Shishenko Sep 22 '15 at 08:06
  • Those two things are not necessarily related. Is there a good reason that they must be in the same language? It's the same logic as compilers are not necessarily written in the same language as the language being compiled – xji Oct 01 '15 at 08:07
  • @JIXiang Sometimes it's very handy to reuse code being built already at build-time. See for example [iwant-demo](https://github.com/wipu/iwant-demo), a demonstration of my build tool [iwant](http://iwant.sourceforge.net) that lets you define builds in fluent java. – Ville Oikarinen Jun 01 '17 at 06:06

5 Answers5

36

The choice of what programming language to use to get anything done must depend on the specific features of that goal and not on the other tasks related to that project.

A build tool does a very specific job, and no matter what language you're using for the main project, a build tool is a software by itself.

Trying to tie the main project and its build tool could be a very bad decision. What you should need with a build tool is mainly a fast process of simple rules, with a fast management of files and IO.

If languages like ruby do use themselves to implement their build tool, is related more on the features of that language than on the assumed need to keep everything written with the same language.

Nicolás
  • 621
  • 6
  • 10
18

It's not impossible to write a build tool that uses a language such as C or Java as their scripting languages. However, build tools benefit from using more declarative scripting languages. Imagine the pain and boilerplate of writing a makefile (or build.xml, or whatever) in C.

Build tools benefit from the use of DSLs, a task for which C isn't a particularly good choice. C is too general for a build tool, and too concerned with error-prone & low-level details. For example, you do not want to be concerned with explicit memory-management in a build tool! So even if using C-like syntax, you'd probably be using garbage collection in your scripting tool, at which point why not simply use a dedicated DSL?

It makes sense that Ruby can be used for this, since it's a higher-level, more declarative language than C or Java. Also see: Gradle, sbt.

Andres F.
  • 5,119
  • 2
  • 29
  • 41
  • 13
    `Imagine the pain and boilerplate of writing a makefile (or build.xml, or whatever) in C.` Imagine the pain and mess of trying to express non-trivial conditional logic (you know, standard imperative stuff) in a declarative XML script. Oh, wait, you don't have to imagine; you've probably had to deal with it, and it was probably a mess and a half, wasn't it? Builds are one of the worst possible choices for a declarative scripting language, because serious problems quickly end up butting up against the limits of declarative-ness, and the ones that don't are simple enough to not need a build tool. – Mason Wheeler Sep 21 '15 at 18:57
  • 8
    @MasonWheeler There are cases where existing build tools have made me suffer, yes. *None* of them would have been helped by using a low-level C-like imperative language. I see something like Ruby as probably ideal: declarative and imperative enough, as needed. C would give me nightmares for this task. To me build systems are a good use case for (mostly) declarative DSLs: you care about *what* to do, not about *how* to do it (most of the time). – Andres F. Sep 21 '15 at 19:02
  • 2
    Fair enough. I've always figured C is one of those "if X is the answer you're asking the wrong question" technologies, to be perfectly honest; it's just that working with XML "scripts" has been a nightmare every time I've had to dive into one. I agree that a higher-level imperative language with DSL capabilities would be ideal. – Mason Wheeler Sep 21 '15 at 19:05
  • @AndresF.: Things like make files require a mixture of approaches. The desired output state is specified declaratively, but the actions necessary to achieve that state are specified imperatively. – supercat Sep 21 '15 at 20:39
  • @supercat Sure, I do not dispute that make files are hybrid systems. Hence "more declarative/high-level", not "exclusively declarative". – Andres F. Sep 21 '15 at 20:46
  • @MasonWheeler I totally agree - having had the nightmare that is a XAML-based declarative build script when I wanted to edit a TFS build, I quickly realised why Jenkins and its "just run a batch script" is vastly superior. And smaller. And easier to maintain (ie possible to maintain). Declarative approaches may seem ideal for 1 or 2 steps but always end up in a horror of nested declarations if you have anything more complex (ie practically everything in the real world) – gbjbaanb Sep 22 '15 at 09:18
  • @MasonWheeler: purely declarative languages will hit problems in build systems, *and* XML-based languages often hit problems for all tasks. XSLT is what convinced me that my original position, "a programmer spends such a tiny proportion of the time typing, that verbose languages aren't problematic for that reason" is not strictly true. So declarative XML build systems hit a double whammy. – Steve Jessop Sep 22 '15 at 09:46
  • @SteveJessop: I would agree with your original position, actually. The problem with XML isn't that it's verbose *per se,* but that it's filled with required syntactical elements that *add no useful information,* and thus detracts from human-readability rather than enhancing it. A lot of times a verbose programming language says stuff that a smart compiler can infer and isn't strictly necessary, but having that information around still makes the code it easier for a human to read. But XML's close tag names just turn it into messy tag soup, and a pretty-printer makes them redundant. – Mason Wheeler Sep 22 '15 at 09:54
  • 1
    @MasonWheeler I think that is as much a problem of the people who designed the config language than XML itself. The common mistake by authors of such languages is to assume that just because something is in XML, it will automatically be human-readable. (I know I've made this mistake more times than I'm comfortable with. It's so tempting.) A well-designed and lean configuration language can work just as well over XML as in any other form. – biziclop Sep 22 '15 at 10:46
12

Let’s give you a real world example.

About 15 years ago I worked on porting a large system written in C from Unix to Windows, it was about 3 million lines of code. To give you some idea of scale, it took over 24hr to compile on some of our unix systems (RS6000), windows could compile the system in about 4hrs.

(We also had 2 million line of code in our own interpreted language, but decided not to use our language for the build systems as it was never designed for file processing. Also we needed a build system to compile the C code that implemented our language.)

At the time the build system was written in a mix of shell scripts and make files, these were not portable to windows – therefore we decided to write our own build system.

We could have used C, however we decided to use python, there was few reasons. (We also rewrote our source code control system in python at the same time, this is was very intergraded with the build system, so object files for checked in modules could be shared by developers.)

  • Most of our code could be built with a few simple rules (only a few thousand lines of python for all platforms, Windows, VMS, and 6 versions of Unix) that were drived from the naming conventions of the files.

  • At the time RegEx was not very standard between C systems on different platforms, Python had built in RegEx.

  • A few modules needed custom build steps, Python allowed class files to be loaded dynamically. We allowed a custom class to be used to build a module (lib), based on having the python file with a magic name in the folder. This was the killer reason to use python.

  • We considered Java, but it was not shipping on all the platforms at that point.

(The UI for our source code control system used a web browser as that was portable across all the platform. This was 6 months before we had a internet connection. We had to download the browser over X25!)

Ian
  • 4,594
  • 18
  • 28
  • Having worked on several bigish systems I sometimes feel like I have a handle on how development scales. Then I read stories like this. Sheesh. – dmckee --- ex-moderator kitten Sep 21 '15 at 22:22
  • @immibis This was the fashionable thing to do 15 years ago. Even more so, this practice was actually endorsed and encouraged by the leading Unix vendors (SGI and DEC in particular, but IBM too). – oakad Sep 22 '15 at 05:56
  • @oaked I just mean it's typo'd as "Unit". (Hence the bold "x") – user253751 Sep 22 '15 at 06:01
  • Let’s add, there was 4 main products built of this code base along with a handful of custom solutions for large customers. Each product or custom solution had different versions of 10s of modules along with 10s of modules that was only used in it. The build system controlled what version of each module was used for each product. (I think there was about 300 modules in all, so even deciding on the version fo each module that when into a give version of a given product was hard.) – Ian Sep 22 '15 at 07:56
  • As to porting to windows. A customer could put a unix work station AND a windows 95 PC on each desk, or just a NT machine. At the time Intel based systems were about a 1/4 of the price of unix systems. The linux project was only just starting. – Ian Sep 22 '15 at 08:01
  • This was DESKTOP software GIS, imaging processing, map editing, road traffic modeling, mobile phone coverage, etc. – Ian Sep 22 '15 at 08:03
  • 1
    People tend to forget that Unix used to mean "expensive". Windows won the desktop/workstation war by being cheaper. It's for the same reason that Linux later won the server war. – slebetman Sep 22 '15 at 08:33
  • 1
    If I want a computer to run software I have written myself, then I want it be flexible and have 101 options how the kernels is compiled etc. However if I am selling software to be installed on the customer’s computer, I want the customer to be running a OS that is not flexible, so that I know it will work on their computer if it works on mine. That was a big factor when looking at Linux as a software vendor - support just looked too expensive. Linux has won the **hosted** server software wars, where the server is provided by the software vendor – e.g. most web deployed software. – Ian Sep 22 '15 at 08:54
  • @Ian nobody compiled custom kernels for AIX. Even with Linux, you can compile your kernel if you want... but hardly anyone who uses Linux does. Don't listen to the powerhackers as representative of 99.999% of Linux users. Why do you think there are long-term-support versions of distros like Redhat and Ubuntu, and locked-down versions like Chromebooks. – gbjbaanb Sep 22 '15 at 09:21
  • @gbjbaanb, agreed today, but go back even only 10 years and there were too many distros to choose from and each customer expected you to support the one they had chosen. Once you sold a system to a customer on a given distro, they expected surport for new versions of that distros over the next 5 or 10 years, even if new customers wanted a different distros. – Ian Sep 22 '15 at 09:42
  • 3
    @slebetman It's only fair - Unix won the previous "war" by being cheaper (compared to the LISPM's and similar machines). It was absolutely horrid, awfully designed and crashing constantly (but it boots up so much faster than a LISPM! :P), using C as primary programming language (quite a high fall from LISP and similar), but it was much cheaper. In fact, many adopted it because it was *free* (there's been many free mutations long before Linux). In fact, I've met lots of old-school LISPers who preferred *MS-DOS* to unixes of the time. Yes, *that* horrible. – Luaan Sep 22 '15 at 13:00
3

The short answer to this question is that this is what a build tool is. A tool that automates the use of other tools, with the aim of building an end product from some source files. If we're writing the build script in the same language as the product itself, we're into the realms of language-specific tools, which would not be considered build tools in the same way.

This raises the question -- why don't compilers provide the sort of build automation that we're used to from tools like make? To which the answer is simply because make exists. The Unix philosophy of "do one thing, and do it well" suggests that it is not the responsibility of the compiler to provide this*. That said, I have personally been through my share of make headaches, and would be interested to try a compiler that provides it's own quote-unquote "native" build system.

For an example of a compiler designed this way, see Jon Blow's (as yet unreleased) Jai, a language intended as a C++ replacement. Links to talks and demos of the compiler are collected here. This language compiles to byte-code that runs within the compiler, with compilation to binary being a standard library provided function, accessible from the byte-code. The intention is to allow for both build automation and meta-programming within the language's native syntax.

* A look at Microsoft's Visual Studio will show you an example of the opposite philosophy. :)

Haddon CD.
  • 31
  • 1
2

The build tool is first and foremost a tool, just like 'gcc', 'ls', 'awk' or 'git'. You feed the tool an input (filename, parameters, etc) and it will do some stuff accordingly.

All of these tools allow you to pass in your input in ways that should be simple enough to write and understand. In the particular case of build tools, the input is a recipe file, a file that describes how your project goes from the source files to the finished product.

If your recipe is as simple as passing in the folder that contains the project and the compiler to use, then a declarative "language" is enough. Once the recipe becomes more and more complicated, with more logic, it becomes troublesome to handle it with declarative languages and more general purpose languages are used.

It should now be apparent that there is no connection between the language used to write your build recipes and the languages used to write your product code simply because they have different purposes and requirements. There is no connection even between the language the build tool is written in and the "language" the build recipes should be written in. Always use the best tool for the job!

Tibos
  • 139
  • 3