20

I used to be a Java developer for a long time, but recently, I joined a Haskell team. In the java world, if you have a large project, with several teams working on it, a common approach is to use an artifact server such as Maven to ease and speed-up the development. Numerous build tools, such as Ant, Maven, Gradle, can build the project and upload a jar file to the artifact server that can be used by the rest of the team without pain. Therefore, by splitting the project into smaller sub-projects, the build time is also drastically reduced.

On the Haskell side, we are using cabal to build the project. Our project takes about 10-15 minutes to build without optimization on. It takes a few hours if compiler optimization is turned on, which is painful.

I wonder, how we can do the same thing as we do in Java here. Is there an easy way to compile and upload the binary of the packages (libraries) to an artifact server and use the prebuilt binaries at the build time? I know that since Haskell generates machine code (rather than byte code in Java) there might be compatibility issues, but we can probably have different binaries for different architectures/OSs stored on the artifact server.

Oxy
  • 309
  • 1
  • 4
  • Not an answer to your question but, do you invoke "cabal build/install" with the -j option? – Daniel Díaz Carrete Mar 05 '15 at 21:30
  • Yes, but I don't see much speedup. CPU usage is around 15%-20% when building. I am not sure where the problem is: `cabal`, `GHC`, `Test.Framework` or the linker. – Oxy Mar 05 '15 at 22:13
  • According to this SO answer http://stackoverflow.com/questions/27173910/why-does-cabal-download-and-compile-from-source the main reason is the native nature of Haskell executables. – Daniel Díaz Carrete Mar 06 '15 at 07:20
  • I am mostly talking about development. It is true that binary may cause incompatibility because of different OSs, CPU architectures and even compiler versions. But, why should I waste so much time when I am developing on the same machine, the same OS and the same compiler every day? – Oxy Mar 06 '15 at 18:46
  • It looks like Halcyon with a cache might be useful: https://halcyon.sh/reference/#private-storage-options – Lubomír Sedlář Mar 06 '15 at 19:09
  • Also not an answer but did you not know that `ghci` can compile to bytecode? Re: [the repository](https://github.com/ghc/ghc/tree/master/compiler/ghci) – Francesco Gramano Apr 26 '15 at 21:46
  • Have you considered using containers and building images of applications on consistent environments? If you have a production environment (some *nix OS), you can create images with all the build tools for your package in that environment and have people use virtualization (like Vagrant) to build and run. That may be able to handle the portability issues of compiling machine code once and distributing the image on a private registry of sorts. Not really Haskell specific but a possible solution. – theWanderer4865 Jul 11 '15 at 14:10

1 Answers1

2

You might consider using Nix, which is a general-purpose, cross-platform package manager with decent support for Haskell.

Nix has a custom programming language for defining packages (which just so happens to be pure, functional and lazy). Defining new packages and extending existing ones is pretty easy (eg. to alter dependencies, get the source from a different git repo, etc.).

Packages are identified by hashes, which include dependencies. Hence multiple versions, or the same version with different dependencies, can live side by side without conflict. Nix can look up the desired hash on a "binary cache" server, to see if that particular package with those particular dependencies has already been built; if so, it'll download the build product rather than compiling.

Currently, the nixpkgs repository includes most/all of Hackage, several GHC versions (7.10.1, 7.8.4 and some JS backends) and a cabal2nix utility which does a pretty good job of generating Nix packages from .cabal files. There's also the Nix-based "hydra" CI server, which you could use to trigger builds based on SCM commits.

Warbo
  • 1,205
  • 7
  • 11