f87cc3a Switches template to use the function form of mkDerivation.
~remexre pushed to ~remexre/clnix git
An alternative packaging for Common Lisp packages in Nix.
Philosophy:
stdenv.mkDerivation as possible.Non-Goals:
aarch64-darwin, aarch64-linux, i686-linux, x86_64-darwin, and x86_64-linux.
However, I personally pretty much only use aarch64-linux and x86_64-linux, so these platforms are the only ones getting significant testing.
This may improve once I set up a Hydra install with access to aarch64-darwin and x86_64-darwin builders; it'd be nice to build the world every commit.
Patches to improve support for other platforms are welcome on the mailing list.$out layout.clnix should probably be considered experimental until it's been able to build all of Quicklisp on all the platforms the flake is available for, on at least two Lisp implementations, for at least 3 months or so. If you choose to use it anyway, bug reports would be extremely welcome on the mailing list.
I'm currently using clnix to build my thesis work, and I plan to support it through at least 2023-01-01.
As of 2022-01-23, clnix has the following issues on the 5050 systems in Quicklisp on x86_64-linux:
This is a 72% build success rate overall.
By comparison, nixpkgs currently has 252 systems in pkgs/development/lisp-modules/quicklisp-to-nix-systems.txt.
clnix builds all of these on aarch64-linux and x86_64-linux.
On aarch64-linux, the numbers are 2, 91, 438, 958, and 71%; and all the systems from quicklisp-to-nix-systems.txt can be built.
More investigation is needed as to which systems fail here in particular.
$ nix repl
Welcome to Nix 2.24.12. Type :? for help.
nix-repl> :lf sourcehut:~remexre/clnix
Added 14 variables.
For each implementation we support, there is an appropriately-named package.
nix-repl> sbcl = outputs.packages.aarch64-linux.sbcl
nix-repl> sbcl
«derivation /nix/store/0842iy3fdq4f15j6r4jvmxkbx6z1v2ks-sbcl-2.4.8.drv»
The implementations are the nixpkgs versions, patched to use recent versions of ASDF.
Each implementation then has a packages attribute, which is a scope (like pythonPackages in nixpkgs) that contains a Nix package for each Lisp system (not project!) on Quicklisp.
Colloquially this is called lispPackages (despite not being an attribute with that exact name).
Some other projects not on Quicklisp are packaged as well; patches to add more are welcomed.
nix-repl> sbcl.packages.alexandria
«derivation /nix/store/x55mdbg8apnwkc5qdbrq5wiib1d0gwxd-alexandria-20231021-git.drv»
nix-repl> sbcl.packages.coalton
«derivation /nix/store/8c6dz8zab84p9kccniiz2ys0kfq3h38g-coalton-20220526-git.drv»
Since lispPackages is a scope, one can use callPackage with it, which behaves as one would expect.
See the Recipes section below for examples of its use.
nix-repl> sbcl.packages.callPackage
«lambda callPackageWith @ /nix/store/v4vrwzcjk2aa9nv7l22j157l6v3dd0vn-source/lib/customisation.nix:128:31»
A mkDerivation wrapper is also provided.
This provides helpful defaults for various phases to try and "do the right thing" when building a system using ASDF.
See the Recipes section below for examples of its use, or lisp-packages/make-derivation.nix for its definition.
nix-repl> sbcl.packages.mkDerivation
{
__functionArgs = { ... };
__functor = «lambda __functor @ /nix/store/v4vrwzcjk2aa9nv7l22j157l6v3dd0vn-source/lib/trivial.nix:430:19»;
override = { ... };
}
A simple helper binary, clnix-swank-server, is provided on the swank attribute of each implementation.
This is also provided in lispPackages.
nix-repl> sbcl.swank
«derivation /nix/store/kr6vwk81g604yqqm3rkrfxc4dlk9lc80-clnix-swank-server.drv»
nix-repl> sbcl.packages.clnix-swank-server
«derivation /nix/store/kr6vwk81g604yqqm3rkrfxc4dlk9lc80-clnix-swank-server.drv»
In flake.nix, or anywhere else:
let sbcl = clnix.packages.${system}.sbcl;
in sbcl.packages.callPackage ./my-package.nix {};
In my-package.nix:
{ alexandria, closer-mop, iterate, mkDerivation }:
mkDerivation {
pname = "my-package";
version = "0.1.0";
src = ./.;
propagatedBuildInputs = [ alexandria closer-mop iterate ];
};
This derivation will compile the ASDF system my-package, which should be defined in my-package.asd inside the src.
If you have multiple systems to compile, or the system is named something different than the derivation, you can use the asdfSystemNames attribute.
This is a list of strings, where each string is the name of a system to be compiled.
The mkDerivation wrapper passes any arguments it doesn't understand along to stdenv.mkDerivation, so it's fine to pass whatever else you want.
Any systems that specify a :build-operation of :program-op are automatically compiled to binaries.
They should also specify :entry-points.
The resulting binaries are in $out/bin, with the same name as the system.
The clnix-swank-server helper binary can be used to start a Swank server.
An example of using it:
{ alexandria, clnix-swank-server, closer-mop, iterate, mkDerivation, mkShell }:
mkDerivation (self: {
pname = "my-package";
version = "0.1.0";
src = ./.;
propagatedBuildInputs = [ alexandria closer-mop iterate ];
passthru.devShell = mkShell {
inputsFrom = [ self ];
nativeBuildInputs = [ clnix-swank-server ];
};
})
This uses the form of mkDerivation that accepts a function, and passes the result of building the package to itself.
This allows defining the devShell without making our own (circular) let binding.
After doing nix develop .#my-package.devShell, you can run clnix-swank-server from the directory my-package.asd is in.
This should start a Swank server on port 4005 that is able to load my-package and any of its dependencies.
The clnix-swank-server binary takes a list of packages to load, if you want to load them immediately rather than doing so from your editor.
Can't create directory /homeless-shelterSomething tried to create a directory inside the home directory.
The right solution is probably to mess with ASDF_OUTPUT_TRANSLATIONS.
This variable controls where the results of building sources go.
See the ASDF manual for more details about it.
An example postConfigure to fix this might be like:
postConfigure = ''
ASDF_OUTPUT_TRANSLATIONS="/tmp/extra-srcs:$out/lib:''${ASDF_OUTPUT_TRANSLATIONS:-}"
'';
Note that:
addToSearchPath.
We typically want to prepend to ASDF_OUTPUT_TRANSLATIONS rather than append.
The older entries (added by dependencies' setup hooks) should be later, since if a dependency defines a system that we also define, ours needs to be taken to avoid ASDF attempting to write compiler output to the dependency's directory in the Nix store.ASDF_OUTPUT_TRANSLATIONS are to continue inheriting configuration.
If any exists, we probably want it!
(This ought to be pretty rare when building packages, but might occur when a user is developing.)Also consider that the source files probably need to be moved to somewhere under $out to be read later; if so, a similar tweak to the above belongs in the setup hook.
Quicklisp's design does not require that the transitive closure of the project dependency relation be a DAG, and indeed, there are several circular dependencies. This mostly occurs because of test systems.
Older versions of clnix tried to explicitly support building multiple projects together to avoid the circular dependencies. However, there's a huge circularity containing many commonly-used packages.
Actually building them this way would result in pulling in a large number of native-code dependencies and having a very long compile in order to use any of them.
Additional care would also be required to split them back out after building, to avoid e.g. cffi having a dependency on libmysqlclient.so!
Implementations are rather conservative in updating ASDF.
Recent versions of ASDF are necessary in order to have features like package-local-nicknames and package-inferred-system work together without drastic hacks that make asdf:find-system side-effecting.
Every system in Quicklisp already depends on asdf, so a more recent version would get loaded anyway as soon as a dependency is loaded.
Patching the implementation to use the same version the Quicklisp-provided systems would use just makes the overall developer experience more consistent.
scripts/ should work inside the devShell.
I use direnv, but nix develop ought to work too.scripts/update-distinfo.pl and written to a JSON file of the form quicklisp/dist-$version.json.
This is then symlinked to quicklisp/dist-latest.json.
We don't yet normalize the JSON to make diffing easier, patches welcome on the mailing list.quicklisp/hashes.json contains the translation.
Run scripts/update-hashes.pl to add every hash mentioned in a JSON'd distinfo file to it.
Note that this entails fetching the referenced files, so it might be somewhat slow.passthru.tests?The code in the clnix repo is offered under the CC0 license. Note that this does not affect the license of any derivation this flake exports that comes from external code; they each have their own licenses.