397 lines
11 KiB
Markdown
397 lines
11 KiB
Markdown
This page contains basic and advanced examples for using mach-nix inside a nix expression
|
|
|
|
|
|
<!--ts-->
|
|
* [Import mach-nix](#import-mach-nix)
|
|
* [mkPython / mkPythonShell](#mkpython--mkpythonshell)
|
|
* [From a list of requirements](#from-a-list-of-requirements)
|
|
* [Include packages from arbitrary sources.](#include-packages-from-arbitrary-sources)
|
|
* [buildPythonPackage / buildPythonApplication](#buildpythonpackage--buildpythonapplication)
|
|
* [Build python package from its source code](#build-python-package-from-its-source-code)
|
|
* [buildPythonPackage from GitHub](#buildpythonpackage-from-github)
|
|
* [buildPythonPackage from GitHub with extras](#buildpythonpackage-from-github-with-extras)
|
|
* [buildPythonPackage from GitHub and add requirements](#buildpythonpackage-from-github-and-add-requirements)
|
|
* [buildPythonPackage from GitHub (explicit source)](#buildpythonpackage-from-github-explicit-source)
|
|
* [buildPythonPackage from GitHub (manual requirements)](#buildpythonpackage-from-github-manual-requirements)
|
|
* [Simplified overrides ('_' argument)](#simplified-overrides-_-argument)
|
|
* [General usage](#general-usage)
|
|
* [Example: add missing build inputs](#example-add-missing-build-inputs)
|
|
* [Overrides (overrides_pre / overrides_post)](#overrides-overrides_pre--overrides_post)
|
|
* [Include poetry2nix overrides](#include-poetry2nix-overrides)
|
|
* [Tensorflow](#tensorflow)
|
|
* [Tensorflow with SSE/AVX/FMA support](#tensorflow-with-sseavxfma-support)
|
|
* [Tensorflow via wheel (newer versions, quicker builds)](#tensorflow-via-wheel-newer-versions-quicker-builds)
|
|
* [PyTorch](#pytorch)
|
|
* [Recent PyTorch with nixpkgs dependencies, and custom python](#recent-pytorch-with-nixpkgs-dependencies-and-custom-python)
|
|
* [JupyterLab](#jupyterlab)
|
|
* [Starting point for a geospatial environment](#starting-point-for-a-geospatial-environment)
|
|
* [Docker](#docker)
|
|
* [JupyterLab Docker Image](#jupyterlab-docker-image)
|
|
* [R and Python](#r-and-python)
|
|
* [Raspberry PI / aarch64 SD Image](#raspberry-pi--aarch64-sd-image)
|
|
|
|
<!-- Added by: grmpf, at: Mon 12 Oct 2020 01:25:02 AM +07 -->
|
|
|
|
<!--te-->
|
|
|
|
### Import mach-nix
|
|
every mach-nix expression should begin like this:
|
|
```nix
|
|
let
|
|
mach-nix = import (builtins.fetchGit {
|
|
url = "https://github.com/DavHau/mach-nix/";
|
|
ref = "refs/tags/3.0.2";
|
|
}) {
|
|
# optionally bring your own nixpkgs
|
|
# pkgs = import <nixpkgs> {};
|
|
|
|
# optionally specify the python version
|
|
# python = "python38";
|
|
|
|
# optionally update pypi data revision from https://github.com/DavHau/pypi-deps-db
|
|
# pypiDataRev = "some_revision";
|
|
# pypiDataSha256 = "some_sha256";
|
|
};
|
|
in
|
|
...
|
|
```
|
|
|
|
### mkPython / mkPythonShell
|
|
#### From a list of requirements
|
|
```nix
|
|
mach-nix.mkPython { # replace with mkPythonShell if shell is wanted
|
|
requirements = builtins.readFile ./requirements.txt;
|
|
}
|
|
```
|
|
|
|
#### Include extra packages.
|
|
`packagesExtra` accepts:
|
|
- python packages built via `mach-nix.buildPythonPackage`.
|
|
- R Packages from `nixpkgs.rPackages` (see R example further down)
|
|
- python package source trees as paths or derivations
|
|
- URLs pointing to a tarball archives containing a python source tree
|
|
```nix
|
|
...
|
|
mach-nix.mkPython {
|
|
requirements = builtins.readFile ./requirements.txt;
|
|
packagesExtra = [
|
|
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
|
|
./some/local/project # from local path
|
|
mach-nix.buildPythonPackage { ... }; # from package
|
|
];
|
|
}
|
|
```
|
|
Alternatively, if requirements are not needed, packagesExtra can be passed directly to mkPython
|
|
```nix
|
|
...
|
|
mach-nix.mkPython [
|
|
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
|
|
./some/local/project # from local path
|
|
mach-nix.buildPythonPackage { ... }; # from package
|
|
]
|
|
```
|
|
|
|
### buildPythonPackage / buildPythonApplication
|
|
These functions can be used to manually build individual python modules or applications. Those can either be used directly, or fed as `packagesExtra` of `mkPython`.
|
|
Whenever `requirements` are not explicitly specified, they will be extracted automatically from the packages setup.py/setup.cfg. The same goes for the `name` and `version`.
|
|
#### Build python package from its source code
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage /python-project-path
|
|
```
|
|
|
|
#### buildPythonPackage from GitHub
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage "https://github.com/psf/requests/tarball/2a7832b5b06d"
|
|
```
|
|
|
|
#### buildPythonPackage from GitHub with extras
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage {
|
|
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
|
|
extras = "socks";
|
|
}
|
|
```
|
|
|
|
#### buildPythonPackage from GitHub and add requirements
|
|
Use `requirementsExtra` in case the auto detected requirements are incomplete
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage {
|
|
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
|
|
requirementsExtra = "pytest";
|
|
}
|
|
```
|
|
|
|
#### buildPythonPackage from GitHub (reproducible source)
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage {
|
|
src = builtins.fetchGit{
|
|
url = "https://github.com/user/projectname";
|
|
ref = "master";
|
|
# rev = "put_commit_hash_here";
|
|
};
|
|
}
|
|
```
|
|
|
|
#### buildPythonPackage from GitHub (manual requirements)
|
|
Use this if automatic requirements extraction doesn't work at all.
|
|
```nix
|
|
...
|
|
mach-nix.buildPythonPackage {
|
|
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
|
|
requirements = ''
|
|
# list of requirements
|
|
'';
|
|
}
|
|
```
|
|
|
|
|
|
## Simplified overrides ('_' argument)
|
|
### General usage
|
|
```
|
|
with mach-nix.nixpkgs;
|
|
mach-nix.mkPython {
|
|
|
|
requirements = "some requirements";
|
|
|
|
_.{package}.buildInputs = [...]; # replace buildInputs
|
|
_.{package}.buildInputs.add = [...]; # add buildInputs
|
|
_.{package}.buildInputs.mod = # modify buildInputs
|
|
oldInputs: filter (inp: ...) oldInputs;
|
|
|
|
_.{package}.patches = [...]; # replace patches
|
|
_.{package}.patches.add = [...]; # add patches
|
|
...
|
|
}
|
|
```
|
|
### Example: add missing build inputs
|
|
For example the package web2ldap depends on another python package `ldap0` which fails to build because of missing dependencies.
|
|
```nix
|
|
...
|
|
with mach-nix.nixpkgs;
|
|
mach-nix.mkPython {
|
|
|
|
requirements = "web2ldap";
|
|
|
|
# add missing dependencies to ldap0
|
|
_.ldap0.buildInputs.add = [ openldap.dev cyrus_sasl.dev ];
|
|
}
|
|
|
|
```
|
|
|
|
## Overrides (overrides_pre / overrides_post)
|
|
### Include poetry2nix overrides
|
|
`imagecodecs` is available via wheel, but if one wants to build it from source, dependencies will be missing since there is no nixpkgs candidate available.
|
|
poetry2nix luckily maintains overrides for this package. They can be included into the mach-nix build like this.
|
|
```nix
|
|
...
|
|
mach-nix.mkPython rec {
|
|
|
|
requirements = ''
|
|
# bunch of other requirements
|
|
imagecodecs
|
|
'';
|
|
|
|
providers = {
|
|
_default = "sdist";
|
|
};
|
|
|
|
# Import overrides from poetry2nix
|
|
# Caution! Use poetry2nix overrides only in `overrides_post`, not `overrides_pre`.
|
|
overridesPost = [
|
|
(
|
|
import (builtins.fetchurl {
|
|
url = "https://raw.githubusercontent.com/nix-community/poetry2nix/1cfaa4084d651d73af137866622e3d0699851008/overrides.nix";
|
|
}) { pkgs = mach-nix.nixpkgs; }
|
|
)
|
|
];
|
|
}
|
|
```
|
|
|
|
## Tensorflow
|
|
|
|
### Tensorflow with SSE/AVX/FMA support
|
|
Tensorflow from pypi does not provide any hardware optimization support. To get a SSE/AVX/FMA enabled version, set the provider for tensorflow to `nixpkgs`.
|
|
|
|
```nix
|
|
...
|
|
mach-nix.mkPython {
|
|
|
|
requirements = ''
|
|
# bunch of other requirements
|
|
tensorflow
|
|
'';
|
|
|
|
# force tensorflow to be taken from nixpkgs
|
|
providers.tensorflow = "nixpkgs";
|
|
}
|
|
```
|
|
This only works if the restrictions in `requirements.txt` allow for the tensorflow version from nixpkgs.
|
|
|
|
### Tensorflow via wheel (newer versions, quicker builds)
|
|
Install recent tensorflow via wheel
|
|
```nix
|
|
...
|
|
mach-nix.mkPython {
|
|
|
|
requirements = ''
|
|
# bunch of other requirements
|
|
tensorflow == 2.2.0rc4
|
|
'';
|
|
# no need to specify provider settings since wheel is the default anyways
|
|
}
|
|
|
|
```
|
|
## PyTorch
|
|
|
|
### Recent PyTorch with nixpkgs dependencies, and custom python
|
|
Recent pytorch version, Build dependencies from source
|
|
```nix
|
|
...
|
|
mach-nix.mkPython rec {
|
|
|
|
requirements = ''
|
|
# bunch of other requirements
|
|
torch == 1.5.0
|
|
'';
|
|
|
|
providers = {
|
|
# disallow wheels by default
|
|
_default = "nixpkgs,sdist";
|
|
# allow wheels only for torch
|
|
torch = "wheel";
|
|
};
|
|
}
|
|
```
|
|
|
|
## JupyterLab
|
|
|
|
### Starting point for a geospatial environment
|
|
```nix
|
|
...
|
|
let
|
|
pyEnv = mach-nix.mkPython rec {
|
|
|
|
requirements = ''
|
|
jupyterlab
|
|
geopandas
|
|
pyproj
|
|
pygeos
|
|
shapely>=1.7.0
|
|
'';
|
|
|
|
providers.shapely = "sdist,nixpkgs";
|
|
};
|
|
in
|
|
mkShell rec {
|
|
|
|
buildInputs = [
|
|
pyEnv
|
|
] ;
|
|
|
|
shellHook = ''
|
|
jupyter lab --notebook-dir=~/
|
|
'';
|
|
}
|
|
```
|
|
|
|
## Docker
|
|
Docker images can be built by using `mkDockerImage` instead of `mkPython`. It accepts the same arguments.
|
|
### JupyterLab Docker Image
|
|
Assuming the following expression under `./jupyter-docker.nix`:
|
|
```nix
|
|
...
|
|
let
|
|
image = mach-nix.mkDockerImage {
|
|
requirements = ''
|
|
jupyterlab
|
|
# add more packages here
|
|
'';
|
|
};
|
|
in
|
|
# The following overrides a call to nixpkgs.dockerTools.buildImage.
|
|
# Find more buildImage examples here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix
|
|
image.override (oldAttrs: {
|
|
name = "jupyterlab";
|
|
config.Cmd = [ "jupyter" "lab" "--notebook-dir=/mnt" "--allow-root" "--ip=0.0.0.0" ];
|
|
})
|
|
```
|
|
Execute the build like:
|
|
```
|
|
nix-build ./jupyter-docker.nix -o ./docker-image
|
|
```
|
|
Afterwards, load the docker image:
|
|
```
|
|
docker load < ./docker-image
|
|
```
|
|
Start the jupyterlab container:
|
|
```
|
|
docker run --rm -it -p 8888:8888 -v $HOME:/mnt jupyterlab
|
|
```
|
|
|
|
## R and Python
|
|
The following is an example for a Python environment mixed with R packages.
|
|
R packages can be added via `packagesExtra`.
|
|
If mach-nix finds R packages inside `packagesExtra`, it will automatically include `rpy2` and add the selected R packages to its buildInputs.
|
|
To get a list of available R packages, execute: `echo "builtins.attrNames(import <nixpkgs> {}).rPackages" | nix repl`
|
|
|
|
```nix
|
|
...
|
|
mach-nix.mkPython {
|
|
requirements = ''
|
|
# some python requirements
|
|
'';
|
|
packagesExtra = with mach-nix.rPackages; [
|
|
data_table
|
|
];
|
|
}
|
|
```
|
|
|
|
|
|
## Raspberry PI / aarch64 SD Image
|
|
This example builds an aarch64 sd image via emulator. For this to work, binfmt support for aarch64 must be installed first. (On NixOS simply set `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`)
|
|
For the SD-image, create a configuration.nix file which adds the mach-nix tool and some default python packages to the system environment.
|
|
**configuration.nix**:
|
|
```nix
|
|
{ config, lib, pkgs, ... }:
|
|
|
|
let
|
|
machNix = import (builtins.fetchGit {
|
|
url = "https://github.com/DavHau/mach-nix/";
|
|
ref = "refs/tags/put_version_here";
|
|
}) { inherit pkgs; };
|
|
|
|
defaultPythonEnv = machNix.mkPython {
|
|
requirements = ''
|
|
cryptography
|
|
'';
|
|
providers.cffi = "nixpkgs";
|
|
};
|
|
|
|
in {
|
|
imports = [
|
|
<nixpkgs/nixos/modules/installer/cd-dvd/sd-image-aarch64.nix>
|
|
];
|
|
environment.systemPackages = [ defaultPythonEnv machNix.mach-nix ];
|
|
sdImage.compressImage = false; # speeds up the build
|
|
}
|
|
```
|
|
with the following **default.nix**:
|
|
```nix
|
|
with import <nixpkgs/nixos> {
|
|
system = "aarch64-linux";
|
|
};
|
|
config.system.build.sdImage
|
|
```
|
|
Execute:
|
|
```bash
|
|
NIXOS_CONFIG=$PWD/configuration.nix nix build -f default.nix
|
|
```
|
|
Or to select a specific channel:
|
|
```bash
|
|
NIXOS_CONFIG=$PWD/configuration.nix nix build -f default.nix -I nixpkgs=channel:nixos-20.03
|
|
```
|