8.2 KiB
- Usage in Nix Expressions:
- Examples for Tensorflow / PyTorch
- Using '_' (simplified override system)
- Using overrides
Usage in Nix Expressions:
import mach-nix
let
mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix/";
ref = "heads/refs/2.3.0";
});
in
...
mkPython / mkPythonShell
From a list of requirements
mach-nix.mkPython { # replace with mkPythonShell if shell is wanted
requirements = builtins.readFile ./requirements.txt;
}
Mix requirements with packages from arbitrary sources.
extra_pkgs accepts python packages built via mach-nix.buildPythonPackage. Alternatively, paths or URLs can be passed which are then automatically wrapped in a mach-nix.buildPythonPackage call.
mach-nix.mkPython {
requirements = builtins.readFile ./requirements.txt;
extra_pkgs = [
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
./some/local/project # from local path
mach-nix.buildPythonPackage { ... }; # from package
];
}
alternatively if requirements are not needed, extra_pkgs can be passed directly to mkPython
mach-nix.mkPython [
"https://github.com/psf/requests/tarball/2a7832b5b06d" # from tarball url
./some/local/project # from local path
mach-nix.buildPythonPackage { ... }; # from package
]
buildPythonPackage / buildPythonApplication
Whenever requirements are not explicitly specified, they will be extracted automatically from the packages setup.py/setup.cfg. The same goes for the name and version.
Build python package from its source code
mach-nix.buildPythonPackage /python-project-path
buildPythonPackage from GitHub
mach-nix.buildPythonPackage "https://github.com/psf/requests/tarball/2a7832b5b06d"
buildPythonPackage from GitHub with extras
mach-nix.buildPythonPackage {
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
extras = "socks";
}
buildPythonPackage from GitHub and add missing requirements
use this in case autdetecting requirements failed
mach-nix.buildPythonPackage {
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
add_requirements = "pytest";
}
buildPythonPackage from GitHub (explicit source)
mach-nix.buildPythonPackage {
src = builtins.fetchGit{
url = "https://github.com/user/projectname";
ref = "master";
# rev = "put_commit_hash_here";
};
}
buildPythonPackage from GitHub (manual requirements)
Use this if automatic requirements extraction doesn't work.
mach-nix.buildPythonPackage {
src = "https://github.com/psf/requests/tarball/2a7832b5b06d";
requirements = ''
# list of requirements
'';
}
Examples for Tensorflow / PyTorch
Tensorflow with SSE/AVX/FMA support
I have a complex set of requirements including tensorflow. I'd like to have tensorflow with the usual nix features enabled like SSE/AVX/FMA which I cannot get from pypi. Therefore I must take tensorflow from nixpkgs. For everything else I keep the default, which means wheels are preferred. This allows for quicker installation of dependencies.
mach-nix.mkPython {
requirements = ''
# bunch of other requirements
tensorflow
'';
providers = {
# force tensorflow to be taken from nixpkgs
tensorflow = "nixpkgs";
};
}
This only works if the restrictions in requirements.txt allow for the tensorflow version from nixpkgs.
Tensorflow via wheel (newer versions, quicker builds)
I'd like to install a more recent version of tensorflow which is not available from nixpkgs. Also I don't like long build times and therefore I want to install tensorflow via wheel. Usually most wheels work pretty well out of the box, but the tensorflow wheel has an issue which I need to fix with an override.
mach-nix.mkPython {
requirements = ''
# bunch of other requirements
tensorflow == 2.2.0rc4
'';
# no need to specify provider settings since wheel is the default anyways
# Fix the tensorflow wheel
_.tensorflow.postInstall = "rm $out/bin/tensorboard";
}
Recent PyTorch with nixpkgs dependencies, and custom python
I'd like to use a recent version of Pytorch from wheel, but I'd like to build the rest of the requirements from sdist or nixpkgs. I want to use python 3.6.
mach-nix.mkPython rec {
requirements = ''
# bunch of other requirements
torch == 1.5.0
'';
providers = {
# disallow wheels by default
_default = "nixpkgs,sdist";
# allow wheels only for torch
torch = "wheel";
};
# Select custom python version (Must be taken from pkgs with the overlay applied)
python = mach-nix.nixpkgs.python36;
}
Using '_' (simplified override system)
General usage
with mach-nix.nixpkgs;
mach-nix.mkPython {
requirements = "some requirements";
_.{package}.buildInputs = [...]; # replace buildInputs
_.{package}.buildInputs.add = [...]; # add buildInputs
_.{package}.buildInputs.mod = # modify buildInputs
oldInputs: filter (inp: ...) oldInputs;
_.{package}.patches = [...]; # replace patches
_.{package}.patches.add = [...]; # add patches
...
}
Example: add missing build inputs
For example the package web2ldap depends on another python package ldap0 which fails to build because of missing dependencies.
with mach-nix.nixpkgs;
mach-nix.mkPython {
requirements = "web2ldap";
# add missing dependencies to ldap0
_.ldap0.buildInputs.add = [ openldap.dev cyrus_sasl.dev ];
}
Using overrides
Include poetry2nix overrides
I have a complex requirements.txt which includes imagecodecs. It is available via wheel, but I prefer to build everything from source. This package has complex build dependencies and is not available from nixpkgs. Luckily poetry2nix` overrides make it work.
mach-nix.mkPython rec {
requirements = ''
# bunch of other requirements
imagecodecs
'';
providers = {
_default = "sdist";
};
# Import overrides from poetry2nix
# Caution! Use poetry2nix overrides only in `overrides_post`, not `overrides_pre`.
overrides_post = [
(
import (builtins.fetchurl {
url = "https://raw.githubusercontent.com/nix-community/poetry2nix/1cfaa4084d651d73af137866622e3d0699851008/overrides.nix";
}) { pkgs = mach-nix.nixpkgs; }
)
];
}