yogsototh/mach-nix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

version 3.4.0

Browse source
This commit is contained in:
DavHau 2022-02-04 12:42:34 +07:00
parent f0f48efdd6
commit fe5255e6fd
4 changed files with 94 additions and 64 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
Changelog.md
View file

@ -1,3 +1,33 @@
# 3.4.0 (29 Jan 2022)
aarch64-darwin, nixpkgs provider improvements, bugfixes
### Features
- support conda packages
- support wheels for apple m1 (aarch64-darwin)
### Changes
- remove support for installing mach-nix via pip
- updated inputs nixpkgs, pypi-deps-db, conda-channels
### Improvements
- support for PEP600 wheels of format: 'manylinux_${GLIBCMAJOR}_${GLIBCMINOR}'
- respect 'python_requires' for sdist packages
- PEP440 compatible pre-release version handling
- improve handling of python packages from nixpkgs
### Fixes
- fix problem where required dependencies were removed from nixpkgs python modules.
- prevent package collisions with dependencies from `packagesExtra`
- fix version comparison of versions with arbitrary length
- various fixes for MacOS
- various fixes for requirements parsing
- various other fixes
### Package Fixes
- libwebp-base: remove colliding binaries in conda package
- pyqt5: fix missing wrapQtAppsHook
# 3.3.0 (22 May 2021)
bugfixes, improvements
@ -12,7 +42,7 @@ bugfixes, improvements
- Mach-nix (used via flakes) will now throw an error if the selected nixpkgs version is newer than the dependency DB since this can cause conflicts in the resulting environment.
- When used via flakes, it was impossible to select the python version because the import function is not used anymore. Now `python` can be passed to `mkPython` alternatively.
- For the flakes cmdline api, collisions are now ignored by default
- The simplified override interface did not deal well with non-existent values.
- The simplified override interface did not deal well with non-existent values.
- Now the `.add` directive automatically assumes an empty list/set/string when the attribute to be extended doesn't exist.
- Now the `.mod` directive will pass `null` to the given function if the attribute to modify doesn't exist instead.
@ -72,7 +102,7 @@ flakes lib, cli improvements, bugfixes
- mkDockerImage produced corrupt images.
- non-python packages passed via `packagesExtra` were not available during runtime. Now they are added to the `PATH`.
- remove `<nixpkgs>` impurity in the dependency extractor used in buildPythonPackage.
# 3.0.2 (27 Oct 2020)
@ -120,16 +150,16 @@ in
```
### Features
- Flakes gateway to pypi. Get a nix shell with arbitrary python packages. Example:
- Flakes gateway to pypi. Get a nix shell with arbitrary python packages. Example:
`nix develop github:davhau/mach-nix#shellWith.requests.tensorflow.aiohttp`
- or a docker image
- or a docker image
`nix build github:davhau/mach-nix#dockerImageWith.package1.package2 ...`
- or a python derivation
- or a python derivation
`nix build github:davhau/mach-nix#with.package1.package2 ...`
- New output formats:
* **mkDockerImage** -> produces layered docker image containing a python environment
* **mkNixpkgs** -> returns nixpkgs which is conform to the given requirements
* **mkOverlay** -> returns an overlay function to make nixpkgs conform to the given requirements
- New output formats:
* **mkDockerImage** -> produces layered docker image containing a python environment
* **mkNixpkgs** -> returns nixpkgs which is conform to the given requirements
* **mkOverlay** -> returns an overlay function to make nixpkgs conform to the given requirements
* **mkPythonOverrides** -> produces pythonOverrides to make python conform to the given requirements.
- New functions **fetchPypiSdist** and **fetchPypiWheel**. Example:
@ -145,13 +175,13 @@ in
```
- R support (experimental): R packages can be passed via `packagesExtra`. Mach-nix will setup rpy2 accordingly. See [usage example](https://github.com/DavHau/mach-nix/blob/master/examples.md#r-and-python).
- Non-python packages can be passed via `packagesExtra` to include them into the environment.
the target platform's python interpreter was used to generate the nix expression, resulting in a failing build
### Improvements
- rework the logic for inheriting dependencies from nixpkgs
- fixes.nix: allow alternative mod function signature with more arguments:
- fixes.nix: allow alternative mod function signature with more arguments:
`key-to-override.mod = pySelf: oldAttrs: oldVal: ...;`
- allow derivations passed as `src` argument to buildPythonPackage
- stop inheriting attribute names from nixpkgs, instead use normalized package names
@ -168,7 +198,7 @@ in
- wheel provider picked wheels incompatible to python version
- unwanted python buildInput inheritance when overriding nixpkgs
- properly parse setup/install_requires if they are strings instead of lists
### Package Fixes
- rpy2: sdist: remove conflicting patch for versions newer than 3.2.6
- pytorch from nixpkgs was not detected as `torch`
@ -184,7 +214,7 @@ bugfixes
- null value error when inheriting passthru for disabled packages
- Wrong provider detected for `sdist` packages in fixes.nix
- overrides from fixes.nix didn't apply for `buildPythonPackage`
### Package Fixes
- pip: allow from `sdist` provider
- pip: remove `reproducible.patch` for versions < 20.0
@ -199,13 +229,13 @@ Global conditional overrides, simple overrides for buildPythonPackage, improved
- Inherit passthru from nixpkgs: Reduces risk of missing attributes like `numpy.blas`.
- Allow passing a string to the `python` argument of mkPython: Values like, for example, `"python38"` are now accepted in which case `pkgs.python38` will be used. The intention is to reduce the risk of accidentally mixing multiple nixpkgs versions.
- Improved error handling while extracting metadata from python sources in buildPythonPackage.
### Fixes
- Selecting `extras` when using `buildPythonPackage` didn't have any effect
- The `passthru` argument for `buildPythonPackage` was ignored
- The `propagatedBuildInputs` argument for `buildPythonPackage` was ignored
- Wheels with multiple python versions in their filename like `PyQt5-...-cp35.cp36.cp37.cp38-...whl` were not selected correctly.
### Package Fixes:
- tensorflow: collision related to tensorboard
- orange3: broken .so file caused by fixupPhase (probably due to shrinking)
@ -215,7 +245,7 @@ Global conditional overrides, simple overrides for buildPythonPackage, improved
simplified override system, autodetect requirements, improved success rate
### Features
- Simplified generic override system via `_` (underscore) argument for `mkPython`.
- Simplified generic override system via `_` (underscore) argument for `mkPython`.
Example: `_.{package}.buildInputs.add = [...]`
- `buildPythonPackage` now automatically detects requirements. Therefore the `requrements` argument becomes optional.
- `buildPythonPackage` now automatically detects package `name` and `version`. Therefore those attributes become optional.
@ -229,7 +259,7 @@ simplified override system, autodetect requirements, improved success rate
- Fix installation of the mach-nix tool via pip. (requirements were missing)
- packages which use a non-normalized version triggered an evaluation error since mach-nix tried to reference their source via normalized version.
- wheels removed from pypi were not removed from the dependency graph which could result in environments failing to build
# 2.2.2 (17 Aug 2020)
@ -238,7 +268,7 @@ simplified override system, autodetect requirements, improved success rate
- Problem generating error message for resolution impossible errors
- `buildPythonPackage` of mach-nix failed if arguments like `pkgs` were passed.
- When overriding packages, mach-nix now falls back to using `overrideAttrs` if `overridePythonAttrs` is not available.
### Package Fixes:
- pip: installation failed. Fixed by forcing `nixpkgs` provider
- gdal: building from sdist doesn't work. Fixed by forcing `nixpkgs` provider
@ -257,7 +287,7 @@ Handle circular dependencies, fix python 3.8 wheels, improve error message
### Fixes
- Fix crash on circular dependencies.
- Python 3.8 wheels have abi tag cp38, not cp38m. This was not considered before which prevented finding suitable manylinux wheels for python 3.8
### Development
- Added integration tests under [./tests/](/tests/)
@ -267,15 +297,15 @@ Improved success rate, MacOS support, bugfixes, optimizations
### Features
- Improved selection of wheel releases. MacOS is now supported and architectures besides x86_64 should be handled correctly.
- Whenever mach-nix resolves dependencies, a visualization of the resulting dependency tree is printed on the terminal.
- Whenever mach-nix resolves dependencies, a visualization of the resulting dependency tree is printed on the terminal.
- The dependency DB is now accessed through a caching layer which reduces the resolver's CPU time significantly for larger environments.
- The python platform context is now generated from the nix build environment variable `system`. This should decrease the chance of impurities during dependency resolution.
### Fixes
- The requires_python attribute of wheels was not respected. This lead to failing builds especially for older python versions. Now `requires_python` is part of the dependency graph and affects resolution.
- Detecting the correct package name for python packages in nixpkgs often failed since the attribute names don't follow a fixed schema. This lead to a handful of different errors in different situations. Now the package names are extracted from the pypi `url` inside the `src` attribute which is much more reliable. For packages which are not fetched from pypi, the `pname` attribute is used as fallback.
- Fixed bug which lead to the error `attribute 'sdist' missing` if a package from the nixpkgs provider was used which doesn't publish it's source on pypi. (For example `tensorflow`)
### Other Changes
- Mach-nix now uses a revision of the nixpkgs-unstable branch instead of nixos-20.03 as base fo the tool and the nixpkgs provider.
- Updated revision of the dependency DB
@ -316,12 +346,12 @@ Improves build-time closure, build success rate and fixes disable_checks option.
- **Extras support**: All providers now fully support python [extras](https://www.python.org/dev/peps/pep-0508/#extras). That means requirements like for example '`requests[security]`' will be correctly resolved.
### Nix Interface changes:
- Removed:
- Removed:
- **mkPythonExpr**: removed in favor of `machNixFile` and `machNix`
- Added:
- Added:
- **machNixFile** which generates a nex expresison
- **machNix** which evaluates the generated nix expression to get `overrides` and `select_pkgs`
- Changed:
- Changed:
- **mkPython**:
- Removed arguments:
- `python_attr` in favor if `python`
@ -337,4 +367,4 @@ Improves build-time closure, build success rate and fixes disable_checks option.
# 1.0.0
Initial Release
Initial Release

42
Readme.md
View file

@ -1,9 +1,9 @@
<p align="center">
<img width="200" src="https://gist.githubusercontent.com/DavHau/9a66b8c66b798254b714cc3ca44ffda8/raw/ef6b947b3753425118c730a5dfe81084c1bcfe86/logo_small.jpg">
<img width="200" src="https://gist.githubusercontent.com/DavHau/9a66b8c66b798254b714cc3ca44ffda8/raw/ef6b947b3753425118c730a5dfe81084c1bcfe86/logo_small.jpg">
</p>
## mach-nix - Create highly reproducible python environments
Mach-nix makes it easy to create and share reproducible python environments or packages. Existing tools for python package management often suffer from reproducibility and complexity issues, requiring a multitude of tools and additional virtualization layers to work sufficiently. Mach-nix aims to solve these problems by providing a simple way to use nix, a revolutionary build system which is known to achieve great reproducibility and portability besides [many other advantages](https://nixos.org/features.html).
Mach-nix makes it easy to create and share reproducible python environments or packages. Existing tools for python package management often suffer from reproducibility and complexity issues, requiring a multitude of tools and additional virtualization layers to work sufficiently. Mach-nix aims to solve these problems by providing a simple way to use nix, a revolutionary build system which is known to achieve great reproducibility and portability besides [many other advantages](https://nixos.org/features.html).
## Who is this meant for?
- Users without nix experience, who want to maintain python environments for their projects which are reliable and easy to reproduce.
@ -57,7 +57,7 @@ Table of Contents
### Installation
```shell
nix-env -if https://github.com/DavHau/mach-nix/tarball/3.3.0 -A mach-nix
nix-env -if https://github.com/DavHau/mach-nix/tarball/3.4.0 -A mach-nix
```
or, if you prefer `nix-shell`:
@ -66,11 +66,11 @@ or, if you prefer `nix-shell`:
```shell
nix shell github:DavHau/mach-nix
```
+ otherwise:
```shell
nix-shell -p '(callPackage (fetchTarball https://github.com/DavHau/mach-nix/tarball/3.3.0) {}).mach-nix'
nix-shell -p '(callPackage (fetchTarball https://github.com/DavHau/mach-nix/tarball/3.4.0) {}).mach-nix'
```
---
@ -82,7 +82,7 @@ This will generate the python environment into `./env`. To activate it, execute:
```bash
nix-shell ./env
```
The `./env` directory contains a portable and reproducible definition of your python environment. To reuse this environment on another system, just copy the `./env` directory
The `./env` directory contains a portable and reproducible definition of your python environment. To reuse this environment on another system, just copy the `./env` directory
and use `nix-shell` to activate it.
---
@ -102,7 +102,7 @@ You can call mach-nix directly from a nix expression
let
mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix";
ref = "refs/tags/3.3.0";
ref = "refs/tags/3.4.0";
}) {};
in
mach-nix.mkPython {
@ -133,7 +133,7 @@ Functions for building python packages or applications:
**buildPythonPackage** and **buildPythonApplication** accept the same arguments as their equally named partners in nixpkgs, plus the arguments of **mkPython**. If name/version/requirements arguments are omitted, mach-nix attempts to detect them automatically. See [./examples.md](/examples.md).
_Note that some dependency declaration formats are missing. For a roadmap, please refer to issue [#132](https://github.com/DavHau/mach-nix/issues/132)._
**mkPython** and all other **mk...** functions take exactly the following arguments:
#### Required Arguments:
@ -147,13 +147,13 @@ _Note that some dependency declaration formats are missing. For a roadmap, pleas
- **overridesPost** (list): (advanced) list of pythonOverrides to apply after the mach-nix overrides. Use this to fixup packages.
- **tests** (bool): Whether to enable tests (default: false)
- **_providerDefaults** (set): builtin provider defaults. Disable them by passing {}
#### Configure Providers
**Providers** allow you to configure the origin for your packages on a granular basis.
The following providers are available:
1. **conda**: Provides packages from anaconda.org. Those packages can contain binaries. Some benefits of anaconda are:
- Different build variants for packages
1. **conda**: Provides packages from anaconda.org. Those packages can contain binaries. Some benefits of anaconda are:
- Different build variants for packages
- Provides all system dependencies for packages
1. **wheel**: Provides all linux compatible wheel releases from pypi. If wheels contain binaries, Mach-nix patches them via patchelf to ensure reproducibility. Wheels are very quick to install and work quite reliable.
1. **sdist**: Provides all setuptools compatible packages from pypi. It still uses nix for building, which allows it to be tweaked in a flexible way. But in some cases problems can occur, if there is not sufficient information available to determine required system depedencies.
@ -184,8 +184,8 @@ Mach-nix will always satisfy the **requirements.txt** fully with the configured
If a mach-nix build fails, most of the time it can be resolved by just switching the provider of a package, which is simple and doesn't require writing a lot of nix code. For some more complex scenarios, checkout the [./examples.md](/examples.md).
## Why nix?
Usually people rely on multiple layers of different package management tools for building their software environments. These tools are often not well integrated with each other and don't offer strong reproducibility. Example: You are on debian/ubuntu and use APT (layer 1) to install python. Then you use venv (layer 2) to overcome some of your layer 1 limitations (not being able to have multiple versions of the same package installed) and afterwards you are using pip (layer 3) to install python packages. You notice that even after pinning all your requirements, your environment behaves differently on your server or your colleagues machine because their underlying system differs from yours. You start using docker (layer 4) to overcome this problem which adds extra complexity to the whole process and gives you some nasty limitations during development. You need to configure your IDE's docker integration and so on. Despite all the effort you put in, still the problem is not fully solved and from time to time your build pipeline just breaks and you need to fix it manually.
Usually people rely on multiple layers of different package management tools for building their software environments. These tools are often not well integrated with each other and don't offer strong reproducibility. Example: You are on debian/ubuntu and use APT (layer 1) to install python. Then you use venv (layer 2) to overcome some of your layer 1 limitations (not being able to have multiple versions of the same package installed) and afterwards you are using pip (layer 3) to install python packages. You notice that even after pinning all your requirements, your environment behaves differently on your server or your colleagues machine because their underlying system differs from yours. You start using docker (layer 4) to overcome this problem which adds extra complexity to the whole process and gives you some nasty limitations during development. You need to configure your IDE's docker integration and so on. Despite all the effort you put in, still the problem is not fully solved and from time to time your build pipeline just breaks and you need to fix it manually.
In contrast to that, the nix package manager provides a from ground up different approach to build software systems. Due to its purely functional approach, nix doesn't require additional layers to make your software reliable. Software environments built with nix are known to be reproducible and portable, which makes many processes during development and deployment easier. Mach-nix leverages that potential by abstracting away the complexity involved in building python environments with nix. Under the hood it just generates and evaluates nix expressions for you.
## How does mach-nix work?
@ -194,8 +194,8 @@ The general mechanism can be broken down into [Dependency resolution](#dependenc
### Dependency resolution
Mach-nix contains a dependency graph of nearly all python packages available on pypi.org. This allows mach-nix to resolve dependencies offline within seconds.
The dependency graph data can be found here: https://github.com/DavHau/pypi-deps-db
The dependency graph is updated on a daily basis by this set of tools: https://github.com/DavHau/pypi-crawlers
The dependency graph data can be found here: https://github.com/DavHau/pypi-deps-db
The dependency graph is updated on a daily basis by this set of tools: https://github.com/DavHau/pypi-crawlers
Despite this graph being updated constantly, mach-nix always pins one specific version of the graph to ensure reproducibility.
@ -208,15 +208,15 @@ After all python dependencies and their providers have been determined by the de
Individual python packages are either built by overriding an existing package definition from nixpkgs, or by creating the package from scratch via nixpkgs' `buildPythonPackage`. Which strategy is used depends on the provider of a package and if it is already packaged in nixpkgs.
Using nixpkgs as a base has the following benefits:
1. **Non-python Dependencies**:
Using nixpkgs as a base has the following benefits:
1. **Non-python Dependencies**:
Many python packages have non-python dependencies like various C libraries. Mach-nix can resolve those dependencies by taking the build inputs from python package definitions in nixpkgs.
1. **Special features**:
Some python packages can be built with special features, like for example SSE/AVX/FMA support in tensorflow. The nixpkgs versions of those python packages often include these features.
1. **Nix specific fixes**:
1. **Nix specific fixes**:
Some python packages might need some additional modification to work with nix. Those are already done in nixpkgs.
If a package is built by overriding nixpkgs, the following attributes are modified:
If a package is built by overriding nixpkgs, the following attributes are modified:
- `src`: updated to the required version
- `name`: modified to match the new version
- `buildInputs`: replaced with mach-nix determined python deps

30
examples.md
View file

@ -46,7 +46,7 @@ every mach-nix expression should begin like this:
let
mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix";
ref = "refs/tags/3.3.0";
ref = "refs/tags/3.4.0";
}) {
# optionally bring your own nixpkgs
# pkgs = import <nixpkgs> {};
@ -69,7 +69,7 @@ in
nixpkgs.url = "github:nixos/nixpkgs";
flake-utils.url = "github:numtide/flake-utils";
mach-nix = {
url = "github:DavHau/mach-nix?ref=3.3.0";
url = "github:DavHau/mach-nix?ref=3.4.0";
inputs = {
nixpkgs.follows = "nixpkgs";
flake-utils.follows = "flake-utils";
@ -198,7 +198,7 @@ mach-nix.mkPython {
_.{package}.buildInputs = [...]; # replace buildInputs
_.{package}.buildInputs.add = [...]; # add buildInputs
_.{package}.buildInputs.mod = # modify buildInputs
oldInputs: filter (inp: ...) oldInputs;
oldInputs: filter (inp: ...) oldInputs;
_.{package}.patches = [...]; # replace patches
_.{package}.patches.add = [...]; # add patches
@ -264,7 +264,7 @@ mach-nix.mkPython {
'';
# force tensorflow to be taken from nixpkgs
providers.tensorflow = "nixpkgs";
providers.tensorflow = "nixpkgs";
}
```
This only works if the restrictions in `requirements.txt` allow for the tensorflow version from nixpkgs.
@ -305,15 +305,15 @@ mach-nix.mkPython rec {
}
```
## Jupyter
## Jupyter
### ...using jupyterWith + mach-nix
In this example, mach-nix is used to resolve our python dependencies and provide them to [jupyterWith](https://github.com/tweag/jupyterWith) which is a Nix-based framework for the definition of declarative and reproducible Jupyter environments.
In this example, mach-nix is used to resolve our python dependencies and provide them to [jupyterWith](https://github.com/tweag/jupyterWith) which is a Nix-based framework for the definition of declarative and reproducible Jupyter environments.
```nix
let
mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix";
ref = "refs/heads/3.3.0"; # update this version
ref = "refs/heads/3.4.0"; # update this version
}) {
python = "python37";
};
@ -407,8 +407,8 @@ docker run --rm -it -p 8888:8888 -v $HOME:/mnt jupyterlab
```
## R and Python
The following is an example for a Python environment mixed with R packages.
R packages can be added via `packagesExtra`.
The following is an example for a Python environment mixed with R packages.
R packages can be added via `packagesExtra`.
If mach-nix finds R packages inside `packagesExtra`, it will automatically include `rpy2` and add the selected R packages to its buildInputs.
To get a list of available R packages, execute: `echo "builtins.attrNames(import <nixpkgs> {}).rPackages" | nix repl`
@ -426,8 +426,8 @@ mach-nix.mkPython {
## Raspberry PI / aarch64 SD Image
This example builds an aarch64 sd image via emulator. For this to work, binfmt support for aarch64 must be installed first. (On NixOS simply set `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`)
For the SD-image, create a configuration.nix file which adds the mach-nix tool and some default python packages to the system environment.
This example builds an aarch64 sd image via emulator. For this to work, binfmt support for aarch64 must be installed first. (On NixOS simply set `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`)
For the SD-image, create a configuration.nix file which adds the mach-nix tool and some default python packages to the system environment.
**configuration.nix**:
```nix
{ config, lib, pkgs, ... }:
@ -474,7 +474,7 @@ NIXOS_CONFIG=$PWD/configuration.nix nix build -f default.nix -I nixpkgs=channel:
### collision between /nix/store/X and /nix/store/Y
In general, there can be two different reasons for this error:
#### 1. Environment contains different versions of same package.
#### 1. Environment contains different versions of same package.
This is the case, if both store paths, mentioned by the error message, **contain the same package name**. For example:
```
collision between
@ -493,9 +493,9 @@ collision between
and
/nix/store/2cfyjnic605mjr5j7zck88gk7bkqwpim-python3.8-opencensus-0.7.12/lib/python3.8/site-packages/opencensus/common/__pycache__/__init__.cpython-38.pyc
```
In this example, both packages `opencensus-context` and `opencensus` contain a file with the exact same import path `[...]/opencensus/common/__pycache__/__init__.cpython-38.pyc`.
Would you be installing these two packages via pip, then the later installed package would overwrite the file of the earlier installed package. This leads to undefined behavior, since the outcome dependes on the order of installation which is not necessarily the same across different installations.
Still, those collisions are common among some python packages, since pip does not raise an error when this happens, and just blindly overwrites files, leaving package maintainers in the dark about the problem.
In this example, both packages `opencensus-context` and `opencensus` contain a file with the exact same import path `[...]/opencensus/common/__pycache__/__init__.cpython-38.pyc`.
Would you be installing these two packages via pip, then the later installed package would overwrite the file of the earlier installed package. This leads to undefined behavior, since the outcome dependes on the order of installation which is not necessarily the same across different installations.
Still, those collisions are common among some python packages, since pip does not raise an error when this happens, and just blindly overwrites files, leaving package maintainers in the dark about the problem.
It should be in the best interest of the packages maintainers to remove these collisions and get rid of undeclared behavior. Therefore it might be a good idea to report the collision issue upstream.
To fix the immediate error with your current mach-nix environment, you have two options:

2
mach_nix/VERSION
View file

@ -1 +1 @@
master
3.4.0