yogsototh/mach-nix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

version 3.4.0

Browse source
This commit is contained in:
DavHau 2022-02-04 12:42:34 +07:00
parent f0f48efdd6
commit fe5255e6fd
4 changed files with 94 additions and 64 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

84
Changelog.md
View file

@ -1,3 +1,33 @@
# 3.4.0 (29 Jan 2022)
aarch64-darwin, nixpkgs provider improvements, bugfixes
### Features
- support conda packages
- support wheels for apple m1 (aarch64-darwin)
### Changes
- remove support for installing mach-nix via pip
- updated inputs nixpkgs, pypi-deps-db, conda-channels
### Improvements
- support for PEP600 wheels of format: 'manylinux_${GLIBCMAJOR}_${GLIBCMINOR}'
- respect 'python_requires' for sdist packages
- PEP440 compatible pre-release version handling
- improve handling of python packages from nixpkgs
### Fixes
- fix problem where required dependencies were removed from nixpkgs python modules.
- prevent package collisions with dependencies from `packagesExtra`
- fix version comparison of versions with arbitrary length
- various fixes for MacOS
- various fixes for requirements parsing
- various other fixes
### Package Fixes
- libwebp-base: remove colliding binaries in conda package
- pyqt5: fix missing wrapQtAppsHook
# 3.3.0 (22 May 2021) # 3.3.0 (22 May 2021)
bugfixes, improvements bugfixes, improvements
@ -12,7 +42,7 @@ bugfixes, improvements
- Mach-nix (used via flakes) will now throw an error if the selected nixpkgs version is newer than the dependency DB since this can cause conflicts in the resulting environment. - Mach-nix (used via flakes) will now throw an error if the selected nixpkgs version is newer than the dependency DB since this can cause conflicts in the resulting environment.
- When used via flakes, it was impossible to select the python version because the import function is not used anymore. Now `python` can be passed to `mkPython` alternatively. - When used via flakes, it was impossible to select the python version because the import function is not used anymore. Now `python` can be passed to `mkPython` alternatively.
- For the flakes cmdline api, collisions are now ignored by default - For the flakes cmdline api, collisions are now ignored by default
- The simplified override interface did not deal well with non-existent values. - The simplified override interface did not deal well with non-existent values.
- Now the `.add` directive automatically assumes an empty list/set/string when the attribute to be extended doesn't exist. - Now the `.add` directive automatically assumes an empty list/set/string when the attribute to be extended doesn't exist.
- Now the `.mod` directive will pass `null` to the given function if the attribute to modify doesn't exist instead. - Now the `.mod` directive will pass `null` to the given function if the attribute to modify doesn't exist instead.
@ -72,7 +102,7 @@ flakes lib, cli improvements, bugfixes
- mkDockerImage produced corrupt images. - mkDockerImage produced corrupt images.
- non-python packages passed via `packagesExtra` were not available during runtime. Now they are added to the `PATH`. - non-python packages passed via `packagesExtra` were not available during runtime. Now they are added to the `PATH`.
- remove `<nixpkgs>` impurity in the dependency extractor used in buildPythonPackage. - remove `<nixpkgs>` impurity in the dependency extractor used in buildPythonPackage.
# 3.0.2 (27 Oct 2020) # 3.0.2 (27 Oct 2020)
@ -120,16 +150,16 @@ in
``` ```
### Features ### Features
- Flakes gateway to pypi. Get a nix shell with arbitrary python packages. Example: - Flakes gateway to pypi. Get a nix shell with arbitrary python packages. Example:
`nix develop github:davhau/mach-nix#shellWith.requests.tensorflow.aiohttp` `nix develop github:davhau/mach-nix#shellWith.requests.tensorflow.aiohttp`
- or a docker image - or a docker image
`nix build github:davhau/mach-nix#dockerImageWith.package1.package2 ...` `nix build github:davhau/mach-nix#dockerImageWith.package1.package2 ...`
- or a python derivation - or a python derivation
`nix build github:davhau/mach-nix#with.package1.package2 ...` `nix build github:davhau/mach-nix#with.package1.package2 ...`
- New output formats: - New output formats:
* **mkDockerImage** -> produces layered docker image containing a python environment * **mkDockerImage** -> produces layered docker image containing a python environment
* **mkNixpkgs** -> returns nixpkgs which is conform to the given requirements * **mkNixpkgs** -> returns nixpkgs which is conform to the given requirements
* **mkOverlay** -> returns an overlay function to make nixpkgs conform to the given requirements * **mkOverlay** -> returns an overlay function to make nixpkgs conform to the given requirements
* **mkPythonOverrides** -> produces pythonOverrides to make python conform to the given requirements. * **mkPythonOverrides** -> produces pythonOverrides to make python conform to the given requirements.
- New functions **fetchPypiSdist** and **fetchPypiWheel**. Example: - New functions **fetchPypiSdist** and **fetchPypiWheel**. Example:
@ -145,13 +175,13 @@ in
``` ```
- R support (experimental): R packages can be passed via `packagesExtra`. Mach-nix will setup rpy2 accordingly. See [usage example](https://github.com/DavHau/mach-nix/blob/master/examples.md#r-and-python). - R support (experimental): R packages can be passed via `packagesExtra`. Mach-nix will setup rpy2 accordingly. See [usage example](https://github.com/DavHau/mach-nix/blob/master/examples.md#r-and-python).
- Non-python packages can be passed via `packagesExtra` to include them into the environment. - Non-python packages can be passed via `packagesExtra` to include them into the environment.
the target platform's python interpreter was used to generate the nix expression, resulting in a failing build the target platform's python interpreter was used to generate the nix expression, resulting in a failing build
### Improvements ### Improvements
- rework the logic for inheriting dependencies from nixpkgs - rework the logic for inheriting dependencies from nixpkgs
- fixes.nix: allow alternative mod function signature with more arguments: - fixes.nix: allow alternative mod function signature with more arguments:
`key-to-override.mod = pySelf: oldAttrs: oldVal: ...;` `key-to-override.mod = pySelf: oldAttrs: oldVal: ...;`
- allow derivations passed as `src` argument to buildPythonPackage - allow derivations passed as `src` argument to buildPythonPackage
- stop inheriting attribute names from nixpkgs, instead use normalized package names - stop inheriting attribute names from nixpkgs, instead use normalized package names
@ -168,7 +198,7 @@ in
- wheel provider picked wheels incompatible to python version - wheel provider picked wheels incompatible to python version
- unwanted python buildInput inheritance when overriding nixpkgs - unwanted python buildInput inheritance when overriding nixpkgs
- properly parse setup/install_requires if they are strings instead of lists - properly parse setup/install_requires if they are strings instead of lists
### Package Fixes ### Package Fixes
- rpy2: sdist: remove conflicting patch for versions newer than 3.2.6 - rpy2: sdist: remove conflicting patch for versions newer than 3.2.6
- pytorch from nixpkgs was not detected as `torch` - pytorch from nixpkgs was not detected as `torch`
@ -184,7 +214,7 @@ bugfixes
- null value error when inheriting passthru for disabled packages - null value error when inheriting passthru for disabled packages
- Wrong provider detected for `sdist` packages in fixes.nix - Wrong provider detected for `sdist` packages in fixes.nix
- overrides from fixes.nix didn't apply for `buildPythonPackage` - overrides from fixes.nix didn't apply for `buildPythonPackage`
### Package Fixes ### Package Fixes
- pip: allow from `sdist` provider - pip: allow from `sdist` provider
- pip: remove `reproducible.patch` for versions < 20.0 - pip: remove `reproducible.patch` for versions < 20.0
@ -199,13 +229,13 @@ Global conditional overrides, simple overrides for buildPythonPackage, improved
- Inherit passthru from nixpkgs: Reduces risk of missing attributes like `numpy.blas`. - Inherit passthru from nixpkgs: Reduces risk of missing attributes like `numpy.blas`.
- Allow passing a string to the `python` argument of mkPython: Values like, for example, `"python38"` are now accepted in which case `pkgs.python38` will be used. The intention is to reduce the risk of accidentally mixing multiple nixpkgs versions. - Allow passing a string to the `python` argument of mkPython: Values like, for example, `"python38"` are now accepted in which case `pkgs.python38` will be used. The intention is to reduce the risk of accidentally mixing multiple nixpkgs versions.
- Improved error handling while extracting metadata from python sources in buildPythonPackage. - Improved error handling while extracting metadata from python sources in buildPythonPackage.
### Fixes ### Fixes
- Selecting `extras` when using `buildPythonPackage` didn't have any effect - Selecting `extras` when using `buildPythonPackage` didn't have any effect
- The `passthru` argument for `buildPythonPackage` was ignored - The `passthru` argument for `buildPythonPackage` was ignored
- The `propagatedBuildInputs` argument for `buildPythonPackage` was ignored - The `propagatedBuildInputs` argument for `buildPythonPackage` was ignored
- Wheels with multiple python versions in their filename like `PyQt5-...-cp35.cp36.cp37.cp38-...whl` were not selected correctly. - Wheels with multiple python versions in their filename like `PyQt5-...-cp35.cp36.cp37.cp38-...whl` were not selected correctly.
### Package Fixes: ### Package Fixes:
- tensorflow: collision related to tensorboard - tensorflow: collision related to tensorboard
- orange3: broken .so file caused by fixupPhase (probably due to shrinking) - orange3: broken .so file caused by fixupPhase (probably due to shrinking)
@ -215,7 +245,7 @@ Global conditional overrides, simple overrides for buildPythonPackage, improved
simplified override system, autodetect requirements, improved success rate simplified override system, autodetect requirements, improved success rate
### Features ### Features
- Simplified generic override system via `_` (underscore) argument for `mkPython`. - Simplified generic override system via `_` (underscore) argument for `mkPython`.
Example: `_.{package}.buildInputs.add = [...]` Example: `_.{package}.buildInputs.add = [...]`
- `buildPythonPackage` now automatically detects requirements. Therefore the `requrements` argument becomes optional. - `buildPythonPackage` now automatically detects requirements. Therefore the `requrements` argument becomes optional.
- `buildPythonPackage` now automatically detects package `name` and `version`. Therefore those attributes become optional. - `buildPythonPackage` now automatically detects package `name` and `version`. Therefore those attributes become optional.
@ -229,7 +259,7 @@ simplified override system, autodetect requirements, improved success rate
- Fix installation of the mach-nix tool via pip. (requirements were missing) - Fix installation of the mach-nix tool via pip. (requirements were missing)
- packages which use a non-normalized version triggered an evaluation error since mach-nix tried to reference their source via normalized version. - packages which use a non-normalized version triggered an evaluation error since mach-nix tried to reference their source via normalized version.
- wheels removed from pypi were not removed from the dependency graph which could result in environments failing to build - wheels removed from pypi were not removed from the dependency graph which could result in environments failing to build
# 2.2.2 (17 Aug 2020) # 2.2.2 (17 Aug 2020)
@ -238,7 +268,7 @@ simplified override system, autodetect requirements, improved success rate
- Problem generating error message for resolution impossible errors - Problem generating error message for resolution impossible errors
- `buildPythonPackage` of mach-nix failed if arguments like `pkgs` were passed. - `buildPythonPackage` of mach-nix failed if arguments like `pkgs` were passed.
- When overriding packages, mach-nix now falls back to using `overrideAttrs` if `overridePythonAttrs` is not available. - When overriding packages, mach-nix now falls back to using `overrideAttrs` if `overridePythonAttrs` is not available.
### Package Fixes: ### Package Fixes:
- pip: installation failed. Fixed by forcing `nixpkgs` provider - pip: installation failed. Fixed by forcing `nixpkgs` provider
- gdal: building from sdist doesn't work. Fixed by forcing `nixpkgs` provider - gdal: building from sdist doesn't work. Fixed by forcing `nixpkgs` provider
@ -257,7 +287,7 @@ Handle circular dependencies, fix python 3.8 wheels, improve error message
### Fixes ### Fixes
- Fix crash on circular dependencies. - Fix crash on circular dependencies.
- Python 3.8 wheels have abi tag cp38, not cp38m. This was not considered before which prevented finding suitable manylinux wheels for python 3.8 - Python 3.8 wheels have abi tag cp38, not cp38m. This was not considered before which prevented finding suitable manylinux wheels for python 3.8
### Development ### Development
- Added integration tests under [./tests/](/tests/) - Added integration tests under [./tests/](/tests/)
@ -267,15 +297,15 @@ Improved success rate, MacOS support, bugfixes, optimizations
### Features ### Features
- Improved selection of wheel releases. MacOS is now supported and architectures besides x86_64 should be handled correctly. - Improved selection of wheel releases. MacOS is now supported and architectures besides x86_64 should be handled correctly.
- Whenever mach-nix resolves dependencies, a visualization of the resulting dependency tree is printed on the terminal. - Whenever mach-nix resolves dependencies, a visualization of the resulting dependency tree is printed on the terminal.
- The dependency DB is now accessed through a caching layer which reduces the resolver's CPU time significantly for larger environments. - The dependency DB is now accessed through a caching layer which reduces the resolver's CPU time significantly for larger environments.
- The python platform context is now generated from the nix build environment variable `system`. This should decrease the chance of impurities during dependency resolution. - The python platform context is now generated from the nix build environment variable `system`. This should decrease the chance of impurities during dependency resolution.
### Fixes ### Fixes
- The requires_python attribute of wheels was not respected. This lead to failing builds especially for older python versions. Now `requires_python` is part of the dependency graph and affects resolution. - The requires_python attribute of wheels was not respected. This lead to failing builds especially for older python versions. Now `requires_python` is part of the dependency graph and affects resolution.
- Detecting the correct package name for python packages in nixpkgs often failed since the attribute names don't follow a fixed schema. This lead to a handful of different errors in different situations. Now the package names are extracted from the pypi `url` inside the `src` attribute which is much more reliable. For packages which are not fetched from pypi, the `pname` attribute is used as fallback. - Detecting the correct package name for python packages in nixpkgs often failed since the attribute names don't follow a fixed schema. This lead to a handful of different errors in different situations. Now the package names are extracted from the pypi `url` inside the `src` attribute which is much more reliable. For packages which are not fetched from pypi, the `pname` attribute is used as fallback.
- Fixed bug which lead to the error `attribute 'sdist' missing` if a package from the nixpkgs provider was used which doesn't publish it's source on pypi. (For example `tensorflow`) - Fixed bug which lead to the error `attribute 'sdist' missing` if a package from the nixpkgs provider was used which doesn't publish it's source on pypi. (For example `tensorflow`)
### Other Changes ### Other Changes
- Mach-nix now uses a revision of the nixpkgs-unstable branch instead of nixos-20.03 as base fo the tool and the nixpkgs provider. - Mach-nix now uses a revision of the nixpkgs-unstable branch instead of nixos-20.03 as base fo the tool and the nixpkgs provider.
- Updated revision of the dependency DB - Updated revision of the dependency DB
@ -316,12 +346,12 @@ Improves build-time closure, build success rate and fixes disable_checks option.
- **Extras support**: All providers now fully support python [extras](https://www.python.org/dev/peps/pep-0508/#extras). That means requirements like for example '`requests[security]`' will be correctly resolved. - **Extras support**: All providers now fully support python [extras](https://www.python.org/dev/peps/pep-0508/#extras). That means requirements like for example '`requests[security]`' will be correctly resolved.
### Nix Interface changes: ### Nix Interface changes:
- Removed: - Removed:
- **mkPythonExpr**: removed in favor of `machNixFile` and `machNix` - **mkPythonExpr**: removed in favor of `machNixFile` and `machNix`
- Added: - Added:
- **machNixFile** which generates a nex expresison - **machNixFile** which generates a nex expresison
- **machNix** which evaluates the generated nix expression to get `overrides` and `select_pkgs` - **machNix** which evaluates the generated nix expression to get `overrides` and `select_pkgs`
- Changed: - Changed:
- **mkPython**: - **mkPython**:
- Removed arguments: - Removed arguments:
- `python_attr` in favor if `python` - `python_attr` in favor if `python`
@ -337,4 +367,4 @@ Improves build-time closure, build success rate and fixes disable_checks option.
# 1.0.0 # 1.0.0
Initial Release Initial Release

42
Readme.md
View file

@ -1,9 +1,9 @@
<p align="center"> <p align="center">
<img width="200" src="https://gist.githubusercontent.com/DavHau/9a66b8c66b798254b714cc3ca44ffda8/raw/ef6b947b3753425118c730a5dfe81084c1bcfe86/logo_small.jpg"> <img width="200" src="https://gist.githubusercontent.com/DavHau/9a66b8c66b798254b714cc3ca44ffda8/raw/ef6b947b3753425118c730a5dfe81084c1bcfe86/logo_small.jpg">
</p> </p>
## mach-nix - Create highly reproducible python environments ## mach-nix - Create highly reproducible python environments
Mach-nix makes it easy to create and share reproducible python environments or packages. Existing tools for python package management often suffer from reproducibility and complexity issues, requiring a multitude of tools and additional virtualization layers to work sufficiently. Mach-nix aims to solve these problems by providing a simple way to use nix, a revolutionary build system which is known to achieve great reproducibility and portability besides [many other advantages](https://nixos.org/features.html). Mach-nix makes it easy to create and share reproducible python environments or packages. Existing tools for python package management often suffer from reproducibility and complexity issues, requiring a multitude of tools and additional virtualization layers to work sufficiently. Mach-nix aims to solve these problems by providing a simple way to use nix, a revolutionary build system which is known to achieve great reproducibility and portability besides [many other advantages](https://nixos.org/features.html).
## Who is this meant for? ## Who is this meant for?
- Users without nix experience, who want to maintain python environments for their projects which are reliable and easy to reproduce. - Users without nix experience, who want to maintain python environments for their projects which are reliable and easy to reproduce.
@ -57,7 +57,7 @@ Table of Contents
### Installation ### Installation
```shell ```shell
nix-env -if https://github.com/DavHau/mach-nix/tarball/3.3.0 -A mach-nix nix-env -if https://github.com/DavHau/mach-nix/tarball/3.4.0 -A mach-nix
``` ```
or, if you prefer `nix-shell`: or, if you prefer `nix-shell`:
@ -66,11 +66,11 @@ or, if you prefer `nix-shell`:
```shell ```shell
nix shell github:DavHau/mach-nix nix shell github:DavHau/mach-nix
``` ```
+ otherwise: + otherwise:
```shell ```shell
nix-shell -p '(callPackage (fetchTarball https://github.com/DavHau/mach-nix/tarball/3.3.0) {}).mach-nix' nix-shell -p '(callPackage (fetchTarball https://github.com/DavHau/mach-nix/tarball/3.4.0) {}).mach-nix'
``` ```
--- ---
@ -82,7 +82,7 @@ This will generate the python environment into `./env`. To activate it, execute:
```bash ```bash
nix-shell ./env nix-shell ./env
``` ```
The `./env` directory contains a portable and reproducible definition of your python environment. To reuse this environment on another system, just copy the `./env` directory The `./env` directory contains a portable and reproducible definition of your python environment. To reuse this environment on another system, just copy the `./env` directory
and use `nix-shell` to activate it. and use `nix-shell` to activate it.
--- ---
@ -102,7 +102,7 @@ You can call mach-nix directly from a nix expression
let let
mach-nix = import (builtins.fetchGit { mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix"; url = "https://github.com/DavHau/mach-nix";
ref = "refs/tags/3.3.0"; ref = "refs/tags/3.4.0";
}) {}; }) {};
in in
mach-nix.mkPython { mach-nix.mkPython {
@ -133,7 +133,7 @@ Functions for building python packages or applications:
**buildPythonPackage** and **buildPythonApplication** accept the same arguments as their equally named partners in nixpkgs, plus the arguments of **mkPython**. If name/version/requirements arguments are omitted, mach-nix attempts to detect them automatically. See [./examples.md](/examples.md). **buildPythonPackage** and **buildPythonApplication** accept the same arguments as their equally named partners in nixpkgs, plus the arguments of **mkPython**. If name/version/requirements arguments are omitted, mach-nix attempts to detect them automatically. See [./examples.md](/examples.md).
_Note that some dependency declaration formats are missing. For a roadmap, please refer to issue [#132](https://github.com/DavHau/mach-nix/issues/132)._ _Note that some dependency declaration formats are missing. For a roadmap, please refer to issue [#132](https://github.com/DavHau/mach-nix/issues/132)._
**mkPython** and all other **mk...** functions take exactly the following arguments: **mkPython** and all other **mk...** functions take exactly the following arguments:
#### Required Arguments: #### Required Arguments:
@ -147,13 +147,13 @@ _Note that some dependency declaration formats are missing. For a roadmap, pleas
- **overridesPost** (list): (advanced) list of pythonOverrides to apply after the mach-nix overrides. Use this to fixup packages. - **overridesPost** (list): (advanced) list of pythonOverrides to apply after the mach-nix overrides. Use this to fixup packages.
- **tests** (bool): Whether to enable tests (default: false) - **tests** (bool): Whether to enable tests (default: false)
- **_providerDefaults** (set): builtin provider defaults. Disable them by passing {} - **_providerDefaults** (set): builtin provider defaults. Disable them by passing {}
#### Configure Providers #### Configure Providers
**Providers** allow you to configure the origin for your packages on a granular basis. **Providers** allow you to configure the origin for your packages on a granular basis.
The following providers are available: The following providers are available:
1. **conda**: Provides packages from anaconda.org. Those packages can contain binaries. Some benefits of anaconda are: 1. **conda**: Provides packages from anaconda.org. Those packages can contain binaries. Some benefits of anaconda are:
- Different build variants for packages - Different build variants for packages
- Provides all system dependencies for packages - Provides all system dependencies for packages
1. **wheel**: Provides all linux compatible wheel releases from pypi. If wheels contain binaries, Mach-nix patches them via patchelf to ensure reproducibility. Wheels are very quick to install and work quite reliable. 1. **wheel**: Provides all linux compatible wheel releases from pypi. If wheels contain binaries, Mach-nix patches them via patchelf to ensure reproducibility. Wheels are very quick to install and work quite reliable.
1. **sdist**: Provides all setuptools compatible packages from pypi. It still uses nix for building, which allows it to be tweaked in a flexible way. But in some cases problems can occur, if there is not sufficient information available to determine required system depedencies. 1. **sdist**: Provides all setuptools compatible packages from pypi. It still uses nix for building, which allows it to be tweaked in a flexible way. But in some cases problems can occur, if there is not sufficient information available to determine required system depedencies.
@ -184,8 +184,8 @@ Mach-nix will always satisfy the **requirements.txt** fully with the configured
If a mach-nix build fails, most of the time it can be resolved by just switching the provider of a package, which is simple and doesn't require writing a lot of nix code. For some more complex scenarios, checkout the [./examples.md](/examples.md). If a mach-nix build fails, most of the time it can be resolved by just switching the provider of a package, which is simple and doesn't require writing a lot of nix code. For some more complex scenarios, checkout the [./examples.md](/examples.md).
## Why nix? ## Why nix?
Usually people rely on multiple layers of different package management tools for building their software environments. These tools are often not well integrated with each other and don't offer strong reproducibility. Example: You are on debian/ubuntu and use APT (layer 1) to install python. Then you use venv (layer 2) to overcome some of your layer 1 limitations (not being able to have multiple versions of the same package installed) and afterwards you are using pip (layer 3) to install python packages. You notice that even after pinning all your requirements, your environment behaves differently on your server or your colleagues machine because their underlying system differs from yours. You start using docker (layer 4) to overcome this problem which adds extra complexity to the whole process and gives you some nasty limitations during development. You need to configure your IDE's docker integration and so on. Despite all the effort you put in, still the problem is not fully solved and from time to time your build pipeline just breaks and you need to fix it manually. Usually people rely on multiple layers of different package management tools for building their software environments. These tools are often not well integrated with each other and don't offer strong reproducibility. Example: You are on debian/ubuntu and use APT (layer 1) to install python. Then you use venv (layer 2) to overcome some of your layer 1 limitations (not being able to have multiple versions of the same package installed) and afterwards you are using pip (layer 3) to install python packages. You notice that even after pinning all your requirements, your environment behaves differently on your server or your colleagues machine because their underlying system differs from yours. You start using docker (layer 4) to overcome this problem which adds extra complexity to the whole process and gives you some nasty limitations during development. You need to configure your IDE's docker integration and so on. Despite all the effort you put in, still the problem is not fully solved and from time to time your build pipeline just breaks and you need to fix it manually.
In contrast to that, the nix package manager provides a from ground up different approach to build software systems. Due to its purely functional approach, nix doesn't require additional layers to make your software reliable. Software environments built with nix are known to be reproducible and portable, which makes many processes during development and deployment easier. Mach-nix leverages that potential by abstracting away the complexity involved in building python environments with nix. Under the hood it just generates and evaluates nix expressions for you. In contrast to that, the nix package manager provides a from ground up different approach to build software systems. Due to its purely functional approach, nix doesn't require additional layers to make your software reliable. Software environments built with nix are known to be reproducible and portable, which makes many processes during development and deployment easier. Mach-nix leverages that potential by abstracting away the complexity involved in building python environments with nix. Under the hood it just generates and evaluates nix expressions for you.
## How does mach-nix work? ## How does mach-nix work?
@ -194,8 +194,8 @@ The general mechanism can be broken down into [Dependency resolution](#dependenc
### Dependency resolution ### Dependency resolution
Mach-nix contains a dependency graph of nearly all python packages available on pypi.org. This allows mach-nix to resolve dependencies offline within seconds. Mach-nix contains a dependency graph of nearly all python packages available on pypi.org. This allows mach-nix to resolve dependencies offline within seconds.
The dependency graph data can be found here: https://github.com/DavHau/pypi-deps-db The dependency graph data can be found here: https://github.com/DavHau/pypi-deps-db
The dependency graph is updated on a daily basis by this set of tools: https://github.com/DavHau/pypi-crawlers The dependency graph is updated on a daily basis by this set of tools: https://github.com/DavHau/pypi-crawlers
Despite this graph being updated constantly, mach-nix always pins one specific version of the graph to ensure reproducibility. Despite this graph being updated constantly, mach-nix always pins one specific version of the graph to ensure reproducibility.
@ -208,15 +208,15 @@ After all python dependencies and their providers have been determined by the de
Individual python packages are either built by overriding an existing package definition from nixpkgs, or by creating the package from scratch via nixpkgs' `buildPythonPackage`. Which strategy is used depends on the provider of a package and if it is already packaged in nixpkgs. Individual python packages are either built by overriding an existing package definition from nixpkgs, or by creating the package from scratch via nixpkgs' `buildPythonPackage`. Which strategy is used depends on the provider of a package and if it is already packaged in nixpkgs.
Using nixpkgs as a base has the following benefits: Using nixpkgs as a base has the following benefits:
1. **Non-python Dependencies**: 1. **Non-python Dependencies**:
Many python packages have non-python dependencies like various C libraries. Mach-nix can resolve those dependencies by taking the build inputs from python package definitions in nixpkgs. Many python packages have non-python dependencies like various C libraries. Mach-nix can resolve those dependencies by taking the build inputs from python package definitions in nixpkgs.
1. **Special features**: 1. **Special features**:
Some python packages can be built with special features, like for example SSE/AVX/FMA support in tensorflow. The nixpkgs versions of those python packages often include these features. Some python packages can be built with special features, like for example SSE/AVX/FMA support in tensorflow. The nixpkgs versions of those python packages often include these features.
1. **Nix specific fixes**: 1. **Nix specific fixes**:
Some python packages might need some additional modification to work with nix. Those are already done in nixpkgs. Some python packages might need some additional modification to work with nix. Those are already done in nixpkgs.
If a package is built by overriding nixpkgs, the following attributes are modified: If a package is built by overriding nixpkgs, the following attributes are modified:
- `src`: updated to the required version - `src`: updated to the required version
- `name`: modified to match the new version - `name`: modified to match the new version
- `buildInputs`: replaced with mach-nix determined python deps - `buildInputs`: replaced with mach-nix determined python deps

30
examples.md
View file

@ -46,7 +46,7 @@ every mach-nix expression should begin like this:
let let
mach-nix = import (builtins.fetchGit { mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix"; url = "https://github.com/DavHau/mach-nix";
ref = "refs/tags/3.3.0"; ref = "refs/tags/3.4.0";
}) { }) {
# optionally bring your own nixpkgs # optionally bring your own nixpkgs
# pkgs = import <nixpkgs> {}; # pkgs = import <nixpkgs> {};
@ -69,7 +69,7 @@ in
nixpkgs.url = "github:nixos/nixpkgs"; nixpkgs.url = "github:nixos/nixpkgs";
flake-utils.url = "github:numtide/flake-utils"; flake-utils.url = "github:numtide/flake-utils";
mach-nix = { mach-nix = {
url = "github:DavHau/mach-nix?ref=3.3.0"; url = "github:DavHau/mach-nix?ref=3.4.0";
inputs = { inputs = {
nixpkgs.follows = "nixpkgs"; nixpkgs.follows = "nixpkgs";
flake-utils.follows = "flake-utils"; flake-utils.follows = "flake-utils";
@ -198,7 +198,7 @@ mach-nix.mkPython {
_.{package}.buildInputs = [...]; # replace buildInputs _.{package}.buildInputs = [...]; # replace buildInputs
_.{package}.buildInputs.add = [...]; # add buildInputs _.{package}.buildInputs.add = [...]; # add buildInputs
_.{package}.buildInputs.mod = # modify buildInputs _.{package}.buildInputs.mod = # modify buildInputs
oldInputs: filter (inp: ...) oldInputs; oldInputs: filter (inp: ...) oldInputs;
_.{package}.patches = [...]; # replace patches _.{package}.patches = [...]; # replace patches
_.{package}.patches.add = [...]; # add patches _.{package}.patches.add = [...]; # add patches
@ -264,7 +264,7 @@ mach-nix.mkPython {
''; '';
# force tensorflow to be taken from nixpkgs # force tensorflow to be taken from nixpkgs
providers.tensorflow = "nixpkgs"; providers.tensorflow = "nixpkgs";
} }
``` ```
This only works if the restrictions in `requirements.txt` allow for the tensorflow version from nixpkgs. This only works if the restrictions in `requirements.txt` allow for the tensorflow version from nixpkgs.
@ -305,15 +305,15 @@ mach-nix.mkPython rec {
} }
``` ```
## Jupyter ## Jupyter
### ...using jupyterWith + mach-nix ### ...using jupyterWith + mach-nix
In this example, mach-nix is used to resolve our python dependencies and provide them to [jupyterWith](https://github.com/tweag/jupyterWith) which is a Nix-based framework for the definition of declarative and reproducible Jupyter environments. In this example, mach-nix is used to resolve our python dependencies and provide them to [jupyterWith](https://github.com/tweag/jupyterWith) which is a Nix-based framework for the definition of declarative and reproducible Jupyter environments.
```nix ```nix
let let
mach-nix = import (builtins.fetchGit { mach-nix = import (builtins.fetchGit {
url = "https://github.com/DavHau/mach-nix"; url = "https://github.com/DavHau/mach-nix";
ref = "refs/heads/3.3.0"; # update this version ref = "refs/heads/3.4.0"; # update this version
}) { }) {
python = "python37"; python = "python37";
}; };
@ -407,8 +407,8 @@ docker run --rm -it -p 8888:8888 -v $HOME:/mnt jupyterlab
``` ```
## R and Python ## R and Python
The following is an example for a Python environment mixed with R packages. The following is an example for a Python environment mixed with R packages.
R packages can be added via `packagesExtra`. R packages can be added via `packagesExtra`.
If mach-nix finds R packages inside `packagesExtra`, it will automatically include `rpy2` and add the selected R packages to its buildInputs. If mach-nix finds R packages inside `packagesExtra`, it will automatically include `rpy2` and add the selected R packages to its buildInputs.
To get a list of available R packages, execute: `echo "builtins.attrNames(import <nixpkgs> {}).rPackages" | nix repl` To get a list of available R packages, execute: `echo "builtins.attrNames(import <nixpkgs> {}).rPackages" | nix repl`
@ -426,8 +426,8 @@ mach-nix.mkPython {
## Raspberry PI / aarch64 SD Image ## Raspberry PI / aarch64 SD Image
This example builds an aarch64 sd image via emulator. For this to work, binfmt support for aarch64 must be installed first. (On NixOS simply set `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`) This example builds an aarch64 sd image via emulator. For this to work, binfmt support for aarch64 must be installed first. (On NixOS simply set `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]`)
For the SD-image, create a configuration.nix file which adds the mach-nix tool and some default python packages to the system environment. For the SD-image, create a configuration.nix file which adds the mach-nix tool and some default python packages to the system environment.
**configuration.nix**: **configuration.nix**:
```nix ```nix
{ config, lib, pkgs, ... }: { config, lib, pkgs, ... }:
@ -474,7 +474,7 @@ NIXOS_CONFIG=$PWD/configuration.nix nix build -f default.nix -I nixpkgs=channel:
### collision between /nix/store/X and /nix/store/Y ### collision between /nix/store/X and /nix/store/Y
In general, there can be two different reasons for this error: In general, there can be two different reasons for this error:
#### 1. Environment contains different versions of same package. #### 1. Environment contains different versions of same package.
This is the case, if both store paths, mentioned by the error message, **contain the same package name**. For example: This is the case, if both store paths, mentioned by the error message, **contain the same package name**. For example:
``` ```
collision between collision between
@ -493,9 +493,9 @@ collision between
and and
/nix/store/2cfyjnic605mjr5j7zck88gk7bkqwpim-python3.8-opencensus-0.7.12/lib/python3.8/site-packages/opencensus/common/__pycache__/__init__.cpython-38.pyc /nix/store/2cfyjnic605mjr5j7zck88gk7bkqwpim-python3.8-opencensus-0.7.12/lib/python3.8/site-packages/opencensus/common/__pycache__/__init__.cpython-38.pyc
``` ```
In this example, both packages `opencensus-context` and `opencensus` contain a file with the exact same import path `[...]/opencensus/common/__pycache__/__init__.cpython-38.pyc`. In this example, both packages `opencensus-context` and `opencensus` contain a file with the exact same import path `[...]/opencensus/common/__pycache__/__init__.cpython-38.pyc`.
Would you be installing these two packages via pip, then the later installed package would overwrite the file of the earlier installed package. This leads to undefined behavior, since the outcome dependes on the order of installation which is not necessarily the same across different installations. Would you be installing these two packages via pip, then the later installed package would overwrite the file of the earlier installed package. This leads to undefined behavior, since the outcome dependes on the order of installation which is not necessarily the same across different installations.
Still, those collisions are common among some python packages, since pip does not raise an error when this happens, and just blindly overwrites files, leaving package maintainers in the dark about the problem. Still, those collisions are common among some python packages, since pip does not raise an error when this happens, and just blindly overwrites files, leaving package maintainers in the dark about the problem.
It should be in the best interest of the packages maintainers to remove these collisions and get rid of undeclared behavior. Therefore it might be a good idea to report the collision issue upstream. It should be in the best interest of the packages maintainers to remove these collisions and get rid of undeclared behavior. Therefore it might be a good idea to report the collision issue upstream.
To fix the immediate error with your current mach-nix environment, you have two options: To fix the immediate error with your current mach-nix environment, you have two options:

2
mach_nix/VERSION
View file

@ -1 +1 @@
master 3.4.0