yogsototh/mach-nix
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge pull request #436 from milahu/patch-1

Browse source 
add section: File resolution (#434)
This commit is contained in:
DavHau 2022-04-25 23:14:54 +02:00 committed by GitHub
commit 26f06e415a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
2 changed files with 60 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

9
Readme.md
View file

@ -44,6 +44,7 @@ Table of Contents
* [Why nix?](#why-nix) * [Why nix?](#why-nix)
* [How does mach-nix work?](#how-does-mach-nix-work) * [How does mach-nix work?](#how-does-mach-nix-work)
* [Dependency resolution](#dependency-resolution) * [Dependency resolution](#dependency-resolution)
* [File resolution](#file-resolution)
* [Generating a nix expression](#generating-a-nix-expression) * [Generating a nix expression](#generating-a-nix-expression)
* [Contributing](#contributing) * [Contributing](#contributing)
* [Limitations](#limitations) * [Limitations](#limitations)
@ -203,6 +204,14 @@ As core for the resolving resolvelib is used: https://github.com/sarugaku/resolv
Mach-nix supports multiple providers to retrieve python packages from. The user can specify which providers should be preferred. Packages from different providers can be mixed. Mach-nix supports multiple providers to retrieve python packages from. The user can specify which providers should be preferred. Packages from different providers can be mixed.
### File resolution
With `pypi-deps-db`, we have built a dependency graph, where each package is defined by name and version, for example `pillow-9.1.0`.
To download a package, we need it's URL and hash. This is where [nix-pypi-fetcher](https://github.com/DavHau/nix-pypi-fetcher) comes in.
`nix-pypi-fetcher` is another database, which allows us to resolve packages to their URL and hash.
For details, see [implementation.md](implementation.md).
### Generating a nix expression ### Generating a nix expression
After all python dependencies and their providers have been determined by the dependency resolver, mach-nix will generate a nix expression defining your python environment. After all python dependencies and their providers have been determined by the dependency resolver, mach-nix will generate a nix expression defining your python environment.

51
implementation.md Normal file
View file

@ -0,0 +1,51 @@
## File resolution
`mach-nix` uses [nix-pypi-fetcher](https://github.com/DavHau/nix-pypi-fetcher) to translate package versions to URLs and hashes.
For example, the declaration "package pillow + version 9.1.0 + python 3.9 + linux" resolves to
```json
"Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": [
"gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=",
"cp39"
],
```
The wheel URL is constructed in [pypi-crawlers/src/crawl_wheel_deps.py](pypi-crawlers/src/crawl_wheel_deps.py)
```py
def construct_url(name, pyver, filename: str):
base_url = "https://files.pythonhosted.org/packages/"
return f"{base_url}{pyver}/{name[0]}/{name}/{filename}"
```
So in this example, the full URL would be
```
https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
```
Now `mach-nix` can call Nix's `fetchurl` like
```nix
builtins.fetchurl {
url = "https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
sha256 = "gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=";
}
```
... and Nix will download the wheel file to `/nix/store`
```console
$ unzip -l /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | head
Archive: /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Length Date Time Name
--------- ---------- ----- ----
0 04-01-2022 10:08 PIL/
0 04-01-2022 10:08 Pillow.libs/
0 04-01-2022 10:08 Pillow-9.1.0.dist-info/
7311 04-01-2022 10:08 PIL/PdfImagePlugin.py
141328 04-01-2022 10:08 PIL/_imagingcms.cpython-39-x86_64-linux-gnu.so
46872 04-01-2022 10:08 PIL/_imagingtk.cpython-39-x86_64-linux-gnu.so
1513 04-01-2022 10:08 PIL/GribStubImagePlugin.py
```