Merge pull request #436 from milahu/patch-1
add section: File resolution (#434)
This commit is contained in:
commit
26f06e415a
2 changed files with 60 additions and 0 deletions
|
@ -44,6 +44,7 @@ Table of Contents
|
|||
* [Why nix?](#why-nix)
|
||||
* [How does mach-nix work?](#how-does-mach-nix-work)
|
||||
* [Dependency resolution](#dependency-resolution)
|
||||
* [File resolution](#file-resolution)
|
||||
* [Generating a nix expression](#generating-a-nix-expression)
|
||||
* [Contributing](#contributing)
|
||||
* [Limitations](#limitations)
|
||||
|
@ -203,6 +204,14 @@ As core for the resolving resolvelib is used: https://github.com/sarugaku/resolv
|
|||
|
||||
Mach-nix supports multiple providers to retrieve python packages from. The user can specify which providers should be preferred. Packages from different providers can be mixed.
|
||||
|
||||
### File resolution
|
||||
With `pypi-deps-db`, we have built a dependency graph, where each package is defined by name and version, for example `pillow-9.1.0`.
|
||||
|
||||
To download a package, we need it's URL and hash. This is where [nix-pypi-fetcher](https://github.com/DavHau/nix-pypi-fetcher) comes in.
|
||||
`nix-pypi-fetcher` is another database, which allows us to resolve packages to their URL and hash.
|
||||
|
||||
For details, see [implementation.md](implementation.md).
|
||||
|
||||
### Generating a nix expression
|
||||
After all python dependencies and their providers have been determined by the dependency resolver, mach-nix will generate a nix expression defining your python environment.
|
||||
|
||||
|
|
51
implementation.md
Normal file
51
implementation.md
Normal file
|
@ -0,0 +1,51 @@
|
|||
## File resolution
|
||||
|
||||
`mach-nix` uses [nix-pypi-fetcher](https://github.com/DavHau/nix-pypi-fetcher) to translate package versions to URLs and hashes.
|
||||
|
||||
For example, the declaration "package pillow + version 9.1.0 + python 3.9 + linux" resolves to
|
||||
|
||||
```json
|
||||
"Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": [
|
||||
"gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=",
|
||||
"cp39"
|
||||
],
|
||||
```
|
||||
|
||||
The wheel URL is constructed in [pypi-crawlers/src/crawl_wheel_deps.py](pypi-crawlers/src/crawl_wheel_deps.py)
|
||||
|
||||
```py
|
||||
def construct_url(name, pyver, filename: str):
|
||||
base_url = "https://files.pythonhosted.org/packages/"
|
||||
return f"{base_url}{pyver}/{name[0]}/{name}/{filename}"
|
||||
```
|
||||
|
||||
So in this example, the full URL would be
|
||||
|
||||
```
|
||||
https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
|
||||
```
|
||||
|
||||
Now `mach-nix` can call Nix's `fetchurl` like
|
||||
|
||||
```nix
|
||||
builtins.fetchurl {
|
||||
url = "https://files.pythonhosted.org/packages/cp39/p/pillow/Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl";
|
||||
sha256 = "gig6+ZwcOluh2kTGcpbVqtGfEcU1tVGlrlUyijF84zE=";
|
||||
}
|
||||
```
|
||||
|
||||
... and Nix will download the wheel file to `/nix/store`
|
||||
|
||||
```console
|
||||
$ unzip -l /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | head
|
||||
Archive: /nix/store/7gbbbrsqkw7f69axyh818abh9yw45fnb-Pillow-9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
|
||||
Length Date Time Name
|
||||
--------- ---------- ----- ----
|
||||
0 04-01-2022 10:08 PIL/
|
||||
0 04-01-2022 10:08 Pillow.libs/
|
||||
0 04-01-2022 10:08 Pillow-9.1.0.dist-info/
|
||||
7311 04-01-2022 10:08 PIL/PdfImagePlugin.py
|
||||
141328 04-01-2022 10:08 PIL/_imagingcms.cpython-39-x86_64-linux-gnu.so
|
||||
46872 04-01-2022 10:08 PIL/_imagingtk.cpython-39-x86_64-linux-gnu.so
|
||||
1513 04-01-2022 10:08 PIL/GribStubImagePlugin.py
|
||||
```
|
Loading…
Reference in a new issue