forked from github.com/pypiserver
cf424c982d
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles * `app.py` talks only to `core.py` with regards to package operations * at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app * `core.py` proxies requests for packages to this `Backend()` * The `Backend` interface/api is defined through three things * methods that an implementation must implement * methods that an implementation may override if it knows better than the defaults * the `PkgFIle` class that is (should be) the main carrier of data * where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility Other things I've done in this PR: * I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means * Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing) * Cleanup some more Python 2 code * Started moving away from `os.path` and `py.path` in favour of `pathlib` Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR. The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs - [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear. - [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it. - [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings. - [ ] Improve testing of the `CacheManager`. ---- * move some functions around in preparation for backend module * rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil * further implement the current filestorage as simple file backend * rename prefix to project, since that's more descriptive * add digester func as attribute to pkgfile * WIP caching backend * WIP make cache better testable * better testability of cache * WIP file backends as plugin * fix typos, run black * Apply suggestions from code review Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com> * add more type hints to pass mypy, fix tox.ini * add package count method to backend * add package count method to backend * minor changes * bugfix when checking invalid whl file * check for existing package recursively, bugfix, some more pathlib * fix unittest * rm dead code * exclude bottle.py from coverage * fix merge mistakes * fix tab indentation * backend as a cli argument * fix cli, add tests * fix mypy * fix more silly mistakes * process feedback * remove dead code Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
73 lines
1.9 KiB
Python
73 lines
1.9 KiB
Python
#! /usr/bin/env py.test
|
|
# -*- coding: utf-8 -*-
|
|
|
|
import logging
|
|
import os
|
|
|
|
import pytest
|
|
|
|
from pypiserver import __main__, core, backend
|
|
from pypiserver.pkg_helpers import (
|
|
normalize_pkgname_for_url,
|
|
)
|
|
from tests.doubles import Namespace
|
|
|
|
|
|
## Enable logging to detect any problems with it
|
|
##
|
|
__main__.init_logging()
|
|
|
|
|
|
def test_listdir_bad_name(tmp_path):
|
|
tmp_path.joinpath("foo.whl").touch()
|
|
res = list(backend.listdir(tmp_path))
|
|
assert res == []
|
|
|
|
|
|
hashes = (
|
|
# empty-sha256
|
|
(
|
|
"sha256",
|
|
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
|
),
|
|
# empty-md5
|
|
("md5", "d41d8cd98f00b204e9800998ecf8427e"),
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(("algo", "digest"), hashes)
|
|
def test_hashfile(tmp_path, algo, digest):
|
|
f = tmp_path.joinpath("empty")
|
|
f.touch()
|
|
assert backend.digest_file(str(f), algo) == f"{algo}={digest}"
|
|
|
|
|
|
@pytest.mark.parametrize("hash_algo", ("md5", "sha256", "sha512"))
|
|
def test_fname_and_hash(tmp_path, hash_algo):
|
|
"""Ensure we are returning the expected hashes for files."""
|
|
|
|
def digester(pkg):
|
|
digest = backend.digest_file(pkg.fn, hash_algo)
|
|
pkg.digest = digest
|
|
return digest
|
|
|
|
f = tmp_path.joinpath("tmpfile")
|
|
f.touch()
|
|
pkgfile = core.PkgFile("tmp", "1.0.0", str(f), f.parent, f.name)
|
|
pkgfile.digester = digester
|
|
|
|
assert pkgfile.fname_and_hash == f"{f.name}#{digester(pkgfile)}"
|
|
|
|
|
|
def test_redirect_project_encodes_newlines():
|
|
"""Ensure raw newlines are url encoded in the generated redirect."""
|
|
request = Namespace(custom_fullpath="/\nSet-Cookie:malicious=1;")
|
|
project = "\nSet-Cookie:malicious=1;"
|
|
newpath = core.get_bad_url_redirect_path(request, project)
|
|
assert "\n" not in newpath
|
|
|
|
|
|
def test_normalize_pkgname_for_url_encodes_newlines():
|
|
"""Ensure newlines are url encoded in package names for urls."""
|
|
assert "\n" not in normalize_pkgname_for_url("/\nSet-Cookie:malicious=1;")
|