github.com
/
pypiserver
mirror of https://github.com/pypiserver/pypiserver
1
0
Fork 1
Code Releases Wiki Activity

Docker improvements (#365) 

* Docker improvements

This addresses much of what was brought up in #359. Specifically, it:

- Significantly improves testing for the Docker image, adding a
  `docker/test_docker.py` file using the regular pytest machinery to
  set up and run docker images for testing
- Hopefully addresses a variety of permissions issues, by being explicit
  about what access pypiserver needs and asking for it, only erroring
  if that access is not available
  - Requires RX permissions on `/data` (R to read files, X to list files
    and to be able to cd into the directory. This is important since
    `/data` is the `WORKDIR`)
  - Requires RWX permissions on `/data/packages`, so that we can list
    packages, write packages, and read packages.
  - When running in the default configuration (as root on Linux or
    as the pypiserver-named rootish user on Mac), with no volumes
    mounted, these requirements are all satisfied
  - Volume mounts still must be readable by the pypiserver user (UID
    9898) in order for the container to run. However, we now error early
    if this is not the case, and direct users to a useful issue.
  - If the container is run as a non-root, non-pypiserver user (e.g.
    because someone ran `docker run --user=<user_id>`, we try to run
    pypiserver as that user). Provided that user has access to the
    necessary directories, it should run fine.
- Fixes issues with running help and similar commands
- Updates the Docker image to use `PYPISERVER_PORT` for port
  specification, while still falling back to `PORT` for backwards
  compatibility
- Moves some docker-related things into a `/docker` directory
- Adds a `Makefile` for building a test fixture package sdist and wheel,
  so that test code can call `make mypkg` and not need to worry about it
  potentially building multiple times

The only issue #359 raises that's not addressed here is the one of
running pypiserver in the Docker container using some non-default server
for performance. I would like to do some benchmarking before deciding on
what to do there.
This commit is contained in:
Matthew Planchard 2021-02-06 11:28:15 -06:00 committed by GitHub
parent df7454ff20
commit d868005e1f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
23 changed files with 781 additions and 91 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.dockerignore
View File

@ -1,8 +1,8 @@
*
!pypiserver
!requirements
!docker-requirements.txt
!entrypoint.sh
!docker/docker-requirements.txt
!docker/entrypoint.sh
!README.rst
!setup.cfg
!setup.py

29
.github/workflows/test.yml vendored
View File

@ -34,7 +34,8 @@ jobs:
test-pypy:
# Run a a separate job so we don't need to mess with conditionally
# splitting the python version from the build matrix.
# splitting the python version from the build matrix. Also the pypy
# tests take freaking forever.
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
@ -64,9 +65,10 @@ jobs:
- name: Check types
# individual mypy files for now, until we get the rest
# of the project typechecking
run: |
mypy \
pypiserver/config.py \
run: >-
mypy
docker/test_docker.py
pypiserver/config.py
tests/test_init.py
- name: Check formatting
run: black --diff --check .
@ -74,8 +76,13 @@ jobs:
run: ./bin/check_readme.sh
# Full-flow docker tests, again not python version dependent
docker:
runs-on: ubuntu-latest
# We _could_ test this on MacOS, but it takes forever to get docker
# installed. I'm going to say for now probably 99% of people using
# the docker image will be doing so from a linux system, e.g. for
# a k8s deploy, and I've verified manually that things work on
# MacOS, so /shrug.
test-docker:
runs-on: "ubuntu-latest"
steps:
- uses: actions/checkout@v2
- name: Set up Python
@ -83,17 +90,19 @@ jobs:
with:
# Pretty much any python version will do
python-version: "3.9"
- name: Install dependencies
run: pip install --use-feature 2020-resolver -U twine
- name: Install test dependencies
run: pip install -r "requirements/test.pip"
- name: Install package
run: pip install -r "requirements/exe.pip"
- name: Run tests
run: ./bin/test-docker.sh
run: "pytest docker/test_docker.py"
tests:
runs-on: "ubuntu-latest"
needs:
- "check"
- "docker"
- "test-docker"
- "test-cpython"
- "test-pypy"
steps:

10
Dockerfile
View File

@ -33,7 +33,7 @@ FROM base AS builder_dependencies
COPY pypiserver /code/pypiserver
COPY requirements /code/requirements
COPY docker-requirements.txt /code
COPY docker/docker-requirements.txt /code
COPY setup.cfg /code
COPY setup.py /code
COPY README.rst /code
@ -50,7 +50,7 @@ FROM base
# Copy the libraries installed via pip
COPY --from=builder_dependencies /install /usr/local
COPY --from=builder_gosu /usr/local/bin/gosu /usr/local/bin/gosu
COPY entrypoint.sh /entrypoint.sh
COPY docker/entrypoint.sh /entrypoint.sh
# Use a consistent user and group ID so that linux users
# can create a corresponding system user and set permissions
@ -65,7 +65,9 @@ RUN apk add bash \
VOLUME /data/packages
WORKDIR /data
ENV PORT=8080
EXPOSE $PORT
ENV PYPISERVER_PORT=8080
# PORT is deprecated. Please use PYPISERVER_PORT instead
ENV PORT=$PYPISERVER_PORT
EXPOSE $PYPISERVER_PORT
ENTRYPOINT ["/entrypoint.sh"]

23
Makefile Normal file
View File

@ -0,0 +1,23 @@
##
# pypiserver
#
# this makefile is used to help with building resources needed for testing
#
# @file
# @version 0.1
SHELL = /bin/sh
MYPKG_SRC = fixtures/mypkg/setup.py $(shell find fixtures/mypkg/mypkg -type f -name '*.py')
# Build the test fixture package.
mypkg: fixtures/mypkg/dist/pypiserver_mypkg-1.0.0.tar.gz
mypkg: fixtures/mypkg/dist/pypiserver_mypkg-1.0.0-py2.py3-none-any.whl
fixtures/mypkg/dist/pypiserver_mypkg-1.0.0.tar.gz: $(MYPKG_SRC)
cd fixtures/mypkg; python setup.py sdist
fixtures/mypkg/dist/pypiserver_mypkg-1.0.0-py2.py3-none-any.whl: $(MYPKG_SRC)
cd fixtures/mypkg; python setup.py bdist_wheel
# end

BIN
bin/pypiserver-1.2.6-py2.py3-none-any.whl
View File

Binary file not shown.

49
bin/test-docker.sh
View File

@ -1,49 +0,0 @@
#!/usr/bin/env sh
# Perform some simple validation to make sure the Docker image works
# Should be run from the repo root.
set -xe # exit on any error, show debug output
DIR="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )"
docker build . -t pypiserver:test
docker run pypiserver:test --help > /dev/null
# Mount our htpasswd file, which contains a test user with a bcrypt-encrypted
# "test" password
CONTAINER_ID=$(docker run \
-d \
-v "${DIR}/test.htpasswd:/data/.htpasswd" \
-p 8080:8080 \
-e PORT=8080 \
pypiserver:test -a "list,update,download" -P /data/.htpasswd packages)
trap "docker container stop $CONTAINER_ID" EXIT
sleep 15 # give the container some time to get going
# Ensure we can authenticate locally
RET=$(curl localhost:8080)
echo $RET
echo $RET | grep -q "pypiserver"
RET=$(curl localhost:8080/packages/)
echo $RET
echo $RET | grep -q "401"
RET=$(curl test:test@localhost:8080/packages/)
echo $RET
echo $RET | grep -q "Index of packages"
twine upload \
-u test \
-p test \
--repository-url http://localhost:8080 \
"${DIR}/pypiserver-1.2.6-py2.py3-none-any.whl"
RET=$(curl test:test@localhost:8080/packages/)
echo $RET
echo $RET | grep -q "pypiserver-1.2.6"

2
bin/test.htpasswd
View File

@ -1,2 +0,0 @@
test:$2y$05$0wU8vmgucWeyLyqxB.mm1OOPf660/exARXPN5uC.gHaWziv7C4t/m

14
docker/README.md Normal file
View File

@ -0,0 +1,14 @@
<!-- -*-GFM-*- -->
# Docker Resources and Tests
This directory contains resources and tests for the docker image.
Note that for these tests to run, the pytest process must be able to run
`docker`. If you are on a system where that requires `sudo`, you will need to
run the tests with `sudo`.
Tests are here rather than in `/tests` because there's no reason to run these
tests as part of the usual `tox` process, which is run in CI against every
supported Python version. We only need to run the Docker tests once.

0
docker-requirements.txt → docker/docker-requirements.txt
View File

130
docker/entrypoint.sh Executable file
View File

@ -0,0 +1,130 @@
#!/usr/bin/env bash
set -euo pipefail
function run() {
# we're not root. Run as who we are.
if [[ "$EUID" -ne 0 ]]; then
eval "$@"
else
gosu pypiserver "$@"
fi
}
if [[ "$EUID" -ne 0 && "$EUID" -ne $(id -u pypiserver) ]]; then
USER_ID="$EUID"
WARN=(
"The pypiserver container was run as a non-root, non-pypiserver user."
"Pypiserver will be run as this user if possible, but this is not"
"officially supported."
)
echo "" 1>&2
echo "${WARN[@]}" 1>&2
echo "" 1>&2
else
USER_ID=$(id -u pypiserver)
fi
function print_permissions_help() {
MSG1=(
"If you are mounting a volume at /data or /data/packages and are running the"
"container on a linux system, you may need to add add a pypiserver"
"group to the host and give it permission to access the directories."
"Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
"details."
)
MSG2=(
"Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
"details."
)
echo "" 1>&2
echo "${MSG1[@]}" 1>&2
echo "" 1>&2
echo "${MSG2[@]}" 1>&2
}
# the user must have read and execute access to the /data directory
# (execute to be able to cd into directory and list content metadata)
if ! run test -r /data -a -x /data; then
chown -R "$USER_ID:pypiserver" /data || true
if ! run test -r /data -a -x /data; then
FAIL_MSG=(
"Cannot start pypiserver:"
"pypiserver user (UID $USER_ID)"
"or pypiserver group (GID $(id -g pypiserver))"
"must have read/execute access to /data"
)
echo "${FAIL_MSG[@]}" 1>&2
echo "" 1>&2
print_permissions_help
exit 1
fi
fi
# The /data/packages directory must exist
# It not existing is very unlikely, possibly impossible, because the VOLUME
# specification in the Dockerfile leads to its being created even if someone is
# mounting a volume at /data that does not contain a /packages subdirectory
if [[ ! -d "/data/packages" ]]; then
if ! run test -w /data; then
FAIL_MSG=(
"Cannot start pypiserver:"
"/data/packages does not exist and"
"pypiserver user (UID $USER_ID)"
"or pypiserver group (GID $(id -g pypiserver))"
"does not have write access to /data to create it"
)
echo "" 1>&2
echo "${FAIL_MSG[@]}" 1>&2
print_permissions_help
exit 1
fi
run mkdir /data/packages
fi
# The pypiserver user needs read/write/execute access to the packages directory
if ! run \
test -w /data/packages \
-a -r /data/packages \
-a -x /data/packages; then
# We'll try to chown as a last resort.
# Don't complain if it fails, since we'll bomb on the next check anyway.
chown -R "$USER_ID:pypiserver" /data/packages || true
if ! run \
test -w /data/packages \
-a -r /data/packages \
-a -x /data/packages; then
FAIL_MSG=(
"Cannot start pypiserver:"
"pypiserver user (UID $USER_ID)"
"or pypiserver group (GID $(id -g pypiserver))"
"must have read/write/execute access to /data/packages"
)
echo "" 1>&2
echo "${FAIL_MSG[@]}" 1>&2
print_permissions_help
exit 1
fi
fi
if [[ "$*" == "" ]]; then
CMD=("run" "-p" "${PYPISERVER_PORT:-$PORT}")
else
# this reassigns the array to the CMD variable
CMD=( "${@}" )
fi
if [[ "$EUID" -ne 0 ]]; then
exec pypi-server "${CMD[@]}"
else
exec gosu pypiserver pypi-server "${CMD[@]}"
fi

538
docker/test_docker.py Normal file
View File

@ -0,0 +1,538 @@
"""Tests for the Pypiserver Docker image."""
import contextlib
import os
import shutil
import socket
import subprocess
import sys
import tempfile
import time
import typing as t
from pathlib import Path
import httpx
import pypiserver
import pytest
THIS_DIR = Path(__file__).parent
ROOT_DIR = THIS_DIR.parent
DOCKERFILE = ROOT_DIR / "Dockerfile"
FIXTURES = ROOT_DIR / "fixtures"
MYPKG_ROOT = FIXTURES / "mypkg"
HTPASS_FILE = FIXTURES / "htpasswd.a.a"
# This is largely useless when using pytest because of the need to use the name
# of the fixture as an argument to the test function or fixture using it
# pylint: disable=redefined-outer-name
#
# Also useless for our test context, where we may want to group test functions
# in a class to share common fixtures, but where we don't care about the
# `self` instance.
# pylint: disable=no-self-use
@pytest.fixture(scope="session")
def image() -> str:
"""Build the docker image for pypiserver.
Return the tag.
"""
tag = "pypiserver:test"
run(
"docker",
"build",
"--file",
str(DOCKERFILE),
"--tag",
tag,
str(ROOT_DIR),
cwd=ROOT_DIR,
)
return tag
@pytest.fixture(scope="session")
def mypkg_build() -> None:
"""Ensure the mypkg test fixture package is build."""
# Use make for this so that it will skip the build step if it's not needed
run("make", "mypkg", cwd=ROOT_DIR)
@pytest.fixture(scope="session")
def mypkg_paths(
mypkg_build: None, # pylint: disable=unused-argument
) -> t.Dict[str, Path]:
"""The path to the mypkg sdist file."""
dist_dir = Path(MYPKG_ROOT) / "dist"
assert dist_dir.exists()
sdist = dist_dir / "pypiserver_mypkg-1.0.0.tar.gz"
assert sdist.exists()
wheel = dist_dir / "pypiserver_mypkg-1.0.0-py2.py3-none-any.whl"
assert wheel.exists()
return {
"dist_dir": dist_dir,
"sdist": sdist,
"wheel": wheel,
}
def wait_for_container(port: int) -> None:
"""Wait for the container to be available."""
for _ in range(60):
try:
httpx.get(f"http://localhost:{port}").raise_for_status()
except (httpx.RequestError, httpx.HTTPStatusError):
time.sleep(1)
else:
return
# If we reach here, we've tried 60 times without success, meaning either
# the container is broken or it took more than about a minute to become
# functional, either of which cases is something we will want to look into.
raise RuntimeError("Could not connect to pypiserver container")
def get_socket() -> int:
"""Find a random, open socket and return it."""
# Close the socket automatically upon exiting the block
with contextlib.closing(
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
) as sock:
# Bind to a random open socket >=1024
sock.bind(("", 0))
# Return the socket number
return sock.getsockname()[1]
class RunReturn(t.NamedTuple):
"""Simple wrapper around a simple subprocess call's results."""
returncode: int
out: str
err: str
def run(
*cmd: str,
capture: bool = False,
raise_on_err: bool = True,
check_code: t.Callable[[int], bool] = lambda c: c == 0,
**popen_kwargs: t.Any,
) -> RunReturn:
"""Run a command to completion."""
stdout = subprocess.PIPE if capture else None
stderr = subprocess.PIPE if capture else None
proc = subprocess.Popen(cmd, stdout=stdout, stderr=stderr, **popen_kwargs)
out, err = proc.communicate()
result = RunReturn(
proc.returncode,
"" if out is None else out.decode(),
"" if err is None else err.decode(),
)
if raise_on_err and not check_code(result.returncode):
raise RuntimeError(result)
return result
def uninstall_pkgs() -> None:
"""Uninstall any packages we've installed."""
res = run("pip", "freeze", capture=True)
if any(
ln.strip().startswith("pypiserver-mypkg") for ln in res.out.splitlines()
):
run("pip", "uninstall", "-y", "pypiserver-mypkg")
@pytest.fixture(scope="session", autouse=True)
def session_cleanup() -> t.Iterator[None]:
"""Deal with any pollution of the local env."""
yield
uninstall_pkgs()
@pytest.fixture()
def cleanup() -> t.Iterator[None]:
"""Clean up after tests that may have affected the env."""
yield
uninstall_pkgs()
class TestCommands:
"""Test commands other than `run`."""
def test_help(self, image: str) -> None:
"""We can get help from the docker container."""
res = run("docker", "run", image, "--help", capture=True)
assert "pypi-server" in res.out
def test_version(self, image: str) -> None:
"""We can get the version from the docker container."""
res = run("docker", "run", image, "--version", capture=True)
assert res.out.strip() == pypiserver.__version__
class TestPermissions:
"""Test permission validation, especially with mounted volumes."""
@pytest.mark.parametrize("perms", (0o706, 0o701, 0o704))
def test_needs_rx_on_data(self, image: str, perms: int) -> None:
"""Read and execute permissions are required on /data."""
# Note we can't run this one as root because then we have to make a file
# that even we can't delete.
with tempfile.TemporaryDirectory() as tmpdir:
# Make sure the directory is not readable for anyone other than
# the owner
data_dir = Path(tmpdir) / "data"
data_dir.mkdir(mode=perms)
res = run(
"docker",
"run",
"--rm",
"--user",
# Run as a not us user ID, so access to /data will be
# determined by the "all other users" setting
str(os.getuid() + 1),
"-v",
# Mount the temporary directory as the /data directory
f"{data_dir}:/data",
image,
capture=True,
# This should error out, so we check that the code is non-zero
check_code=lambda c: c != 0,
)
assert "must have read/execute access" in res.err
@pytest.mark.parametrize(
"extra_args",
(("--user", str(os.getuid())), ("--user", str(os.getuid() + 1))),
)
def test_needs_rwx_on_packages(self, image: str, extra_args: tuple) -> None:
"""RWX permission is required on /data/packages."""
with tempfile.TemporaryDirectory() as tmpdir:
td_path = Path(tmpdir)
# Make the /data directory read/writable by anyone
td_path.chmod(0o777)
# Add the /data/packages directory, and make it readable by anyone,
# but writable only by the owner
(td_path / "packages").mkdir(mode=0o444)
res = run(
"docker",
"run",
"--rm",
*extra_args,
"-v",
# Mount the temporary directory as the /data directory
f"{tmpdir}:/data",
image,
capture=True,
# We should error out in this case
check_code=lambda c: c != 0,
)
assert "must have read/write/execute access" in res.err
def test_runs_as_pypiserver_user(self, image: str) -> None:
"""Test that the default run uses the pypiserver user."""
host_port = get_socket()
res = run(
"docker",
"run",
"--rm",
"--detach",
"--publish",
f"{host_port}:8080",
image,
capture=True,
)
container_id = res.out.strip()
try:
wait_for_container(host_port)
res = run(
"docker",
"container",
"exec",
container_id,
"ps",
"a",
capture=True,
)
proc_line = next(
filter(
# grab the process line for the pypi-server process
lambda ln: "pypi-server" in ln,
res.out.splitlines(),
)
)
user = proc_line.split()[1]
# the ps command on these alpine containers doesn't always show the
# full user name, so we only check for the first bit
assert user.startswith("pypi")
finally:
run("docker", "container", "rm", "-f", container_id)
class TestBasics:
"""Test basic pypiserver functionality in a simple unauthed container."""
HOST_PORT = get_socket()
@pytest.fixture(scope="class")
def container(self, image: str) -> t.Iterator[str]:
"""Run the pypiserver container.
Returns the container ID.
"""
res = run(
"docker",
"run",
"--rm",
"--publish",
f"{self.HOST_PORT}:8080",
"--detach",
image,
"run",
"--passwords",
".",
"--authenticate",
".",
capture=True,
)
wait_for_container(self.HOST_PORT)
container_id = res.out.strip()
yield container_id
run("docker", "container", "rm", "-f", container_id)
@pytest.fixture(scope="class")
def upload_mypkg(
self,
container: str, # pylint: disable=unused-argument
mypkg_paths: t.Dict[str, Path],
) -> None:
"""Upload mypkg to the container."""
run(
sys.executable,
"-m",
"twine",
"upload",
"--repository-url",
f"http://localhost:{self.HOST_PORT}",
"--username",
"",
"--password",
"",
f"{mypkg_paths['dist_dir']}/*",
)
@pytest.mark.usefixtures("upload_mypkg")
def test_download(self) -> None:
"""Download mypkg from the container."""
with tempfile.TemporaryDirectory() as tmpdir:
run(
sys.executable,
"-m",
"pip",
"download",
"--index-url",
f"http://localhost:{self.HOST_PORT}/simple",
"--dest",
tmpdir,
"pypiserver_mypkg",
)
assert any(
"pypiserver_mypkg" in path.name
for path in Path(tmpdir).iterdir()
)
@pytest.mark.usefixtures("upload_mypkg", "cleanup")
def test_install(self) -> None:
"""Install mypkg from the container.
Note this also ensures that name normalization is working,
since we are requesting the package name with a dash, rather
than an underscore.
"""
run(
sys.executable,
"-m",
"pip",
"install",
"--index-url",
f"http://localhost:{self.HOST_PORT}/simple",
"pypiserver-mypkg",
)
run("python", "-c", "'import pypiserver_mypkg; mypkg.pkg_name()'")
@pytest.mark.usefixtures("container")
def test_welcome(self) -> None:
"""View the welcome page."""
resp = httpx.get(f"http://localhost:{self.HOST_PORT}")
assert resp.status_code == 200
assert "pypiserver" in resp.text
class TestAuthed:
"""Test basic pypiserver functionality in a simple unauthed container."""
HOST_PORT = get_socket()
@pytest.fixture(scope="class")
def container(self, image: str) -> t.Iterator[str]:
"""Run the pypiserver container.
Returns the container ID.
"""
with tempfile.TemporaryDirectory() as tmpdir:
dirpath = Path(tmpdir)
shutil.copy2(HTPASS_FILE, dirpath / "htpasswd")
pkg_path = dirpath / "packages"
pkg_path.mkdir(mode=0o777)
res = run(
"docker",
"run",
"--rm",
"--publish",
f"{self.HOST_PORT}:8080",
"-v",
f"{dirpath / 'htpasswd'}:/data/htpasswd",
"--detach",
image,
"run",
"--passwords",
"/data/htpasswd",
"--authenticate",
"download, update",
capture=True,
)
wait_for_container(self.HOST_PORT)
container_id = res.out.strip()
yield container_id
run("docker", "container", "rm", "-f", container_id)
@pytest.fixture(scope="class")
def upload_mypkg(
self,
container: str, # pylint: disable=unused-argument
mypkg_paths: t.Dict[str, Path],
) -> None:
"""Upload mypkg to the container."""
run(
sys.executable,
"-m",
"twine",
"upload",
"--repository-url",
f"http://localhost:{self.HOST_PORT}",
"--username",
"a",
"--password",
"a",
f"{mypkg_paths['dist_dir']}/*",
)
def test_upload_failed_auth(
self,
container: str, # pylint: disable=unused-argument
mypkg_paths: t.Dict[str, Path],
) -> None:
"""Upload mypkg to the container."""
run(
sys.executable,
"-m",
"twine",
"upload",
"--repository-url",
f"http://localhost:{self.HOST_PORT}",
f"{mypkg_paths['dist_dir']}/*",
check_code=lambda c: c != 0,
)
@pytest.mark.usefixtures("upload_mypkg")
def test_download(self) -> None:
"""Download mypkg from the container."""
with tempfile.TemporaryDirectory() as tmpdir:
run(
sys.executable,
"-m",
"pip",
"download",
"--index-url",
f"http://a:a@localhost:{self.HOST_PORT}/simple",
"--dest",
tmpdir,
"pypiserver_mypkg",
)
assert any(
"pypiserver_mypkg" in path.name
for path in Path(tmpdir).iterdir()
)
@pytest.mark.usefixtures("upload_mypkg")
def test_download_failed_auth(self) -> None:
"""Download mypkg from the container."""
with tempfile.TemporaryDirectory() as tmpdir:
run(
sys.executable,
"-m",
"pip",
"download",
"--index-url",
f"http://localhost:{self.HOST_PORT}/simple",
"--dest",
tmpdir,
"pypiserver_mypkg",
check_code=lambda c: c != 0,
)
@pytest.mark.usefixtures("upload_mypkg", "cleanup")
def test_install(self) -> None:
"""Install mypkg from the container.
Note this also ensures that name normalization is working,
since we are requesting the package name with a dash, rather
than an underscore.
"""
run(
sys.executable,
"-m",
"pip",
"install",
"--index-url",
f"http://a:a@localhost:{self.HOST_PORT}/simple",
"pypiserver-mypkg",
)
run("python", "-c", "'import pypiserver_mypkg; mypkg.pkg_name()'")
@pytest.mark.usefixtures("upload_mypkg", "cleanup")
def test_install_failed_auth(self) -> None:
"""Install mypkg from the container.
Note this also ensures that name normalization is working,
since we are requesting the package name with a dash, rather
than an underscore.
"""
run(
sys.executable,
"-m",
"pip",
"install",
"--no-cache",
"--index-url",
f"http://localhost:{self.HOST_PORT}/simple",
"pypiserver-mypkg",
check_code=lambda c: c != 0,
)
def test_welcome(self) -> None:
"""View the welcome page."""
resp = httpx.get(f"http://localhost:{self.HOST_PORT}")
assert resp.status_code == 200
assert "pypiserver" in resp.text

11
entrypoint.sh
View File

@ -1,11 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# chown the VOLUME mount set in the dockerfile
# If you're using an alternative directory for packages,
# you'll need to ensure that pypiserver has read and
# write access to that directory
chown -R pypiserver:pypiserver /data/packages
exec gosu pypiserver pypi-server -p "$PORT" $@

0
tests/htpasswd.a.a → fixtures/htpasswd.a.a
View File

6
fixtures/mypkg/mypkg/__init__.py Normal file
View File

@ -0,0 +1,6 @@
"""A very simple python file to package for testing."""
def pkg_name() -> None:
"""Print the package name."""
print("mypkg")

7
fixtures/mypkg/setup.cfg Normal file
View File

@ -0,0 +1,7 @@
[wheel]
universal=1
[mypy]
follow_imports = silent
ignore_missing_imports = True

10
fixtures/mypkg/setup.py Normal file
View File

@ -0,0 +1,10 @@
"""A simple setup file for this test package."""
from setuptools import setup
setup(
name="pypiserver_mypkg",
description="Test pkg",
version="1.0.0",
packages=["mypkg"],
)

5
requirements/test.pip
View File

@ -2,9 +2,10 @@
# running tests
gevent>=1.1b4; python_version >= '3'
pip>=7
httpx
pip
passlib>=1.6
pytest>=2.3
pytest>=6
pytest-cov
setuptools
tox

3
setup.cfg
View File

@ -27,3 +27,6 @@ warn_unused_ignores = True
[mypy-tests.*]
disallow_untyped_decorators = False
[mypy-test_docker.*]
disallow_untyped_decorators = False

2
tests/test_config.py
View File

@ -14,7 +14,7 @@ from pypiserver.config import DEFAULTS, Config, RunConfig, UpdateConfig
FILE_DIR = pathlib.Path(__file__).parent.resolve()
# Username and password stored in the htpasswd.a.a test file.
HTPASS_TEST_FILE = str(FILE_DIR / "htpasswd.a.a")
HTPASS_TEST_FILE = str(FILE_DIR / "../fixtures/htpasswd.a.a")
HTPASS_TEST_USER = "a"
HTPASS_TEST_PASS = "a"

2
tests/test_init.py
View File

@ -16,7 +16,7 @@ import pypiserver
logger = logging.getLogger(__name__)
TEST_DIR = pathlib.Path(__file__).parent
HTPASS_FILE = TEST_DIR / "htpasswd.a.a"
HTPASS_FILE = TEST_DIR / "../fixtures/htpasswd.a.a"
WELCOME_FILE = TEST_DIR / "sample_msg.html"

2
tests/test_main.py
View File

@ -13,7 +13,7 @@ from pypiserver.bottle import Bottle
THIS_DIR = pathlib.Path(__file__).parent
HTPASS_FILE = THIS_DIR / "htpasswd.a.a"
HTPASS_FILE = THIS_DIR / "../fixtures/htpasswd.a.a"
IGNORELIST_FILE = THIS_DIR / "test-ignorelist"

19
tests/test_server.py
View File

@ -18,6 +18,7 @@ import shutil
import socket
import sys
import time
import typing as t
from collections import namedtuple
from pathlib import Path
from shlex import split
@ -40,7 +41,9 @@ Srv = namedtuple("Srv", ("port", "root"))
@contextlib.contextmanager
def run_server(root, authed=False, other_cli=""):
"""Run a server, optionally with partial auth enabled."""
htpasswd = CURRENT_PATH.joinpath("htpasswd.a.a").expanduser().resolve()
htpasswd = (
CURRENT_PATH.joinpath("../fixtures/htpasswd.a.a").expanduser().resolve()
)
pswd_opt_choices = {
True: f"-P {htpasswd} -a update,download",
False: "-P. -a.",
@ -91,7 +94,7 @@ def _kill_proc(proc):
proc.kill()
def build_url(port, user="", pswd=""):
def build_url(port: t.Union[int, str], user: str = "", pswd: str = "") -> str:
auth = f"{user}:{pswd}@" if user or pswd else ""
return f"http://{auth}localhost:{port}"
@ -196,12 +199,18 @@ def empty_packdir(tmp_path_factory):
return tmp_path_factory.mktemp("dists")
def pip_download(cmd, port, install_dir, user=None, pswd=None):
def pip_download(
cmd: str,
port: t.Union[int, str],
install_dir: str,
user: str = None,
pswd: str = None,
) -> int:
url = build_url(port, user, pswd)
return _run_pip(f"-vv download -d {install_dir} -i {url} {cmd}")
def _run_pip(cmd):
def _run_pip(cmd: str) -> int:
ncmd = (
"pip --no-cache-dir --disable-pip-version-check "
f"--retries 0 --timeout 5 --no-input {cmd}"
@ -260,7 +269,7 @@ def authed_pypirc(authed_server):
yield path
def run_twine(command, package, conf):
def run_twine(command: str, package: str, conf: str) -> None:
proc = Popen(
split(
f"twine {command} --repository test --config-file {conf} {package}"

2
tox.ini
View File

@ -8,7 +8,7 @@ allowlist_externals=
sitepackages=False
commands=
/bin/sh -c "{env:PYPISERVER_SETUP_CMD:true}"
pytest --cov=pypiserver {posargs}
pytest --cov=pypiserver {posargs:tests}
[pytest]