Docker improvements (#365)

* Docker improvements

This addresses much of what was brought up in #359. Specifically, it:

- Significantly improves testing for the Docker image, adding a
  `docker/test_docker.py` file using the regular pytest machinery to
  set up and run docker images for testing
- Hopefully addresses a variety of permissions issues, by being explicit
  about what access pypiserver needs and asking for it, only erroring
  if that access is not available
  - Requires RX permissions on `/data` (R to read files, X to list files
    and to be able to cd into the directory. This is important since
    `/data` is the `WORKDIR`)
  - Requires RWX permissions on `/data/packages`, so that we can list
    packages, write packages, and read packages.
  - When running in the default configuration (as root on Linux or
    as the pypiserver-named rootish user on Mac), with no volumes
    mounted, these requirements are all satisfied
  - Volume mounts still must be readable by the pypiserver user (UID
    9898) in order for the container to run. However, we now error early
    if this is not the case, and direct users to a useful issue.
  - If the container is run as a non-root, non-pypiserver user (e.g.
    because someone ran `docker run --user=<user_id>`, we try to run
    pypiserver as that user). Provided that user has access to the
    necessary directories, it should run fine.
- Fixes issues with running help and similar commands
- Updates the Docker image to use `PYPISERVER_PORT` for port
  specification, while still falling back to `PORT` for backwards
  compatibility
- Moves some docker-related things into a `/docker` directory
- Adds a `Makefile` for building a test fixture package sdist and wheel,
  so that test code can call `make mypkg` and not need to worry about it
  potentially building multiple times

The only issue #359 raises that's not addressed here is the one of
running pypiserver in the Docker container using some non-default server
for performance. I would like to do some benchmarking before deciding on
what to do there.

.dockerignore

@@ -1,8 +1,8 @@
 *
 !pypiserver
 !requirements
-!docker-requirements.txt
+!docker/docker-requirements.txt
-!entrypoint.sh
+!docker/entrypoint.sh
 !README.rst
 !setup.cfg
 !setup.py

.github/workflows/test.yml

@@ -34,7 +34,8 @@ jobs:
   test-pypy:
 
     # Run a a separate job so we don't need to mess with conditionally
-    # splitting the python version from the build matrix.
+    # splitting the python version from the build matrix. Also the pypy
+    # tests take freaking forever.
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
@@ -64,9 +65,10 @@ jobs:
     - name: Check types
       # individual mypy files for now, until we get the rest
       # of the project typechecking
-      run: |
+      run: >-
-        mypy \
+        mypy
-         pypiserver/config.py \
+        docker/test_docker.py
+        pypiserver/config.py 
         tests/test_init.py
     - name: Check formatting
       run: black --diff --check .
@@ -74,8 +76,13 @@ jobs:
       run: ./bin/check_readme.sh
 
   # Full-flow docker tests, again not python version dependent
-  docker:
+  # We _could_ test this on MacOS, but it takes forever to get docker
-    runs-on: ubuntu-latest
+  # installed. I'm going to say for now probably 99% of people using
+  # the docker image will be doing so from a linux system, e.g. for
+  # a k8s deploy, and I've verified manually that things work on
+  # MacOS, so /shrug.
+  test-docker:
+    runs-on: "ubuntu-latest"
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python
@@ -83,17 +90,19 @@ jobs:
       with:
         # Pretty much any python version will do
         python-version: "3.9"
-    - name: Install dependencies
+    - name: Install test dependencies
-      run: pip install --use-feature 2020-resolver -U twine
+      run: pip install -r "requirements/test.pip"
+    - name: Install package
+      run: pip install -r "requirements/exe.pip"
     - name: Run tests
-      run: ./bin/test-docker.sh
+      run: "pytest docker/test_docker.py"
 
 
   tests:
     runs-on: "ubuntu-latest"
     needs:
       - "check"
-      - "docker"
+      - "test-docker"
       - "test-cpython"
       - "test-pypy"
     steps:

Dockerfile

@@ -33,7 +33,7 @@ FROM base AS builder_dependencies
 
 COPY pypiserver /code/pypiserver
 COPY requirements /code/requirements
-COPY docker-requirements.txt /code
+COPY docker/docker-requirements.txt /code
 COPY setup.cfg /code
 COPY setup.py /code
 COPY README.rst /code
@@ -50,7 +50,7 @@ FROM base
 # Copy the libraries installed via pip
 COPY --from=builder_dependencies /install /usr/local
 COPY --from=builder_gosu /usr/local/bin/gosu /usr/local/bin/gosu
-COPY entrypoint.sh /entrypoint.sh
+COPY docker/entrypoint.sh /entrypoint.sh
 
 # Use a consistent user and group ID so that linux users
 # can create a corresponding system user and set permissions
@@ -65,7 +65,9 @@ RUN apk add bash \
 
 VOLUME /data/packages
 WORKDIR /data
-ENV PORT=8080
+ENV PYPISERVER_PORT=8080
-EXPOSE $PORT
+# PORT is deprecated. Please use PYPISERVER_PORT instead
+ENV PORT=$PYPISERVER_PORT
+EXPOSE $PYPISERVER_PORT
 
 ENTRYPOINT ["/entrypoint.sh"]

Makefile

@@ -0,0 +1,23 @@
+##
+# pypiserver
+#
+# this makefile is used to help with building resources needed for testing
+#
+# @file
+# @version 0.1
+
+SHELL = /bin/sh
+
+MYPKG_SRC = fixtures/mypkg/setup.py $(shell find fixtures/mypkg/mypkg -type f -name '*.py')
+
+# Build the test fixture package.
+mypkg: fixtures/mypkg/dist/pypiserver_mypkg-1.0.0.tar.gz
+mypkg: fixtures/mypkg/dist/pypiserver_mypkg-1.0.0-py2.py3-none-any.whl
+fixtures/mypkg/dist/pypiserver_mypkg-1.0.0.tar.gz: $(MYPKG_SRC)
+	cd fixtures/mypkg; python setup.py sdist
+
+fixtures/mypkg/dist/pypiserver_mypkg-1.0.0-py2.py3-none-any.whl: $(MYPKG_SRC)
+	cd fixtures/mypkg; python setup.py bdist_wheel
+
+
+# end

bin/test-docker.sh

@@ -1,49 +0,0 @@
-#!/usr/bin/env sh
-
-# Perform some simple validation to make sure the Docker image works
-# Should be run from the repo root.
-
-set -xe  # exit on any error, show debug output
-
-DIR="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )"
-
-docker build . -t pypiserver:test
-
-docker run pypiserver:test --help > /dev/null
-
-# Mount our htpasswd file, which contains a test user with a bcrypt-encrypted
-# "test" password
-CONTAINER_ID=$(docker run \
-    -d \
-    -v "${DIR}/test.htpasswd:/data/.htpasswd" \
-    -p 8080:8080 \
-    -e PORT=8080 \
-    pypiserver:test -a "list,update,download" -P /data/.htpasswd packages)
-
-trap "docker container stop $CONTAINER_ID" EXIT
-
-sleep 15  # give the container some time to get going
-
-# Ensure we can authenticate locally
-RET=$(curl localhost:8080)
-echo $RET
-echo $RET | grep -q "pypiserver"
-
-RET=$(curl localhost:8080/packages/)
-echo $RET
-echo $RET | grep -q "401"
-
-
-RET=$(curl test:test@localhost:8080/packages/)
-echo $RET
-echo $RET | grep -q "Index of packages"
-
-twine upload \
-    -u test \
-    -p test \
-    --repository-url http://localhost:8080 \
-    "${DIR}/pypiserver-1.2.6-py2.py3-none-any.whl"
-
-RET=$(curl test:test@localhost:8080/packages/)
-echo $RET
-echo $RET | grep -q "pypiserver-1.2.6"

bin/test.htpasswd

@@ -1,2 +0,0 @@
-test:$2y$05$0wU8vmgucWeyLyqxB.mm1OOPf660/exARXPN5uC.gHaWziv7C4t/m
-

docker/README.md

@@ -0,0 +1,14 @@
+<!-- -*-GFM-*- -->
+
+# Docker Resources and Tests
+
+This directory contains resources and tests for the docker image.
+
+Note that for these tests to run, the pytest process must be able to run
+`docker`. If you are on a system where that requires `sudo`, you will need to
+run the tests with `sudo`.
+
+Tests are here rather than in `/tests` because there's no reason to run these
+tests as part of the usual `tox` process, which is run in CI against every
+supported Python version. We only need to run the Docker tests once.
+

docker/entrypoint.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+
+set -euo pipefail
+
+function run() {
+    # we're not root. Run as who we are.
+    if [[ "$EUID" -ne 0 ]]; then
+        eval "$@"
+    else
+        gosu pypiserver "$@"
+    fi
+}
+
+if [[ "$EUID" -ne 0 && "$EUID" -ne $(id -u pypiserver) ]]; then
+    USER_ID="$EUID"
+    WARN=(
+        "The pypiserver container was run as a non-root, non-pypiserver user."
+        "Pypiserver will be run as this user if possible, but this is not"
+        "officially supported."
+    )
+    echo "" 1>&2
+    echo "${WARN[@]}" 1>&2
+    echo "" 1>&2
+else
+    USER_ID=$(id -u pypiserver)
+fi
+
+
+function print_permissions_help() {
+    MSG1=(
+        "If you are mounting a volume at /data or /data/packages and are running the"
+        "container on a linux system, you may need to add add a pypiserver"
+        "group to the host and give it permission to access the directories."
+        "Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
+        "details."
+    )
+    MSG2=(
+        "Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
+        "details."
+    )
+    echo "" 1>&2
+    echo "${MSG1[@]}" 1>&2
+    echo "" 1>&2
+    echo "${MSG2[@]}" 1>&2
+}
+
+
+# the user must have read and execute access to the /data directory
+# (execute to be able to cd into directory and list content metadata)
+if ! run test -r /data -a -x /data; then
+
+    chown -R "$USER_ID:pypiserver" /data || true
+
+    if ! run test -r /data -a -x /data; then
+        FAIL_MSG=(
+            "Cannot start pypiserver:"
+            "pypiserver user (UID $USER_ID)"
+            "or pypiserver group (GID $(id -g pypiserver))"
+            "must have read/execute access to /data"
+        )
+        echo "${FAIL_MSG[@]}" 1>&2
+        echo "" 1>&2
+        print_permissions_help
+        exit 1
+    fi
+
+fi
+
+# The /data/packages directory must exist
+# It not existing is very unlikely, possibly impossible, because the VOLUME
+# specification in the Dockerfile leads to its being created even if someone is
+# mounting a volume at /data that does not contain a /packages subdirectory
+if [[ ! -d "/data/packages" ]]; then
+    if ! run test -w /data; then
+        FAIL_MSG=(
+            "Cannot start pypiserver:"
+            "/data/packages does not exist and"
+            "pypiserver user (UID $USER_ID)"
+            "or pypiserver group (GID $(id -g pypiserver))"
+            "does not have write access to /data to create it"
+        )
+        echo "" 1>&2
+        echo "${FAIL_MSG[@]}" 1>&2
+        print_permissions_help
+        exit 1
+    fi
+    run mkdir /data/packages
+fi
+
+# The pypiserver user needs read/write/execute access to the packages directory
+if ! run \
+    test -w /data/packages \
+    -a -r /data/packages \
+    -a -x /data/packages; then
+
+    # We'll try to chown as a last resort.
+    # Don't complain if it fails, since we'll bomb on the next check anyway.
+    chown -R "$USER_ID:pypiserver" /data/packages || true
+
+    if ! run \
+        test -w /data/packages \
+        -a -r /data/packages \
+        -a -x /data/packages; then
+        FAIL_MSG=(
+            "Cannot start pypiserver:"
+            "pypiserver user (UID $USER_ID)"
+            "or pypiserver group (GID $(id -g pypiserver))"
+            "must have read/write/execute access to /data/packages"
+        )
+        echo "" 1>&2
+        echo "${FAIL_MSG[@]}" 1>&2
+        print_permissions_help
+        exit 1
+    fi
+
+fi
+
+
+if [[ "$*" == "" ]]; then
+    CMD=("run" "-p" "${PYPISERVER_PORT:-$PORT}")
+else
+    # this reassigns the array to the CMD variable
+    CMD=( "${@}" )
+fi
+
+if [[ "$EUID" -ne 0 ]]; then
+    exec pypi-server "${CMD[@]}"
+else
+    exec gosu pypiserver pypi-server "${CMD[@]}"
+fi

docker/test_docker.py

@@ -0,0 +1,538 @@
+"""Tests for the Pypiserver Docker image."""
+
+import contextlib
+import os
+import shutil
+import socket
+import subprocess
+import sys
+import tempfile
+import time
+import typing as t
+from pathlib import Path
+
+import httpx
+import pypiserver
+import pytest
+
+
+THIS_DIR = Path(__file__).parent
+ROOT_DIR = THIS_DIR.parent
+DOCKERFILE = ROOT_DIR / "Dockerfile"
+FIXTURES = ROOT_DIR / "fixtures"
+MYPKG_ROOT = FIXTURES / "mypkg"
+HTPASS_FILE = FIXTURES / "htpasswd.a.a"
+
+
+# This is largely useless when using pytest because of the need to use the name
+# of the fixture as an argument to the test function or fixture using it
+# pylint: disable=redefined-outer-name
+#
+# Also useless for our test context, where we may want to group test functions
+# in a class to share common fixtures, but where we don't care about the
+# `self` instance.
+# pylint: disable=no-self-use
+
+
+@pytest.fixture(scope="session")
+def image() -> str:
+    """Build the docker image for pypiserver.
+
+    Return the tag.
+    """
+    tag = "pypiserver:test"
+    run(
+        "docker",
+        "build",
+        "--file",
+        str(DOCKERFILE),
+        "--tag",
+        tag,
+        str(ROOT_DIR),
+        cwd=ROOT_DIR,
+    )
+    return tag
+
+
+@pytest.fixture(scope="session")
+def mypkg_build() -> None:
+    """Ensure the mypkg test fixture package is build."""
+    # Use make for this so that it will skip the build step if it's not needed
+    run("make", "mypkg", cwd=ROOT_DIR)
+
+
+@pytest.fixture(scope="session")
+def mypkg_paths(
+    mypkg_build: None,  # pylint: disable=unused-argument
+) -> t.Dict[str, Path]:
+    """The path to the mypkg sdist file."""
+    dist_dir = Path(MYPKG_ROOT) / "dist"
+    assert dist_dir.exists()
+
+    sdist = dist_dir / "pypiserver_mypkg-1.0.0.tar.gz"
+    assert sdist.exists()
+
+    wheel = dist_dir / "pypiserver_mypkg-1.0.0-py2.py3-none-any.whl"
+    assert wheel.exists()
+
+    return {
+        "dist_dir": dist_dir,
+        "sdist": sdist,
+        "wheel": wheel,
+    }
+
+
+def wait_for_container(port: int) -> None:
+    """Wait for the container to be available."""
+    for _ in range(60):
+        try:
+            httpx.get(f"http://localhost:{port}").raise_for_status()
+        except (httpx.RequestError, httpx.HTTPStatusError):
+            time.sleep(1)
+        else:
+            return
+
+    # If we reach here, we've tried 60 times without success, meaning either
+    # the container is broken or it took more than about a minute to become
+    # functional, either of which cases is something we will want to look into.
+    raise RuntimeError("Could not connect to pypiserver container")
+
+
+def get_socket() -> int:
+    """Find a random, open socket and return it."""
+    # Close the socket automatically upon exiting the block
+    with contextlib.closing(
+        socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    ) as sock:
+        # Bind to a random open socket >=1024
+        sock.bind(("", 0))
+        # Return the socket number
+        return sock.getsockname()[1]
+
+
+class RunReturn(t.NamedTuple):
+    """Simple wrapper around a simple subprocess call's results."""
+
+    returncode: int
+    out: str
+    err: str
+
+
+def run(
+    *cmd: str,
+    capture: bool = False,
+    raise_on_err: bool = True,
+    check_code: t.Callable[[int], bool] = lambda c: c == 0,
+    **popen_kwargs: t.Any,
+) -> RunReturn:
+    """Run a command to completion."""
+    stdout = subprocess.PIPE if capture else None
+    stderr = subprocess.PIPE if capture else None
+    proc = subprocess.Popen(cmd, stdout=stdout, stderr=stderr, **popen_kwargs)
+    out, err = proc.communicate()
+    result = RunReturn(
+        proc.returncode,
+        "" if out is None else out.decode(),
+        "" if err is None else err.decode(),
+    )
+    if raise_on_err and not check_code(result.returncode):
+        raise RuntimeError(result)
+    return result
+
+
+def uninstall_pkgs() -> None:
+    """Uninstall any packages we've installed."""
+    res = run("pip", "freeze", capture=True)
+    if any(
+        ln.strip().startswith("pypiserver-mypkg") for ln in res.out.splitlines()
+    ):
+        run("pip", "uninstall", "-y", "pypiserver-mypkg")
+
+
+@pytest.fixture(scope="session", autouse=True)
+def session_cleanup() -> t.Iterator[None]:
+    """Deal with any pollution of the local env."""
+    yield
+    uninstall_pkgs()
+
+
+@pytest.fixture()
+def cleanup() -> t.Iterator[None]:
+    """Clean up after tests that may have affected the env."""
+    yield
+    uninstall_pkgs()
+
+
+class TestCommands:
+    """Test commands other than `run`."""
+
+    def test_help(self, image: str) -> None:
+        """We can get help from the docker container."""
+        res = run("docker", "run", image, "--help", capture=True)
+        assert "pypi-server" in res.out
+
+    def test_version(self, image: str) -> None:
+        """We can get the version from the docker container."""
+        res = run("docker", "run", image, "--version", capture=True)
+        assert res.out.strip() == pypiserver.__version__
+
+
+class TestPermissions:
+    """Test permission validation, especially with mounted volumes."""
+
+    @pytest.mark.parametrize("perms", (0o706, 0o701, 0o704))
+    def test_needs_rx_on_data(self, image: str, perms: int) -> None:
+        """Read and execute permissions are required on /data."""
+        # Note we can't run this one as root because then we have to make a file
+        # that even we can't delete.
+        with tempfile.TemporaryDirectory() as tmpdir:
+            # Make sure the directory is not readable for anyone other than
+            # the owner
+            data_dir = Path(tmpdir) / "data"
+            data_dir.mkdir(mode=perms)
+
+            res = run(
+                "docker",
+                "run",
+                "--rm",
+                "--user",
+                # Run as a not us user ID, so access to /data will be
+                # determined by the "all other users" setting
+                str(os.getuid() + 1),
+                "-v",
+                # Mount the temporary directory as the /data directory
+                f"{data_dir}:/data",
+                image,
+                capture=True,
+                # This should error out, so we check that the code is non-zero
+                check_code=lambda c: c != 0,
+            )
+
+            assert "must have read/execute access" in res.err
+
+    @pytest.mark.parametrize(
+        "extra_args",
+        (("--user", str(os.getuid())), ("--user", str(os.getuid() + 1))),
+    )
+    def test_needs_rwx_on_packages(self, image: str, extra_args: tuple) -> None:
+        """RWX permission is required on /data/packages."""
+        with tempfile.TemporaryDirectory() as tmpdir:
+            td_path = Path(tmpdir)
+            # Make the /data directory read/writable by anyone
+            td_path.chmod(0o777)
+            # Add the /data/packages directory, and make it readable by anyone,
+            # but writable only by the owner
+            (td_path / "packages").mkdir(mode=0o444)
+
+            res = run(
+                "docker",
+                "run",
+                "--rm",
+                *extra_args,
+                "-v",
+                # Mount the temporary directory as the /data directory
+                f"{tmpdir}:/data",
+                image,
+                capture=True,
+                # We should error out in this case
+                check_code=lambda c: c != 0,
+            )
+            assert "must have read/write/execute access" in res.err
+
+    def test_runs_as_pypiserver_user(self, image: str) -> None:
+        """Test that the default run uses the pypiserver user."""
+        host_port = get_socket()
+        res = run(
+            "docker",
+            "run",
+            "--rm",
+            "--detach",
+            "--publish",
+            f"{host_port}:8080",
+            image,
+            capture=True,
+        )
+        container_id = res.out.strip()
+        try:
+            wait_for_container(host_port)
+            res = run(
+                "docker",
+                "container",
+                "exec",
+                container_id,
+                "ps",
+                "a",
+                capture=True,
+            )
+            proc_line = next(
+                filter(
+                    # grab the process line for the pypi-server process
+                    lambda ln: "pypi-server" in ln,
+                    res.out.splitlines(),
+                )
+            )
+            user = proc_line.split()[1]
+            # the ps command on these alpine containers doesn't always show the
+            # full user name, so we only check for the first bit
+            assert user.startswith("pypi")
+        finally:
+            run("docker", "container", "rm", "-f", container_id)
+
+
+class TestBasics:
+    """Test basic pypiserver functionality in a simple unauthed container."""
+
+    HOST_PORT = get_socket()
+
+    @pytest.fixture(scope="class")
+    def container(self, image: str) -> t.Iterator[str]:
+        """Run the pypiserver container.
+
+        Returns the container ID.
+        """
+        res = run(
+            "docker",
+            "run",
+            "--rm",
+            "--publish",
+            f"{self.HOST_PORT}:8080",
+            "--detach",
+            image,
+            "run",
+            "--passwords",
+            ".",
+            "--authenticate",
+            ".",
+            capture=True,
+        )
+        wait_for_container(self.HOST_PORT)
+        container_id = res.out.strip()
+        yield container_id
+        run("docker", "container", "rm", "-f", container_id)
+
+    @pytest.fixture(scope="class")
+    def upload_mypkg(
+        self,
+        container: str,  # pylint: disable=unused-argument
+        mypkg_paths: t.Dict[str, Path],
+    ) -> None:
+        """Upload mypkg to the container."""
+        run(
+            sys.executable,
+            "-m",
+            "twine",
+            "upload",
+            "--repository-url",
+            f"http://localhost:{self.HOST_PORT}",
+            "--username",
+            "",
+            "--password",
+            "",
+            f"{mypkg_paths['dist_dir']}/*",
+        )
+
+    @pytest.mark.usefixtures("upload_mypkg")
+    def test_download(self) -> None:
+        """Download mypkg from the container."""
+        with tempfile.TemporaryDirectory() as tmpdir:
+            run(
+                sys.executable,
+                "-m",
+                "pip",
+                "download",
+                "--index-url",
+                f"http://localhost:{self.HOST_PORT}/simple",
+                "--dest",
+                tmpdir,
+                "pypiserver_mypkg",
+            )
+            assert any(
+                "pypiserver_mypkg" in path.name
+                for path in Path(tmpdir).iterdir()
+            )
+
+    @pytest.mark.usefixtures("upload_mypkg", "cleanup")
+    def test_install(self) -> None:
+        """Install mypkg from the container.
+
+        Note this also ensures that name normalization is working,
+        since we are requesting the package name with a dash, rather
+        than an underscore.
+        """
+        run(
+            sys.executable,
+            "-m",
+            "pip",
+            "install",
+            "--index-url",
+            f"http://localhost:{self.HOST_PORT}/simple",
+            "pypiserver-mypkg",
+        )
+        run("python", "-c", "'import pypiserver_mypkg; mypkg.pkg_name()'")
+
+    @pytest.mark.usefixtures("container")
+    def test_welcome(self) -> None:
+        """View the welcome page."""
+        resp = httpx.get(f"http://localhost:{self.HOST_PORT}")
+        assert resp.status_code == 200
+        assert "pypiserver" in resp.text
+
+
+class TestAuthed:
+    """Test basic pypiserver functionality in a simple unauthed container."""
+
+    HOST_PORT = get_socket()
+
+    @pytest.fixture(scope="class")
+    def container(self, image: str) -> t.Iterator[str]:
+        """Run the pypiserver container.
+
+        Returns the container ID.
+        """
+        with tempfile.TemporaryDirectory() as tmpdir:
+            dirpath = Path(tmpdir)
+            shutil.copy2(HTPASS_FILE, dirpath / "htpasswd")
+            pkg_path = dirpath / "packages"
+            pkg_path.mkdir(mode=0o777)
+
+            res = run(
+                "docker",
+                "run",
+                "--rm",
+                "--publish",
+                f"{self.HOST_PORT}:8080",
+                "-v",
+                f"{dirpath / 'htpasswd'}:/data/htpasswd",
+                "--detach",
+                image,
+                "run",
+                "--passwords",
+                "/data/htpasswd",
+                "--authenticate",
+                "download, update",
+                capture=True,
+            )
+            wait_for_container(self.HOST_PORT)
+            container_id = res.out.strip()
+            yield container_id
+            run("docker", "container", "rm", "-f", container_id)
+
+    @pytest.fixture(scope="class")
+    def upload_mypkg(
+        self,
+        container: str,  # pylint: disable=unused-argument
+        mypkg_paths: t.Dict[str, Path],
+    ) -> None:
+        """Upload mypkg to the container."""
+        run(
+            sys.executable,
+            "-m",
+            "twine",
+            "upload",
+            "--repository-url",
+            f"http://localhost:{self.HOST_PORT}",
+            "--username",
+            "a",
+            "--password",
+            "a",
+            f"{mypkg_paths['dist_dir']}/*",
+        )
+
+    def test_upload_failed_auth(
+        self,
+        container: str,  # pylint: disable=unused-argument
+        mypkg_paths: t.Dict[str, Path],
+    ) -> None:
+        """Upload mypkg to the container."""
+        run(
+            sys.executable,
+            "-m",
+            "twine",
+            "upload",
+            "--repository-url",
+            f"http://localhost:{self.HOST_PORT}",
+            f"{mypkg_paths['dist_dir']}/*",
+            check_code=lambda c: c != 0,
+        )
+
+    @pytest.mark.usefixtures("upload_mypkg")
+    def test_download(self) -> None:
+        """Download mypkg from the container."""
+        with tempfile.TemporaryDirectory() as tmpdir:
+            run(
+                sys.executable,
+                "-m",
+                "pip",
+                "download",
+                "--index-url",
+                f"http://a:a@localhost:{self.HOST_PORT}/simple",
+                "--dest",
+                tmpdir,
+                "pypiserver_mypkg",
+            )
+            assert any(
+                "pypiserver_mypkg" in path.name
+                for path in Path(tmpdir).iterdir()
+            )
+
+    @pytest.mark.usefixtures("upload_mypkg")
+    def test_download_failed_auth(self) -> None:
+        """Download mypkg from the container."""
+        with tempfile.TemporaryDirectory() as tmpdir:
+            run(
+                sys.executable,
+                "-m",
+                "pip",
+                "download",
+                "--index-url",
+                f"http://localhost:{self.HOST_PORT}/simple",
+                "--dest",
+                tmpdir,
+                "pypiserver_mypkg",
+                check_code=lambda c: c != 0,
+            )
+
+    @pytest.mark.usefixtures("upload_mypkg", "cleanup")
+    def test_install(self) -> None:
+        """Install mypkg from the container.
+
+        Note this also ensures that name normalization is working,
+        since we are requesting the package name with a dash, rather
+        than an underscore.
+        """
+        run(
+            sys.executable,
+            "-m",
+            "pip",
+            "install",
+            "--index-url",
+            f"http://a:a@localhost:{self.HOST_PORT}/simple",
+            "pypiserver-mypkg",
+        )
+        run("python", "-c", "'import pypiserver_mypkg; mypkg.pkg_name()'")
+
+    @pytest.mark.usefixtures("upload_mypkg", "cleanup")
+    def test_install_failed_auth(self) -> None:
+        """Install mypkg from the container.
+
+        Note this also ensures that name normalization is working,
+        since we are requesting the package name with a dash, rather
+        than an underscore.
+        """
+        run(
+            sys.executable,
+            "-m",
+            "pip",
+            "install",
+            "--no-cache",
+            "--index-url",
+            f"http://localhost:{self.HOST_PORT}/simple",
+            "pypiserver-mypkg",
+            check_code=lambda c: c != 0,
+        )
+
+    def test_welcome(self) -> None:
+        """View the welcome page."""
+        resp = httpx.get(f"http://localhost:{self.HOST_PORT}")
+        assert resp.status_code == 200
+        assert "pypiserver" in resp.text

entrypoint.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-set -euo pipefail
-
-# chown the VOLUME mount set in the dockerfile
-# If you're using an alternative directory for packages,
-# you'll need to ensure that pypiserver has read and
-# write access to that directory
-chown -R pypiserver:pypiserver /data/packages
-
-exec gosu pypiserver pypi-server -p "$PORT" $@

fixtures/mypkg/mypkg/__init__.py

@@ -0,0 +1,6 @@
+"""A very simple python file to package for testing."""
+
+
+def pkg_name() -> None:
+    """Print the package name."""
+    print("mypkg")

fixtures/mypkg/setup.cfg

@@ -0,0 +1,7 @@
+[wheel]
+universal=1
+
+[mypy]
+follow_imports = silent
+ignore_missing_imports = True
+

fixtures/mypkg/setup.py

@@ -0,0 +1,10 @@
+"""A simple setup file for this test package."""
+
+from setuptools import setup
+
+setup(
+    name="pypiserver_mypkg",
+    description="Test pkg",
+    version="1.0.0",
+    packages=["mypkg"],
+)

requirements/test.pip

@@ -2,9 +2,10 @@
 # running tests
 
 gevent>=1.1b4; python_version >= '3'
-pip>=7
+httpx
+pip
 passlib>=1.6
-pytest>=2.3
+pytest>=6
 pytest-cov
 setuptools
 tox

setup.cfg

@@ -27,3 +27,6 @@ warn_unused_ignores = True
 
 [mypy-tests.*]
 disallow_untyped_decorators = False
+
+[mypy-test_docker.*]
+disallow_untyped_decorators = False

tests/test_config.py

@@ -14,7 +14,7 @@ from pypiserver.config import DEFAULTS, Config, RunConfig, UpdateConfig
 FILE_DIR = pathlib.Path(__file__).parent.resolve()
 
 # Username and password stored in the htpasswd.a.a test file.
-HTPASS_TEST_FILE = str(FILE_DIR / "htpasswd.a.a")
+HTPASS_TEST_FILE = str(FILE_DIR / "../fixtures/htpasswd.a.a")
 HTPASS_TEST_USER = "a"
 HTPASS_TEST_PASS = "a"
 

tests/test_init.py

@@ -16,7 +16,7 @@ import pypiserver
 logger = logging.getLogger(__name__)
 
 TEST_DIR = pathlib.Path(__file__).parent
-HTPASS_FILE = TEST_DIR / "htpasswd.a.a"
+HTPASS_FILE = TEST_DIR / "../fixtures/htpasswd.a.a"
 WELCOME_FILE = TEST_DIR / "sample_msg.html"
 
 

tests/test_main.py

@@ -13,7 +13,7 @@ from pypiserver.bottle import Bottle
 
 
 THIS_DIR = pathlib.Path(__file__).parent
-HTPASS_FILE = THIS_DIR / "htpasswd.a.a"
+HTPASS_FILE = THIS_DIR / "../fixtures/htpasswd.a.a"
 IGNORELIST_FILE = THIS_DIR / "test-ignorelist"
 
 

tests/test_server.py

@@ -18,6 +18,7 @@ import shutil
 import socket
 import sys
 import time
+import typing as t
 from collections import namedtuple
 from pathlib import Path
 from shlex import split
@@ -40,7 +41,9 @@ Srv = namedtuple("Srv", ("port", "root"))
 @contextlib.contextmanager
 def run_server(root, authed=False, other_cli=""):
     """Run a server, optionally with partial auth enabled."""
-    htpasswd = CURRENT_PATH.joinpath("htpasswd.a.a").expanduser().resolve()
+    htpasswd = (
+        CURRENT_PATH.joinpath("../fixtures/htpasswd.a.a").expanduser().resolve()
+    )
     pswd_opt_choices = {
         True: f"-P {htpasswd} -a update,download",
         False: "-P. -a.",
@@ -91,7 +94,7 @@ def _kill_proc(proc):
         proc.kill()
 
 
-def build_url(port, user="", pswd=""):
+def build_url(port: t.Union[int, str], user: str = "", pswd: str = "") -> str:
     auth = f"{user}:{pswd}@" if user or pswd else ""
     return f"http://{auth}localhost:{port}"
 
@@ -196,12 +199,18 @@ def empty_packdir(tmp_path_factory):
     return tmp_path_factory.mktemp("dists")
 
 
-def pip_download(cmd, port, install_dir, user=None, pswd=None):
+def pip_download(
+    cmd: str,
+    port: t.Union[int, str],
+    install_dir: str,
+    user: str = None,
+    pswd: str = None,
+) -> int:
     url = build_url(port, user, pswd)
     return _run_pip(f"-vv download -d {install_dir} -i {url} {cmd}")
 
 
-def _run_pip(cmd):
+def _run_pip(cmd: str) -> int:
     ncmd = (
         "pip --no-cache-dir --disable-pip-version-check "
         f"--retries 0 --timeout 5 --no-input {cmd}"
@@ -260,7 +269,7 @@ def authed_pypirc(authed_server):
         yield path
 
 
-def run_twine(command, package, conf):
+def run_twine(command: str, package: str, conf: str) -> None:
     proc = Popen(
         split(
             f"twine {command} --repository test --config-file {conf} {package}"

tox.ini

@@ -8,7 +8,7 @@ allowlist_externals=
 sitepackages=False
 commands=
     /bin/sh -c "{env:PYPISERVER_SETUP_CMD:true}"
-    pytest --cov=pypiserver {posargs}
+    pytest --cov=pypiserver {posargs:tests}
 
 
 [pytest]