Docker improvements (#365)
* Docker improvements
This addresses much of what was brought up in #359. Specifically, it:
- Significantly improves testing for the Docker image, adding a
`docker/test_docker.py` file using the regular pytest machinery to
set up and run docker images for testing
- Hopefully addresses a variety of permissions issues, by being explicit
about what access pypiserver needs and asking for it, only erroring
if that access is not available
- Requires RX permissions on `/data` (R to read files, X to list files
and to be able to cd into the directory. This is important since
`/data` is the `WORKDIR`)
- Requires RWX permissions on `/data/packages`, so that we can list
packages, write packages, and read packages.
- When running in the default configuration (as root on Linux or
as the pypiserver-named rootish user on Mac), with no volumes
mounted, these requirements are all satisfied
- Volume mounts still must be readable by the pypiserver user (UID
9898) in order for the container to run. However, we now error early
if this is not the case, and direct users to a useful issue.
- If the container is run as a non-root, non-pypiserver user (e.g.
because someone ran `docker run --user=<user_id>`, we try to run
pypiserver as that user). Provided that user has access to the
necessary directories, it should run fine.
- Fixes issues with running help and similar commands
- Updates the Docker image to use `PYPISERVER_PORT` for port
specification, while still falling back to `PORT` for backwards
compatibility
- Moves some docker-related things into a `/docker` directory
- Adds a `Makefile` for building a test fixture package sdist and wheel,
so that test code can call `make mypkg` and not need to worry about it
potentially building multiple times
The only issue #359 raises that's not addressed here is the one of
running pypiserver in the Docker container using some non-default server
for performance. I would like to do some benchmarking before deciding on
what to do there.
2021-02-06 18:28:15 +01:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
|
|
|
function run() {
|
|
|
|
# we're not root. Run as who we are.
|
|
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
|
|
eval "$@"
|
|
|
|
else
|
|
|
|
gosu pypiserver "$@"
|
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
|
|
|
if [[ "$EUID" -ne 0 && "$EUID" -ne $(id -u pypiserver) ]]; then
|
|
|
|
USER_ID="$EUID"
|
|
|
|
WARN=(
|
|
|
|
"The pypiserver container was run as a non-root, non-pypiserver user."
|
|
|
|
"Pypiserver will be run as this user if possible, but this is not"
|
|
|
|
"officially supported."
|
|
|
|
)
|
|
|
|
echo "" 1>&2
|
|
|
|
echo "${WARN[@]}" 1>&2
|
|
|
|
echo "" 1>&2
|
|
|
|
else
|
|
|
|
USER_ID=$(id -u pypiserver)
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
|
|
function print_permissions_help() {
|
|
|
|
MSG1=(
|
|
|
|
"If you are mounting a volume at /data or /data/packages and are running the"
|
|
|
|
"container on a linux system, you may need to add add a pypiserver"
|
|
|
|
"group to the host and give it permission to access the directories."
|
|
|
|
"Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
|
|
|
|
"details."
|
|
|
|
)
|
|
|
|
MSG2=(
|
|
|
|
"Please see https://github.com/pypiserver/pypiserver/issues/256 for more"
|
|
|
|
"details."
|
|
|
|
)
|
|
|
|
echo "" 1>&2
|
|
|
|
echo "${MSG1[@]}" 1>&2
|
|
|
|
echo "" 1>&2
|
|
|
|
echo "${MSG2[@]}" 1>&2
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
# the user must have read and execute access to the /data directory
|
|
|
|
# (execute to be able to cd into directory and list content metadata)
|
|
|
|
if ! run test -r /data -a -x /data; then
|
|
|
|
|
|
|
|
chown -R "$USER_ID:pypiserver" /data || true
|
|
|
|
|
|
|
|
if ! run test -r /data -a -x /data; then
|
|
|
|
FAIL_MSG=(
|
|
|
|
"Cannot start pypiserver:"
|
|
|
|
"pypiserver user (UID $USER_ID)"
|
|
|
|
"or pypiserver group (GID $(id -g pypiserver))"
|
|
|
|
"must have read/execute access to /data"
|
|
|
|
)
|
|
|
|
echo "${FAIL_MSG[@]}" 1>&2
|
|
|
|
echo "" 1>&2
|
|
|
|
print_permissions_help
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
# The /data/packages directory must exist
|
|
|
|
# It not existing is very unlikely, possibly impossible, because the VOLUME
|
|
|
|
# specification in the Dockerfile leads to its being created even if someone is
|
|
|
|
# mounting a volume at /data that does not contain a /packages subdirectory
|
|
|
|
if [[ ! -d "/data/packages" ]]; then
|
|
|
|
if ! run test -w /data; then
|
|
|
|
FAIL_MSG=(
|
|
|
|
"Cannot start pypiserver:"
|
|
|
|
"/data/packages does not exist and"
|
|
|
|
"pypiserver user (UID $USER_ID)"
|
|
|
|
"or pypiserver group (GID $(id -g pypiserver))"
|
|
|
|
"does not have write access to /data to create it"
|
|
|
|
)
|
|
|
|
echo "" 1>&2
|
|
|
|
echo "${FAIL_MSG[@]}" 1>&2
|
|
|
|
print_permissions_help
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
run mkdir /data/packages
|
|
|
|
fi
|
|
|
|
|
|
|
|
# The pypiserver user needs read/write/execute access to the packages directory
|
|
|
|
if ! run \
|
|
|
|
test -w /data/packages \
|
|
|
|
-a -r /data/packages \
|
|
|
|
-a -x /data/packages; then
|
|
|
|
|
|
|
|
# We'll try to chown as a last resort.
|
|
|
|
# Don't complain if it fails, since we'll bomb on the next check anyway.
|
|
|
|
chown -R "$USER_ID:pypiserver" /data/packages || true
|
|
|
|
|
|
|
|
if ! run \
|
|
|
|
test -w /data/packages \
|
|
|
|
-a -r /data/packages \
|
|
|
|
-a -x /data/packages; then
|
|
|
|
FAIL_MSG=(
|
|
|
|
"Cannot start pypiserver:"
|
|
|
|
"pypiserver user (UID $USER_ID)"
|
|
|
|
"or pypiserver group (GID $(id -g pypiserver))"
|
|
|
|
"must have read/write/execute access to /data/packages"
|
|
|
|
)
|
|
|
|
echo "" 1>&2
|
|
|
|
echo "${FAIL_MSG[@]}" 1>&2
|
|
|
|
print_permissions_help
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
|
|
if [[ "$*" == "" ]]; then
|
2021-02-08 00:04:06 +01:00
|
|
|
# Use the gunicorn server by default, since it's more performant than
|
|
|
|
# bottle's default server
|
|
|
|
CMD=("run" "-p" "${PYPISERVER_PORT:-$PORT}" "--server" "gunicorn")
|
Docker improvements (#365)
* Docker improvements
This addresses much of what was brought up in #359. Specifically, it:
- Significantly improves testing for the Docker image, adding a
`docker/test_docker.py` file using the regular pytest machinery to
set up and run docker images for testing
- Hopefully addresses a variety of permissions issues, by being explicit
about what access pypiserver needs and asking for it, only erroring
if that access is not available
- Requires RX permissions on `/data` (R to read files, X to list files
and to be able to cd into the directory. This is important since
`/data` is the `WORKDIR`)
- Requires RWX permissions on `/data/packages`, so that we can list
packages, write packages, and read packages.
- When running in the default configuration (as root on Linux or
as the pypiserver-named rootish user on Mac), with no volumes
mounted, these requirements are all satisfied
- Volume mounts still must be readable by the pypiserver user (UID
9898) in order for the container to run. However, we now error early
if this is not the case, and direct users to a useful issue.
- If the container is run as a non-root, non-pypiserver user (e.g.
because someone ran `docker run --user=<user_id>`, we try to run
pypiserver as that user). Provided that user has access to the
necessary directories, it should run fine.
- Fixes issues with running help and similar commands
- Updates the Docker image to use `PYPISERVER_PORT` for port
specification, while still falling back to `PORT` for backwards
compatibility
- Moves some docker-related things into a `/docker` directory
- Adds a `Makefile` for building a test fixture package sdist and wheel,
so that test code can call `make mypkg` and not need to worry about it
potentially building multiple times
The only issue #359 raises that's not addressed here is the one of
running pypiserver in the Docker container using some non-default server
for performance. I would like to do some benchmarking before deciding on
what to do there.
2021-02-06 18:28:15 +01:00
|
|
|
else
|
|
|
|
# this reassigns the array to the CMD variable
|
|
|
|
CMD=( "${@}" )
|
|
|
|
fi
|
|
|
|
|
|
|
|
if [[ "$EUID" -ne 0 ]]; then
|
|
|
|
exec pypi-server "${CMD[@]}"
|
|
|
|
else
|
|
|
|
exec gosu pypiserver pypi-server "${CMD[@]}"
|
|
|
|
fi
|