Fix the `build` calls to use `--no-isolation`, in order to fix
regression in running tests in an offline environment.
Unlike the previous `setup.py` calls, `build` defaults to creating a new
virtual environment to perform the build. This, in turn, requires
fetching the build dependencies from the Internet, effectively making
the tests error out in an offline environment. Passing `--no-isolation`
makes `build` use the already-installed host dependencies instead.
Co-authored-by: Mitja O <dmtree.dev@yahoo.com>
* Update _app.py
* format code
* Revert "format code"
This reverts commit c5b0c9e20a72c284cb96022f8f9916bc65892ac6.
* format code
* format code with black
* update test_json_info
* Update tests/test_app.py
Co-authored-by: Mitja O <dmtree.dev@yahoo.com>
* Update tests/test_app.py
Co-authored-by: Mitja O <dmtree.dev@yahoo.com>
---------
Co-authored-by: Mitja O <dmtree.dev@yahoo.com>
* feat: markdown conversion
logo and badges
* feat: markdown conversion
fix logo path
* feat: markdown
table changes
* feat: markdown
table alignment
* feat: markdown
check toc
* feat: markdown
toc additions
* feat: markdown
quickstart section
* feat: dependabot
more details section in quick start
* feat: dependabot
correct bold
* feat: markdown
client side config, pip
* feat: markdown
typo in title
* feat: markdown
typo in configuring pip
* feat: markdown
apache like authentication section
an initial cut to view the markdown
in github
* feat: markdown
typo in markdown link
* feat: markdown
remove trailing colon
* feat: markdown
typo in shell markdown
* feat: markdown
standardize on 4 space indent in shell code block
* feat: markdown
complete markdown for section up to alternate installation methods
* feat: markdown
add more of the contents to test with
* feat: markdown
contents
* feat: markdown
contents
* feat: markdown
contents
* feat: markdown
dquote>
dquote> recipes
* feat: markdown
dquote>
dquote> recipes
* feat: markdown
dquote>
dquote> up to licensing
* feat: markdown
dquote>
dquote> contents
* Update README.md
Missing exclamation mark
* Update README.md
missing link
* Update README.md
remove duplicated text
* Update README.md
bold differences it Table of contents
* Update README.md
additional bold changes in table of contents
* Update README.md
broken link
* Update README.md
typo in link fix
* Update README.md
change code block to text as shell highlighting was showing some items in red
* Update README.md
code block shell to text
* Update README.md
correct pypi-server update section
* feat: markdown
dquote>
dquote> link back to TOC title
* Update README.md
change link to TOC title
* Update README.md
link test
* Update README.md
link update
* Update README.md
link update
* Update README.md
link update
* feat: markdown
links
* Update README.md
change the level of indent for uploading packages remotely
* Update README.md
add link to python-pam
* feat: markdown
apache link to TOC not working.
* Update README.md
grammar
* Update README.md
typo bold
* feat: markdown
undo bolded text in TOC and titles as linking does
not work
* feat: markdown
remove bold from TOC
* feat: feature
more link issues
* feat: markdown
fixing broken links
* feat: markdown
change text slightly as markdown only links to plain text
* feat: markdown
typo
* feat: markdown
more link typos
* Update README.md
typo in link
* Update README.md
link will not work with braces in the titles
* feat: markdown
run mdformat and apply changes, :) lint!
* feat: markdown
- check via mdformat
- remove old check script
- update test-requirements.txt
* feat: markdown
correct the errors in the mdformat run command
* feat: markdown
for testing remove all the actual jobs
* feat: markdown
re-run mdformat
* feat: markdown
put the jobs back in after testing the mdformat
cmd for passing and failing via workflow dispatch
* feat: markdown
remove references to README.md
* feat: markdown
change action to workflow dispatch for testing
* feat: markdown
- update docker igore
- alter unit test to look for version number after md changes
* feat: markdown
black linting
* feat: markdown
update comments
* feat: markdown
update bumpver to look at md rather than rst file
* feat: markdown
replace workflow dispatch with pull request
to get ready for the final PR
* feat: markdown-delete-original
delete the original rst file
* feat: markdown-delete-original
change ci to workflow dispatch for testing
* feat: markdown-delete-original
revert workflow dispatch
* feat: markdown-badge-links
set the links back to the original
URLs.
* feat: markdown-badge-links
fix brackets
* feat: markdown
update the version and date
* feat: markdown conversion
markdown changes to conform to mdformat tooling.
* feat: 🩺 allow customized health check endpoint
Get the liveness endpoint from the environment variable `HEALTH_ENDPOINT` and verify it. If the customized endpoint is invalied, it will fallback to the DEFAULT_HEALTH_ENDPOINT.
* test: ✅ Test customized endpoint feature
* fix: 🚨 fix check
* feat: Use CLI interface to set health endpoint
* style: 💄 fix black format
* Separate 'build app' and 'add routes'
https://github.com/pypiserver/pypiserver/pull/442#discussion_r973771421
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* keep DEFAULTS in config.py
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* style alignment
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* make CLI arg description more clear
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* style: 🎨 style alignment
* refactor: ✅ SRP, add routes after app created, instead of patching in app_from_config
* style: 🎨 format CLI help
* test: ✅ add test_setup_routes_from_config
* fix: 🐛 test name doesn't work as expected because of using the wrong ids generator.
* test: 🧪 add config error cases for health endpoint
* test: ✅ fix health_endpoint_arg tests
* fix: ✅ Do not fallback to default silently, should raise error
* test: 🧪 add test_health_endpoint in test_main
* test: ✅ setup routes in main
* docs: 📝 Update the help command output in the Quickstart
* docs: 🐛 missing space
* docs: 📝 Add 'Custom Health Check Endpoint' to 'Recipes'
* docs: 📝 refine README
* revert: ⏪ revert auto isoft
* build: 💚 fix mypy, missing return types
* Update README.rst
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* Update README.rst
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* Update pypiserver/config.py
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* Update README.rst
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* style: 💄 black format
* Update README.rst
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
Co-authored-by: Dmitrii Orlov <dmtree.dev@yahoo.com>
* Add extremely basic /health endpoint
* Replace robot-sounds with Ok.
* add test case for /health endpoint
* Apply suggestion to _app.py:
- remove dot from Ok response in health endpoint
* Fix tests for the health endpoint
* Formatting _app.py
Updates the Docker configuration to use the gunicorn server with gevent
workers by default. Adds `waitress` to the docker container, so that if
no server is specified, we will fall back to that rather than `wsgiref`.
Making this happen brought a few other issues to light, which are also
addressed here.
- Docker log output not immediately being flushed to stdout (#358):
resolved by setting the `PYTHONUNBUFFERED` env var to `t` in the
docker container
- When the WSGIRef server is selected, its access logs are written
directly to stderr, rather than going through the logging machinery:
resolved by adding a new `WsgiHandler` class and passing in to
bottle's `run()` method when running the wsgi server. This required a
new `ServerCheck` class to determine whether the wsgi server is
selected when the `auto` option is used
- When using `gunicorn` along with the watchdog cache, package uplaods
were not being picked up by the watcher. Updated the `add_package`
and `remove_package` methods on the `CachingFileBackend` to bust the
cache
* Docker improvements
This addresses much of what was brought up in #359. Specifically, it:
- Significantly improves testing for the Docker image, adding a
`docker/test_docker.py` file using the regular pytest machinery to
set up and run docker images for testing
- Hopefully addresses a variety of permissions issues, by being explicit
about what access pypiserver needs and asking for it, only erroring
if that access is not available
- Requires RX permissions on `/data` (R to read files, X to list files
and to be able to cd into the directory. This is important since
`/data` is the `WORKDIR`)
- Requires RWX permissions on `/data/packages`, so that we can list
packages, write packages, and read packages.
- When running in the default configuration (as root on Linux or
as the pypiserver-named rootish user on Mac), with no volumes
mounted, these requirements are all satisfied
- Volume mounts still must be readable by the pypiserver user (UID
9898) in order for the container to run. However, we now error early
if this is not the case, and direct users to a useful issue.
- If the container is run as a non-root, non-pypiserver user (e.g.
because someone ran `docker run --user=<user_id>`, we try to run
pypiserver as that user). Provided that user has access to the
necessary directories, it should run fine.
- Fixes issues with running help and similar commands
- Updates the Docker image to use `PYPISERVER_PORT` for port
specification, while still falling back to `PORT` for backwards
compatibility
- Moves some docker-related things into a `/docker` directory
- Adds a `Makefile` for building a test fixture package sdist and wheel,
so that test code can call `make mypkg` and not need to worry about it
potentially building multiple times
The only issue #359 raises that's not addressed here is the one of
running pypiserver in the Docker container using some non-default server
for performance. I would like to do some benchmarking before deciding on
what to do there.
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
I gave test_server.py some much needed attention. This file now take ~30 seconds on my machine to run (down from 130 seconds), and I cleaned up the code a little. Let's see how this goes in CI
Commits:
-------------
* minimize time.sleep, convert to pathlib
* refactor, dry code
* run black
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
Thanks @elfjes for pointing out that I'd missed this! I also went ahead
and bumped the version in the README to 2.0.0dev1, so that it's clear
that what's in master shouldn't be what people expect from pypi or in the
docker image.
Adds an argparse config that, while adding subcommands (`pypi-server run` and `pypi-server update`), retains full commandline backwards compatibility with the existing config parsing logic.
There's a bit of hackery required to do this, so this also issues a warning if using the non-subcommand arguments, allowing us to potentially remove support for the old form in our next next major version bump (i.e. 3.0).
Also adds a `.pyproject.toml` with a black config, and a mypy config block to `setup.cfg`.
`mypy` is now called in `tox`, currently only for `config.py`, because nothing else typechecks successfully.
----
* WIP: argparse config
* Complete config
* Test all the config options
* Another test and a note re: being unused
* mypy config, call mypy in tox
* No mypy on pypy
* Fix tox config
* Add venv to black ignore
* fix tox config (again)
* Fix formatting, simplify error handling
* FMT: Run black on changed files
* Cleanup setup.py
* remove explicit inheritance from object
* convert most string interpolations to f-strings
Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl>
* run black on codebase
* add black check to travis ci
* add pyproject.toml, revert black on bottle.py
Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl>
* log to stdout
* add stdout logging to config and test it
* remove non-implemented parameter from docs
* configure log stream based on config, somehow this change got lost
* fix unittests for other python versions
* option to specify log stream
* Be more explicit in usage text
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* remove redundant arguments
* be more consistent in usage text
* add test for disabling stream logging
* fix side-effect of unittests
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* Add the option to specify list of modules we don't want to update
Signed-off-by: Peter Slovak <peter.slovak@websupport.sk>
* Fix docs
Signed-off-by: Peter Slovak <peter.slovak@websupport.sk>
* Minimize the number of strip() calls
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* Log an exception when we fail to open/read the package blacklist file
* Abort server startup if we fail to read the blacklist file
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* [268] Ensure remove_pkg removes all pkg instances
Reported by @esciara in #268.
Previously, the `remove_pkg` command was only removing the first
matching package that it found so if, for example, there were a .tar.gz
file and a .whl file, it would only remove one of them.
Of course, it could be run in succession to accomplish full removal, but
the expected behavior is that removal will remove the package entirely.
Here, I've grouped `remove_pkg` related tests into a test class, added
some tests that verify the expected behavior, and updated the
`remove_pkg` method to remove all matching packages.
* CHANGES.rst
Resolves#237
Previously, we were not running any sort of URL escaping on values
passed in from the client that were used for redirects. This allowed
injection attacks via URL encoded newlines in the original request.
This update ensures that all user-supplied paths that are used as
components of redirects are passed through `urllib.parse.quote()`
(or the python 2 equivalent) prior to being used in a redirect
response.
Also specified 127.0.0.1 rather than 0.0.0.0 (the default) in server
tests to avoid triggering firewall dialogs when testing on MacOS
Resolves#226
Cleaned up `test_server.py` a bit:
* Call `twine` via `Popen` instead of using their internal API, which
should help us from having to fix twine tests as often
* Cleaned up fixtures
* Some PEP8 updates and general formatting
* Grouped test fixtures and functions all before the tests themselves
* Refactored `updae()` into several single-responsibility functions
* Added tests for said functions
* Some PEP 008 cleanup
* Imported standard `__future__` imports in `manage.py` and its test
module
Resolves#205 - pypi.python.org shutting down
* Updated the default fallback URL to `pypi.org/simple` rather than
`pypi.python.org/simple`
* Scrubbed references and links to `pypi.python.org`
* Fixed tests breaking due to the removal of `pip.main()` in pip 10.0 -
see pypa/pip#5080 for more info
The ability to propagate configuration values from a paste config
file was introduced in #156. However, as pointed out in #125
by @redbaron4, the string strip method introduced in #156 was
problematic in Python 2.
This resolves that issue while also creating a test that fails
on the current master and passes with updates, demonstrating the
issue.
In addition to including tests from @jameshiebert with slight
updates, I have also resolved an issue with newer versions of twine,
which require an extra parameter in the calls to `upload` and
`register`.
Resolves#179
Due to a pypi API change (reasoning
[here](https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html),
thanks to @natefoo for the link), the redirect links that we were
generating for distributions not present in the package index were
failing with `403` errors.
@ankostis, I have not had time to look into why the standalone tests are
failing. Perhaps we should just ignore them in order to get a new
version out once this is merged to master?
`pypi.python.org/simple` will respond to HTTP requests with `403: SSL is required.`
This will change the default fallback_url to use HTTPS. This should fix#179.
