Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
* Cleanup setup.py
* remove explicit inheritance from object
* convert most string interpolations to f-strings
Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl>
* run black on codebase
* add black check to travis ci
* add pyproject.toml, revert black on bottle.py
Co-authored-by: Pelle Koster <pelle.koster@nginfra.nl>
* log to stdout
* add stdout logging to config and test it
* remove non-implemented parameter from docs
* configure log stream based on config, somehow this change got lost
* fix unittests for other python versions
* option to specify log stream
* Be more explicit in usage text
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* remove redundant arguments
* be more consistent in usage text
* add test for disabling stream logging
* fix side-effect of unittests
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* Add the option to specify list of modules we don't want to update
Signed-off-by: Peter Slovak <peter.slovak@websupport.sk>
* Fix docs
Signed-off-by: Peter Slovak <peter.slovak@websupport.sk>
* Minimize the number of strip() calls
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* Log an exception when we fail to open/read the package blacklist file
* Abort server startup if we fail to read the blacklist file
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
Resolves#237
Previously, we were not running any sort of URL escaping on values
passed in from the client that were used for redirects. This allowed
injection attacks via URL encoded newlines in the original request.
This update ensures that all user-supplied paths that are used as
components of redirects are passed through `urllib.parse.quote()`
(or the python 2 equivalent) prior to being used in a redirect
response.
Also specified 127.0.0.1 rather than 0.0.0.0 (the default) in server
tests to avoid triggering firewall dialogs when testing on MacOS
+ FIX#104: Stopped parsing invalid package-versions prefixed with `v`;
they are invalid according to :pep-reference:`0440`.
+ Also support versions with epochs separated by `!` like `package-1!1.1.0`.
+ Move bottle-filename check on app module.
+ Standalone executable based on wheel.
+ Properly use `pkg_resources` so reading `welcome-msg` file
works even from within zip.
+ Mark `zip_safe=True` in setup.py.
+ Delete forgotten distutils startup script.
+ Build standalone before installing anything else,
to check if any deps are missing.
+ Restore py25 in Travis.
* Use stabdard python's logging lib.
* Log http-request/response/errors.
* Cmd-line options for logging-format and filename.
* Cmd-line options for request /response/error requests/responses/errors
props to log.
* Add `-v` option controlling verbosity.
* Add docs about new options.
* TCs only `-v` & `--log_file ` (logging statements used throughout all
tests).
The fix in commit 7f97612 for supporting the package naming used by the
pytz module caused a regression if the package name contained a dash
followed by a number. We fix this by splitting on all dashes followed by
numbers and recreating the package name from all components but the
last.
instead we use an iterator that returns all relevant information. that
simplifies the code quite a bit because we now mostly operate on the
iterators result instead of handling filenames, versions, prefixes...
