Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
I gave test_server.py some much needed attention. This file now take ~30 seconds on my machine to run (down from 130 seconds), and I cleaned up the code a little. Let's see how this goes in CI
Commits:
-------------
* minimize time.sleep, convert to pathlib
* refactor, dry code
* run black
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
Adds an argparse config that, while adding subcommands (`pypi-server run` and `pypi-server update`), retains full commandline backwards compatibility with the existing config parsing logic.
There's a bit of hackery required to do this, so this also issues a warning if using the non-subcommand arguments, allowing us to potentially remove support for the old form in our next next major version bump (i.e. 3.0).
Also adds a `.pyproject.toml` with a black config, and a mypy config block to `setup.cfg`.
`mypy` is now called in `tox`, currently only for `config.py`, because nothing else typechecks successfully.
----
* WIP: argparse config
* Complete config
* Test all the config options
* Another test and a note re: being unused
* mypy config, call mypy in tox
* No mypy on pypy
* Fix tox config
* Add venv to black ignore
* fix tox config (again)
* Fix formatting, simplify error handling
* FMT: Run black on changed files
Resolves#205 - pypi.python.org shutting down
* Updated the default fallback URL to `pypi.org/simple` rather than
`pypi.python.org/simple`
* Scrubbed references and links to `pypi.python.org`
* Fixed tests breaking due to the removal of `pip.main()` in pip 10.0 -
see pypa/pip#5080 for more info
Resolves#179
Due to a pypi API change (reasoning
[here](https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html),
thanks to @natefoo for the link), the redirect links that we were
generating for distributions not present in the package index were
failing with `403` errors.
@ankostis, I have not had time to look into why the standalone tests are
failing. Perhaps we should just ignore them in order to get a new
version out once this is merged to master?
* Updated .travis.yml to fix PEP 440 warnings
* Fixed twine calls
We were getting test failures on multiple branches in
`test_server.py`. I first investigated a warning message
popping up in every test run:
```
PEP440Warning,
/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/pkg_resources/__init__.py:2510: PEP440Warning: 'setuptools (git-0.4.0)' is being parsed as a legacy, non PEP 440, version. You may find odd behavior and sort order. In particular it will be sorted as less than 0.0. It is recommend to migrate to PEP 440 compatible versions.
```
Moving the installation of setuptools, pip, sphinx, and tox
into the `install` key for Travis resolved that issue, but
`test_server.py` tests were still failing. It turns out that
Twine 1.7.0 added support for SSL cert specification and,
in the process, changed the call signature for the `upload`
and `register` internal methods.
This PR fixes the calls so that they align with Twine's new
function signature. Note that tests now fail on Twine <1.7.0,
so I have also updated the dev requirements file.
Dmitrii Orlov