2011-08-30 03:38:59 +02:00
|
|
|
#! /usr/bin/env py.test
|
|
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
# Builtin imports
|
2015-09-16 23:54:41 +02:00
|
|
|
import logging
|
2019-09-18 03:29:48 +02:00
|
|
|
import os
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
import pathlib
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
import xmlrpc.client as xmlrpclib
|
|
|
|
|
from html import unescape
|
2016-05-12 03:53:35 +02:00
|
|
|
|
|
|
|
|
# Third party imports
|
|
|
|
|
import pytest
|
|
|
|
|
import webtest
|
|
|
|
|
|
|
|
|
|
# Local Imports
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
from tests.test_pkg_helpers import files, invalid_files
|
|
|
|
|
from pypiserver import __main__, bottle, core, Bottle
|
|
|
|
|
from pypiserver.backend import CachingFileBackend, SimpleFileBackend
|
2011-08-30 03:38:59 +02:00
|
|
|
|
2015-09-17 19:58:22 +02:00
|
|
|
# Enable logging to detect any problems with it
|
2014-11-15 02:37:40 +01:00
|
|
|
##
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
|
2020-10-03 23:30:49 +02:00
|
|
|
__main__.init_logging()
|
2012-04-07 23:10:49 +02:00
|
|
|
|
2015-09-17 19:58:22 +02:00
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def app(tmpdir):
|
2012-04-07 23:10:49 +02:00
|
|
|
from pypiserver import app
|
2019-09-18 03:29:48 +02:00
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
return app(
|
|
|
|
|
roots=[pathlib.Path(tmpdir.strpath)],
|
|
|
|
|
authenticate=[],
|
|
|
|
|
password_file=".",
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
backend_arg="simple-dir",
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
)
|
2011-08-30 03:38:59 +02:00
|
|
|
|
2012-04-03 22:49:47 +02:00
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def testapp(app):
|
2016-05-12 03:53:35 +02:00
|
|
|
"""Return a webtest TestApp initiated with pypiserver app"""
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
bottle.debug(True)
|
2012-12-02 01:15:18 +01:00
|
|
|
return webtest.TestApp(app)
|
2012-04-03 22:49:47 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def root(tmpdir):
|
2016-05-12 03:53:35 +02:00
|
|
|
"""Return a pytest temporary directory"""
|
2012-12-02 01:15:18 +01:00
|
|
|
return tmpdir
|
2012-04-03 22:49:47 +02:00
|
|
|
|
2011-08-30 03:38:59 +02:00
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def priv(app):
|
|
|
|
|
b = bottle.Bottle()
|
|
|
|
|
b.mount("/priv/", app)
|
|
|
|
|
return b
|
2011-09-01 01:08:32 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def testpriv(priv):
|
|
|
|
|
return webtest.TestApp(priv)
|
2011-08-30 03:38:59 +02:00
|
|
|
|
|
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
@pytest.fixture
|
|
|
|
|
def search_xml():
|
|
|
|
|
"""Return an xml dom suitable for passing to search"""
|
2019-09-18 03:29:48 +02:00
|
|
|
xml = "<xml><methodName>search</methodName><string>test</string></xml>"
|
2016-05-12 03:53:35 +02:00
|
|
|
return xml
|
|
|
|
|
|
|
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
@pytest.fixture(
|
|
|
|
|
params=[
|
|
|
|
|
" ", # Mustcontain test below fails when string is empty.
|
|
|
|
|
"Hey there!",
|
|
|
|
|
"<html><body>Hey there!</body></html>",
|
|
|
|
|
]
|
|
|
|
|
)
|
2015-02-23 01:45:35 +01:00
|
|
|
def welcome_file_no_vars(request, root):
|
2016-05-12 03:53:35 +02:00
|
|
|
"""Welcome file fixture
|
|
|
|
|
|
|
|
|
|
:param request: pytest builtin fixture
|
|
|
|
|
:param root: root temporary directory
|
|
|
|
|
"""
|
2015-02-23 01:45:35 +01:00
|
|
|
wfile = root.join("testwelcome.html")
|
2015-02-23 02:19:58 +01:00
|
|
|
wfile.write(request.param)
|
2015-09-16 23:54:41 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
return wfile
|
|
|
|
|
|
|
|
|
|
|
2015-09-16 23:54:41 +02:00
|
|
|
@pytest.fixture()
|
2015-02-23 01:45:35 +01:00
|
|
|
def welcome_file_all_vars(request, root):
|
2015-09-17 19:58:22 +02:00
|
|
|
msg = """
|
2015-02-23 01:45:35 +01:00
|
|
|
{{URL}}
|
|
|
|
|
{{VERSION}}
|
|
|
|
|
{{NUMPKGS}}
|
|
|
|
|
{{PACKAGES}}
|
|
|
|
|
{{SIMPLE}}
|
|
|
|
|
"""
|
|
|
|
|
wfile = root.join("testwelcome.html")
|
2015-02-23 02:19:58 +01:00
|
|
|
wfile.write(msg)
|
2015-09-16 23:54:41 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
return wfile
|
|
|
|
|
|
|
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
@pytest.fixture
|
|
|
|
|
def add_file_to_root(app):
|
|
|
|
|
def file_adder(root, filename, content=""):
|
|
|
|
|
root.join(filename).write(content)
|
|
|
|
|
backend = app.config.backend
|
|
|
|
|
if isinstance(backend, CachingFileBackend):
|
|
|
|
|
backend.cache_manager.invalidate_root_cache(root)
|
|
|
|
|
|
|
|
|
|
return file_adder
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_root_count(root, testapp, add_file_to_root):
|
2016-05-12 03:53:35 +02:00
|
|
|
"""Test that the welcome page count updates with added packages
|
|
|
|
|
|
|
|
|
|
:param root: root temporary directory fixture
|
|
|
|
|
:param testapp: webtest TestApp
|
|
|
|
|
"""
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/")
|
|
|
|
|
resp.mustcontain("PyPI compatible package index serving 0 packages")
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
add_file_to_root(root, "Twisted-11.0.0.tar.bz2")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/")
|
|
|
|
|
resp.mustcontain("PyPI compatible package index serving 1 packages")
|
2011-08-30 03:38:59 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_root_hostname(testapp):
|
|
|
|
|
resp = testapp.get("/", headers={"Host": "systemexit.de"})
|
2020-10-06 04:04:22 +02:00
|
|
|
resp.mustcontain(
|
|
|
|
|
"easy_install --index-url http://systemexit.de/simple/ PACKAGE"
|
|
|
|
|
)
|
2012-12-02 01:15:18 +01:00
|
|
|
# go("http://systemexit.de/")
|
2011-08-30 03:38:59 +02:00
|
|
|
|
|
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
def test_root_welcome_msg_no_vars(root, welcome_file_no_vars):
|
|
|
|
|
from pypiserver import app
|
2019-09-18 03:29:48 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
app = app(root=root.strpath, welcome_file=welcome_file_no_vars.strpath)
|
|
|
|
|
testapp = webtest.TestApp(app)
|
|
|
|
|
resp = testapp.get("/")
|
|
|
|
|
from pypiserver import __version__ as pver
|
2019-09-18 03:29:48 +02:00
|
|
|
|
2015-02-23 02:19:58 +01:00
|
|
|
resp.mustcontain(welcome_file_no_vars.read(), no=pver)
|
2015-02-23 01:45:35 +01:00
|
|
|
|
2014-11-14 00:36:32 +01:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
def test_root_welcome_msg_all_vars(root, welcome_file_all_vars):
|
2014-11-14 00:36:32 +01:00
|
|
|
from pypiserver import app
|
2019-09-18 03:29:48 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
app = app(root=root.strpath, welcome_file=welcome_file_all_vars.strpath)
|
2014-11-14 00:36:32 +01:00
|
|
|
testapp = webtest.TestApp(app)
|
|
|
|
|
resp = testapp.get("/")
|
2015-09-16 23:54:41 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
from pypiserver import __version__ as pver
|
2019-09-18 03:29:48 +02:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
resp.mustcontain(pver)
|
|
|
|
|
|
2014-11-14 00:36:32 +01:00
|
|
|
|
2015-02-21 00:34:37 +01:00
|
|
|
def test_root_welcome_msg_antiXSS(testapp):
|
2015-02-23 01:45:35 +01:00
|
|
|
"""https://github.com/pypiserver/pypiserver/issues/77"""
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.get("/?<alert>Red</alert>", headers={"Host": "somehost.org"})
|
2015-02-21 00:34:37 +01:00
|
|
|
resp.mustcontain("alert", "somehost.org", no="<alert>")
|
2014-11-14 00:36:32 +01:00
|
|
|
|
2015-02-23 01:45:35 +01:00
|
|
|
|
|
|
|
|
def test_root_remove_not_found_msg_antiXSS(testapp):
|
|
|
|
|
"""https://github.com/pypiserver/pypiserver/issues/77"""
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
expect_errors=True,
|
|
|
|
|
headers={"Host": "somehost.org"},
|
|
|
|
|
params={
|
|
|
|
|
":action": "remove_pkg",
|
|
|
|
|
"name": "<alert>Red</alert>",
|
|
|
|
|
"version": "1.1.1",
|
|
|
|
|
},
|
|
|
|
|
)
|
2015-02-23 01:45:35 +01:00
|
|
|
resp.mustcontain("alert", "somehost.org", no="<alert>")
|
|
|
|
|
|
|
|
|
|
|
2016-05-15 04:45:46 +02:00
|
|
|
def test_packages_redirect(testapp):
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/packages")
|
2016-05-15 04:45:46 +02:00
|
|
|
assert resp.status_code >= 300
|
|
|
|
|
assert resp.status_code < 400
|
2019-09-18 03:29:48 +02:00
|
|
|
assert resp.location.endswith("/packages/")
|
2016-05-15 04:45:46 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_packages_empty(testapp):
|
|
|
|
|
resp = testapp.get("/packages/")
|
2012-12-02 01:15:18 +01:00
|
|
|
assert len(resp.html("a")) == 0
|
2011-08-31 21:22:52 +02:00
|
|
|
|
|
|
|
|
|
2022-11-02 12:32:20 +01:00
|
|
|
def test_health_default_endpoint(testapp):
|
2022-07-26 23:28:48 +02:00
|
|
|
resp = testapp.get("/health")
|
|
|
|
|
assert resp.status_int == 200
|
|
|
|
|
assert "Ok" in resp
|
|
|
|
|
|
|
|
|
|
|
2022-11-02 12:32:20 +01:00
|
|
|
def test_health_customized_endpoint(root):
|
|
|
|
|
from pypiserver import app
|
|
|
|
|
|
|
|
|
|
_app = app(root=root.strpath, health_endpoint="/healthz")
|
|
|
|
|
testapp = webtest.TestApp(_app)
|
|
|
|
|
resp = testapp.get("/healthz")
|
|
|
|
|
assert resp.status_int == 200
|
|
|
|
|
assert "Ok" in resp
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_health_invalid_customized_endpoint(root):
|
|
|
|
|
from pypiserver import app
|
|
|
|
|
|
|
|
|
|
with pytest.raises(RuntimeError, match="overlaps with existing routes"):
|
|
|
|
|
app(root=root.strpath, health_endpoint="/simple")
|
|
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_favicon(testapp):
|
|
|
|
|
testapp.get("/favicon.ico", status=404)
|
2011-08-31 21:22:52 +02:00
|
|
|
|
2011-08-31 22:00:09 +02:00
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
def test_fallback(testapp):
|
|
|
|
|
assert not testapp.app._pypiserver_config.disable_fallback
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/pypiserver/", status=302)
|
2018-06-12 03:27:09 +02:00
|
|
|
assert resp.headers["Location"] == "https://pypi.org/simple/pypiserver/"
|
2011-08-31 22:00:09 +02:00
|
|
|
|
|
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
def test_no_fallback(testapp):
|
|
|
|
|
testapp.app._pypiserver_config.disable_fallback = True
|
2012-12-02 01:15:18 +01:00
|
|
|
testapp.get("/simple/pypiserver/", status=404)
|
2011-09-01 00:07:40 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_serve_no_dotfiles(root, testapp):
|
2011-09-01 00:07:40 +02:00
|
|
|
root.join(".foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
testapp.get("/packages/.foo-1.0.zip", status=404)
|
2011-09-01 00:07:40 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_packages_list_no_dotfiles(root, testapp):
|
2011-09-01 00:07:40 +02:00
|
|
|
root.join(".foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/packages/")
|
|
|
|
|
assert "foo" not in resp
|
2011-09-01 00:07:40 +02:00
|
|
|
|
|
|
|
|
|
2016-05-15 04:45:46 +02:00
|
|
|
def test_simple_redirect(testapp):
|
|
|
|
|
resp = testapp.get("/simple")
|
|
|
|
|
assert resp.status_code >= 300
|
|
|
|
|
assert resp.status_code < 400
|
2019-09-18 03:29:48 +02:00
|
|
|
assert resp.location.endswith("/simple/")
|
2016-05-15 04:45:46 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_list_no_dotfiles(root, testapp):
|
2011-09-01 00:07:40 +02:00
|
|
|
root.join(".foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert "foo" not in resp
|
2011-09-01 00:07:40 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_list_no_dotfiles2(root, testapp):
|
2011-09-01 00:07:40 +02:00
|
|
|
root.join(".foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert resp.html("a") == []
|
2011-09-01 00:19:14 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_serve_no_dotdir(root, testapp):
|
2011-10-07 20:31:51 +02:00
|
|
|
root.mkdir(".subdir").join("foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
testapp.get("/packages/.subdir/foo-1.0.zip", status=404)
|
2011-10-07 20:31:51 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_packages_list_no_dotdir(root, testapp):
|
2011-10-07 20:31:51 +02:00
|
|
|
root.mkdir(".subdir").join("foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/packages/")
|
|
|
|
|
assert "foo" not in resp
|
2011-10-07 20:31:51 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_list_no_dotdir(root, testapp):
|
2011-10-07 20:31:51 +02:00
|
|
|
root.mkdir(".subdir").join("foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert "foo" not in resp
|
2011-10-07 20:31:51 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_list_no_dotdir2(root, testapp):
|
2011-10-07 20:31:51 +02:00
|
|
|
root.mkdir(".subdir").join("foo-1.0.zip").write("secret")
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/foo/")
|
|
|
|
|
assert resp.html("a") == []
|
2011-10-07 20:31:51 +02:00
|
|
|
|
|
|
|
|
|
2016-05-15 04:45:46 +02:00
|
|
|
def test_simple_name_redirect(testapp):
|
|
|
|
|
resp = testapp.get("/simple/foobar")
|
|
|
|
|
assert resp.status_code >= 300
|
|
|
|
|
assert resp.status_code < 400
|
2019-09-18 03:29:48 +02:00
|
|
|
assert resp.location.endswith("/simple/foobar/")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"package,normalized",
|
|
|
|
|
[
|
|
|
|
|
("FooBar", "foobar"),
|
|
|
|
|
("Foo.Bar", "foo-bar"),
|
|
|
|
|
("foo_bar", "foo-bar"),
|
|
|
|
|
("Foo-Bar", "foo-bar"),
|
|
|
|
|
("foo--_.bar", "foo-bar"),
|
|
|
|
|
],
|
|
|
|
|
)
|
2016-05-15 04:45:46 +02:00
|
|
|
def test_simple_normalized_name_redirect(testapp, package, normalized):
|
|
|
|
|
resp = testapp.get("/simple/{0}/".format(package))
|
|
|
|
|
assert resp.status_code >= 300
|
|
|
|
|
assert resp.status_code < 400
|
2019-09-18 03:29:48 +02:00
|
|
|
assert resp.location.endswith("/simple/{0}/".format(normalized))
|
2016-05-15 04:45:46 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_index(root, testapp):
|
2011-09-01 00:19:14 +02:00
|
|
|
root.join("foobar-1.0.zip").write("")
|
|
|
|
|
root.join("foobar-1.1.zip").write("")
|
|
|
|
|
root.join("foobarbaz-1.1.zip").write("")
|
|
|
|
|
root.join("foobar.baz-1.1.zip").write("")
|
|
|
|
|
|
2016-05-15 04:45:46 +02:00
|
|
|
resp = testapp.get("/simple/foobar/")
|
2012-12-02 01:15:18 +01:00
|
|
|
assert len(resp.html("a")) == 2
|
2011-09-01 00:19:14 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_index_list(root, testapp):
|
2011-09-01 00:19:14 +02:00
|
|
|
root.join("foobar-1.0.zip").write("")
|
|
|
|
|
root.join("foobar-1.1.zip").write("")
|
|
|
|
|
root.join("foobarbaz-1.1.zip").write("")
|
|
|
|
|
root.join("foobar.baz-1.1.zip").write("")
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert len(resp.html("a")) == 3
|
2011-09-01 00:19:14 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_simple_index_case(root, testapp):
|
2011-09-01 00:19:14 +02:00
|
|
|
root.join("FooBar-1.0.zip").write("")
|
|
|
|
|
root.join("FooBar-1.1.zip").write("")
|
2016-05-15 04:45:46 +02:00
|
|
|
resp = testapp.get("/simple/foobar/")
|
2012-12-02 01:15:18 +01:00
|
|
|
assert len(resp.html("a")) == 2
|
2012-04-03 22:49:47 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 01:15:18 +01:00
|
|
|
def test_nonroot_root(testpriv):
|
|
|
|
|
resp = testpriv.get("/priv/", headers={"Host": "nonroot"})
|
2020-10-06 04:04:22 +02:00
|
|
|
resp.mustcontain(
|
|
|
|
|
"easy_install --index-url http://nonroot/priv/simple/ PACKAGE"
|
|
|
|
|
)
|
2012-04-03 22:49:47 +02:00
|
|
|
|
|
|
|
|
|
2019-04-29 11:22:35 +02:00
|
|
|
def test_nonroot_root_with_x_forwarded_host(testapp):
|
|
|
|
|
resp = testapp.get("/", headers={"X-Forwarded-Host": "forward.ed/priv/"})
|
2020-10-06 04:04:22 +02:00
|
|
|
resp.mustcontain(
|
|
|
|
|
"easy_install --index-url http://forward.ed/priv/simple/ PACKAGE"
|
|
|
|
|
)
|
2019-04-29 11:22:35 +02:00
|
|
|
resp.mustcontain("""<a href="/priv/packages/">here</a>""")
|
|
|
|
|
|
|
|
|
|
|
2019-05-02 16:14:31 +02:00
|
|
|
def test_nonroot_root_with_x_forwarded_host_without_trailing_slash(testapp):
|
|
|
|
|
resp = testapp.get("/", headers={"X-Forwarded-Host": "forward.ed/priv"})
|
2020-10-06 04:04:22 +02:00
|
|
|
resp.mustcontain(
|
|
|
|
|
"easy_install --index-url http://forward.ed/priv/simple/ PACKAGE"
|
|
|
|
|
)
|
2019-04-29 11:22:35 +02:00
|
|
|
resp.mustcontain("""<a href="/priv/packages/">here</a>""")
|
|
|
|
|
|
|
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
def test_nonroot_simple_index(root, testpriv, add_file_to_root):
|
|
|
|
|
add_file_to_root(root, "foobar-1.0.zip", "123")
|
2016-05-15 04:45:46 +02:00
|
|
|
resp = testpriv.get("/priv/simple/foobar/")
|
|
|
|
|
links = resp.html("a")
|
|
|
|
|
assert len(links) == 1
|
|
|
|
|
assert links[0]["href"].startswith("/priv/packages/foobar-1.0.zip#")
|
2012-04-03 22:49:47 +02:00
|
|
|
|
|
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
def test_nonroot_simple_index_with_x_forwarded_host(
|
|
|
|
|
root, testapp, add_file_to_root
|
|
|
|
|
):
|
|
|
|
|
add_file_to_root(root, "foobar-1.0.zip", "123")
|
|
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.get(
|
|
|
|
|
"/simple/foobar/", headers={"X-Forwarded-Host": "forwarded.ed/priv/"}
|
|
|
|
|
)
|
2019-04-29 11:22:35 +02:00
|
|
|
links = resp.html("a")
|
|
|
|
|
assert len(links) == 1
|
|
|
|
|
assert links[0]["href"].startswith("/priv/packages/foobar-1.0.zip#")
|
|
|
|
|
|
|
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
def test_nonroot_simple_packages(root, testpriv, add_file_to_root):
|
|
|
|
|
add_file_to_root(root, "foobar-1.0.zip", "123")
|
2016-05-15 04:45:46 +02:00
|
|
|
resp = testpriv.get("/priv/packages/")
|
|
|
|
|
links = resp.html("a")
|
|
|
|
|
assert len(links) == 1
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
assert "/priv/packages/foobar-1.0.zip#" in links[0]["href"]
|
|
|
|
|
|
2012-12-02 23:07:48 +01:00
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
def test_nonroot_simple_packages_with_x_forwarded_host(
|
|
|
|
|
root, testapp, add_file_to_root
|
|
|
|
|
):
|
|
|
|
|
add_file_to_root(root, "foobar-1.0.zip", "123")
|
2012-12-02 23:07:48 +01:00
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.get(
|
|
|
|
|
"/packages/", headers={"X-Forwarded-Host": "forwarded/priv/"}
|
|
|
|
|
)
|
2019-04-29 11:22:35 +02:00
|
|
|
links = resp.html("a")
|
|
|
|
|
assert len(links) == 1
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
assert "/priv/packages/foobar-1.0.zip#" in links[0]["href"]
|
2019-04-29 11:22:35 +02:00
|
|
|
|
|
|
|
|
|
2012-12-02 23:07:48 +01:00
|
|
|
def test_root_no_relative_paths(testpriv):
|
2015-02-27 23:02:07 +01:00
|
|
|
"""https://github.com/pypiserver/pypiserver/issues/25"""
|
2012-12-02 23:07:48 +01:00
|
|
|
resp = testpriv.get("/priv/")
|
|
|
|
|
hrefs = [x["href"] for x in resp.html("a")]
|
2019-09-18 03:29:48 +02:00
|
|
|
assert hrefs == [
|
|
|
|
|
"/priv/packages/",
|
|
|
|
|
"/priv/simple/",
|
|
|
|
|
"https://pypi.org/project/pypiserver/",
|
|
|
|
|
]
|
2013-04-02 01:49:15 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_simple_index_list_no_duplicates(root, testapp):
|
|
|
|
|
root.join("foo-bar-1.0.tar.gz").write("")
|
|
|
|
|
root.join("foo_bar-1.0-py2.7.egg").write("")
|
|
|
|
|
|
|
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert len(resp.html("a")) == 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_simple_index_list_name_with_underscore(root, testapp):
|
|
|
|
|
root.join("foo_bar-1.0.tar.gz").write("")
|
|
|
|
|
root.join("foo_bar-1.0-py2.7.egg").write("")
|
|
|
|
|
|
|
|
|
|
resp = testapp.get("/simple/")
|
|
|
|
|
assert len(resp.html("a")) == 1
|
|
|
|
|
hrefs = [x["href"] for x in resp.html("a")]
|
2016-05-15 04:45:46 +02:00
|
|
|
assert hrefs == ["foo-bar/"]
|
2013-04-02 01:49:15 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_simple_index_egg_and_tarball(root, testapp):
|
|
|
|
|
root.join("foo-bar-1.0.tar.gz").write("")
|
|
|
|
|
root.join("foo_bar-1.0-py2.7.egg").write("")
|
|
|
|
|
|
2016-05-15 04:45:46 +02:00
|
|
|
resp = testapp.get("/simple/foo-bar/")
|
2013-04-02 01:49:15 +02:00
|
|
|
assert len(resp.html("a")) == 2
|
2013-04-02 22:03:50 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_simple_index_list_name_with_underscore_no_egg(root, testapp):
|
|
|
|
|
root.join("foo_bar-1.0.tar.gz").write("")
|
|
|
|
|
root.join("foo-bar-1.1.tar.gz").write("")
|
|
|
|
|
|
|
|
|
|
resp = testapp.get("/simple/")
|
2016-05-15 04:45:46 +02:00
|
|
|
assert len(resp.html("a")) == 1
|
2013-04-02 22:03:50 +02:00
|
|
|
hrefs = set([x["href"] for x in resp.html("a")])
|
2017-12-18 12:55:30 +01:00
|
|
|
assert hrefs == {"foo-bar/"}
|
2015-01-10 18:18:47 +01:00
|
|
|
|
|
|
|
|
|
2021-08-12 07:46:42 +02:00
|
|
|
def test_json_info(root, testapp):
|
|
|
|
|
root.join("foobar-1.0.zip").write("")
|
|
|
|
|
root.join("foobar-1.1.zip").write("")
|
2023-09-25 09:58:11 +02:00
|
|
|
root.join("foobar-1.1-linux.zip").write("")
|
2021-08-12 07:46:42 +02:00
|
|
|
root.join("foobarX-1.1.zip").write("")
|
|
|
|
|
|
|
|
|
|
resp = testapp.get("/foobar/json")
|
|
|
|
|
assert "info" in resp.json
|
|
|
|
|
assert "releases" in resp.json
|
|
|
|
|
assert len(resp.json["info"]) == 1
|
|
|
|
|
assert len(resp.json["releases"]) == 2
|
2023-09-25 09:58:11 +02:00
|
|
|
assert len(resp.json["releases"]["1.0"]) == 1
|
|
|
|
|
assert len(resp.json["releases"]["1.1"]) == 2
|
2021-08-12 07:46:42 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_json_info_package_not_existing(root, testapp):
|
|
|
|
|
resp = testapp.get("/foobar/json", status=404)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"package,normalized",
|
|
|
|
|
[
|
|
|
|
|
("FooBar", "foobar"),
|
|
|
|
|
("Foo.Bar", "foo-bar"),
|
|
|
|
|
("foo_bar", "foo-bar"),
|
|
|
|
|
("Foo-Bar", "foo-bar"),
|
|
|
|
|
("foo--_.bar", "foo-bar"),
|
|
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
def test_json_info_normalized_name_redirect(testapp, package, normalized):
|
|
|
|
|
resp = testapp.get("/{0}/json".format(package))
|
|
|
|
|
assert resp.status_code >= 300
|
|
|
|
|
assert resp.status_code < 400
|
|
|
|
|
assert resp.location.endswith("/{0}/json".format(normalized))
|
|
|
|
|
|
|
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
def test_no_cache_control_set(root, testapp):
|
|
|
|
|
assert not testapp.app._pypiserver_config.cache_control
|
2015-01-10 18:18:47 +01:00
|
|
|
root.join("foo_bar-1.0.tar.gz").write("")
|
|
|
|
|
resp = testapp.get("/packages/foo_bar-1.0.tar.gz")
|
|
|
|
|
assert "Cache-Control" not in resp.headers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_cache_control_set(root):
|
|
|
|
|
from pypiserver import app
|
2019-09-18 03:29:48 +02:00
|
|
|
|
2015-01-10 18:18:47 +01:00
|
|
|
AGE = 86400
|
2017-11-14 15:41:26 +01:00
|
|
|
app_with_cache = webtest.TestApp(app(root=root.strpath, cache_control=AGE))
|
2015-01-10 18:18:47 +01:00
|
|
|
root.join("foo_bar-1.0.tar.gz").write("")
|
|
|
|
|
resp = app_with_cache.get("/packages/foo_bar-1.0.tar.gz")
|
|
|
|
|
assert "Cache-Control" in resp.headers
|
2020-10-08 03:45:51 +02:00
|
|
|
assert resp.headers["Cache-Control"] == f"public, max-age={AGE}"
|
2016-01-17 21:57:16 +01:00
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
def test_upload_noAction(testapp):
|
2016-01-17 21:57:16 +01:00
|
|
|
resp = testapp.post("/", expect_errors=1)
|
2019-09-18 03:29:48 +02:00
|
|
|
assert resp.status == "400 Bad Request"
|
2019-09-02 21:31:59 +02:00
|
|
|
assert "Missing ':action' field!" in unescape(resp.text)
|
2016-01-17 21:57:16 +01:00
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
|
Use argparse config throughout app (#349)
This PR is a pretty substantial refactor of the entrypoints of pypiserver (`__main__` and `__init__`) to use the argparse-based config added in #339.
- Updated `RunConfig` and `UpdateConfig` classes to have exclusive init kwargs, instead of taking an namespace. This turned out to be much easier when working with the library-style app initialization in `__init__`, both for direct instantiation and via paste config
- Added an `iter_packages()` method to the `RunConfig` to iterate over packages specified by the configuration (note @elfjes, I think that replacing this with e.g. a `backend` reference will be a nice way to tie in #348)
- Added a general-purpose method to map legacy keyword arguments to the `app()` and `paste_app_factory()` functions to updated forms
- Refactored the `paste_app_factory()` to not mutate the incoming dictionary
- Removed all argument-parsing and config-related code from `__main__` and `core`
- Moved `_logwrite` from `__init__` to `__main__`, since that was the only place it was being used after the updates to `core`
- Updated `digest_file` to use `hashlib.new(algo)` instead of `getattr(hashlib, algo)`, because the former supports more algorithms
- Updated `setup.py` to, instead of calling `eval()` on the entirety of `__init__`, to instead just evaluate the line that defines the version
- Assigned the config to a `._pypiserver_config` attribute on the `Bottle` instance to reduce hacky test workarounds
- Fixed the tox config, which I broke in #339
* Config: add auth & absolute path resolution
* Config: check pkg dirs on config creation
* Instantiate config with kwargs, not namespace
* WIP: still pulling the threads
* Init seems to be working
* tests passing locally, still need to update cache
* Fix tox command
* unused import
* Fix typing
* Be more selective in exec() in setup.py
* Require accurate casing for hash algos
* Remove old comment
* Comments, minor updates and simplifications
* move _logwrite to a more reasonable place
* Update config to work with cache
* Type cachemanager listdir in core
* Update config module docstring, rename method
* Add more comments re: paste config
* Add comments to main, remove unneded check
* Remove commented code
* Use {posargs} instead of [] for clarity in tox
* Add dupe check for kwarg updater
* Remove unused references on app instance
* Fix typo
* Remove redundancy in log level parsing
2020-10-26 00:48:28 +01:00
|
|
|
def test_upload_badAction(testapp):
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.post("/", params={":action": "BAD"}, expect_errors=1)
|
|
|
|
|
assert resp.status == "400 Bad Request"
|
2019-09-02 21:31:59 +02:00
|
|
|
assert "Unsupported ':action' field: BAD" in unescape(resp.text)
|
2016-01-17 21:57:16 +01:00
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
@pytest.mark.parametrize(
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
"package", [f[0] for f in files if f[1] and "/" not in f[0]]
|
2019-09-18 03:29:48 +02:00
|
|
|
)
|
2016-01-17 21:57:16 +01:00
|
|
|
def test_upload(package, root, testapp):
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
params={":action": "file_upload"},
|
|
|
|
|
upload_files=[("content", package, b"")],
|
|
|
|
|
)
|
2016-01-17 21:57:16 +01:00
|
|
|
assert resp.status_int == 200
|
|
|
|
|
uploaded_pkgs = [f.basename for f in root.listdir()]
|
|
|
|
|
assert len(uploaded_pkgs) == 1
|
|
|
|
|
assert uploaded_pkgs[0].lower() == package.lower()
|
|
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
def test_upload_conflict_on_existing(root, testapp):
|
|
|
|
|
package = "foo_bar-1.0.tar.gz"
|
|
|
|
|
root.join("foo_bar-1.0.tar.gz").write("")
|
|
|
|
|
|
|
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
params={":action": "file_upload"},
|
|
|
|
|
upload_files=[("content", package, b"")],
|
|
|
|
|
status=409,
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
assert resp.status_int == 409
|
|
|
|
|
assert "Package 'foo_bar-1.0.tar.gz' already exists!" in unescape(resp.text)
|
|
|
|
|
|
|
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
@pytest.mark.parametrize(
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
"package", [f[0] for f in files if f[1] and "/" not in f[0]]
|
2019-09-18 03:29:48 +02:00
|
|
|
)
|
2016-01-19 18:46:51 +01:00
|
|
|
def test_upload_with_signature(package, root, testapp):
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
params={":action": "file_upload"},
|
|
|
|
|
upload_files=[
|
|
|
|
|
("content", package, b""),
|
2020-10-08 03:45:51 +02:00
|
|
|
("gpg_signature", f"{package}.asc", b""),
|
2019-09-18 03:29:48 +02:00
|
|
|
],
|
|
|
|
|
)
|
2016-01-19 18:46:51 +01:00
|
|
|
assert resp.status_int == 200
|
2016-04-21 17:08:18 +02:00
|
|
|
uploaded_pkgs = [f.basename.lower() for f in root.listdir()]
|
2016-01-19 18:46:51 +01:00
|
|
|
assert len(uploaded_pkgs) == 2
|
2016-04-21 17:08:18 +02:00
|
|
|
assert package.lower() in uploaded_pkgs
|
2020-10-08 03:45:51 +02:00
|
|
|
assert f"{package.lower()}.asc" in uploaded_pkgs
|
2016-01-19 18:46:51 +01:00
|
|
|
|
2016-05-12 03:53:35 +02:00
|
|
|
|
Refactor storage operations into separate Backend classes (#348)
Following the discussion in #253 and #325 I've created a first iteration on what a `Backend` interface could look like and how the current file storage operations may be refactored into this interface. It goes from the following principles
* `app.py` talks only to `core.py` with regards to package operations
* at configuration time, a `Backend` implementation is chosen and created for the lifetime of the configured app
* `core.py` proxies requests for packages to this `Backend()`
* The `Backend` interface/api is defined through three things
* methods that an implementation must implement
* methods that an implementation may override if it knows better than the defaults
* the `PkgFIle` class that is (should be) the main carrier of data
* where possible, implementation details must be hidden from concrete `Backend`s to promote extensibility
Other things I've done in this PR:
* I've tried to talk about packages and projects, rather than files and prefixes, since these are the domain terms PEP503 uses, and imho it's also more clear what it means
* Better testability of the `CacheManager` (no more race conditions when `watchdog` is installed during testing)
* Cleanup some more Python 2 code
* Started moving away from `os.path` and `py.path` in favour of `pathlib`
Furthermore I've created a `plugin.py` with a sample of how I think plugin system could look like. This sampIe assumes we use `argparse` and allows for the extension of cli arguments that a plugin may need. I think the actual implementation of such a plugin system is beyond the scope of this PR, but I've used it as a target for the Backend refactoring. If requested, I'll remove it from this PR.
The following things still need to be done / discussed. These can be part of this PR or moved into their own, separate PRs
- [ ] Simplify the `PgkFile` class. It currently consists of a number of attributes that don't necessarily belong with it, and not all attributes are aptly named (imho). I would like to minimalize the scope of `PkgFile` so that its only concern is being a data carrier between the app and the backends, and make its use more clear.
- [ ] Add a `PkgFile.metadata` that backend implementations may use to store custom data for packages. For example the current `PkgFile.root` attribute is an implementation detail of the filestorage backends, and other Backend implementations should not be bothered by it.
- [ ] Use `pathlib` wherever possible. This may also result in less attributes for `PkgFile`, since some things may be just contained in a single `Path` object, instead of multtiple strings.
- [ ] Improve testing of the `CacheManager`.
----
* move some functions around in preparation for backend module
* rename pkg_utils to pkg_helpers to prevent confusion with stdlib pkgutil
* further implement the current filestorage as simple file backend
* rename prefix to project, since that's more descriptive
* add digester func as attribute to pkgfile
* WIP caching backend
* WIP make cache better testable
* better testability of cache
* WIP file backends as plugin
* fix typos, run black
* Apply suggestions from code review
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
* add more type hints to pass mypy, fix tox.ini
* add package count method to backend
* add package count method to backend
* minor changes
* bugfix when checking invalid whl file
* check for existing package recursively, bugfix, some more pathlib
* fix unittest
* rm dead code
* exclude bottle.py from coverage
* fix merge mistakes
* fix tab indentation
* backend as a cli argument
* fix cli, add tests
* fix mypy
* fix more silly mistakes
* process feedback
* remove dead code
Co-authored-by: Matthew Planchard <mplanchard@users.noreply.github.com>
2021-02-02 18:44:29 +01:00
|
|
|
@pytest.mark.parametrize("package", invalid_files)
|
2016-01-17 21:57:16 +01:00
|
|
|
def test_upload_badFilename(package, root, testapp):
|
2019-09-18 03:29:48 +02:00
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
params={":action": "file_upload"},
|
|
|
|
|
upload_files=[("content", package, b"")],
|
|
|
|
|
expect_errors=1,
|
|
|
|
|
)
|
|
|
|
|
assert resp.status == "400 Bad Request"
|
2020-10-08 03:45:51 +02:00
|
|
|
assert f"Bad filename: {package}" in resp.text
|
2016-01-19 12:52:34 +01:00
|
|
|
|
|
|
|
|
|
2019-09-18 03:29:48 +02:00
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"pkgs,matches",
|
|
|
|
|
[
|
|
|
|
|
([], []),
|
|
|
|
|
(["test-1.0.tar.gz"], [("test", "1.0")]),
|
|
|
|
|
(
|
|
|
|
|
["test-1.0.tar.gz", "test-test-2.0.1.tar.gz"],
|
|
|
|
|
[("test", "1.0"), ("test-test", "2.0.1")],
|
|
|
|
|
),
|
|
|
|
|
(["test-1.0.tar.gz", "other-2.0.tar.gz"], [("test", "1.0")]),
|
|
|
|
|
(["test-2.0-py2.py3-none-any.whl"], [("test", "2.0")]),
|
|
|
|
|
(["other-2.0.tar.gz"], []),
|
|
|
|
|
],
|
|
|
|
|
)
|
2016-05-12 03:53:35 +02:00
|
|
|
def test_search(root, testapp, search_xml, pkgs, matches):
|
|
|
|
|
"""Test the search functionality at the RPC2 endpoint
|
|
|
|
|
|
|
|
|
|
Calls the handle_rpc function by posting to the WebTest server with
|
|
|
|
|
the string returned by the ``search_xml`` fixture as the request
|
|
|
|
|
body. The result is parsed for convenience by the xmlrpc.client
|
|
|
|
|
(xmlrpclib in Python 2.x). The parsed result is a 2-tuple. The
|
|
|
|
|
second item is the method called, in this case always "search". The
|
|
|
|
|
first item is a 1-tuple which contains a list of match information
|
|
|
|
|
as dicts, e.g:
|
|
|
|
|
|
|
|
|
|
``(([{'version': '2.0', '_pypi_ordering': 0,
|
|
|
|
|
'name': 'test', 'summary': '2.0'}],), 'search')``
|
|
|
|
|
|
|
|
|
|
The pkgs parameter is a list of items to write to the packages
|
|
|
|
|
directory, which should then be available for the subsequent
|
|
|
|
|
search. The matches parameter is a list of 2-tuples of the form
|
|
|
|
|
(``name``, ``version``), where ``name`` and ``version`` are the
|
|
|
|
|
expected name and version matches for a search for the "test"
|
|
|
|
|
package as specified by the search_xml fixture.
|
|
|
|
|
|
2022-05-23 10:49:18 +02:00
|
|
|
:param root: root temporary directory fixture; used as packages dir
|
2016-05-12 03:53:35 +02:00
|
|
|
for testapp
|
|
|
|
|
:param testapp: webtest TestApp
|
|
|
|
|
:param str search_xml: XML string roughly equivalent to a pip search
|
|
|
|
|
for "test"
|
|
|
|
|
:param pkgs: package file names to be written into packages
|
|
|
|
|
directory
|
|
|
|
|
:param matches: a list of 2-tuples containing expected (name,
|
|
|
|
|
version) matches for the "test" query
|
|
|
|
|
"""
|
|
|
|
|
for pkg in pkgs:
|
2019-09-18 03:29:48 +02:00
|
|
|
root.join(pkg).write("")
|
|
|
|
|
resp = testapp.post("/RPC2", search_xml)
|
2016-05-12 03:53:35 +02:00
|
|
|
parsed = xmlrpclib.loads(resp.text)
|
2019-09-18 03:29:48 +02:00
|
|
|
assert len(parsed) == 2 and parsed[1] == "search"
|
2016-05-12 03:53:35 +02:00
|
|
|
if not matches:
|
|
|
|
|
assert len(parsed[0]) == 1 and not parsed[0][0]
|
|
|
|
|
else:
|
|
|
|
|
assert len(parsed[0][0]) == len(matches) and parsed[0][0]
|
|
|
|
|
for returned in parsed[0][0]:
|
|
|
|
|
print(returned)
|
2019-09-18 03:29:48 +02:00
|
|
|
assert returned["name"] in [match[0] for match in matches]
|
|
|
|
|
assert returned["version"] in [match[1] for match in matches]
|
2016-05-12 03:53:35 +02:00
|
|
|
|
|
|
|
|
|
2019-09-18 04:30:30 +02:00
|
|
|
class TestRemovePkg:
|
|
|
|
|
"""The API allows removal of packages."""
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"pkg, name, ver",
|
|
|
|
|
(
|
|
|
|
|
("test-1.0.tar.gz", "test", "1.0"),
|
|
|
|
|
("test-1.0-py2-py3-none-any.whl", "test", "1.0"),
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
def test_remove_pkg(self, root, testapp, pkg, name, ver):
|
|
|
|
|
"""Packages can be removed via POST."""
|
|
|
|
|
root.join(pkg).write("")
|
|
|
|
|
assert len(os.listdir(str(root))) == 1
|
|
|
|
|
params = {":action": "remove_pkg", "name": name, "version": ver}
|
|
|
|
|
testapp.post("/", params=params)
|
|
|
|
|
assert len(os.listdir(str(root))) == 0
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"pkg, name, ver",
|
|
|
|
|
(
|
|
|
|
|
("test-1.0.tar.gz", "test", "1.0"),
|
|
|
|
|
("test-1.0-py2-py3-none-any.whl", "test", "1.0"),
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"other",
|
|
|
|
|
(
|
|
|
|
|
"test-2.0.tar.gz",
|
|
|
|
|
"test-2.0-py2-py3-none-any.whl",
|
|
|
|
|
"other-1.0.tar.gz",
|
|
|
|
|
"other-1.0-py2-py3-none-any.whl",
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
def test_remove_pkg_only_targeted(
|
|
|
|
|
self, root, testapp, pkg, name, ver, other
|
|
|
|
|
):
|
|
|
|
|
"""Only the targeted package is removed."""
|
|
|
|
|
root.join(pkg).write("")
|
|
|
|
|
root.join(other).write("")
|
|
|
|
|
assert len(os.listdir(str(root))) == 2
|
|
|
|
|
params = {":action": "remove_pkg", "name": name, "version": ver}
|
|
|
|
|
testapp.post("/", params=params)
|
|
|
|
|
assert len(os.listdir(str(root))) == 1
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"pkgs, name, ver",
|
|
|
|
|
(
|
|
|
|
|
(
|
|
|
|
|
("test-1.0.tar.gz", "test-1.0-py2-py3-none-any.whl"),
|
|
|
|
|
"test",
|
|
|
|
|
"1.0",
|
|
|
|
|
),
|
|
|
|
|
),
|
|
|
|
|
)
|
|
|
|
|
def test_all_instances_removed(self, root, testapp, pkgs, name, ver):
|
|
|
|
|
"""Test that all instances of the target are removed."""
|
|
|
|
|
for pkg in pkgs:
|
|
|
|
|
root.join(pkg).write("")
|
|
|
|
|
assert len(os.listdir(str(root))) == len(pkgs)
|
|
|
|
|
params = {":action": "remove_pkg", "name": name, "version": ver}
|
|
|
|
|
testapp.post("/", params=params)
|
|
|
|
|
assert len(os.listdir(str(root))) == 0
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
("name", "version"),
|
|
|
|
|
[
|
|
|
|
|
(None, None),
|
|
|
|
|
(None, ""),
|
|
|
|
|
("", None),
|
|
|
|
|
(None, "1"),
|
|
|
|
|
("pkg", None),
|
|
|
|
|
("", "1"),
|
|
|
|
|
("pkg", ""),
|
|
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
def test_remove_pkg_missingNaveVersion(self, name, version, root, testapp):
|
2020-10-08 03:45:51 +02:00
|
|
|
msg = f"Missing 'name'/'version' fields: name={name}, version={version}"
|
2019-09-18 04:30:30 +02:00
|
|
|
params = {":action": "remove_pkg", "name": name, "version": version}
|
|
|
|
|
params = dict((k, v) for k, v in params.items() if v is not None)
|
|
|
|
|
resp = testapp.post("/", expect_errors=1, params=params)
|
|
|
|
|
|
|
|
|
|
assert resp.status == "400 Bad Request"
|
2020-10-08 03:45:51 +02:00
|
|
|
assert msg in unescape(resp.text)
|
2019-09-18 04:30:30 +02:00
|
|
|
|
|
|
|
|
def test_remove_pkg_notFound(self, root, testapp):
|
|
|
|
|
resp = testapp.post(
|
|
|
|
|
"/",
|
|
|
|
|
expect_errors=1,
|
|
|
|
|
params={":action": "remove_pkg", "name": "foo", "version": "123"},
|
|
|
|
|
)
|
|
|
|
|
assert resp.status == "404 Not Found"
|
|
|
|
|
assert "foo (123) not found" in unescape(resp.text)
|