py3: make get_current_authuser handle missing tg context consistently and explicitly
tg context handling ends up using tg.support.registry.StackedObjectProxy._current_obj for attribute access ... which if no context has been pushed will end up in: raise TypeError( 'No object (name: %s) has been registered for this ' 'thread' % self.____name__)
utils2.get_current_authuser used code like: if hasattr(tg.tmpl_context, 'authuser'):
Python 2 hasattr will call __getattr__ and return False if it throws any exception. (It would thus catch the TypeError and silently fall through to use the default user None.) This hasattr behavior is confusing and hard to use correctly. Here, it was used incorrectly. It has been common practice to work around by using something like: getattr(x, y, None) is not None
Python 3 hasattr fixed this flaw and only catches AttributeError. The TypeError would thus (rightfully) be propagated. That is a change that must be handled when introducing py3 support.
The get_current_authuser code could more clearly and simple and py3-compatible be written as: return getattr(tmpl_context, 'authuser', None) - but then we also have to handle the TypeError explicitly ... which we are happy to do.
celery: introduce make_app instead of creating app at import time
It is dirty to instantiate things at import time (unless it really is basic singletons).
In 0.5.1 (and earlier), such dirtyness made partial test execution fail when other things had global side effects and things didn't use the usual import order:
$ py.test kallithea/lib/ collecting ... ――― kallithea/lib/celerypylons/__init__.py ――― kallithea/lib/celerypylons/__init__.py:58: in <module> app.config_from_object(celery_config(tg.config)) kallithea/lib/celerypylons/__init__.py:28: in celery_config assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded' data/env/lib/python2.7/site-packages/tg/configuration/tgconfig.py:31: in __getitem__ return self.config_proxy.current_conf()[key] E KeyError: 'celery.imports'
Avoid that by running a "factory" function when the celery app actually is needed.
config: fix pyflakes warning about unused tg import
app_cfg had an unused 'import tg'. tg.hooks was used, but through a separate import. Clean that up by consistently using tg (which always makes tg.hooks available) and dropping the separate hooks import.
(localrepo might already always be available in the mercurial namespace due to side effects from other imports, but it is still better to do it explicit ... and also to please pytype.)
Since a while, the test suite shows following warning:
kallithea/tests/__init__.py:29 /home/tdescham/repo/contrib/kallithea/kallithea-review/kallithea/tests/__init__.py:29: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: kallithea.tests pytest.register_assert_rewrite('kallithea.tests')
The problem can be fixed by moving the register_assert_rewrite call from kallithea/tests/__init__.py to the root-level conftest.py, outside of the 'kallithea' module.
tests: remove race condition in test_forgot_password
One in so many times, test_forgot_password failed with:
kallithea/tests/functional/test_login.py:427: in test_forgot_password assert '\n%s\n' % token in body E assert ('\n%s\n' % 'd71ad3ed3c6ca637ad00b7098828d33c56579201') in "Password Reset Request\n\nHello passwd reset,\n\nWe have received a request to reset the password for your account.\n\nTo s...7e89326ca372ade1d424dafb106d824cddb\n\nIf it weren't you who requested the password reset, just disregard this message.\n"
i.e. the expected token is not the one in the email.
The token is calculated based on a timestamp (among others). And the token is calculated twice: once in the real code and once in the test, each time on a slightly different timestamp. Even though there is flooring of the timestamp to a second resolution, there will always be a race condition where the two timestamps floor to a different second, e.g. 4.99 vs 5.01.
The problem can be reproduced reliably by adding a sleep of e.g. 2 seconds before generating the password reset mail (after the test has already calculated the expected token).
Solve this problem by mocking the time.time() used to generate the timestamp, so that the timestamp used for the real token is the same as the one used for the expected token in the test.
scripts: handle "Python 2.7 reached the end of its life" message
The script failed with:
Error: pip detected following problems: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Thus, clarify the use of i18n.lang (refining f2f7a8c1281e and 8931078f70db) and set 'en' as default value on app startup.
TurboGears requires an (empty) translation for the source language which is default for i18n.lang . The empty .mo for en is created as the 4 magic .mo bytes followed by lengths of 0: printf '\x95\x04\x12\xde\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' > kallithea/i18n/en/LC_MESSAGES/kallithea.mo
login: fix incorrect CSRF rejection of "Reset Your Password" form (Issue #350)
htmlfill would remove the CSRF token from the form when substituting the query parameters, causing password reset to break.
By default, htmlfill will clear all input fields that doesn't have a new "default" value provided. It could be fixed by setting force_defaults to False - see http://www.formencode.org/en/1.2-branch/modules/htmlfill.html . It could also be fixed by providing the CSRF token in the defaults to be substituted in the form.
Instead, refactor password_reset_confirmation to have more explicitly safe handling of query parameters. Replace htmlfill with the usual template variables.
The URLs are generated in kallithea/model/user.py send_reset_password_email() and should only contain email, timestamp (integer as digit string) and a hex token from get_reset_password_token() .
setup: bump sqlalchemy minimum version to 1.2.9 to get rid of py3 warning
Avoid verbose warning:
build/minimum-dependency-versions-venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py:637 .../site-packages/sqlalchemy/util/langhelpers.py:637: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly d_args = inspect.formatargspec(spec[0][1:])
setup: bump alembic minimum version to 1.0.10 to get rid of py3 warning
Avoid verbose warning:
.../site-packages/alembic/util/langhelpers.py:92: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly formatvalue=lambda x: '=' + x)
setup: bump decorator minimum version to 4.2.1 to get rid of py3 warning
Avoid verbose warning:
.../site-packages/decorator.py:95: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly formatvalue=lambda val: "", *argspec)[1:-1]
setup: bump dulwich minimum version to 0.19.0 to get good py3 support
Fix failure in test_compare_forks_on_branch_extra_commits_git ... but make a major bump to make sure we get good py3 support in this cruicial and complex low level library.
setup: bump webtest minimum version to 2.0.6 to get py3 stdlib support
/usr/lib64/python3.7/http/cookiejar.py:723: in is_third_party if not domain_match(req_host, reach(request.origin_req_host)): E AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'
File ".../lib/python3.7/site-packages/whoosh/reading.py", line 241, in expand_prefix if fn != fieldname or not text.startswith(prefix): TypeError: startswith first arg must be bytes or a tuple of bytes, not str
setup: bump mako minimum version to 0.9.1 to get py3 support
Avoid test_api_get_pullrequest crash:
data/env3/lib/python3.7/site-packages/mako/template.py:653: in _compile_text code = compile(source, cid, 'exec') E File "email_templates_button_html", line 15 E def render_body(context,url,title='',padding_top=,padding_bottom=,**pageargs): E ^ E SyntaxError: invalid syntax
setup: bump python-dateutil minimum version to 2.1.0 to get py3 support
Avoid py2 syntax:
E File "/home/mk/kallithea-py3/build/minimum-dependency-versions-venv/lib/python3.7/site-packages/dateutil/tz.py", line 78 E `self._name`, E ^ E SyntaxError: invalid syntax
setup: bump formencode minimum version to 1.3.1 to get py3 support
1.3.0 failed with py3:
build/minimum-dependency-versions-venv/lib/python3.7/site-packages/formencode/api.py:153: if unicode is not str: # Python 2 E NameError: name 'unicode' is not defined
E File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/tg/request_local.py", line 37, in languages_best_match E items = [i for i, q in sorted(al._parsed, key=lambda iq: -iq[1])] E TypeError: 'NoneType' object is not iterable
b075693b3214 introduced use of acceptable_offers which with WebOb < 1.8 would fail with:
File ".../kallithea/lib/middleware/pygrack.py", line 189, in __call__ elif req.accept.acceptable_offers(self.valid_accepts): AttributeError: 'MIMEAccept' object has no attribute 'acceptable_offers'
cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly
The configuration and type fixing will be invoked from make_base_app, and we will thus not have to do it:
File "kallithea/config/middleware.py", line 31, in make_app_without_logging return make_base_app(global_conf, full_stack=full_stack, **app_conf) File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app wrap_app) File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app app = TGApp(conf) File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__ app_wrapper = wrapper(self.wrapped_dispatch, self.config) File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__ self.options = parse_cache_config_options(config) File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options
This will fix a py3 problem where setup_cache_regions was run *after* beaker had coerced types, thus introducing string types in the config where beaker expected the integers it had put there.
Visiting a /statistics with py3 would fail with: ... in statistics sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10] TypeError: '<' not supported between instances of 'dict' and 'dict'
The "summary" computation didn't have that problem. And it put '?' as description for unknown extensions. And it had stable output as it also sorted on the file extension as secondary key. Just use that.
lib: clean up ext_json and how it is used - avoid monkey patching
Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded as str. But we generally want JSON as bytes (which json.loads also can read), so also wrap the result with ascii_bytes in many places.
vcs: fix get_file_annotate - consistently bind sha so it has the right value when executing
The Git implementation did *not* save the sha value in the lambda expression for the "changeset lazy loader". Thus, if the generator had moved on and assigned a different value to sha when the expression was executed, it would use the "wrong" sha.
Fixed by doing as the Hg implementation: bind the sha value as value of a default parameter when defining the lambda expression.
The Hg implementation did however also save the line - it is not used, and there is no need for that.
vcs: tweak how revisions and repo names are shown in error messages
Decode bytes to str, and show repo name instead of repr or full server file system path. In some places, it will only report the "basename" of the repository, without any "group names" that also would be nice to have. The easy alternative would be to show the full file system path ... but it would be unfortunate to leak absolute server side paths to end users.
logging: always invoke fileConfig with '__file__' and 'here'
WSGI servers tend to provide '__file__' and 'here' as 'defaults' when invoking fileConfig, so '%(here)s' string interpolation also can be used in logging configuration.
Make sure we do the same when we initialize logging without using a WSGI server.
It is annoying to have to do this, and it will only in rare cases make any difference ... but it seems like the best option.
logging: drop fileConfig initialization in make_app - backout 0d4dd9380a45
0d4dd9380a45 was a bit harmful, as it might overwrite existing good logging configuration.
0d4dd9380a45 no longer seems relevant: Testing shows that logging for `gearbox serve` *is* activated anyway. gearbox/commands/serve.py will invoke "setup_logging" right before "loadapp".
We must and can assume that logging has been initialized before make_app.
Reported and based on analysis by Wolfgang Scherer.
Essentially a backout of d2a97f73fa1f and the 4851d15bc437_db_migration_step_after_95c01895c006_ alembic step.
We can't reliably have full index on fields with unbounded length. The upgrade step has been reported to fail on MySQL [1]:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1170, "BLOB/TEXT column 'public_key' used in key specification without a key length") [SQL: u'CREATE INDEX usk_public_key_idx ON user_ssh_keys (public_key)'] (Background on this error at: http://sqlalche.me/e/e3q8)
And we really don't need this index ... especially now when we use fingerprints for key deletion instead of looking up by the full public key.
ssh: make it clear that SshKeyModel.delete has user as mandatory parameter
It is already provided in the two uses: kallithea/controllers/admin/my_account.py: SshKeyModel().delete(fingerprint, request.authuser.user_id) kallithea/controllers/admin/users.py: SshKeyModel().delete(fingerprint, c.user.user_id)
validator: fix ASCII password check to verify if it can be *encoded* in ascii
In Python 2, unicode strings have a .decode method (which really doesn't make sense). Python 3 has more strict typing by design, and unicode strings don't have a .decode method.
A Unicode string "is ASCII" if it can be encoded as ASCII. The check should thus *encode* to ASCII - not decode.
gist: make it a bit more clear how gist_access_id is used ... and how it is different from gist_id
A gist has a gist_access_id which gives access to it. For private Gists, it is a multi-letter secure random string.
gist_id is the primary key in the database and thus an automatically incrementing integer. It is also used as the not-so-secret gist_access_id for public gists.
This gets rid of one odd safe_unicode applied to an int.
lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()
Before, h.flash would trust any input to contain html ... and callers would convert exceptions to string, often with a simple str() or unicode() ... which really didn't deserve to be trusted.
Instead, only trust messages that have a __html__ and escape anything else ... but also apply str/unicode on the parameter so the caller doesn't have to but *can* pass an exception directly.