i18n: make sure 'en' in Accept-Language is recognized as having 100% coverage - i18n.lang is for source language

https://github.com/cdent/paste/blob/3.2.3/paste/wsgiwrappers.py#L89 describes
how paste solved the problem. TG solves the same problem (with less
explanation) in
https://github.com/TurboGears/tg2/blob/tg2.4.2/tg/request_local.py#L36 with
fallback language specified in i18n.lang .

Thus, clarify the use of i18n.lang (refining f2f7a8c1281e and 8931078f70db) and
set 'en' as default value on app startup.

TurboGears requires an (empty) translation for the source language which is
default for i18n.lang . The empty .mo for en is created as the 4 magic .mo
bytes followed by lengths of 0:
printf '\x95\x04\x12\xde\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' > kallithea/i18n/en/LC_MESSAGES/kallithea.mo

Based on report and analysis by Wolfgang Scherer. Also discussed upstream on
https://github.com/TurboGears/tg2/pull/115 .

auth: accept sha256 passwords on all platforms - not only on Windows

Give less surprises when changing platform.

Still, bcrypt is only supported and used on Posix.

bcrypt "hashes" will have length 60 and start with '$' and will thus
immediately skip the sha256 check.

The change should be safe: Users can't influence what kind of hashed key will
be in the database and can thus not influence the auth method.

(We really should use bcrypt on Windows too ... or change to something more
state of the art.)

login: fix incorrect CSRF rejection of "Reset Your Password" form (Issue #350)

htmlfill would remove the CSRF token from the form when substituting the query
parameters, causing password reset to break.

By default, htmlfill will clear all input fields that doesn't have a new
"default" value provided. It could be fixed by setting force_defaults to False
- see http://www.formencode.org/en/1.2-branch/modules/htmlfill.html . It could
also be fixed by providing the CSRF token in the defaults to be substituted in
the form.

Instead, refactor password_reset_confirmation to have more explicitly safe
handling of query parameters. Replace htmlfill with the usual template
variables.

The URLs are generated in kallithea/model/user.py send_reset_password_email()
and should only contain email, timestamp (integer as digit string) and a hex
token from get_reset_password_token() .

setup: bump sqlalchemy minimum version to 1.2.9 to get rid of py3 warning

Avoid verbose warning:

build/minimum-dependency-versions-venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py:637
.../site-packages/sqlalchemy/util/langhelpers.py:637: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
d_args = inspect.formatargspec(spec[0][1:])

setup: bump alembic minimum version to 1.0.10 to get rid of py3 warning

Avoid verbose warning:

.../site-packages/alembic/util/langhelpers.py:92: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
formatvalue=lambda x: '=' + x)

setup: bump decorator minimum version to 4.2.1 to get rid of py3 warning

Avoid verbose warning:

.../site-packages/decorator.py:95: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
formatvalue=lambda val: "", *argspec)[1:-1]

setup: bump dulwich minimum version to 0.19.0 to get good py3 support

Fix failure in test_compare_forks_on_branch_extra_commits_git ... but make a
major bump to make sure we get good py3 support in this cruicial and complex
low level library.

setup: bump webtest minimum version to 2.0.6 to get py3 stdlib support

/usr/lib64/python3.7/http/cookiejar.py:723: in is_third_party
if not domain_match(req_host, reach(request.origin_req_host)):
E AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

setup: bump whoosh minimum version to 2.7.1 to get py3 support

Fix:

- kallithea/tests/functional/test_search_indexing.py:111 TestSearchControllerIndexing.test_repository_tokenization[content-this_should_be_unique_content-1-group/*]

File ".../lib/python3.7/site-packages/whoosh/reading.py", line 241, in expand_prefix
if fn != fieldname or not text.startswith(prefix):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

setup: bump mako minimum version to 0.9.1 to get py3 support

Avoid test_api_get_pullrequest crash:

data/env3/lib/python3.7/site-packages/mako/template.py:653: in _compile_text
code = compile(source, cid, 'exec')
E File "email_templates_button_html", line 15
E def render_body(context,url,title='',padding_top=,padding_bottom=,**pageargs):
E ^
E SyntaxError: invalid syntax

setup: bump python-dateutil minimum version to 2.1.0 to get py3 support

Avoid py2 syntax:

E File "/home/mk/kallithea-py3/build/minimum-dependency-versions-venv/lib/python3.7/site-packages/dateutil/tz.py", line 78
E `self._name`,
E ^
E SyntaxError: invalid syntax

setup: bump beaker minimum version to 1.10.1 to get py3 support

1.10.0 failed on py3 because it was using the reserved async keyword:

File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/beaker/synchronization.py", line 289
self.async = 0
^
SyntaxError: invalid syntax

setup: bump formencode minimum version to 1.3.1 to get py3 support

1.3.0 failed with py3:

build/minimum-dependency-versions-venv/lib/python3.7/site-packages/formencode/api.py:153:
if unicode is not str: # Python 2
E NameError: name 'unicode' is not defined

setup: bump ipaddr minium version to 2.2.0 to get py3 support

2.2.0 is also the latest version, but released in 2017.

Previous versions failed installation on py3 with:

File "/tmp/pip-install-o31ka351/ipaddr/ipaddr.py", line 1450
ip_int = 0L
^
SyntaxError: invalid syntax
----------------------------------------

setup: bump TurboGears minimum version to 2.4

Avoid failures starting with test_api_wrong_key:

E File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/tg/request_local.py", line 37, in languages_best_match
E items = [i for i, q in sorted(al._parsed, key=lambda iq: -iq[1])]
E TypeError: 'NoneType' object is not iterable

setup: bump WebOb minimum version to 1.8

b075693b3214 introduced use of acceptable_offers which with WebOb < 1.8 would
fail with:

File ".../kallithea/lib/middleware/pygrack.py", line 189, in __call__
elif req.accept.acceptable_offers(self.valid_accepts):
AttributeError: 'MIMEAccept' object has no attribute 'acceptable_offers'

cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly

The configuration and type fixing will be invoked from make_base_app, and we
will thus not have to do it:

File "kallithea/config/middleware.py", line 31, in make_app_without_logging
return make_base_app(global_conf, full_stack=full_stack, **app_conf)
File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app
wrap_app)
File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app
app = TGApp(conf)
File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__
app_wrapper = wrapper(self.wrapped_dispatch, self.config)
File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__
self.options = parse_cache_config_options(config)
File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options

This will fix a py3 problem where setup_cache_regions was run *after* beaker
had coerced types, thus introducing string types in the config where beaker
expected the integers it had put there.

git: initialize hook app environment with paste as we do in other places

We want to use the whole paste / tg stack because it will do more
initialization for us - consistently, and in the right order.

This will allow a next change to avoid a problem in our custom beaker
initialization.

Going through paste might be a bit more heavy and slower than before, but it is
only run once for each git push.

Ultimately, we should probably get rid of the special hook entry point, and
just use kallithea-cli as the only entry point.

py3: open files as binary or not, depending on how we want to use them

The difference will matter when bytes and str are different.

summary: compute lang_stats consistently

Visiting a /statistics with py3 would fail with:
... in statistics
sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10]
TypeError: '<' not supported between instances of 'dict' and 'dict'

The "summary" computation didn't have that problem. And it put '?' as
description for unknown extensions. And it had stable output as it also sorted
on the file extension as secondary key. Just use that.

summary: drop no_data in all possible ways

The controllers some odd "is" checks did and removed has been.

The javascript variables were never used.

summary.html didn't use no_data at all.

The only real use was in statistics.html ... where it seems like a better fit
to just use c.stats_percentage .

lib: fix mail address encodings and add test coverage

Now it also works on Py3 ... apparently in more cases and more correctly than
before.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

vcs: drop unused NodeGeneratorBase()()

Creating a whole list seems inefficient ... but it is fortunately not used.

vcs: drop GitChangeset.id

It seems to be like raw_id, except the odd feature that it sometimes would
return 'tip'.

lib: de-obfuscate nested functions in pygmentize_annotation and how they always pass an annotate_from_changeset_func to AnnotateHtmlFormatter

These functions contain the only call to annotate_highlight, which contains
the only call to AnnotateHtmlFormatter.

annotate_highlight is thus clearly always passed an
annotate_from_changeset_func.

vcs: drop unused kallithea/lib/vcs/utils/annotate.py

Some of it seems to live on in kallithea/lib/annotate.py.

vcs: fix get_file_annotate - consistently bind sha so it has the right value when executing

The Git implementation did *not* save the sha value in the lambda expression
for the "changeset lazy loader". Thus, if the generator had moved on and
assigned a different value to sha when the expression was executed, it would
use the "wrong" sha.

Fixed by doing as the Hg implementation: bind the sha value as value of a
default parameter when defining the lambda expression.

The Hg implementation did however also save the line - it is not used, and
there is no need for that.

vcs: tweak how revisions and repo names are shown in error messages

Decode bytes to str, and show repo name instead of repr or full server file
system path. In some places, it will only report the "basename" of the
repository, without any "group names" that also would be nice to have. The easy
alternative would be to show the full file system path ... but it would be
unfortunate to leak absolute server side paths to end users.

vcs: minor cleanup

Make it more friendly to humans and py3.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

git: more elegant handling of installed pre/post-receive hook failing on direct repo access

The hook would fail with a long backtrace when get_hook_environment raise an error exception.

Instead, as first thing in the entry point from the hook, catch that situation
and report it nicely before "quietly" skipping the hook:

[mk@here myrepo]$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 204 bytes | 204.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Skipping Kallithea Git post-recieve hook 'hooks/post-receive'.
remote: Git was apparently not invoked by Kallithea: Environment variable KALLITHEA_EXTRAS not found
To /tmp/somerepo
* [new branch] master -> master
[mk@here myrepo]$

We could be paranoid and let it (and the pre hook) fail ... but that doesn't
seem helpful.

Reported by Edmund Wong at [1].

[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003071.html

logging: always invoke fileConfig with '__file__' and 'here'

WSGI servers tend to provide '__file__' and 'here' as 'defaults' when invoking
fileConfig, so '%(here)s' string interpolation also can be used in logging
configuration.

Make sure we do the same when we initialize logging without using a WSGI server.

It is annoying to have to do this, and it will only in rare cases make any
difference ... but it seems like the best option.

Patch also modified by Mads Kiilerich.

logging: drop fileConfig initialization in make_app - backout 0d4dd9380a45

0d4dd9380a45 was a bit harmful, as it might overwrite existing good logging
configuration.

0d4dd9380a45 no longer seems relevant: Testing shows that logging for `gearbox
serve` *is* activated anyway. gearbox/commands/serve.py will invoke
"setup_logging" right before "loadapp".

We must and can assume that logging has been initialized before make_app.

Reported and based on analysis by Wolfgang Scherer.

ssh: drop usk_public_key_idx again

Essentially a backout of d2a97f73fa1f and the
4851d15bc437_db_migration_step_after_95c01895c006_ alembic step.

We can't reliably have full index on fields with unbounded length. The
upgrade step has been reported to fail on MySQL [1]:

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError)
(1170, "BLOB/TEXT column 'public_key' used in key specification without
a key length") [SQL: u'CREATE INDEX usk_public_key_idx ON user_ssh_keys
(public_key)'] (Background on this error at: http://sqlalche.me/e/e3q8)

And we really don't need this index ... especially now when we use
fingerprints for key deletion instead of looking up by the full public key.


[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003068.html

ssh: make it clear that SshKeyModel.delete has user as mandatory parameter

It is already provided in the two uses:
kallithea/controllers/admin/my_account.py: SshKeyModel().delete(fingerprint, request.authuser.user_id)
kallithea/controllers/admin/users.py: SshKeyModel().delete(fingerprint, c.user.user_id)

ssh: use fingerprint when deleting public keys

Avoid relying on a database index of the full public key string.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

cleanup: fix db.py "import *"

pyflakes rightfully complains about wildcard imports. But we like to keep the
short names, so let's just be explicit about which names we import.

py3: fix error in template from introducing safe_unicode in 9203621cae03

Add missing test coverage of actual edit form.

validator: fix ASCII password check to verify if it can be *encoded* in ascii

In Python 2, unicode strings have a .decode method (which really doesn't make
sense). Python 3 has more strict typing by design, and unicode strings don't
have a .decode method.

A Unicode string "is ASCII" if it can be encoded as ASCII. The check should
thus *encode* to ASCII - not decode.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

py3: drop __unicode__ ... and generally move towards just providing a good __repr__

We should never show str() (or similar) to end users.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

controllers: use unicode for internal rendering of template snippets

Like 8e2313be042d.

gist: make it a bit more clear how gist_access_id is used ... and how it is different from gist_id

A gist has a gist_access_id which gives access to it. For private Gists, it is
a multi-letter secure random string.

gist_id is the primary key in the database and thus an automatically
incrementing integer. It is also used as the not-so-secret gist_access_id for
public gists.

This gets rid of one odd safe_unicode applied to an int.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

lib: refactor _get_access_path as get_path_info

We will need it later when it gets more tricky to get the path from environ ...

lib: avoid comparing ordering with None

Fails in py3.

lib: mktime needs a tuple - not a list

Py3 is more strict.

lib: base64 followup from 82b1eaec25f5

Needed for py3.

lib: avoid import string.replace

Odd way to do it ... and gone in py3.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

vcs: make hg get_changesets compatible with py3

If start was 0, we would end up with start_pos None and getting a compare with
None which would fail in py3.

Instead, make sure start_pos is None iff start is None.

Also, make the actual check more robust.

vcs: change check_integrity to work on paths

Py3 failed because nodes can't be hashed.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

vcs: change get_diff to use _run_git_command directly

This will allow run_git_command to return strings while get_diff still return
raw bytes.

lib: let get_git_version invoke git as all other commands do, without special options

There is no need for _bare or _safe. It is fine to have '-c
core.quotepath=false' before '--version', and it is perfectly fine to get a
RepositoryError if things go terribly wrong.

vcs: use more correct git revision identifiers

It will have the right types and make a difference for py3.

vcs: replace __getslice__ with slice handling in __getitem__

Needed for py3.

vcs: fix __eq__ to return false if types are different

Follow up to ae12fabba699.

vcs: drop Repository.name_unicode

It wouldn't make much sense in py3 ... and is barely used.

vcs: replace "has_key" with "in" in decorator skip criteria

Needed for py3.

tests: fix ordering of get_inline_comments output

Py3 would give different ordering.

tests: fix test_files.py json comparison to load json to struct so it is independent of serialization order

Py3 would fail the test because different ordering.

tests: sort groups in test_enforce_groups check to make it independent of ordering

Py3 would give another ordering.

tests: fix ordering of options of "ret_type must be one of ..." returned from api

Py3 would fail becauase it use another ordering.

tests: avoid unicode u'' repr in test_diff_markup output

It will go away with py3 anyway.

tests: refactor test_redirection_to_login_form_preserves_get_args to test more correctly

Preparing for py3.

lib: introduce string conversion functions for ASCII without further encoding concerns

Avoid the trial-and-error and vagueness of the "safe" functions.

This should replace some use of safe_unicode and safe_str. It will mostly be a
noop in py2 but will be crucial in py3.

lib: establish py3 compatible strategy for string handling: introducing safe_bytes and deprecating safe_str

The meaning of safe_str will change when moving to py3. All use of safe_str is
thus tech debt that we have to chop off, mostly by moving to either
safe_unicode or safe_bytes ... or dropping because we know what we are doing
and rely on the improved type safety in py3.

vcs: drop unused _diff_name_status

It popped up as something to do for py3. Don't waste time on it.

lib: some cleanup of utils.py imports

Utils really shouldn't rely on model ...