summary: compute lang_stats consistently

Visiting a /statistics with py3 would fail with:
... in statistics
sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10]
TypeError: '<' not supported between instances of 'dict' and 'dict'

The "summary" computation didn't have that problem. And it put '?' as
description for unknown extensions. And it had stable output as it also sorted
on the file extension as secondary key. Just use that.

summary: drop no_data in all possible ways

The controllers some odd "is" checks did and removed has been.

The javascript variables were never used.

summary.html didn't use no_data at all.

The only real use was in statistics.html ... where it seems like a better fit
to just use c.stats_percentage .

lib: fix mail address encodings and add test coverage

Now it also works on Py3 ... apparently in more cases and more correctly than
before.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

vcs: drop unused NodeGeneratorBase()()

Creating a whole list seems inefficient ... but it is fortunately not used.

vcs: drop GitChangeset.id

It seems to be like raw_id, except the odd feature that it sometimes would
return 'tip'.

lib: de-obfuscate nested functions in pygmentize_annotation and how they always pass an annotate_from_changeset_func to AnnotateHtmlFormatter

These functions contain the only call to annotate_highlight, which contains
the only call to AnnotateHtmlFormatter.

annotate_highlight is thus clearly always passed an
annotate_from_changeset_func.

vcs: drop unused kallithea/lib/vcs/utils/annotate.py

Some of it seems to live on in kallithea/lib/annotate.py.

vcs: fix get_file_annotate - consistently bind sha so it has the right value when executing

The Git implementation did *not* save the sha value in the lambda expression
for the "changeset lazy loader". Thus, if the generator had moved on and
assigned a different value to sha when the expression was executed, it would
use the "wrong" sha.

Fixed by doing as the Hg implementation: bind the sha value as value of a
default parameter when defining the lambda expression.

The Hg implementation did however also save the line - it is not used, and
there is no need for that.

vcs: tweak how revisions and repo names are shown in error messages

Decode bytes to str, and show repo name instead of repr or full server file
system path. In some places, it will only report the "basename" of the
repository, without any "group names" that also would be nice to have. The easy
alternative would be to show the full file system path ... but it would be
unfortunate to leak absolute server side paths to end users.

vcs: minor cleanup

Make it more friendly to humans and py3.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

git: more elegant handling of installed pre/post-receive hook failing on direct repo access

The hook would fail with a long backtrace when get_hook_environment raise an error exception.

Instead, as first thing in the entry point from the hook, catch that situation
and report it nicely before "quietly" skipping the hook:

[mk@here myrepo]$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 204 bytes | 204.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Skipping Kallithea Git post-recieve hook 'hooks/post-receive'.
remote: Git was apparently not invoked by Kallithea: Environment variable KALLITHEA_EXTRAS not found
To /tmp/somerepo
* [new branch] master -> master
[mk@here myrepo]$

We could be paranoid and let it (and the pre hook) fail ... but that doesn't
seem helpful.

Reported by Edmund Wong at [1].

[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003071.html

logging: always invoke fileConfig with '__file__' and 'here'

WSGI servers tend to provide '__file__' and 'here' as 'defaults' when invoking
fileConfig, so '%(here)s' string interpolation also can be used in logging
configuration.

Make sure we do the same when we initialize logging without using a WSGI server.

It is annoying to have to do this, and it will only in rare cases make any
difference ... but it seems like the best option.

Patch also modified by Mads Kiilerich.

logging: drop fileConfig initialization in make_app - backout 0d4dd9380a45

0d4dd9380a45 was a bit harmful, as it might overwrite existing good logging
configuration.

0d4dd9380a45 no longer seems relevant: Testing shows that logging for `gearbox
serve` *is* activated anyway. gearbox/commands/serve.py will invoke
"setup_logging" right before "loadapp".

We must and can assume that logging has been initialized before make_app.

Reported and based on analysis by Wolfgang Scherer.

ssh: drop usk_public_key_idx again

Essentially a backout of d2a97f73fa1f and the
4851d15bc437_db_migration_step_after_95c01895c006_ alembic step.

We can't reliably have full index on fields with unbounded length. The
upgrade step has been reported to fail on MySQL [1]:

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError)
(1170, "BLOB/TEXT column 'public_key' used in key specification without
a key length") [SQL: u'CREATE INDEX usk_public_key_idx ON user_ssh_keys
(public_key)'] (Background on this error at: http://sqlalche.me/e/e3q8)

And we really don't need this index ... especially now when we use
fingerprints for key deletion instead of looking up by the full public key.


[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003068.html

ssh: make it clear that SshKeyModel.delete has user as mandatory parameter

It is already provided in the two uses:
kallithea/controllers/admin/my_account.py: SshKeyModel().delete(fingerprint, request.authuser.user_id)
kallithea/controllers/admin/users.py: SshKeyModel().delete(fingerprint, c.user.user_id)

ssh: use fingerprint when deleting public keys

Avoid relying on a database index of the full public key string.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

cleanup: fix db.py "import *"

pyflakes rightfully complains about wildcard imports. But we like to keep the
short names, so let's just be explicit about which names we import.

py3: fix error in template from introducing safe_unicode in 9203621cae03

Add missing test coverage of actual edit form.

validator: fix ASCII password check to verify if it can be *encoded* in ascii

In Python 2, unicode strings have a .decode method (which really doesn't make
sense). Python 3 has more strict typing by design, and unicode strings don't
have a .decode method.

A Unicode string "is ASCII" if it can be encoded as ASCII. The check should
thus *encode* to ASCII - not decode.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

py3: drop __unicode__ ... and generally move towards just providing a good __repr__

We should never show str() (or similar) to end users.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

controllers: use unicode for internal rendering of template snippets

Like 8e2313be042d.

gist: make it a bit more clear how gist_access_id is used ... and how it is different from gist_id

A gist has a gist_access_id which gives access to it. For private Gists, it is
a multi-letter secure random string.

gist_id is the primary key in the database and thus an automatically
incrementing integer. It is also used as the not-so-secret gist_access_id for
public gists.

This gets rid of one odd safe_unicode applied to an int.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

lib: refactor _get_access_path as get_path_info

We will need it later when it gets more tricky to get the path from environ ...

lib: avoid comparing ordering with None

Fails in py3.

lib: mktime needs a tuple - not a list

Py3 is more strict.

lib: base64 followup from 82b1eaec25f5

Needed for py3.

lib: avoid import string.replace

Odd way to do it ... and gone in py3.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

vcs: make hg get_changesets compatible with py3

If start was 0, we would end up with start_pos None and getting a compare with
None which would fail in py3.

Instead, make sure start_pos is None iff start is None.

Also, make the actual check more robust.

vcs: change check_integrity to work on paths

Py3 failed because nodes can't be hashed.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

vcs: change get_diff to use _run_git_command directly

This will allow run_git_command to return strings while get_diff still return
raw bytes.

lib: let get_git_version invoke git as all other commands do, without special options

There is no need for _bare or _safe. It is fine to have '-c
core.quotepath=false' before '--version', and it is perfectly fine to get a
RepositoryError if things go terribly wrong.

vcs: use more correct git revision identifiers

It will have the right types and make a difference for py3.

vcs: replace __getslice__ with slice handling in __getitem__

Needed for py3.

vcs: fix __eq__ to return false if types are different

Follow up to ae12fabba699.

vcs: drop Repository.name_unicode

It wouldn't make much sense in py3 ... and is barely used.

vcs: replace "has_key" with "in" in decorator skip criteria

Needed for py3.

tests: fix ordering of get_inline_comments output

Py3 would give different ordering.

tests: fix test_files.py json comparison to load json to struct so it is independent of serialization order

Py3 would fail the test because different ordering.

tests: sort groups in test_enforce_groups check to make it independent of ordering

Py3 would give another ordering.

tests: fix ordering of options of "ret_type must be one of ..." returned from api

Py3 would fail becauase it use another ordering.

tests: avoid unicode u'' repr in test_diff_markup output

It will go away with py3 anyway.

tests: refactor test_redirection_to_login_form_preserves_get_args to test more correctly

Preparing for py3.

lib: introduce string conversion functions for ASCII without further encoding concerns

Avoid the trial-and-error and vagueness of the "safe" functions.

This should replace some use of safe_unicode and safe_str. It will mostly be a
noop in py2 but will be crucial in py3.

lib: establish py3 compatible strategy for string handling: introducing safe_bytes and deprecating safe_str

The meaning of safe_str will change when moving to py3. All use of safe_str is
thus tech debt that we have to chop off, mostly by moving to either
safe_unicode or safe_bytes ... or dropping because we know what we are doing
and rely on the improved type safety in py3.

vcs: drop unused _diff_name_status

It popped up as something to do for py3. Don't waste time on it.

lib: some cleanup of utils.py imports

Utils really shouldn't rely on model ...

hg: bump minimum version to 5.1

We will soon move to Python 3 which only will support 5.1 or later.

Remove old hacks and tech debt.

Also avoids future warning:
DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

py3: only use safe_str for string conversion - not for arbitrary __str__ invocation

Usually, we actually just want unicode string, and it is better to just unicode().

lib: only maintain one copy of safe_str / safe_unicode

The standalone-ish nature of vcs gets a bit in the way. It already depends on
some very generic Kallithea functionality. But for now, avoid code duplication,
and let Kallithea use vcs functionality instead of duplicating it.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

feeds: replace webhelpers.feedgenerator with simple mako templating for rendering RSS/Atom XML

Most of the complexity in RSS libraries is in dynamically supporting all kinds
of attributes. For our use, we have a small static set of attributes we use,
and it is simpler to just use mako.

Also, webhelpers is dead, and the alternatives seem quite heavy.

repo: don't just report user name and email in one field - separate things properly

In the repo RSS feed, report author as
<author>name@example.com (User Name)</author>
instead of using
<dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">User Name &lt;name@example.com&gt;</dc:creator>

And in the ATOM feed with name and email separate:
<author>
<name>User Name</name>
<email>name@example.com</email>
</author>
Instead of
<name>User Name &lt;name@example.com&gt;</name>

journal: don't include email in author name - avoid double data

In journal RSS feed, report author as:
<author>name@example.com (User Name)</author>
instead of double email due to:
<author>name@example.com (User Name &lt;name@example.com&gt;)</author>

In the journal ATOM feed, report author as:
<author>
<name>user Name</name>
<email>test_admin@example.com</email>
</author>
instead of using double email due to:
<name>name@example.com (User Name &lt;name@example.com&gt;)</name>

i18n: updated translation for Russian

Currently translated at 100.0% (691 of 691 strings)

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

py3: prepare for types.NotImplementedType going away

From 2to3 types.