tests: update test_filenode_path for py3

Unicode is now handled very differently. There are no encoded unicode str, and
the test barely makes sense any more.

tests: accept py3 output in test_list_valued_setting_update

The old behaviour seems completely broken and useless and thus evidently unused
... but the new result seems slightly better.

py3: fix non-ASCII URLs - decode unicode correctly before passing them to controllers as unicode strings

This is needed for supporting localized repo path names in the path of URLs.

Some references:

https://www.python.org/dev/peps/pep-0333/#unicode-issues
https://bugs.python.org/issue16679
http://lucumr.pocoo.org/2010/5/25/wsgi-on-python-3/
https://bugs.launchpad.net/pecan/+bug/1451842
https://github.com/tipabu/eventlet/commit/a5a7751b013fe99b6d30acbca79e819770e9ae5d

py3: update safe_* functions for how unicode pretty much has been renamed to str

safe_str and safe_unicode is thus the same thing.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: use inspect.getfullargspec instead of deprecated inspect.getargspec

Avoid some pytest warnings.

py3: fix kallithea-cli ini parsing after ConfigParser got strict=True

ConfigParser in py3 defaults to strict=True and would thus reject our ssh
logging hack of renaming config sections ... which cause duplicate section
names.

Fortunately, fileConfig now also allows passing a ConfigParser, and we can
avoid using io.StringIO .

py3: update ssh for base64.b64decode raising binascii.Error instead of TypeError

A command like:
python -c 'import base64; base64.b64decode("QQ")'
would fail in Python2 with:
TypeError: Incorrect padding
but in python3:
binascii.Error: Incorrect padding

py3: use html.escape instead of cgi.escape

DeprecationWarning: cgi.escape is deprecated, use html.escape instead

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: support new stdlib module names configparser, urllib and http lib

From "2to3 -f imports".

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

py3: use global next() function instead of .next() method

From 2to3 -f next.

py3: use .__func__ instead of .im_func

Based on 2to3 -f methodattrs ... but manually dropping a __func__ because py3 unbound methods *are* the func itself.

py3: support __bool__

From 2to3 -f nonzero.

py3: drop support for long - just use int instead

From 2to3 long.

py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way

Bump Mercurial minimum version to 5.2 - the first version that claim stable py3
support.

setup: support Mercurial 5.3

Kallithea has been updated for the only known API change in 5.3.

The 5.3 upgrade notes do not indicate other relevant changes.

vcs: drop unused utils/archivers.py

Don't ever bother to try to consider maintaining it.

pull-request: refactor iteration version bump

Avoid relying on exceptions thrown when re.match fails ... and thus please
pytype and make the code better.

summary: use calendar.monthrange instead of internal calendar.mday

Fix pytype warning:
No attribute 'mdays' on module 'calendar' [module-attr]

This also takes care of handling leap-years correctly. Not that it matters, but
very nice ...

auth: make crowd logging simpler

There is no point in creating dicts and then logging them as json. Also, json
can't handle py3 bytes and it would fail on py3. (ext_json could perhaps handle
bytes, but it seems better to keep it simple and explicit.)

If the default repr isn't good enough, it would be better to use pprint. But
repr is good enough.

py3: fix recaptcha Request parameter type

urllib.parse.urlencode will (after making '%' encoding of unicode characters)
return a str instead of the obviously correct 7-bit ascii bytes.

When we need bytes, we thus have to encode to bytes.

py3: use exception .args instead of .message

Args seems slightly more fragile and *could* introduce problems for trivial use
if args is empty. But .message is gone.

py3: make get_current_authuser handle missing tg context consistently and explicitly

tg context handling ends up using
tg.support.registry.StackedObjectProxy._current_obj for attribute access ...
which if no context has been pushed will end up in:
raise TypeError(
'No object (name: %s) has been registered for this '
'thread' % self.____name__)

utils2.get_current_authuser used code like:
if hasattr(tg.tmpl_context, 'authuser'):

Python 2 hasattr will call __getattr__ and return False if it throws any
exception. (It would thus catch the TypeError and silently fall through to use
the default user None.) This hasattr behavior is confusing and hard to use
correctly. Here, it was used incorrectly. It has been common practice to work
around by using something like:
getattr(x, y, None) is not None

Python 3 hasattr fixed this flaw and only catches AttributeError. The TypeError
would thus (rightfully) be propagated. That is a change that must be handled
when introducing py3 support.

The get_current_authuser code could more clearly and simple and py3-compatible
be written as:
return getattr(tmpl_context, 'authuser', None)
- but then we also have to handle the TypeError explicitly ... which we are
happy to do.

celery: introduce make_app instead of creating app at import time

It is dirty to instantiate things at import time (unless it really is basic
singletons).

In 0.5.1 (and earlier), such dirtyness made partial test execution fail when
other things had global side effects and things didn't use the usual import
order:

$ py.test kallithea/lib/
collecting ...
――― kallithea/lib/celerypylons/__init__.py ―――
kallithea/lib/celerypylons/__init__.py:58: in <module>
app.config_from_object(celery_config(tg.config))
kallithea/lib/celerypylons/__init__.py:28: in celery_config
assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded'
data/env/lib/python2.7/site-packages/tg/configuration/tgconfig.py:31: in __getitem__
return self.config_proxy.current_conf()[key]
E KeyError: 'celery.imports'

Avoid that by running a "factory" function when the celery app actually is
needed.

scripts: use explicit relative imports of contributor_data

Avoid any ambiguity ... and avoid pytype failing to find
scripts/contributor_data.py when scripts isn't in sys.path.

config: fix pyflakes warning about unused tg import

app_cfg had an unused 'import tg'. tg.hooks was used, but through a separate
import. Clean that up by consistently using tg (which always makes tg.hooks
available) and dropping the separate hooks import.

cli: fix missing import of kallithea.config.middleware

Don't rely on other imports making it available as side effect.

vcs: fix missing import of mercurial.localrepo

(localrepo might already always be available in the mercurial namespace due to
side effects from other imports, but it is still better to do it explicit ...
and also to please pytype.)

pytest: fix register_assert_rewrite call

Since a while, the test suite shows following warning:

kallithea/tests/__init__.py:29
/home/tdescham/repo/contrib/kallithea/kallithea-review/kallithea/tests/__init__.py:29: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: kallithea.tests
pytest.register_assert_rewrite('kallithea.tests')

The problem can be fixed by moving the register_assert_rewrite call from
kallithea/tests/__init__.py to the root-level conftest.py, outside of the
'kallithea' module.

vcs: drop unused get_hook_location (Issue #353)

Also, the hg implementation was odd - the '.hgrc' directory name looks very
similar to 'hgrc' but could be anything, such as 'hooks'.

setup: bump sqlalchemy minimum version to 1.2.9 to get rid of py3 warning

Avoid verbose warning:

build/minimum-dependency-versions-venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py:637
.../site-packages/sqlalchemy/util/langhelpers.py:637: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
d_args = inspect.formatargspec(spec[0][1:])

setup: bump alembic minimum version to 1.0.10 to get rid of py3 warning

Avoid verbose warning:

.../site-packages/alembic/util/langhelpers.py:92: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
formatvalue=lambda x: '=' + x)

setup: bump decorator minimum version to 4.2.1 to get rid of py3 warning

Avoid verbose warning:

.../site-packages/decorator.py:95: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
formatvalue=lambda val: "", *argspec)[1:-1]

setup: bump dulwich minimum version to 0.19.0 to get good py3 support

Fix failure in test_compare_forks_on_branch_extra_commits_git ... but make a
major bump to make sure we get good py3 support in this cruicial and complex
low level library.

setup: bump webtest minimum version to 2.0.6 to get py3 stdlib support

/usr/lib64/python3.7/http/cookiejar.py:723: in is_third_party
if not domain_match(req_host, reach(request.origin_req_host)):
E AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

setup: bump whoosh minimum version to 2.7.1 to get py3 support

Fix:

- kallithea/tests/functional/test_search_indexing.py:111 TestSearchControllerIndexing.test_repository_tokenization[content-this_should_be_unique_content-1-group/*]

File ".../lib/python3.7/site-packages/whoosh/reading.py", line 241, in expand_prefix
if fn != fieldname or not text.startswith(prefix):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

setup: bump mako minimum version to 0.9.1 to get py3 support

Avoid test_api_get_pullrequest crash:

data/env3/lib/python3.7/site-packages/mako/template.py:653: in _compile_text
code = compile(source, cid, 'exec')
E File "email_templates_button_html", line 15
E def render_body(context,url,title='',padding_top=,padding_bottom=,**pageargs):
E ^
E SyntaxError: invalid syntax

setup: bump python-dateutil minimum version to 2.1.0 to get py3 support

Avoid py2 syntax:

E File "/home/mk/kallithea-py3/build/minimum-dependency-versions-venv/lib/python3.7/site-packages/dateutil/tz.py", line 78
E `self._name`,
E ^
E SyntaxError: invalid syntax

setup: bump beaker minimum version to 1.10.1 to get py3 support

1.10.0 failed on py3 because it was using the reserved async keyword:

File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/beaker/synchronization.py", line 289
self.async = 0
^
SyntaxError: invalid syntax

setup: bump formencode minimum version to 1.3.1 to get py3 support

1.3.0 failed with py3:

build/minimum-dependency-versions-venv/lib/python3.7/site-packages/formencode/api.py:153:
if unicode is not str: # Python 2
E NameError: name 'unicode' is not defined

setup: bump ipaddr minium version to 2.2.0 to get py3 support

2.2.0 is also the latest version, but released in 2017.

Previous versions failed installation on py3 with:

File "/tmp/pip-install-o31ka351/ipaddr/ipaddr.py", line 1450
ip_int = 0L
^
SyntaxError: invalid syntax
----------------------------------------

setup: bump TurboGears minimum version to 2.4

Avoid failures starting with test_api_wrong_key:

E File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/tg/request_local.py", line 37, in languages_best_match
E items = [i for i, q in sorted(al._parsed, key=lambda iq: -iq[1])]
E TypeError: 'NoneType' object is not iterable

setup: bump WebOb minimum version to 1.8

b075693b3214 introduced use of acceptable_offers which with WebOb < 1.8 would
fail with:

File ".../kallithea/lib/middleware/pygrack.py", line 189, in __call__
elif req.accept.acceptable_offers(self.valid_accepts):
AttributeError: 'MIMEAccept' object has no attribute 'acceptable_offers'

cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly

The configuration and type fixing will be invoked from make_base_app, and we
will thus not have to do it:

File "kallithea/config/middleware.py", line 31, in make_app_without_logging
return make_base_app(global_conf, full_stack=full_stack, **app_conf)
File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app
wrap_app)
File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app
app = TGApp(conf)
File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__
app_wrapper = wrapper(self.wrapped_dispatch, self.config)
File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__
self.options = parse_cache_config_options(config)
File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options

This will fix a py3 problem where setup_cache_regions was run *after* beaker
had coerced types, thus introducing string types in the config where beaker
expected the integers it had put there.

git: initialize hook app environment with paste as we do in other places

We want to use the whole paste / tg stack because it will do more
initialization for us - consistently, and in the right order.

This will allow a next change to avoid a problem in our custom beaker
initialization.

Going through paste might be a bit more heavy and slower than before, but it is
only run once for each git push.

Ultimately, we should probably get rid of the special hook entry point, and
just use kallithea-cli as the only entry point.

py3: open files as binary or not, depending on how we want to use them

The difference will matter when bytes and str are different.

summary: compute lang_stats consistently

Visiting a /statistics with py3 would fail with:
... in statistics
sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10]
TypeError: '<' not supported between instances of 'dict' and 'dict'

The "summary" computation didn't have that problem. And it put '?' as
description for unknown extensions. And it had stable output as it also sorted
on the file extension as secondary key. Just use that.

summary: drop no_data in all possible ways

The controllers some odd "is" checks did and removed has been.

The javascript variables were never used.

summary.html didn't use no_data at all.

The only real use was in statistics.html ... where it seems like a better fit
to just use c.stats_percentage .

lib: fix mail address encodings and add test coverage

Now it also works on Py3 ... apparently in more cases and more correctly than
before.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

vcs: drop unused NodeGeneratorBase()()

Creating a whole list seems inefficient ... but it is fortunately not used.

vcs: drop GitChangeset.id

It seems to be like raw_id, except the odd feature that it sometimes would
return 'tip'.

lib: de-obfuscate nested functions in pygmentize_annotation and how they always pass an annotate_from_changeset_func to AnnotateHtmlFormatter

These functions contain the only call to annotate_highlight, which contains
the only call to AnnotateHtmlFormatter.

annotate_highlight is thus clearly always passed an
annotate_from_changeset_func.

vcs: drop unused kallithea/lib/vcs/utils/annotate.py

Some of it seems to live on in kallithea/lib/annotate.py.

vcs: fix get_file_annotate - consistently bind sha so it has the right value when executing

The Git implementation did *not* save the sha value in the lambda expression
for the "changeset lazy loader". Thus, if the generator had moved on and
assigned a different value to sha when the expression was executed, it would
use the "wrong" sha.

Fixed by doing as the Hg implementation: bind the sha value as value of a
default parameter when defining the lambda expression.

The Hg implementation did however also save the line - it is not used, and
there is no need for that.

vcs: tweak how revisions and repo names are shown in error messages

Decode bytes to str, and show repo name instead of repr or full server file
system path. In some places, it will only report the "basename" of the
repository, without any "group names" that also would be nice to have. The easy
alternative would be to show the full file system path ... but it would be
unfortunate to leak absolute server side paths to end users.

vcs: minor cleanup

Make it more friendly to humans and py3.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

cleanup: fix db.py "import *"

pyflakes rightfully complains about wildcard imports. But we like to keep the
short names, so let's just be explicit about which names we import.

py3: fix error in template from introducing safe_unicode in 9203621cae03

Add missing test coverage of actual edit form.

validator: fix ASCII password check to verify if it can be *encoded* in ascii

In Python 2, unicode strings have a .decode method (which really doesn't make
sense). Python 3 has more strict typing by design, and unicode strings don't
have a .decode method.

A Unicode string "is ASCII" if it can be encoded as ASCII. The check should
thus *encode* to ASCII - not decode.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

py3: drop __unicode__ ... and generally move towards just providing a good __repr__

We should never show str() (or similar) to end users.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

controllers: use unicode for internal rendering of template snippets

Like 8e2313be042d.