vcs: slight clean up of _check_url

Make static void function static and without return value.

lib/diffs: make sure that trailing tabs are indicated

Backport of 53142fd5af4e - the problem also existed before f79c40759d6f.

Eat trailing newline after trailing tab as we already do for trailing spaces.

The change diff mainly shows re group numbers shifting.

celery: change template to use celery 4 setting name celery.result_backend

Missed in e1d4a0d8520f ...

front-end: use 'bin' path for node commands instead of '.bin'

license-checker is using relative paths for importing other modules - that
worked fine when .bin/license-checker was a symlink, but not on filesystems
without symlinks support:

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module '../lib/index'

git: disallow odd characters in path of git:// URLs

Mitigate https://blog.harold.kim/2020/11/invalid-url-on-git-clone-leading-to-ssrf
until the problem is fixed properly in Git.

The checks might be more strict than necessary but should not have any impact
on real world use cases.

Thanks to stypr of Flatt Security for raising this.

git: make sure _check_url only accept the protocols accepted by is_valid_repo_uri

Avoid unnecessary flexibility, ambiguity, and complexity.

The file protocol was never used. But when cloning existing managed repos,
is_valid_repo_url would be skipped and _check_url would be called with absolute
paths.

repo_groups: fix select of parent group when adding repo group

h.select was passed a list of repo groups where group_id was integer, but
parent_group in the request was a string - thus no match.

Do as in repos controller create_repository (and in error handling): leave it
to htmlfill to patch up the generated HTML using defaults ... but make sure we
always have a default.

repo_groups: extra escape of names when used in select drop-downs

The lack of escaping could be a problem *if* it was possible to create repo
groups with dangerous names.

This was seen for example when specifying parent group of repos and repo
groups.

We want to keep groups_choices as HTML literals so paths can use » as
separator.

repos: extra HTML escaping of repo and repo group names shown in DataTables

These names will already have been "slugged" and can thus not contain anything
that can be used for any attack. But let's be explicitly safe and escape them
anyway.

raw_name without escaping would cause XSS *if* it was possible to create unsafe
repo names.

just_name must be escaped in order to make search work correctly - for example
if searching for '<' ... *if* it was possible for names to contain that.

repogroups: fix HTML markup of descriptions

Repogroup descriptions were not urlified like repo descriptions are. That
caused incorrect rendering with posibility of XSS.

The problem was introduced in 0.4.0 with 6db3122e4d75.

Thanks to stypr of Flatt Security for reporting this vulnerability.

model/user: 'body' parameter to NotificationModel().create() is unused so remove it

The email template for new user registration mails does not use the 'body'
parameter, so don't pass anything.

cleanup: get rid of redundant assignments 'foo = foo'

Redundant assignments were searched with:

grep '^ *\(.*\) = \1$' `hg files`

Some of these cases have resulted from earlier cleanup, like:

foo = safe_unicode(foo)

and later batch removing safe_unicode usage. See e.g. commit
e35373106528b29461b7f5144780cb9ec79f765d .


Note: two false positives occur:

kallithea/tests/models/test_settings.py:39:
setting.app_settings_value = setting.app_settings_value
kallithea/tests/models/test_settings.py:42:
setting.app_settings_value = setting.app_settings_value

which is not a plain assignment but where 'app_settings_value' is a property
method with side effects.

model/comment: drop unused fields in email_kwargs

Keep things as simple as possible.

model: remove unused 'subject' parameter of NotificationModel.create()

The subject of emails is determined with EmailNotificationModel._subj_map,
based on the notification type. The 'subject' parameter passed to
NotificationModel.create is completely unused.

Remove this parameter and update its callers, removing code that is now no
longer used.

model/comment: inline _get_notification_data

This method was only used in one place, from code which was very related.
Inline it to group things together and reduce complexity.

model/comment: extract notification-related code into a separate method

Preparation for grouping with _get_notification_data next, and keeping clear
separation between creating the comment itself, and creating the
notification.

mail: migrate to modern email.message instead of using old compat32 functionality

Before, the HTML part could be detected as pure 7-bit and sent as:
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Now, it will use:
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

That should work fine too.

mail: fix duplicate "From" headers

Problem introduced in 9a0c41175e66: When iterating the headers dict and setting
"msg[key] = value", it wasn't replacing the header but performing add_header so
we sometimes ended up with two From headers.

It is also a general problem that while the headers dict only can contain each
key once, it can contain entries that only differ in casing and thus will fold
to the same message header, making it possible to end up adding duplicate
headers.

"msg.replace_header(key, value)" is not a simple solution to the problem: it
will raise KeyError if no such previous key exists.

Now, make the problem more clear by explicitly using add_header.

Avoid the duplication problem by deleting the key (no matter which casing)
before invoking add_header. Delete promises that "No exception is raised if the
named field isn’t present in the headers".

front-end: use 'bin' path for node commands instead of '.bin'

license-checker is using relative paths for importing other modueles - that
worked fine when .bin/license-checker was a symlink, but not on filesystems
without symlinks support:

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module '../lib/index'

middleware: drop appenlight integration

It seems trivial and is currently not hooked up and has not been tested with py3. It is thus better to drop it.

hooks: move the vcs hook entry points and setup code out of lib

Mercurial hooks are running in a process that already has been initialized, so
they invoke the hooks lib directly. Git hooks are binaries and need a lot of
initialization before they can do the same. Move this extra setup code
elsewhere.

Having this high level code in bin is perhaps also not ideal, but it also
doesn't seem that bad: that is where other command line entry points invoke
make_app.

(It seems like it could be adventageous to somehow use "real" bin commands for
hooks ... but for now we use the home-made templates.)

Note: As a side effect of this change, all git hooks *must* be re-installed
when upgrading.

hooks: import whole hooks module

Minimize impact if there should be cycles.

vcs: move ssh handlers out of "backends"

The ssh handlers are much more high-level than the low-level backends ... and
not used as a part of the "backends" concept at all.

git: detect existing symlink hooks before overwriting - only update plain files

If the existing hook is a symlink, the hook is not what we put in place, and we
should stay away - no matter if it points at something that looks like a
Kallithea hook.

git: write Git hook files atomically

Make sure we don't follow symlinks or inherit permissions from previously
installed hook.

git: minor refactor and clean-up of hook installation code

Make "pre-receive" and "post-receive" more explicit and grepable.

scm: simplify InMemoryChangeset handling

Avoid conditional imports.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

model: changes toward import whole modules

*If* there should be circular dependencies, importing 'from' another module
could fail because the module at that time only was partially imported. That
had to be worked around by importing at runtime instead of globally.

Instead, try to always import whole modules. (But we should still try to avoid
cycles.)

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

lib: move link_to_ref from helpers to utils2

Less reason to import the messy helpers.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

lib: consistently use webutils.url instead of h.url

Templates are still the valid use case for accessing through h.

Simplify mocking of url() function.

lib: drop sparsely used h.not_mapped_error

It might be a good idea, but then we should use it much more consistently ...
and it should probably be done differently. Let's keep it simple and be
consistent.

lib: move locale.py to locales.py to avoid shadowing of standard module

"Fix" spurious problem, seen for example as:

$ python kallithea/lib/annotate.py
Traceback (most recent call last):
File ".../lib64/python3.8/site-packages/mercurial/encoding.py", line 107, in <module>
encoding = locale.getpreferredencoding().encode('ascii') or b'ascii'
AttributeError: module 'locale' has no attribute 'getpreferredencoding'

That happened when something in some other module tried to import stdlib locale
... but somehow would pick up the kallithea locale module and things would
fail.

Stay out of that kind of trouble by using a name that doesn't collide.

diffs: remove unused argument enable_comments and class no-comment

enable_comments was only used to set/not-set the 'no-comment' CSS class.
While this class was emitted, no CSS rule nor any JavaScript logic was
actually using it. Last real usage of that class was removed with commit
e87baa8f1c5bd2488aefc23b95c0db3a04bc8431.

Cleanup the code by not emitting 'no-comment' and remove the
'enable_comments' flag.

style: mark failed comment submissions with red panel heading

Make it more obvious to the user that a comment submission failed: mark the
panel of the failed comment as "panel-danger" so the color of the comment panel
heading changes to red.

Previously, only the user and comment text would fade a bit.

lib/diffs: make sure that trailing tabs are indicated

Between the initial submission and final version of commit f79c40759d6f,
changes were made that turn out to be incorrect. The changes assume that the
later match on trailing tabs will 'win' from the plain 'tab' match. However,
Python 're' documentation says:

As the target string is scanned, REs separated by '|' are tried from
left to right. When one pattern completely matches, that branch is
accepted. This means that once A matches, B will not be tested further,
even if it would produce a longer overall match. In other words, the '|'
operator is never greedy.
https://docs.python.org/3.8/library/re.html

As a result, a trailing tab is seen as a plain tab and not highlighted in a
special way.

Unify the tab handling to make it unambiguous how they should be parsed.

The change diff mainly shows re group numbers shifting.

diff: refactor to avoid complexity of passing unnecessary trailing newline around

_escaper tried to handle these optional newlines ... but did not always preserve
them. Keep it simple.

In normal operations, all diff lines end with \n . We only saw un-terminated
lines in a couple of old test cases.

hg: support update with Mercurial 5.6

Fix:
TypeError: update() takes from 1 to 3 positional arguments but 5 were given

git: fix upload-pack content-type with py3

It is checked earlier that git_command is one of two string constants, and with
py3, things are much simpler and we don't have to consider string coersion.

Reported and debugged by Valentin.

diffs: add doctests for _escaper, also showing incorrect behavior

Add some doctests for the _escaper function.
Note: some tests now show incorrect behavior that will be fixed soon.

cli: fix db-create after removal of 'tests' argument in DbManage

Broken by commit 3afc6bff84bb.

[Thomas De Schampheleire: edited commit message]

lib: drop timerproxy

This module has been unused for forever and does no good.

mail: use plain standard library for sending mails instead of rcmail

Avoid a lot of custom complexity that adds no value.

A few trivial lines from rcmail are inlined.

mail: simplify celery task for sending mail

Name variables after their config setting, and drop undocumented debug setting.

db: consistently use the name settings for the return value from get_app_settings

Get rid of yet another old rc reference.

(The "settings" in auth is slightly different, but pretty much the same.)

lib: fix bad references to utils3

A problem introduced in when rebasing 5e46f73f0d1c after renaming the temporary
utils3 name to webutils.

html: make mako markup structure more well-formed

And avoid using mako control structures inside tags.

html: put 'use strict' on separate lines

use.py:
import re
import sys
for fn in sys.argv[1:]:
with open(fn) as f:
s = f.read()
s = re.sub(r'''(<script>)('use strict';)\n( *)''', r'''\1\n\3\2\n\3''', s)
with open(fn, 'w') as f:
f.write(s)

python use.py $(hg loc 'kallithea/templates/**.html')

docs: drop documentation for installing without virtualenv

Installing globally as root require an unjustified trust in our build/install
process ... and makes it very hard to uninstall or clean up.

Installing as a user is slightly better but also more error-prone and not
recommended.

Those who really want to do it and know how to do it also don't need this
description.

config: move WSGI middleware apps from lib to config

These middlewares are full WSGI applications - that is not so lib-ish. The
middleware is referenced from the application in config - that seems like a
good place for them to live.

config: move config.conf to kallithea.lib

kallithea.config is mainly the WSGI entry point with TG stuff, and thus a high
level controller thing - not a place to store configuration.

Note: The content of conf.py is kind of similar to what we have in
kallithea/__init__.py . These two should perhaps be merged somehow.

config: move various py templates to kallithea/templates/py/

For some reason, we had some python templates in kallithea/config .
kallithea.config is mainly the TG entry point, and thus a high level controller
thing - not a place to store templates.

Instead, use the templates directory and introduce a new py subdirectory.

With git hook templates in a templates directory, there is no need for tmpl in
the name.

routing: move config.routing to kallithea.controllers

Routing doesn't belong in config. Having it there caused unfortunate
dependencies.

We do routing the old way. If we did it the new way, it would be defined in the
root controller. But for now, we just place it next to the root controller.

routing: move ADMIN_PREFIX constant from routing to top level kallithea module

Perhaps not the ideal place, but that is how we handle somewhat similar data.
This avoids an unfortunate dependency from the model to routing.

routing: separate url handling from routing - move it to webutils

This is a helper method relying on the thread local tg.request. We didn't have
a good place to put it. Now we do.

This (re)moves unfortunate dependencies to the routing module (which almost is
a controller).

vcs: move changeset diff from controller to vcs

Remove unfortunate model dependency on controller ... and put the VCS details
where they belong, with less VCS specific knowledge in the controllers.

lib: move extract_mentioned_users out of utils2

Avoid model.db reference to keep utils2.py independent of Kallithea classes.

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

db_manage: drop special DbManage mode for tests

There is no need for a special mode - just use force_ask to answer yes to
"destroy old database".