mail: drop unicode filter from templates

Unicode is the default in py3, and we no longer pass any non-string arguments
that need stringification before rendering.

This is backing out a part of 4fbab9d5be4d.

pytype: introduce configuration for running pytype

Install pytype and dependencies on Python 3.8 with:
pip install --upgrade -r pytype_requirements.txt

Checks can then be run with:
pytype

pytype: add assertion to guide pytype through Whoosh usage

Mute pytype warnings:

File "kallithea/controllers/admin/admin.py", line 105, in _journal_filter: No attribute 'fieldname' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 106, in _journal_filter: No attribute 'text' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'startdate' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'enddate' on None [attribute-error]
In Optional[Any]

pytype: add some assertions to guide pytype through const dict with tricky typing structure

These assertions also make the code more explicit and slightly more readable.

Mute pytype warnings:

File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'path' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]
File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'content' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]

pytype: add Python type annotations where necessary to guide pytype

Mute pytype warnings:

File "kallithea/lib/auth.py", line 142, in _cached_perms_data: No attribute 'DEFAULT_USER_ID' on module 'kallithea' [module-attr]
File "kallithea/lib/vcs/backends/base.py", line 73, in ...: No attribute '...' on BaseRepository [attribute-error]
File "kallithea/lib/vcs/backends/base.py", line 405, in ...: No attribute '...' on BaseChangeset [attribute-error]
File "kallithea/tests/api/api_base.py", line 2397, in test_api_get_changeset: No attribute 'TEST_REVISION' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_DST' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_SRC' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2467, in test_api_get_pullrequest: No attribute 'TEST_PR_REVISIONS' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 67, in api_call: No attribute 'app' on _BaseTestApi [attribute-error]
File "kallithea/tests/base.py", line 154, in log_user: No attribute 'app' on TestController [attribute-error]
File "kallithea/tests/base.py", line 169, in _get_logged_user: No attribute '_logged_username' on TestController [attribute-error]

pytype: mute errors from import of optional or platform specific modules

Mute pytype warnings:

File "kallithea/bin/kallithea_cli_iis.py", line 69, in iis_install: Can't find module 'isapi_wsgi'. [import-error]
File "kallithea/config/post_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/config/pre_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/lib/compat.py", line 59, in kill: No attribute 'windll' on module 'ctypes' [module-attr]
File "kallithea/lib/utils.py", line 242, in is_valid_repo_uri: Can't find module 'hgsubversion.svnrepo'. [import-error]
File "kallithea/tests/scripts/manual_test_concurrency.py", line 203, in <module>: No attribute '_RandomNameSequence' on module 'tempfile' [module-attr]

bin: add missing imports

Spotted by pytype --strict-import:
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'utils' on module 'kallithea.lib' [module-attr]
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'scm' on module 'kallithea.model' [module-attr]

It happened to work anyway due to the import chain ... but it is better to be
explicit.

ssh: handle case where hg serve_forever doesn't serve forever

serve_forever will no longer exit in Mercurial 5.7 .
This changed with https://mercurial-scm.org/repo/hg/rev/b7b8a1538161 .

hooks: on Git, invoke hooks/post-receive-custom from hooks/post-receive

Make it possible for admins to install all kinds of hooks.

Based on a patch by Tim Ooms.

A more generic solution would be nice, but for now we aim for this minimal
solution.

hooks: clarify in UI and documentation how the hook control essentially is Mercurial only

The Git hook model doesn't directly allow multiple hooks of the same kind.

hooks: when overwriting non-Kallithea hooks, move the old hook to .bak

Note: This will only happen when actually overwriting an unknown hook. Repeated
hook installation will thus not clobber the existing .bak file until a
non-Kallithea hook is found again.

hooks: be more consistent in only using active Ui entries

There is no UI to control or display the ui_active value for custom hooks, but
*if* they are inactive, they will be ignored in make_ui, and it will be
misleading and confusing to show them in the list of active custom hooks.

There *should* never be any inactive hooks entries, but let's be consistent in
handling the case *if* it should happen. (It happened for me while hacking
around.)

hooks: put repo_size and update hooks in kallithea namespace

Keep things separate.

Include missing migration steps for 642847355a10.

hooks: use internal Mercurial module for update after push

Don't rely on having a working 'hg' command in $PATH.

hooks: fix incorrect visual nesting in the admin UI

A form-group was missing.

celery: upgrade to Celery 5.0 ... and adjust for Click API

Celery 5 has apparently no relevant API or config changes.

Celery is however switching to click. run_from_argv goes away, and there is no
simple way to do as before and start the worker with our Celery app but still
use Celery's own command line parser.

Apply hacks to make sure it still is possible to run like:

kallithea-cli celery-run -c my.ini -- --help

kallithea-cli celery-run -c my.ini -- --loglevel=ERROR broker_url=amqp://u:p@localhost:5672/v

celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: move repo create tasks to repo model - that's where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

The whole thing about model classes and instantiation is just trouble - we will
get rid of that soon.

celery: move Whoosh indexing task to whoosh library where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: let async tasks choose at runtime if they should use immediate execution or dispatch to the Celery worker

Make it completely safe to use task annotation at import time, before global
config has been set.

celery: celery-run should only initialize app and sqlalchemy after workers have been forked

If app and SqlAlchemy were initialized before launching celery, the forked
workers would inherit the database connection ... and that doesn't work.

This could be handled by disposing the engine after forking the worker or
before each task ... but it remains unnecessary and wrong to initialize the
engine early when it isn't used, and then fork it.

celery: change celery-run to not take a whole app - take a config file, and then initialize app

A step towards a better future ...

celery: don't rely on CELERY_APP only being set (from app_cfg) when use_celery

We would rather always have CELERY_APP set.

Kind of backing out 894a662b12b3.

cli: in runtime_wrapper for config file handling, don't return unused command handler value

Return values are not used - we should use sys.exit or similar to set command
return codes.

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

celery: drop traces of support for task_always_eager

It was broken - retrieval of results didn't work.

Also, task_always_eager still require a working celery.broker_url even though
it is unused. It is thus much better to just set use_celery=false .

celery: always drop db session after running a task

Drop dbsession annotation - it could easily be forgotten. This seems simpler
and more reliable.

celery: use regular automatic session handling for tasks

There were no point in having a special celery get_session() .

celery: use explicit task names - avoid automatic naming with "kallithea.lib.celerylib." prefix

We wrap async functions in a local f_async wrapper, defined in
kallithea/lib/celerylib/__init__.py . For a function Foo.X, even though we
changed the wrapper's __name__ to X, the tasks would be named
kallithea.lib.celerylib.X , without using the actual module name of X for
namespacing.

Drop modifying __name__, and just specify the name explicitly, without trying
to namespace it.

TurboGears: drop workaround for < 2.4

Backout a38e05a0c79e and tweak.

api: fix update_repo check for permission to create top level repos

The hg.create.repository permission only apply to creation (or renaming) of top
level repos - it is not relevant for other kinds of renaming.

Moving or renaming repos in other locations is now covered by other checks.

api: fix repo group permission check for repo creation

hg.create.repository is only controlling whether all users have write access at
top level. For other repo locations, check for write permission to the repo
group.

Note: This also covers "repo creation" by forking or by moving another repo.

api: fix repo creation inside repo groups when using Celery

Repo creation would fail when Celery tried serializing a whole RepoGroup
object. It only worked when not using Celery.

model: simplify how get_commits_stats task group on author

Avoid using the caching h.person . We want to get rid of the model
dependency on helpers.

The stats are persisted, and any temporary incorrectness in the long term
cached h.person will thus remain forever. It is thus arguably better to avoid
using it in this place.

get_commits_stats is also a long running task, so speed is not *that* critical.
And generally, processing commits in order will have a lot of the same
committers, so a local cache will have a good hit rate.

(Alternatively, h.person could perhaps be in user model ... but that's not how
it is now.)

lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

indexers: drop sys.path mangling in daemon

The original comment in b153a51b1d3b was "to get the pylons_app import" ...
which seems like a bad workaround for not having installed the app properly
before running.

tests: drop vcs setup_package

Repos are always created before running vcs tests, and the SCMFetcher code was
thus effectively dead.

tests: always pass username as cur_user

Passing a whole User wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

tests: fix create_fork to actually pass a repo id as fork_parent_id

Passing a whole repo wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

api: don't pass owner to create_repo in form_data

create_repo gets owner as 'cur_user', but 'owner' in form_data was unused.

celery: use kallithea.CELERY_APP.AsyncResult instead of celery.result import

This is more spot-on and more correct.

celery: drop kallithea.CELERY_EAGER - it is more spot-on to look directly at task_always_eager

The global flag was set as a side effect in the make_app function. That's not
pretty.

celery: fix CELERY_APP creation after breakage in ea1c608efa3a

A part of change still under development had sneaked in.

vcs: fix subprocessio process termination

659ecd26002c had a cut'n'paste error so it dropped process termination in the
SubprocessIOChunker close method.

api: fix repo creation from API when using celery

Repo creation would fail when celery tried serializing a whole User object. It
only worked when not using celery.

changelog: fix crash on display of empty commit messages

Also handle commit messages with leading whitespace.

.splitlines() is different from .split('\n') and will give an empty list when
given an empty string.

Problem reported and initial patch proposed by Brett Smith.

lib: clean up pointless use of h.person on db.User just to get .username

After cleaning up, h.person will never be invoked with a db.User .

indexer: drop "akc" cleanup

It was introduced in 0eceb478c720 but it is unclear what it does. We would
rather understand and fix any js encoding errors.

vcs: drop repo contact ... and the odd uses of it

A half-baked and essentially unused hgweb feature.

The reference in get_commits_stats doesn't seem reachable in any relevant use
cases.

lib: move urlification to webutils

Less use of helpers in model.

notifications: only render body with mentions if there is a body

Notifications for registering a user would pass body=None. It is unfortunate to
pass None all the way to render_w_mentions - it seems fair enough that this
function expects a str.

lib: drop is_hg and is_git

It is just as simple to be explicit.

templates: pass repo type to repolabel as str

Keep it simple and avoid passing complex repo objects around.

lib: move js escaping from helpers to webutils

It was no real problem it lived in helpers: it was only used from templates
anyway.

But for completeness and correctness, move all web utils to webutils.

celery: rename celerypylons to more descriptive celery_app

This module is all about creating a Celery app.

docs: fix broken ref

7982eac0703d removed too much.

repo-scan: recommend running this to reinstall Git hooks after upgrading

It would be too invasive to fully automate this.

repo-scan: log error and continue after failure creating a scanned repo

Continue after for example:
Exception: invalid repo name Foo

repo-scan: overwrite_git_hooks also implies install_git_hooks

(Otherwise, we should probably report an error when ignoring it.)

docs: describe, visualize, and verify internal code structure and layering

Try to describe something that isn't entirely there yet.

deps.py will help track and minimize violations through checks and
visualization in deps.svg .

scripts: handle running with pending deleted files

Don't fail on files in manifest missing in working directory.

diff: handle some escaped characters in Git diffs

There are some odd characters (like \r and \n) that the Kallithea UI doesn't
allow in filenames in repos. Kallithea (through the routes module) will fail to
generate URLs when browsing Files. That is a known limitation with minimal
real-world impact, non-trivial to work around or fix.

There are very few relevant use cases for tracking files with odd filenames. \t
is valid but is hard to render in a meaningful way in the UI. And ASCII
characters like \ and " are not usable on Windows and should just be avoided.

Kallithea would parse Git diffs with odd characers incorrectly or fail, even
before hitting the known limitation. With this change, Kallithea will parse
diffs with odd filenames correctly (and then hit the limitation).

Git will quote odd filenames and escape the odd characters when emitting diffs.
(Mercurial does by design not allow \r and \n , and Mercurial will thus never
have to quote file names in diffs.)

Quotes are already handled (and ignored). With this change, Kallithea will
handle \ unescaping of \\ and \", the usual letters like \r and \n and \t, and
octal numbers like \033 (for ESC) .

Filenames with \ and " will work perfectly (when not on Windows).

Filenames with \t and ESC will work fine, but without helpful display in the
UI.

Filenames with \r and \n will still make the UI fail when trying to generate
URLs.

Thanks to stypr of Flatt Security for raising this.

diff: improved handling of Git diffs with " quoting

Kallithea would intentionally and explicitly fail with an ugly exception when
trying to parse Git diffs with quoted filenames.

Improve this by parsing quotes ... and ignore them, as long as they are
matching. The content inside the quotes might be \-escaped ... but that could
potentially also be the case without quoting. We will fix that later.

Adding some test cases that before would have failed to parse and raised an
exception.

Thanks to stypr of Flatt Security for raising this.

hg: make sure _check_url only accept the protocols accepted by is_valid_repo_uri

Avoid unnecessary flexibility, ambiguity, and complexity.

This is similar to what 3ea3d3a2b3e3 did for Git.

vcs: use urlparse in _check_url

This makes URL checking more standard and more explicit.

vcs: always pass str to _check_url

hg would sometimes pass bytes.