i18n: updated translation for French

Currently translated at 99.8% (1079 of 1081 strings)

i18n: update normalized translations

Refactorings moved things around.

i18n: fix duplicate entries after fixing odd space before ? in "Don't have an account ?"

Dropping the odd spaces in 462064bd9489 gave some duplicate translation strings
... but there are no actual conflicting translations, so it can be resolved
"trivially".

ini: set translate_forwarded_server = False when using paste prefix middleware

Paste Deploy PrefixMiddleware will be default trust HTTP headers that a proxy
server might have set. That can be a problem if there is no proxy server or if
the proxy just pass these headers through.

Change the configuration example to disable this translation.

git: fix whitespace in previous commit

scripts/whitespacecleanup.sh will remove tabs introduced in f0fbb0fe4462 ...
but that is fine; it was just an example.

git: update check for invalid URL characters to work with Python versions that include an attempt at fixing the very same problem

With changes like
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
making it to Python 3.10 and being backported to previous Python versions, the
approach in a8a51a3bdb61 no longer works when combined with
urllib.parse.urlparse in d2f59de17bef: path will never contain the invalid
characters.

To catch this case anyway, add a new check to verify that the parsed URL can
roundtrip back to the original representation with urllib.parse.urlunparse .

The actual exception might vary, but one of them should always fire.

There is a risk that the new check will reject some URLs that somehow isn't
normalized. No such cases have been found yet.

auth: only use X- headers instead of wsgi.url_scheme if explicitly told so in url_scheme_header - drop https_fixup setting

Before, several X- headers would be trusted to overrule the actual connection
protocol (http or https) seen by the Kallithea WSGI server. That was mainly
when https_fixup were set, but it incorrectly also kicked in if https_fixup or
use_htsts were configured. The ambiguity of which headers were used also made
it less reliable. The proxy server not only had to be configured to set one of
the headers correctly, it also had to make sure other headers were not passed
on from the client. It would thus in some cases be possible for clients to fake
the connection scheme, and thus potentially be possible to bypass restrictions
configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
protocol. Users can configure url_scheme_header to for example
HTTP_X_FORWARDED_PROTO instead of using the default wsgi.url_scheme .

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

The semantics of the old https_fixup were unsafe, so it has been dropped.
Admins that are upgrading must change their configuration to use the new
url_scheme_header option.

auth: only use X- headers instead of REMOTE_ADDR if explicitly told so in remote_addr_header

Before, X-Forwarded-For (and others) headers would *always* be trusted blindly,
also in setups without a proxy server. It would thus in some cases be
possible for users to fake their IP, and thus potentially be possible to bypass
IP restrictions configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
remote address. Users can configure remote_addr_header to for example
HTTP_X_FORWARDED_FOR instead of using the default REMOTE_ADDR.

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

auth: don't trust clients too much - only trust the *last* IP in the X-Forwarded-For header

The X-Forwarded-For header contains a list of IP addresses, where each
proxy server appends the IP they see their request coming from.
See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For .

Trusting the *first* IP in HTTP_X_FORWARDED_FOR would allow clients to claim
any IP, which could be used to bypass IP restrictions configured in Kallithea.

Instead, only trust the last proxy in the chain, and thus only use the *last*
IP in HTTP_X_FORWARDED_FOR. (In setups where more than last IP should be
trusted, the last proxy server in the chain must be configured rewrite the
header accordingly.)

setup: make chardet a mandatory dependency

It was made mandatory in 9685f50a69d0. At that time it seemed like it always
were present due to indirect dependencies, but apparently that is no longer the
case.

hg: read hgrc, also in repo_scan

This might make a difference in some real world cases.

db: read hgrc in Repository.scm_instance_no_cache and Gist.scm_instance

This will make sure .hg/hgrc is read in some code paths where it was missing.
That might make a difference in some real world cases.

hg: support Mercurial 5.8

Some 'remote:' messages changed to appear on stderr.

pytype: drop pytype_requirements.txt appenlight dependency

appenlight support was dropped in 09826e98b064.

tests: address pedantic pytype 2021.4.1 warning

File "kallithea/tests/api/api_base.py", line 1315, in test_api_fork_repo_non_admin_no_permission_to_fork: Name 'fork_name' is not defined [name-error]

diffs: let DiffProcessor inject context_lineno in chunk changes

Make it possible to use these numbers reliably, also outside as_html.

diffs: drop pointless check for "\ No newline at end of file"

Such lines would have been caught on '\' as "unknown command" by the command
check a few lines above.

diffs: drop pointless check for adding context line

The length of the match was always 5.

comments: don't open comment form on page load for comments on line 0

Data attributes have magic parsing of strings so '0' becomes 0 which is false.

Further, 0 == '' in JavaScript, so we have to use !== '' to catch empty strings.

diffs: DiffParser should only HTML escape lines when generating HTML

Move DiffParser HTML escaping so it only is applied when generating HTML.

diffs: add test coverage of inline diff with other markup

Add trailing whitespaces ... but also whitespace before changes (which might
seem trailing but isn't).

mail: drop unicode filter from templates

Unicode is the default in py3, and we no longer pass any non-string arguments
that need stringification before rendering.

This is backing out a part of 4fbab9d5be4d.

pytype: introduce configuration for running pytype

Install pytype and dependencies on Python 3.8 with:
pip install --upgrade -r pytype_requirements.txt

Checks can then be run with:
pytype

pytype: add assertion to guide pytype through Whoosh usage

Mute pytype warnings:

File "kallithea/controllers/admin/admin.py", line 105, in _journal_filter: No attribute 'fieldname' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 106, in _journal_filter: No attribute 'text' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'startdate' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'enddate' on None [attribute-error]
In Optional[Any]

pytype: add some assertions to guide pytype through const dict with tricky typing structure

These assertions also make the code more explicit and slightly more readable.

Mute pytype warnings:

File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'path' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]
File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'content' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]

pytype: add Python type annotations where necessary to guide pytype

Mute pytype warnings:

File "kallithea/lib/auth.py", line 142, in _cached_perms_data: No attribute 'DEFAULT_USER_ID' on module 'kallithea' [module-attr]
File "kallithea/lib/vcs/backends/base.py", line 73, in ...: No attribute '...' on BaseRepository [attribute-error]
File "kallithea/lib/vcs/backends/base.py", line 405, in ...: No attribute '...' on BaseChangeset [attribute-error]
File "kallithea/tests/api/api_base.py", line 2397, in test_api_get_changeset: No attribute 'TEST_REVISION' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_DST' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_SRC' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2467, in test_api_get_pullrequest: No attribute 'TEST_PR_REVISIONS' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 67, in api_call: No attribute 'app' on _BaseTestApi [attribute-error]
File "kallithea/tests/base.py", line 154, in log_user: No attribute 'app' on TestController [attribute-error]
File "kallithea/tests/base.py", line 169, in _get_logged_user: No attribute '_logged_username' on TestController [attribute-error]

pytype: mute errors from import of optional or platform specific modules

Mute pytype warnings:

File "kallithea/bin/kallithea_cli_iis.py", line 69, in iis_install: Can't find module 'isapi_wsgi'. [import-error]
File "kallithea/config/post_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/config/pre_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/lib/compat.py", line 59, in kill: No attribute 'windll' on module 'ctypes' [module-attr]
File "kallithea/lib/utils.py", line 242, in is_valid_repo_uri: Can't find module 'hgsubversion.svnrepo'. [import-error]
File "kallithea/tests/scripts/manual_test_concurrency.py", line 203, in <module>: No attribute '_RandomNameSequence' on module 'tempfile' [module-attr]

bin: add missing imports

Spotted by pytype --strict-import:
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'utils' on module 'kallithea.lib' [module-attr]
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'scm' on module 'kallithea.model' [module-attr]

It happened to work anyway due to the import chain ... but it is better to be
explicit.

ssh: handle case where hg serve_forever doesn't serve forever

serve_forever will no longer exit in Mercurial 5.7 .
This changed with https://mercurial-scm.org/repo/hg/rev/b7b8a1538161 .

hooks: on Git, invoke hooks/post-receive-custom from hooks/post-receive

Make it possible for admins to install all kinds of hooks.

Based on a patch by Tim Ooms.

A more generic solution would be nice, but for now we aim for this minimal
solution.

hooks: clarify in UI and documentation how the hook control essentially is Mercurial only

The Git hook model doesn't directly allow multiple hooks of the same kind.

hooks: when overwriting non-Kallithea hooks, move the old hook to .bak

Note: This will only happen when actually overwriting an unknown hook. Repeated
hook installation will thus not clobber the existing .bak file until a
non-Kallithea hook is found again.

hooks: be more consistent in only using active Ui entries

There is no UI to control or display the ui_active value for custom hooks, but
*if* they are inactive, they will be ignored in make_ui, and it will be
misleading and confusing to show them in the list of active custom hooks.

There *should* never be any inactive hooks entries, but let's be consistent in
handling the case *if* it should happen. (It happened for me while hacking
around.)

hooks: put repo_size and update hooks in kallithea namespace

Keep things separate.

Include missing migration steps for 642847355a10.

hooks: use internal Mercurial module for update after push

Don't rely on having a working 'hg' command in $PATH.

hooks: fix incorrect visual nesting in the admin UI

A form-group was missing.

celery: upgrade to Celery 5.0 ... and adjust for Click API

Celery 5 has apparently no relevant API or config changes.

Celery is however switching to click. run_from_argv goes away, and there is no
simple way to do as before and start the worker with our Celery app but still
use Celery's own command line parser.

Apply hacks to make sure it still is possible to run like:

kallithea-cli celery-run -c my.ini -- --help

kallithea-cli celery-run -c my.ini -- --loglevel=ERROR broker_url=amqp://u:p@localhost:5672/v

celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: move repo create tasks to repo model - that's where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

The whole thing about model classes and instantiation is just trouble - we will
get rid of that soon.

celery: move Whoosh indexing task to whoosh library where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: let async tasks choose at runtime if they should use immediate execution or dispatch to the Celery worker

Make it completely safe to use task annotation at import time, before global
config has been set.

celery: celery-run should only initialize app and sqlalchemy after workers have been forked

If app and SqlAlchemy were initialized before launching celery, the forked
workers would inherit the database connection ... and that doesn't work.

This could be handled by disposing the engine after forking the worker or
before each task ... but it remains unnecessary and wrong to initialize the
engine early when it isn't used, and then fork it.

celery: change celery-run to not take a whole app - take a config file, and then initialize app

A step towards a better future ...

celery: don't rely on CELERY_APP only being set (from app_cfg) when use_celery

We would rather always have CELERY_APP set.

Kind of backing out 894a662b12b3.

cli: in runtime_wrapper for config file handling, don't return unused command handler value

Return values are not used - we should use sys.exit or similar to set command
return codes.

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

celery: drop traces of support for task_always_eager

It was broken - retrieval of results didn't work.

Also, task_always_eager still require a working celery.broker_url even though
it is unused. It is thus much better to just set use_celery=false .

celery: always drop db session after running a task

Drop dbsession annotation - it could easily be forgotten. This seems simpler
and more reliable.

celery: use regular automatic session handling for tasks

There were no point in having a special celery get_session() .

celery: use explicit task names - avoid automatic naming with "kallithea.lib.celerylib." prefix

We wrap async functions in a local f_async wrapper, defined in
kallithea/lib/celerylib/__init__.py . For a function Foo.X, even though we
changed the wrapper's __name__ to X, the tasks would be named
kallithea.lib.celerylib.X , without using the actual module name of X for
namespacing.

Drop modifying __name__, and just specify the name explicitly, without trying
to namespace it.

TurboGears: drop workaround for < 2.4

Backout a38e05a0c79e and tweak.

model: simplify how get_commits_stats task group on author

Avoid using the caching h.person . We want to get rid of the model
dependency on helpers.

The stats are persisted, and any temporary incorrectness in the long term
cached h.person will thus remain forever. It is thus arguably better to avoid
using it in this place.

get_commits_stats is also a long running task, so speed is not *that* critical.
And generally, processing commits in order will have a lot of the same
committers, so a local cache will have a good hit rate.

(Alternatively, h.person could perhaps be in user model ... but that's not how
it is now.)

lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

indexers: drop sys.path mangling in daemon

The original comment in b153a51b1d3b was "to get the pylons_app import" ...
which seems like a bad workaround for not having installed the app properly
before running.

tests: drop vcs setup_package

Repos are always created before running vcs tests, and the SCMFetcher code was
thus effectively dead.

tests: always pass username as cur_user

Passing a whole User wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

tests: fix create_fork to actually pass a repo id as fork_parent_id

Passing a whole repo wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

api: don't pass owner to create_repo in form_data

create_repo gets owner as 'cur_user', but 'owner' in form_data was unused.

celery: use kallithea.CELERY_APP.AsyncResult instead of celery.result import

This is more spot-on and more correct.

celery: drop kallithea.CELERY_EAGER - it is more spot-on to look directly at task_always_eager

The global flag was set as a side effect in the make_app function. That's not
pretty.