i18n: updated translation for Greek

Currently translated at 94.9% (1032 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for Polish

Currently translated at 64.1% (697 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 94.9% (1032 of 1087 strings)

i18n: updated translation for German

Currently translated at 69.0% (751 of 1087 strings)

i18n: updated translation for French

Currently translated at 99.7% (1084 of 1087 strings)

i18n: updated translation for Portuguese

Currently translated at 45.8% (498 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 99.3% (1080 of 1087 strings)

hg: restore make_ui reading of .hg/hgrc

This regressed with f83326e2e66c, released in 0.6.0 .

Fix problem reported by Felipe Cardoso Resende.

api: fix update_repo check for permission to create top level repos

The hg.create.repository permission only apply to creation (or renaming) of top
level repos - it is not relevant for other kinds of renaming.

Moving or renaming repos in other locations is now covered by other checks.

api: fix repo group permission check for repo creation

hg.create.repository is only controlling whether all users have write access at
top level. For other repo locations, check for write permission to the repo
group.

Note: This also covers "repo creation" by forking or by moving another repo.

api: fix repo creation inside repo groups when using Celery

Repo creation would fail when Celery tried serializing a whole RepoGroup
object. It only worked when not using Celery.

api: fix repo creation from API when using celery

Repo creation would fail when celery tried serializing a whole User object. It
only worked when not using celery.

changelog: fix crash on display of empty commit messages

Also handle commit messages with leading whitespace.

.splitlines() is different from .split('\n') and will give an empty list when
given an empty string.

Problem reported and initial patch proposed by Brett Smith.

lib/diffs: make sure that trailing tabs are indicated

Backport of 53142fd5af4e - the problem also existed before f79c40759d6f.

Eat trailing newline after trailing tab as we already do for trailing spaces.

The change diff mainly shows re group numbers shifting.

celery: change template to use celery 4 setting name celery.result_backend

Missed in e1d4a0d8520f ...

front-end: use 'bin' path for node commands instead of '.bin'

license-checker is using relative paths for importing other modules - that
worked fine when .bin/license-checker was a symlink, but not on filesystems
without symlinks support:

internal/modules/cjs/loader.js:883
throw err;
^

Error: Cannot find module '../lib/index'

git: disallow odd characters in path of git:// URLs

Mitigate https://blog.harold.kim/2020/11/invalid-url-on-git-clone-leading-to-ssrf
until the problem is fixed properly in Git.

The checks might be more strict than necessary but should not have any impact
on real world use cases.

Thanks to stypr of Flatt Security for raising this.

git: make sure _check_url only accept the protocols accepted by is_valid_repo_uri

Avoid unnecessary flexibility, ambiguity, and complexity.

The file protocol was never used. But when cloning existing managed repos,
is_valid_repo_url would be skipped and _check_url would be called with absolute
paths.

repo_groups: fix select of parent group when adding repo group

h.select was passed a list of repo groups where group_id was integer, but
parent_group in the request was a string - thus no match.

Do as in repos controller create_repository (and in error handling): leave it
to htmlfill to patch up the generated HTML using defaults ... but make sure we
always have a default.

repo_groups: extra escape of names when used in select drop-downs

The lack of escaping could be a problem *if* it was possible to create repo
groups with dangerous names.

This was seen for example when specifying parent group of repos and repo
groups.

We want to keep groups_choices as HTML literals so paths can use » as
separator.

repos: extra HTML escaping of repo and repo group names shown in DataTables

These names will already have been "slugged" and can thus not contain anything
that can be used for any attack. But let's be explicitly safe and escape them
anyway.

raw_name without escaping would cause XSS *if* it was possible to create unsafe
repo names.

just_name must be escaped in order to make search work correctly - for example
if searching for '<' ... *if* it was possible for names to contain that.

repogroups: fix HTML markup of descriptions

Repogroup descriptions were not urlified like repo descriptions are. That
caused incorrect rendering with posibility of XSS.

The problem was introduced in 0.4.0 with 6db3122e4d75.

Thanks to stypr of Flatt Security for reporting this vulnerability.

git: fix upload-pack content-type with py3

It is checked earlier that git_command is one of two string constants, and with
py3, things are much simpler and we don't have to consider string coersion.

Reported and debugged by Valentin.

i18n: updated translation for Polish

Currently translated at 56.3% (612 of 1087 strings)

i18n: updated translation for Norwegian Bokmål

Currently translated at 34.5% (376 of 1087 strings)

i18n: updated translation for French

Currently translated at 99.1% (1078 of 1087 strings)

git: fix pull request deletion - don't crash on deleting refs to PR heads

The refs name was passed as unicode string, and that would cause failure like:
File ".../site-packages/dulwich/repo.py", line 720, in __delitem__
if name.startswith(b"refs/") or name == b"HEAD":
TypeError: startswith first arg must be str or a tuple of str, not bytes

Fixed by correctly passing the ref name as bytes, as we do when creating the PR
refs.

Tests added by Mads Kiilerich.

tests: add coverage of Git pullrequests

Git is so different than Mercurial that it is easier to use separate tests.
Some of the tests that are relevant for Mercurial doesn't apply to the Git
support.

Also fix crash an odd corner case of creating PRs from a repo that has no
branches at all.

api: fix 'kallithea-api --save-config'

Commit eca0cb56a822 attempted to fix a type inconsistency, which caused
failure in the 'kallithea-api' tool when using '--save-config', but this
unfortunately did not fix the problem completely.

Following error still appeared:

Traceback (most recent call last):
File ".../bin/kallithea-api", line 33, in <module>
sys.exit(load_entry_point('Kallithea', 'console_scripts', 'kallithea-api')())
File ".../bin/kallithea_api.py", line 84, in main
'apihost': args.apihost})
File ".../bin/base.py", line 104, in __init__
self.make_config(config)
File ".../bin/base.py", line 132, in make_config
ext_json.dump(config, f, indent=4)
File "/usr/lib/python3.7/json/__init__.py", line 180, in dump
fp.write(chunk)
TypeError: a bytes-like object is required, not 'str'

The json module documentation says:
https://docs.python.org/3.7/library/json.html#basic-usage
"The json module always produces str objects, not bytes objects. Therefore,
fp.write() must support str input."

Therefore, instead of opening the file in binary mode and writing bytes,
open it in text mode and write strings.

For symmetry reasons, we make the same change when _loading_ the config
file, but this code worked regardless.

py3: fix rc_types error

Spotted by pytypes:

File "kallithea/bin/base.py", line 133, in make_config: Function BinaryIO.write was called with the wrong arguments [wrong-arg-types]
Expected: (self, s: Union[bytearray, bytes, memoryview])
Actually passed: (self, s: str)

vcs: fix cloning remote repository with HTTP authentication (Issue #379)

Using a remote clone URI of
http://user:pass@host/...
triggered an exception:

...
E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri
E GitRepository._check_url(url)
E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url
E passmgr.add_password(*authinfo)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr>
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri
E host, port = splitport(authority)
E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport
E match = _portprog.fullmatch(host)
E TypeError: cannot use a string pattern on a bytes-like object


The authinfo tuple is obtained via mercurial.util.url, which unfortunately
returns a tuple of bytes whereas urllib expects strings.
It seems that mercurial internally has some more hacking around urllib as
urllibcompat.py, which we don't use.

Therefore, transform the bytes into strings before passing authinfo to
urllib. As the realm can be None, we need to check it specifically otherwise
safe_str would return a string 'None'.

A basic test that catches the mentioned problem is added, even though it
does not actually test that cloning with auth info will actually work (it
only tests that it fails cleanly if the URI is not reachable).

Additionally, one use of 'test_uri' in hg/repository.py still needed to be
transformed from bytes to string. For git this was already ok.

vcs: extract get_urllib_request_handlers out of {hg,git}/repository.py

Avoid some duplication between hg and git backends.
A subsequent commit will need bugfixes in this area.

docs: mention the need for worker process restarts to avoid "memory leaks" (Issue #346)

It might not be ideal to mention such details in the overview, but that seems
like the best place for generic advice.

(Kallithea should still aim for keeping memory growth under control.)

hg: fix crash on displaying bookmarks - add lost hex conversion when computing bookmark property

The crash was seen for example when using "Switch To" with bookmarks.

The problem was introduced with py3 conversion in e7dbe089e10d.

setup: exclude celery 4.4.4 which is broken due to unexpressed dependency

Celery 4.4.4 introduced the use of the 'future' package but forgot to
express it in its dependencies.

We could add the missing dependency on 'future' in Kallithea, but since the
problem is already fixed upstream shortly after 4.4.4 was released [1], we can
be sure that the next release (presumably 4.4.5) will contain the fix.

[1] https://github.com/celery/celery/pull/6146

cleanup: drop last traces of kallithea-config

The kallithea-config sources were removed in commit
213085032127e941a3bd93d0e510828a9d87bf32 but an entry point was still
created by setup.py.
Moreover, the ini file still referenced this, instead of kallithea-cli
(config-create).

html: fix invalid html / mako structure

Fix bad nesting of tags and mako conditional and drop superfluous '}' outside
the mako block.

tests: actually test something useful in test_edit for gists (Issue #376)

Even though there was a test for editing gists, it did not catch the basic
loading problem reported in issue #376. In fact, the test just loaded the
edit page, but since no user was actually logged in, it just loaded the
login screen. As a result, no real gist editing code was tested at all.

Instead, explicitly check the redirection to a login screen, then proceed
with logging in and check that the edit page can be loaded.

Additionally, don't rely on the magic gist id '1' but create an actual gist
first.

Note: the test does not yet really edit a gist.

ini: explain how % must be escaped as %% (Issue #375)

Gearbox / paste would fail with:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '('

The special handling of % is standard configparser behaviour, unless explicitly
fetching raw settings.

db: drop double specification of unique constraint (Issue #374)

Consistently use "unique=True" for single column constraints. Only use
UniqueConstraint for composite constraints.

MySQL did not like this double specification, while PostgreSQL and SQLite
did not care.

ini: put kallithea-config version in the ini file it creates (Issue #373)

Help keeping track of how the .ini was created.

docs: clarify the installation steps and how things fit together

Hint that create-db user information is for creating the initial local user
and that the database connection (which also might use some kind of user) is
something else.

tests: coverage of Git repo feeds

Coverage of the case reported in Issue #372.

feeds: fix failure getting feed for Git repos (Issue #372)

GitChangeset.diff() did
return ''.join(self.repository.get_diff(...))
even though get_diff returned a string. It worked, but was unnecessary and
inefficient.

That fails in py3: get_diff returns bytes ... and iterating doesn't give
characters but integers and we would get:
TypeError: sequence item 0: expected a bytes-like object, int found

Fixed by dropping the unnecessary iteration and joining.

This function is only used for feeds statistics.

auth: simplify handling of Crowd json response (Issue #370)

Correct error where Crowd authentication didn't work due to urllib.readlines()
returning bytes and thus failing to be joined with a string.

json.loads is however happy to take bytes directly.

Fix error handling to also handle bytes without crashing.

contributors: map different names of J. Lavoie onto one

Assuming the last used format is the right one.

i18n: updated translation for German

Currently translated at 68.9% (750 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.9% (1076 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.9% (1076 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.1% (1067 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.0% (1066 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 94.8% (1031 of 1087 strings)

i18n: updated translation for Luxembourgish

Currently translated at 6.1% (67 of 1087 strings)

issue: log error if still using issue_pat_XXX configuration after it was removed in d24051ce961c

Show a helpful:
found unsupported issue_prefix_pr = 'PR' - use issue_sub_pr instead
before bailing out with:
skipping incomplete issue pattern 'issue_pat_pr': '(?:PR\\s*)(\\d+)' -> 'https://kallithea-scm.org/repos/kallithea/pull-request/{id}' None

scripts/i18n: update i18n howto with recipe for use of scripts/i18n tooling

For now, just adding a new section. It should be integrated with the existing
documentation. Some steps can perhaps be simplified or automated so we need
less words.

inifile: add doctest coverage of comment corner cases for trailing space and value of None

Shows room for improvement - doesn't show any bugs.

The value of None is undefined behaviour.

tests: fix TestPermissions.test_inactive_user_group_does_not_affect_repo_group_permissions_inverse

Handle L1_new leak from test_rename_top_level_group_in_nested_setup .

tests: fix test_auth_ldap dependency on test order to avoid failure on uniqueness constraint

It would perhaps be even better to have each test clean up ... but let's only
aim for only avoiding collision on leaked test data.

tests: fix test_admin_settings test dependencies

Failure could be reproduced with
py.test kallithea/tests/functional/test_admin_settings.py -k x