js: introduce an html_escape function

In analogy to (python) kallithea.lib.helpers.html_escape, introduce the
equivalent for use in JavaScript.

i18n: updated translation for German

Currently translated at 60.6% (688 of 1136 strings)

i18n: updated translation for Ukrainian

Currently translated at 1.2% (14 of 1136 strings)

i18n: updated translation for Norwegian Bokmål

Currently translated at 12.5% (143 of 1136 strings)

i18n: updated translation for Ukrainian

Currently translated at 0.1% (1 of 1136 strings)

i18n: updated translation for Norwegian Bokmål

Currently translated at 12.5% (143 of 1136 strings)

lib: use bleach to sanitize HTML generated from markdown - fix XSS issue when repo front page shows README.md

Reported by Bob Hogg <wombat@rwhogg.site> .

hg: make protocol access control more explicit

Enumerate all currently known commands, and default to require 'push' access
for all unknown commands.

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

hg: explicit handling of the 'batch' protocol command - consider it a "push" command if any of the batch commands are

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

db: drop constraint that started failing with MariaDB 10.2 / MySQL 5.7 (Issue #324)

The constraint was to prevent simple recursive parent references, but it only
checked direct parents. We thus have to rely on the high level application
maintaining the invariant anyway.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

repos: introduce low level slug check of repo and group names

The high level web forms already slug-ify repo and repo group names. It might
thus not create the exact repo that was created, but the name will be "safe".

For API, we would rather have it fail than not doing exactly what was requested.

Thus, always verify at low level that the provided name wouldn't be modified by
slugification. This makes sure the API provide allow the same actual names as
the web UI.

This will only influence creation and renaming of repositories and repo groups.
Existing repositories will continue working as before.

This is a slight API change, but it makes the system more stable and can
prevent some security issues - especially XSS attacks.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

tests: clarify that fixture.create_repo_group takes a name, not a path

Don't pass a path on to the model layer - it can't and won't handle it correctly.

repos: only allow api repo creation in existing groups

Fix problem with '../something' paths being allowed; '..' will always exist and
can't be created.

This also introduce a small API change: Repository groups must now exist before
repositories can be created. This makes the API more explicit and simpler.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

repos: introduce low level check of clone URIs to prevent direct file system access to local repos

This is already checked in web form validation, but also check at low level to
make sure API access enforce the same invariants.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

tests: fix some incorrect test use of create_repo to create forks

This would create problems later when we start checking clone_uri.

repos: add missing access control check for repository permission management

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

i18n: updated translation for French

Currently translated at 100.0% (1136 of 1136 strings)

make-release: use a fresh virtualenv for building the release

A user-provided virtualenv could contain extra packages or different
versions, which may mask problems inside Kallithea.

make-release: remove instruction about updating copyright/contributors

The copyright script is now run by the script itself, so a warning to run it
is no longer necessary.

make-release: import version and copyright updates from default branch (dba4e770d4b6)

make-release has been improved on the default branch in commit dba4e770d4b6,
and that version of the script also almost works on stable.

There are some parts we don't have in stable branch: dev_requirements.txt,
run-all-cleanup, and a working dist file content check:

The MANIFEST.in file on the stable branch is not complete.
Most of the missing entries could simply be added
(.travis.yml, requirements.txt, tox.ini, and scripts/*) but there is one
problematic entry: kallithea/i18n/en/LC_MESSAGES/kallithea.mo.
In the check command, all .mo entries are filtered out, causing a remaining
diff for the English .mo.

As that file has been removed on the default branch already, and the missing
entries are not important for the release anyway, simply drop the check
for now.

vcs: Fix internal server error when trying to get diff from Mercurial for paths that include globbing patterns (Issue #308):

- Treat passed-in paths as exact matches, dissallowing any type of
globbing/regex/pattern matching.
- Added accompanying tests.

vcs: sanitize diff context values (Issue #306)

- Updated Git repository implementation to ensure context falls within
0 to 2**31-1 range (inclusive) when fetching a diff.
- Added tests for Git repositories for checking passed-in negative and
overflowing contexts (for the --unified option).
- Updated Mercurial repository implementation to ensure context is not
negative when fetching a diff.
- Added tests for Mercurial repositories for checking passed-in
negative context (for the --unified option).

setup: updated dependencies to accept latest Pylons release (1.0.3).

A couple of weeks ago the Pylons project has released version 1.0.3, removing
the previously stable version 1.0.2 in the process. In turn, this would mean
that new installation of Kallithea will end-up with older version of Pylons
than what is actually available, and in particular a much older release than
1.0.2.

setup-db: print completion message to avoid confusion (issue #303)

There are cases where the last message of setup-db is a warning, giving the
impression that the overall command failed which may not actually be the
case.

For example, when git is not installed, warnings are given, but they are not
an actual error. Kallithea will work fine for Mercurial repositories.

To avoid any confusion, print a completion message at the end.
Any real errors will abort the command and not make it this far.

login: fix crash when entering non-ASCII password for login (Issue #300)

Avoid errors like
UnicodeEncodeError: 'ascii' codec can't encode characters in position X: ordinal not in range(128)
when the user enters non-ASCII passwords for existing internal accounts in the
login prompt.

The password forms have "always" rejected non-ASCII passwords with
Invalid characters (non-ASCII) in password

Added tag 0.3.3 for changeset 64ea7ea09236
Added signature for changeset 64ea7ea09236

scripts: add update-copyrights.py

This script will automate how the current attribution and copyright listings
are generated, thus also enabling it to be maintained from repository history
in the future.

This doesn't change anything about who has or hasn't contributed or deserves
credit, it just does a best effort attempt at maintaining existing attribution
we inherited and automating it going forward. The automation might expose
existing problems - they should be fixed by adding or removing exceptions in
the script.

hg: remove _rcpath override

It is unclear what the purpose of this snippet was, and Mercurial 4.2 has no
_rcpath anymore.

AFAICS, since we use ui.ui() instead of ui.ui.load(), the config files are not
used anyway.

email: don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)

For example, on Linux, running `gearbox serve` with LANG=C, would crash in:

File ".../kallithea/lib/celerylib/tasks.py", line 307, in send_email
% (' '.join(recipients), headers, subject, body, html_body))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 224: ordinal not in range(128)

Replacing render with render_unicode works in this case ... but there might be
other problems elsewhere.

journal: make "repository:" filtering condition work as expected (Issue #261)

Before this revision, journal filtering conditions like as below never
match against any entry, even if there are corresponded repositories.

- repository:foo/bar
- repository:foo-bar

Whoosh library, which is used to parse filtering condition, does:

- treat almost all non-alphanumeric characters as delimiter at
parsing condition
- join each conditions at filtering by "AND", by default

For example, filtering condition "repository:foo/bar" is translated as
"repository:foo AND repository:bar". This combined condition never
matches against any entry, because it is impossible that "repository"
field in DBMS table "user_logs" has both "foo" and "bar" values at
same time.

Using TEXT for "repository" of JOURNAL_SCHEMA causes this issue,
because TEXT assumes tokenization at parsing.

In addition to it, using TEXT also causes unintentional ignorance of
"stop words" in filtering conditions. For example, "this", "a", "you",
and so on are ignored at parsing, because these are too generic words
(from point of view of generic "text search").

To make "repository:" filtering condition work as expected, this
revision uses ID instead of TEST for "repository" of
JOURNAL_COLUMN. ID avoids both tokenization and removing "stop words".

This replacement should be safe with already existing DBMS instance,
because:

- JOURNAL_SCHEMA is used only to parse filtering condition
- DBMS table "user_logs" itself is defined by UserLog class
(in kallithea/model/db.py)

BTW, using ID also avoids normalization by lowercase-ing. But this
doesn't violate current case-insensitive search policy, because
LOWER-ing in actual SQL query is achieved by get_filterion() or so in
kallithea/controllers/admin/admin.py.

admin: better handling of rescan/delete and invalidate at the same time (Issue #252)

It would try to invalidate deleted repositories before rescan would delete them.

Instead, scan first, then invalidate.

Also, handle VCS error from invalidation and report success as a flash message.

git: clarify that non-bare git repos not are supported (Issue #254)

Non-bare repos must have a branch checked out ... and then you can't push to
the branch.

repo-scan: rewrite get_filesystem_repos to use os.walk instead of stupid recursion

I think this is more readable. It is also faster. Perhaps because the more
readable implementation makes it easier to optimize.

email: add X-Kallithea-Notification-Type header

To simplify email filtering, add a header indicating the type of email being
sent. The 'type_' value corresponds to one of the types defined in class
Notification in kallithea/model/db.py, e.g. 'cs_comment', 'pull_request',
'pull_request_comment', ...

config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.

annotate: invoke hg annotate in a way that is forward compatible with Mercurial 3.9

https://selenic.com/hg/rev/576ff900fcc7 changed the behaviour when
linenumber=None.

linenumber=False has been tested to work with our oldest supported Mercurial
version.

git: include an LF at the end of the service advertisement (Fixes #230)

This fixes hg-git/Dulwich and possibly other conservative Git clients,
which do not ignore the absence of the LF.

The original comment was a guess based on reverse engineering the protocol
not specified in the documentation yet at that moment. Now that the
documentation exists and states this explicitly, just do as it says.

lock: fix for Mercurial 3.6+ - wrap hgweb to catch Locked exceptions from hooks

With Mercurial 3.6, the handling of WSGI responses changed. The hook exceptions
are no longer raised directly when app(environ, start_response) is called so
the 'except HTTPLockedRC as e' block in _handle_request (a few lines above )
does not work anymore because the exception happens later.

Therefore I created a wrapper class that can catch the exceptions.

This makes locking work again and fixes lock related tests like
TestVCSOperations.test_clone_after_repo_was_locked_hg which expect certain
output of the hg client in case of an HTTPLockedRC exception.

Depending on how https://bz.mercurial-scm.org/show_bug.cgi?id=5232 gets
handled, this fix might become obsolete in the future.

(Modified by Mads Kiilerich)

i18n: updated translation for Belarusian

Currently translated at 78.5% (889 of 1132 strings)

i18n: updated translation for Chinese (Traditional)

Currently translated at 15.1% (171 of 1132 strings)

i18n: updated translation for Dutch (Belgium)

Currently translated at 0.8% (10 of 1132 strings)

setup: don't use setuptools 34 - it has indirect conflicts with the celery version supported on the stable branch (Issue #266)

Setuptools==34 requires packaging>=16.8 which has an unconstrained requirement
of pyparsing ... but actually it doesn't work with pyparsing==1.5.7 ... which
is required by celery<2.3 ... which this version of Kallithea requires.

Celery has been upgraded on the development branch but we don't want to do that
on the stable branch.

i18n: updated translation for Russian

Currently translated at 73.4% (831 of 1132 strings)

middleware: replace references to Errormator with AppEnlight.

Errormator has been rebranded as AppEnlight a while back.

Errormator is no longer available. This is just a trivial rename that can't
make things worse.

docs: update links to Mercurial's website and wiki

Also Mercurial itself is not a coding standard, so add possessive suffix.

protocols: fix assertion error when accessing repositories with "permanent" urls (Issue #202)

I am not aware of any good way to test this, so it is tested with a Mercurial only hack.

api: fix forking of repos in repo groups (Issue #210)

Patch by Alexandre Beaulieu.

api: avoid duplicating group name when updating repo (Issue #37)

The api incorrectly passed repo.repo_name as repo_name, and the model update
function incorrectly always required repo_name.

Added tag 0.3.2 for changeset a84d40e9481f
Added signature for changeset a84d40e9481f

auth: further sanitize requests to prevent GET CSRF (CVE-2016-3691)

Routes allows GET requests to override the HTTP method, which breaks
the Kallithea CSRF protection (which only applies to POST requests).

This commit blocks such GET request, preventing CSRF attacks.

auth: properly invoke PermFunctions (CVE-2016-3114)

This fixes a vulnerability that allowed logged-in users to edit or
delete open pull requests associated with any repository to which
they had read access, plus a related vulnerability allowing logged-in
users to delete any comment from any repository, provided they could
determine the comment ID and had read access to just one repository.

changelog: fix range selection from changelog in git repos (Issue #190)

The Mercurial backend can handle short revision ids so the problem does not
occur with hg repositories. Git needs the full hash.

vcs: fix repo size calculation

tip.walk already 'dives into' the dirs - don't do that twice.

api: avoid sending double Content-Length header (Issue #201)

WSGI will take of Content-Length - handling it explicitly was misguided.

docs: improve mod_wsgi documentation (Issue #203)

Based on feedback from Graham Dumpleton.

* Recommend setting global Apache config WSGIRestrictEmbedded On
* Don't set the default value for processes=1 - it has side effects
* Use python-home instead of python-path
* Fix confusing mentioning of root and clarify the use of specifying user and group
* Include WSGIProcessGroup in configuration excerpt

docs: add notes about IIS, Windows Authentication and Mercurial

(The original patch from Konstantin has been heavily copyedited and modified by
Mads Kiilerich but is still [based on] Konstantin's feedback and contribution.)

Added tag 0.3.1 for changeset 9bf8eb837e78
Added signature for changeset 9bf8eb837e78

auth: support both old and new python-pam API

python-pam 1.8.* provides a new API replacing the previously
available authenticate() function. The latest unreleased
version brings it back, but until it's available, use a
custom shim instead.

users: fix crash when creating users with non ASCII characters

This was already changed by the cleaned up on the default branch in
330c671dd451 but happened to also fix this issue on the stable branch.