docs: upgrade: add section on updating git hooks

Commit 39bdf0ab2862 has changed the Git hooks. When upgrading Kallithea to
this commit, but not explicitly reinstalling the Git hooks, pushing to Git
repos will fail.

Add a section to the documentation to explain that a reinstall of the hooks
may be needed, and how to do it.

config: abort early if the environment doesn't allow Python to pass Unicode strings to the file system layer

Dulwich passes unicode paths to the file system layer and it will thus not work
if run with LC_ALL=C ... which is quite common for services.

Also try to detect different cases and give helpful error messages about what
environment variable could be changed to what.

notification: don't repeat common actions for each email recipient

The emails sent upon PR creation or comments are identical for all
recipients, except for the 'To:' header added by the send_email method.
Calculation of subject, body, etc. can thus be moved outside of the
loop.

Move the manipulation of the recipient list down to where it is used, for
clarity.

urls: allow canonical_url to contain more than just a hostname

Although the .ini file gives the example:

canonical_url = https://kallithea.example.com/repos

it does not actually work. The '/repos' part is stripped off by the
canonical_url method.

The 'host' entry in the arguments passed to routes.url does not strictly
need to be a pure hostname. At least, the implementation does no validation
of this fact, it is concatenated verbatim between the protocol and the rest
of the URL.

As mapping Kallithea to a subpath of a base hostname is a valid
implementation, the canonical_url feature should allow it.

tests: workaround for pytest 3.7.1 breaking autouse of classmethods

See https://github.com/pytest-dev/pytest/issues/3778 .

setup: bump all upper pip dependency versions to minor updates of what currently is available and testable on pypi

Based on manual editing after:
pip freeze | sed -n 's/==/ /gp' | while read p v; do sed -i -e "/\<$p[ \"]/s/\(\",\|$\)/, < $v\1/gi" setup.py dev_requirements.txt; done

These updates have been tested with automated tests and some amount of manual
testing. Remaining problems will hopefully be caught by additional testing
before the branch is declared stable.

setup: clarify that we require Routes less than 2.0

The next version after 1.13 is currently 2.0 and breaks test_files.py
test_file_annotation and test_file_annotation_git .

celery: clarify that celery 4 doesn't work

The latest supported Celery version, 3.1.26.post2, is the latest in the whole 3
series. It works fine. But celery-4.0.0 doesn't work at all when testing. For
now, just make it explicit that this is the limit.

ishell: specify ipython minimum version 4

`gearbox ishell` has an "optional" dependency on ipython as it fails with:
Kallithea ishell requires the IPython Python package
Before the previous change, after installing ipython < 4, it would however
still fail the same way.
With the previous change, it would fail in a slightly more helpful way with:
ImportError: No module named traitlets.config.loader

With ipython < 4, after installing the missing traitlets (4.3.2) as hinted,
ishell still failed with:
TraitError: The 'config' trait of an InteractiveShellEmbed instance must be a Config or None, but a value of class 'traitlets.config.loader.Config' (i.e. {'InteractiveShellEmbed': {'confirm_exit': False}}) was specified.

With ipython >= 4, traitlets is installed as dependency, and ishell works.
Tested with both ipython 4.0 and current latest version 5.8 .

Thus, just clarify that we only support ipython >= 4 .

ishell: only report 'Kallithea ishell requires the IPython Python package' when ipython import fails

Move everything but the actual ipython import outside the 'Kallithea ishell
requires the IPython Python package' "try".

auth: strip RFC4007 zone identifiers from IPv6 addresses before doing access control

If using IPv6, the request IP address might contain a '%' that the ipaddr
module that is used for IP filtering can't handle.

https://tools.ietf.org/html/rfc4007#section-11 specifies how IPv6 addresses can
have zone identifiers like trailing '%13' or '%eth0'. The zone identifier is
used to help distinguish *if* the same address should be available on multiple
interfaces. It *could* potentially have security implications in the odd case
where the same address is different on different interfaces. The IP whitelist
functionality does however not support zone filters, so there is no way users
can expect the zone to be relevant for IP filtering. We can thus safely strip
the zone index and only check for match on the other parts of the address.

setup: sort gearbox.commands

There did not seem to be any particular order, so alphabetical sorting seems
most appropriate.

paster_commands: remove unused imports

Remove some unused imports, using 'autoflake' followed by manual redacting.
find kallithea/lib/paster_commands/ -name "*.py" \
| xargs autoflake --remove-all-unused-imports -i

This same command could be applied gradually throughout the code base as
areas are touched. Since autoflake may be too greedy in special cases, it is
more difficult to do a big-bang action over all code.

git: fix push to empty repo (Issue 323)

Git would fail to log revisions when the list of heads to exclude included an
empty string (in place of the pushed ref).

To avoid that, skip the skipped revision instead of making it an empty string.
`git log --not` works fine without providing any revisions to "not".

Verify in test_push_new_repo_git that it actually logged the push.

hooks: slight cleanup of handling of scm_repo.run_git_command return values

It would still be nice to have better error handling, but this makes it
slightly more clear what is going on and what error handling we don't have..

middleware: clean-up of Git _git_stored_op

This seems to do the same in a simpler way.

It is unclear what _git_first_op did. But no matter if it is a typo or
something else, it is unused.

cache: move cache invalidation from web handler to post push hook

We need the post push hook anyway ... and having the cache invalidation here
will also work for ssh pushes in the future.

The name log_push_action is thus no longer spot-on. That might change later,
but requires some care as it also is used directly as hook name.

Note that having cache invalidation in the hook will do that debug logging no
longer will appear in the server log.

Based on a patch by Dominik Ruf.

middleware: inline the _set_extras invocation from the __inject_extras method, leaving behind only functionality for augmenting ui from hgrc

I don't know about the logic for merging hg configs ... but for now it is left untouched.

hooks: clarify return value from Git hooks

don't rely on the implicit None

hooks: use os.environ directly for KALLITHEA_EXTRAS instead of sometimes passing it as parameter

It is better to be consistent ... even if that means always using the global
environment that is hard to avoid.

hooks: empower the Git entry points and avoid the big handle_git_receive dispatch

Use the new _hook_environment and enjoy the small and clean hooks.

Just moving code - no change.

hooks: extract a separate _hook_environment from handle_git_receive

We want to split handle_git_receive up, and this self-contained chunk is what
will be shared.

Just moving code - not changing comments or anything.

git: introduce test_push_new_repo_git to reproduce a hook crash when pushing to empty repos

The crash will be fixed by a later changeset.

Reported on https://bitbucket.org/conservancy/kallithea/issues/323/git-hook-error-on-push-of-first-commit

remote: Traceback (most recent call last):
remote: File "hooks/post-receive", line 36, in <module>
remote: main()
remote: File "hooks/post-receive", line 32, in main
remote: sys.exit(kallithea.lib.hooks.handle_git_post_receive(repo_path, git_stdin_lines))
remote: File "kallithea/lib/hooks.py", line 453, in handle_git_post_receive
remote: git_revs += scm_repo.run_git_command(cmd)[0].splitlines()
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 164, in run_git_command
remote: return self._run_git_command(cmd, **opts)
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 151, in _run_git_command
remote: raise RepositoryError(tb_err)
remote: kallithea.lib.vcs.exceptions.RepositoryError: Couldn't run git command (['git', '-c', 'core.quotepath=false', 'log', 'b991c8d9ae7e66e165fc5eeb297c6843d21915e0', '--reverse', '--pretty=format:%H', '--not', '']).
remote: Original error was:Subprocess exited due to an error:
remote: fatal: ambiguous argument '': unknown revision or path not in the working tree.
remote:
To http://127.0.0.1:44433/new_git_DHAsQQ
* [new branch] master -> master

hg: fix 4.7 compatibility in checkout_branch

Test coverage by workdir tests.

setup: normalize casing of dependencies and formatting of constraints

The package name casing might seem random, but it is more correct than our own random casing.

These changes mainly come from:
pip freeze | sed -n '/==/s/==.*//gp' | while read a; do sed -i "s/$a/$a/i" setup.py dev_requirements.txt ; done
sed -i -e 's/>=/ >= /' -e 's/ >=/ >=/' -e 's/>= />= /g' -e 's/,</, </' -e 's/</< /' -e 's/< /< /g' -e 's/< =/<= /g' setup.py dev_requirements.txt

hg: be more strict about which kinds of revision lookup we want from hg

Comply with
https://www.mercurial-scm.org/repo/hg/file/4.6/mercurial/context.py#l380
to support recent Mercurial versions.

hg: don't use custom diff arguments when generating diffstats for rss feed

With Mercurial 4.7, it is no longer possible to pass arguments like `git=True`
to `ctx.diff()`.

This will change the reported data slightly, but the difference isn't
important. Also, the code should be refactored to do something similar to what
we do in other places.

hg: bump minimum version to 4.1.1

522cfb2be9e1 introduced test_custom_hooks_preoutgoing, but thorough testing now
reveals that the test failed with Mercurial 4.0 to 4.1.0 . Instead of
investigating and fixing code or test, just accept this more than one year old
version as minimum.

ishell: use traitlets.config and require ipython >= 4.0.0

Since version 4 was released in 2015, ishell has warned:
IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated. You should import from traitlets.config instead.
"You should import from traitlets.config instead.", ShimWarning)

docs: move authentication info to separate file

Goal is to reduce the gigantic 'setup' page in size, and make the
information more understandable. Currently, the different topics do not
really belong together.
This is one small step towards that goal.

docs: vcs_setup: add note about Git HTTP POST buffer

As suggested by https://bitbucket.org/conservancy/kallithea/issues/229/bad-file-descriptor#comment-30073565

docs: split vcs_support into admin/vcs_setup and usage/vcs_notes

The existing page on VCS support was a mix of information needed to setup
Kallithea with respect to version control systems, with information
regarding using version control systems (or specific aspects of it) with
Kallithea.

Move the first part to the Administrator Guide, and rebrand the second part
as VCS Usage Notes.

In vcs_notes.rst, the general info is moved above the Mercurial-specific
part, but otherwise left untouched.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

db: drop constraint that started failing with MariaDB 10.2 / MySQL 5.7 (Issue #324)

The constraint was to prevent simple recursive parent references, but it only
checked direct parents. We thus have to rely on the high level application
maintaining the invariant anyway.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

docs: reorganize index page

Reorganize the index page of the manual into three main sections:
administrator guide, user guide, developer guide. In principle,
administrators, users and developers are independent roles that can be
fulfilled by different people.

'Administrators' install, configure and maintain Kallithea. They have
'admin' privilege in the web interface.

'Users' do not have access to the Kallithea installation files. They can
only access the web interface, and generally do not have 'admin' privilege.
They may be the owner of repositories, repository groups, or user groups,
and as such modify the settings of these objects, even though they cannot
modify the settings of Kallithea itself.

'Developers' modify the Kallithea source code. They may or may not
contribute these changes back to the Kallithea community.



The 'Readme' section is a bit outside of this classification. As it serves
as a basic introduction to Kallithea, it is kept first.

The 'Index' and 'Search' functionality are not really 'part of' the manual,
so should not be under 'Other topics'. Instead they are put at the very top
so they are readily visible.


This commit is only changing the index page. Subsequent commits can move and
split some documentation files in the repository to mirror this structure.
For example, the 'Version control systems support' page has information for
administrators (ini settings and importing existing repositories) as well as
for users (handling Mercurial subrepositories).

templates: remove references to non-existing commit-message class

As there already is monospace styling on textarea.form-control, there is no
need for a replacement.

style: use monospace for changeset messages

Similar to commits 9005c37dbe21 and 6e65c2153baa that make PR descriptions
display in a monospace font, apply monospace on changeset messages.

Note that in the files template, the reference to the non-existing 'commit'
class can be replaced instead of appending to it.

utils: move repo_name_slug to utils2 to prevent import cycle on setup_db

After commit 57a733313e4f, 'gearbox setup-db -c my.ini' fails with an import
cycle as follows:

Traceback (most recent call last):
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/bin/gearbox", line 11, in <module>
sys.exit(main())
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 199, in main
return gearbox.run(args)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 145, in run
return self._run_subcommand(remainder)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 149, in _run_subcommand
subcommand = self.command_manager.find_command(argv)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commandmanager.py", line 78, in find_command
cmd_factory = cmd_ep.resolve()
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/paster_commands/setup_db.py", line 27, in <module>
from kallithea.lib.db_manage import DbManage
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/db_manage.py", line 47, in <module>
from kallithea.model.repo_group import RepoGroupModel
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/model/repo_group.py", line 35, in <module>
import kallithea.lib.utils
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 48, in <module>
from kallithea.model.repo_group import RepoGroupModel
ImportError: cannot import name RepoGroupModel


i.e. kallithea.model.repo_group wants to import kallithea.lib.utils which
in turn wants to import kallithea.model.repo_group.

In fact there exists kallithea.lib.utils and kallithea.lib.utils2.
The current split is that 'utils2' contains 'simple' utilities, none of which
depend on kallithea models, controllers, ... In contrast, 'utils' does rely
on such kallithea classes.

As kallithea.model.repo_group was only include kallithea.lib.utils for its
repo_name_slug method, which has no dependency on other kallithea classes,
move that method (and its dependent recursive_replace) to
kallithea.lib.utils2 instead. This fixes the import cycle.

hg: set encoding to utf-8 by default to always show unicode characters correctly

Unicode characters would be shown as '?' if Kallithea was launched in a LANG=C
environment (or similar).

The problem could be solved by setting HGENCODING before launching Kallithea or
before importing Mercurial. These are often not good solutions.

Instead, introduce a hgencoding config setting that triggers monkey patching of
Mercurial.

pullrequests: don't show empty "additional changesets" (issue #280)

When opening a pullrequest on a revision range including the tipmost
revision, and then pushing a new revision on top of that, the PR page shows:
This pullrequest can be updated with changes on ...:
and then nothing.

This is because the available revisions looped over are in 'avail_cs' but
the guard around the loop is checking on 'avail_revs'. The former, while
derived from avail_revs, can become empty under circumstances like this one.

Fix the problem by changing the guard checking avail_cs rather than
avail_revs, and making sure the printed message is aligned to 'No additional
changes found'.

repos: make edit_permissions_revoke controller return an actual (empty) body on success

The code 204 was slightly confusing even though technically OK.

changelog: tweak positioning of checkboxes for changeset selection

Avoid the default vertical alignment on baseline. Instead align on text bottom
- that makes it show up more nicely aligned with the text.

Also override the Bootstrap top margin - it made the checkbox the largest
element in rows and it was thus controlling the height for no good reason.

repos: introduce low level slug check of repo and group names

The high level web forms already slug-ify repo and repo group names. It might
thus not create the exact repo that was created, but the name will be "safe".

For API, we would rather have it fail than not doing exactly what was requested.

Thus, always verify at low level that the provided name wouldn't be modified by
slugification. This makes sure the API provide allow the same actual names as
the web UI.

This will only influence creation and renaming of repositories and repo groups.
Existing repositories will continue working as before.

This is a slight API change, but it makes the system more stable and can
prevent some security issues - especially XSS attacks.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

tests: clarify that fixture.create_repo_group takes a name, not a path

Don't pass a path on to the model layer - it can't and won't handle it correctly.

repos: only allow api repo creation in existing groups

Fix problem with '../something' paths being allowed; '..' will always exist and
can't be created.

This also introduce a small API change: Repository groups must now exist before
repositories can be created. This makes the API more explicit and simpler.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

repos: introduce low level check of clone URIs to prevent direct file system access to local repos

This is already checked in web form validation, but also check at low level to
make sure API access enforce the same invariants.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

tests: fix some incorrect test use of create_repo to create forks

This would create problems later when we start checking clone_uri.

repos: add missing access control check for repository permission management

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

repos: introduce low level slug check of repo and group names

The high level web forms already slug-ify repo and repo group names. It might
thus not create the exact repo that was created, but the name will be "safe".

For API, we would rather have it fail than not doing exactly what was requested.

Thus, always verify at low level that the provided name wouldn't be modified by
slugification. This makes sure the API provide allow the same actual names as
the web UI.

This will only influence creation and renaming of repositories and repo groups.
Existing repositories will continue working as before.

This is a slight API change, but it makes the system more stable and can
prevent some security issues - especially XSS attacks.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

tests: clarify that fixture.create_repo_group takes a name, not a path

Don't pass a path on to the model layer - it can't and won't handle it correctly.

repos: only allow api repo creation in existing groups

Fix problem with '../something' paths being allowed; '..' will always exist and
can't be created.

This also introduce a small API change: Repository groups must now exist before
repositories can be created. This makes the API more explicit and simpler.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

repos: introduce low level check of clone URIs to prevent direct file system access to local repos

This is already checked in web form validation, but also check at low level to
make sure API access enforce the same invariants.

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

tests: fix some incorrect test use of create_repo to create forks

This would create problems later when we start checking clone_uri.

repos: add missing access control check for repository permission management

This issue was found and reported by
Kacper Szurek
https://security.szurek.pl/

admin: hooks: shorten input field of custom hooks to make place for delete button

The delete button is currently on another line than the custom hook name and
value. Reduce the size of the value input field to fix that.

admin: hooks: handle overflow of long hook names

Some hook names are quite long and are visually truncated in the admin
section. Use the same principle as on the System Info: hide overflow and add
ellipsis, use a title attribute to let users discover the full name on
hover.

admin: system info: add title attribute for truncated items

On the System Info page, the item names may be truncated, e.g. "Kallithea
configur...". In these cases, it is not possible for a user to retrieve the
full name.

Add a title attribute that will show the full name on hover.

admin: hooks: move informative span to 'custom hooks'

Explaining what hooks are is mostly useful on the 'custom hooks' section, as
the built-in hooks are read-only.

admin: hooks: only flash 'Updated hooks' if there are changes

Clicking 'Save' on the hook administration page currently always renders the
flash message 'Updated hooks' even if nothing was changed.
This may be particularly confusing when the action you intended to do got an
error, e.g. adding a hook that already exists, adding a builtin hook, ...

Instead, compare the old and new value when editing a hook, and only save
and create the flash if they are different.

For this to be work correctly in test, the old value needs to be passed as
well like in the real situation, otherwise the 'zip' operation will return
an empty list.

admin: hooks: prevent editing of builtin hooks (issue #226)

Builtin hooks are supposed to be read-only, but it was still possible to
'add' a new hook with the same name as an existing built-in one, changing
its value.

admin: hooks: prevent creation of existing custom hook

Trying to add a hook that already exists does not currently give an error
but does not work.

Detect the situation and report via a flash.

admin: hooks: restore delete functionality as intended

Commit 9d34bea3059d9abd0d912f37a2475ee67c8e2918 ("style: various minor-ish
markup changes, preparing for Bootstrap") partially broke the delete
functionality of hooks. When clicking the delete button, the hook is deleted
via AJAX and then the corresponding form group is intended to be removed
visually. This relies on an 'id' attribute on the form-group div.

The mentioned commit moved the div with the 'id' attribute outside the loop
iterating over the different hooks, so that there no longer is a div with
the id expected by the delete button. The hook would still be deleted, but
the page visually still looks the same until refresh/Save.

Move the diff back into the loop. This causes a little more visual
separation between the different hooks, but still looks OK.
The layout of the built-in hooks (that can't be removed) is left untouched.
Making the same change there would be possible but there the extra vertical
whitespace seems somewhat unnecessary, although this is personal.

admin: hooks: check session flashes in tests

Add checking of the session flash when a new hook is added.

auth: raise log level of 'permission denied' from DEBUG to INFO (issue #243)

Denying an access can be quite relevant to an administrator.
A log level of INFO is more reasonable than DEBUG.

tests: fix caching issue in test_ip_restriction_git

Following test failure is observed in
TestVCSOperations.test_ip_restriction_git:

―――――――――――――― TestVCSOperations.test_ip_restriction_git ――――――――――――――――――
kallithea/tests/other/test_vcs_operations.py:584: in test_ip_restriction_git
assert re.search(r'\b403\b', stderr)
E assert None
E + where None = <function search at 0x7fb9772da578>('\\b403\\b', "Cloning into '/tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ'...\n")
E + where <function search at 0x7fb9772da578> = re.search
------------------------- Captured stdout call ----------------------------
*** CMD git clone http://test_admin:test12@127.0.0.1:45291/vcs_test_git /tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ ***
stderr: "Cloning into '/tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ'...\n"



The test is setting up IP restrictions, verifying that access is no longer
possible, then clears the restriction. There already were sleeps after
clearing the restrictions, in order for the cache to expire and have the
setting take effect.
But there was no sleep on the enabling of the IP restriction, allowing
situations where the code would still run without restriction, and thus
allow the access, failing the test.

The failure has only been observed on test_ip_restriction_git, but the
change is also made for test_ip_restriction_hg.

The existing sleeps after restriction clearing are moved up to the 'finally'
clause to make it clear to which code they belong.

setup: support Mercurial 4.6

A number of issues were found but have been fixed.

hg: support more strict branch name checking in Mercurial 4.6 - be more careful about not passing unicode branch names

TestHgApi.test_api_get_changesets_with_branch would fail.

hg: support introduction of wsgiresponse in Mercurial 4.6

Lock tests would fail without this.

hg: support rename of obsolete related attributes in Mercurial 4.6

The internals and the UI should perhaps also change at some point.

hg: use localrepo indexing for revision number lookup instead of the lookup function

Mercurial 4.6 (0a1fb171dc1d) changed lookup and would fail if given a revision number.