controllers: forward pullrequests.delete_comment to changeset

Remove duplication between pullrequests and changeset.
We move the code outside ChangesetController to make it callable from
PullrequestsController.

Note:
- instead of keeping the method pullrequests.delete_comment itself and
letting it forward to changeset.delete_comment, an alternative solution
would have been to change the routing table directly. However, the chosen
solution makes it more explicit which operations are supported on each
controller.

controllers: align pullrequests.delete_comment with changeset.delete_comment

This commit purely serves to highlight the differences.
The subsequent commit will remove the duplication.

tests: test status change in comments and pull requests

Status changes when commenting was not yet covered.
This commits adds the basic logic, but is not full coverage.

tests: rename test_changeset_comments to test_changeset_pullrequests_comments

As the scope of this file now is both changeset and pullrequest commenting,
the filename should change as well.

tests: add tests for commenting on pullrequests

Starting from the existing tests for commenting on a changeset, adapt them
to test the commenting feature on pullrequests.

tests: small improvements to test_changeset_comments

- response checking is incorrectly placed under 'Test DB' comment
- no response check on the comment text was done
- related: use different text for the comments in different tests
- commit_id should be comment_id
- remove duplicate checking of comment count in DB

utils2: Move ask_ok out of paster_commands

The ask_ok function is still a mess and should be cleaned up ...

cli: convert 'gearbox celeryd' into 'kallithea-cli celery-run'

Note:
- '--' is never explicitly present in the arguments when using Click.
The click parser will take care of '--' as separator between
dash-dash-arguments and positional arguments, following standard UNIX
conventions.

cli: convert 'gearbox make-rcext' in 'kallithea-cli extensions-create'

Note: 'extensions' instead of 'rcextensions' as first step to get away from
the 'rc' prefix.

cli: convert 'gearbox cleanup-repos' into 'kallithea-cli repo-purge-deleted'

Some changes to user-facing text are done.

cli: convert 'gearbox update-repoinfo' into 'kallithea-cli repo-update-metadata'

The --update-only option is replaced by listing repository names as separate command line arguments.

The --invalidate-cache option is dropped and we just always invalidate all caches.

cli: convert 'gearbox make-config' into 'kallithea-cli config-create'

... and update documentation.

cli: initial introduction of 'kallithea-cli' command

This commit adds a command 'kallithea-cli' that intends to replace the
existing set of 'gearbox' commands that relate to setting up kallithea.
Gearbox would still be used for 'gearbox serve', but other commands like
'make-config', 'setup-db', etc. would be converted to 'kallithea-cli'
commands.

The python package 'Click' is used to generate the CLI. Using decorators,
this package makes it very easy to generate a CLI out of simple methods.
See: http://click.pocoo.org/7/

Using Gearbox for custom commands is possible, but documentation on this
topic is limited. As the added value of Gearbox for that use case is not
clear, we can well switch to something else if it seems better/easier.

lib: use bleach to sanitize HTML generated from markdown - fix XSS issue when repo front page shows README.md

Reported by Bob Hogg <wombat@rwhogg.site> .

hg: make protocol access control more explicit

Enumerate all currently known commands, and default to require 'push' access
for all unknown commands.

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

hg: explicit handling of the 'batch' protocol command - consider it a "push" command if any of the batch commands are

This change mitigates some privilege escalation problems like CVE-2018-1000132
which was fixed in Mercurial 4.5.1 and currently is described on
https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.5.1_.2F_4.5.2_.282018-03-06.29 .

graph: don't try to get obsolete status for fake parent revisions

Fake parents use negative revision numbers as internal placeholders.
Previous Mercurial versions could handle negative non-existing revision
numbers, but Mercurial 4.8 will fail on this invalid input when getting the
obsolete status.
Instead, make sure we only get obsolete status for actual revisions.

hg: use localrepo.instance instead of invoking localrepository directly

The instance method has been available for very long time and does almost the
same thing.

Also, for Mercurial 4.8, the create functionality is only available on the
instance factory.

docs: update i18n doc after TG migration changed lang to i18n.lang and test.ini is generated

These descriptions are not elegant, but now we make them less wrong.

scripts/manifest: drop unused file

Usage of this file has seized since commit 87496864d4c6.

tests: fix vcs Hg helper function get_config_value to not modify global state when using a custom config file

This leak was what made test_get_user_name and test_get_user_email pass when
test_get_config_value had been run first.

tests: fix vcs Hg helper functions get_user_name and get_user_email to actually use provided config_file

Running just test_get_user_name and test_get_user_email would fail when run as:
py.test kallithea/tests/vcs/test_repository.py -k test_get_user

The config file was not used at all, and the tests would only pass when
test_get_config_value had been run first and as an unfortunate side-effect had
read the same file.

style: vertically align comment bubble icon

The icon of the comment bubble is slightly off-center vertically (too low).
This problem can be solved by reducing the line height instead of using the
default of 20px.

style: remove vertical scrollbar on hover of last line's comment bubble

When hovering the comment bubble of the last line of a diff, a vertical
scrollbar appears. This behavior is not present on the stable branch.

Attempts to fix the problem by changing element sizes or padding did not
solve the problem. As an alternative, this commit hides overflow - this makes
no visual difference.

model/scm: fix incorrect reporting of pull from remote git repo

In issue #327 it was reported that the pull of a remote git repo causes the
following inaccurate journal entry:
"committed via Kallithea into repository"

The problem is caused by the following flow:

1. pull_changes calls _handle_push for git repos, passing
action='push_remote'.

2. _handle_push calls _handle_rc_scm_extras, but does not pass 'action'

3. _handle_rc_scm_extras uses a default value of 'push_local' for action.

4. The action string 'push_local' will result in the mentioned journal
entry, via action_map in action_parser() (kallithea/lib/helpers.py).


To get a more accurate log entry, _handle_push should pass the action that
it got from pull_changes. In that case, the log entry will become
"pulled from remote into repository"

make-config: reorganize code to make it clear that --show-defaults can't be combined with custom key=value

The key=value arguments could only be specified after specifying an ini file,
and that couldn't be combined with --show-defaults.

docs: upgrade: add section on updating git hooks

Commit 39bdf0ab2862 has changed the Git hooks. When upgrading Kallithea to
this commit, but not explicitly reinstalling the Git hooks, pushing to Git
repos will fail.

Add a section to the documentation to explain that a reinstall of the hooks
may be needed, and how to do it.

config: abort early if the environment doesn't allow Python to pass Unicode strings to the file system layer

Dulwich passes unicode paths to the file system layer and it will thus not work
if run with LC_ALL=C ... which is quite common for services.

Also try to detect different cases and give helpful error messages about what
environment variable could be changed to what.

notification: don't repeat common actions for each email recipient

The emails sent upon PR creation or comments are identical for all
recipients, except for the 'To:' header added by the send_email method.
Calculation of subject, body, etc. can thus be moved outside of the
loop.

Move the manipulation of the recipient list down to where it is used, for
clarity.

urls: allow canonical_url to contain more than just a hostname

Although the .ini file gives the example:

canonical_url = https://kallithea.example.com/repos

it does not actually work. The '/repos' part is stripped off by the
canonical_url method.

The 'host' entry in the arguments passed to routes.url does not strictly
need to be a pure hostname. At least, the implementation does no validation
of this fact, it is concatenated verbatim between the protocol and the rest
of the URL.

As mapping Kallithea to a subpath of a base hostname is a valid
implementation, the canonical_url feature should allow it.

tests: workaround for pytest 3.7.1 breaking autouse of classmethods

See https://github.com/pytest-dev/pytest/issues/3778 .

setup: bump all upper pip dependency versions to minor updates of what currently is available and testable on pypi

Based on manual editing after:
pip freeze | sed -n 's/==/ /gp' | while read p v; do sed -i -e "/\<$p[ \"]/s/\(\",\|$\)/, < $v\1/gi" setup.py dev_requirements.txt; done

These updates have been tested with automated tests and some amount of manual
testing. Remaining problems will hopefully be caught by additional testing
before the branch is declared stable.

setup: clarify that we require Routes less than 2.0

The next version after 1.13 is currently 2.0 and breaks test_files.py
test_file_annotation and test_file_annotation_git .

celery: clarify that celery 4 doesn't work

The latest supported Celery version, 3.1.26.post2, is the latest in the whole 3
series. It works fine. But celery-4.0.0 doesn't work at all when testing. For
now, just make it explicit that this is the limit.

ishell: specify ipython minimum version 4

`gearbox ishell` has an "optional" dependency on ipython as it fails with:
Kallithea ishell requires the IPython Python package
Before the previous change, after installing ipython < 4, it would however
still fail the same way.
With the previous change, it would fail in a slightly more helpful way with:
ImportError: No module named traitlets.config.loader

With ipython < 4, after installing the missing traitlets (4.3.2) as hinted,
ishell still failed with:
TraitError: The 'config' trait of an InteractiveShellEmbed instance must be a Config or None, but a value of class 'traitlets.config.loader.Config' (i.e. {'InteractiveShellEmbed': {'confirm_exit': False}}) was specified.

With ipython >= 4, traitlets is installed as dependency, and ishell works.
Tested with both ipython 4.0 and current latest version 5.8 .

Thus, just clarify that we only support ipython >= 4 .

ishell: only report 'Kallithea ishell requires the IPython Python package' when ipython import fails

Move everything but the actual ipython import outside the 'Kallithea ishell
requires the IPython Python package' "try".

auth: strip RFC4007 zone identifiers from IPv6 addresses before doing access control

If using IPv6, the request IP address might contain a '%' that the ipaddr
module that is used for IP filtering can't handle.

https://tools.ietf.org/html/rfc4007#section-11 specifies how IPv6 addresses can
have zone identifiers like trailing '%13' or '%eth0'. The zone identifier is
used to help distinguish *if* the same address should be available on multiple
interfaces. It *could* potentially have security implications in the odd case
where the same address is different on different interfaces. The IP whitelist
functionality does however not support zone filters, so there is no way users
can expect the zone to be relevant for IP filtering. We can thus safely strip
the zone index and only check for match on the other parts of the address.

setup: sort gearbox.commands

There did not seem to be any particular order, so alphabetical sorting seems
most appropriate.

paster_commands: remove unused imports

Remove some unused imports, using 'autoflake' followed by manual redacting.
find kallithea/lib/paster_commands/ -name "*.py" \
| xargs autoflake --remove-all-unused-imports -i

This same command could be applied gradually throughout the code base as
areas are touched. Since autoflake may be too greedy in special cases, it is
more difficult to do a big-bang action over all code.

git: fix push to empty repo (Issue 323)

Git would fail to log revisions when the list of heads to exclude included an
empty string (in place of the pushed ref).

To avoid that, skip the skipped revision instead of making it an empty string.
`git log --not` works fine without providing any revisions to "not".

Verify in test_push_new_repo_git that it actually logged the push.

hooks: slight cleanup of handling of scm_repo.run_git_command return values

It would still be nice to have better error handling, but this makes it
slightly more clear what is going on and what error handling we don't have..

middleware: clean-up of Git _git_stored_op

This seems to do the same in a simpler way.

It is unclear what _git_first_op did. But no matter if it is a typo or
something else, it is unused.

cache: move cache invalidation from web handler to post push hook

We need the post push hook anyway ... and having the cache invalidation here
will also work for ssh pushes in the future.

The name log_push_action is thus no longer spot-on. That might change later,
but requires some care as it also is used directly as hook name.

Note that having cache invalidation in the hook will do that debug logging no
longer will appear in the server log.

Based on a patch by Dominik Ruf.

middleware: inline the _set_extras invocation from the __inject_extras method, leaving behind only functionality for augmenting ui from hgrc

I don't know about the logic for merging hg configs ... but for now it is left untouched.

hooks: clarify return value from Git hooks

don't rely on the implicit None

hooks: use os.environ directly for KALLITHEA_EXTRAS instead of sometimes passing it as parameter

It is better to be consistent ... even if that means always using the global
environment that is hard to avoid.

hooks: empower the Git entry points and avoid the big handle_git_receive dispatch

Use the new _hook_environment and enjoy the small and clean hooks.

Just moving code - no change.

hooks: extract a separate _hook_environment from handle_git_receive

We want to split handle_git_receive up, and this self-contained chunk is what
will be shared.

Just moving code - not changing comments or anything.

git: introduce test_push_new_repo_git to reproduce a hook crash when pushing to empty repos

The crash will be fixed by a later changeset.

Reported on https://bitbucket.org/conservancy/kallithea/issues/323/git-hook-error-on-push-of-first-commit

remote: Traceback (most recent call last):
remote: File "hooks/post-receive", line 36, in <module>
remote: main()
remote: File "hooks/post-receive", line 32, in main
remote: sys.exit(kallithea.lib.hooks.handle_git_post_receive(repo_path, git_stdin_lines))
remote: File "kallithea/lib/hooks.py", line 453, in handle_git_post_receive
remote: git_revs += scm_repo.run_git_command(cmd)[0].splitlines()
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 164, in run_git_command
remote: return self._run_git_command(cmd, **opts)
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 151, in _run_git_command
remote: raise RepositoryError(tb_err)
remote: kallithea.lib.vcs.exceptions.RepositoryError: Couldn't run git command (['git', '-c', 'core.quotepath=false', 'log', 'b991c8d9ae7e66e165fc5eeb297c6843d21915e0', '--reverse', '--pretty=format:%H', '--not', '']).
remote: Original error was:Subprocess exited due to an error:
remote: fatal: ambiguous argument '': unknown revision or path not in the working tree.
remote:
To http://127.0.0.1:44433/new_git_DHAsQQ
* [new branch] master -> master

hg: fix 4.7 compatibility in checkout_branch

Test coverage by workdir tests.

setup: normalize casing of dependencies and formatting of constraints

The package name casing might seem random, but it is more correct than our own random casing.

These changes mainly come from:
pip freeze | sed -n '/==/s/==.*//gp' | while read a; do sed -i "s/$a/$a/i" setup.py dev_requirements.txt ; done
sed -i -e 's/>=/ >= /' -e 's/ >=/ >=/' -e 's/>= />= /g' -e 's/,</, </' -e 's/</< /' -e 's/< /< /g' -e 's/< =/<= /g' setup.py dev_requirements.txt

hg: be more strict about which kinds of revision lookup we want from hg

Comply with
https://www.mercurial-scm.org/repo/hg/file/4.6/mercurial/context.py#l380
to support recent Mercurial versions.

hg: don't use custom diff arguments when generating diffstats for rss feed

With Mercurial 4.7, it is no longer possible to pass arguments like `git=True`
to `ctx.diff()`.

This will change the reported data slightly, but the difference isn't
important. Also, the code should be refactored to do something similar to what
we do in other places.

hg: bump minimum version to 4.1.1

522cfb2be9e1 introduced test_custom_hooks_preoutgoing, but thorough testing now
reveals that the test failed with Mercurial 4.0 to 4.1.0 . Instead of
investigating and fixing code or test, just accept this more than one year old
version as minimum.

ishell: use traitlets.config and require ipython >= 4.0.0

Since version 4 was released in 2015, ishell has warned:
IPython/config.py:13: ShimWarning: The `IPython.config` package has been deprecated. You should import from traitlets.config instead.
"You should import from traitlets.config instead.", ShimWarning)

docs: move authentication info to separate file

Goal is to reduce the gigantic 'setup' page in size, and make the
information more understandable. Currently, the different topics do not
really belong together.
This is one small step towards that goal.

docs: vcs_setup: add note about Git HTTP POST buffer

As suggested by https://bitbucket.org/conservancy/kallithea/issues/229/bad-file-descriptor#comment-30073565

docs: split vcs_support into admin/vcs_setup and usage/vcs_notes

The existing page on VCS support was a mix of information needed to setup
Kallithea with respect to version control systems, with information
regarding using version control systems (or specific aspects of it) with
Kallithea.

Move the first part to the Administrator Guide, and rebrand the second part
as VCS Usage Notes.

In vcs_notes.rst, the general info is moved above the Mercurial-specific
part, but otherwise left untouched.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

db: drop constraint that started failing with MariaDB 10.2 / MySQL 5.7 (Issue #324)

The constraint was to prevent simple recursive parent references, but it only
checked direct parents. We thus have to rely on the high level application
maintaining the invariant anyway.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

docs: reorganize index page

Reorganize the index page of the manual into three main sections:
administrator guide, user guide, developer guide. In principle,
administrators, users and developers are independent roles that can be
fulfilled by different people.

'Administrators' install, configure and maintain Kallithea. They have
'admin' privilege in the web interface.

'Users' do not have access to the Kallithea installation files. They can
only access the web interface, and generally do not have 'admin' privilege.
They may be the owner of repositories, repository groups, or user groups,
and as such modify the settings of these objects, even though they cannot
modify the settings of Kallithea itself.

'Developers' modify the Kallithea source code. They may or may not
contribute these changes back to the Kallithea community.



The 'Readme' section is a bit outside of this classification. As it serves
as a basic introduction to Kallithea, it is kept first.

The 'Index' and 'Search' functionality are not really 'part of' the manual,
so should not be under 'Other topics'. Instead they are put at the very top
so they are readily visible.


This commit is only changing the index page. Subsequent commits can move and
split some documentation files in the repository to mirror this structure.
For example, the 'Version control systems support' page has information for
administrators (ini settings and importing existing repositories) as well as
for users (handling Mercurial subrepositories).

templates: remove references to non-existing commit-message class

As there already is monospace styling on textarea.form-control, there is no
need for a replacement.

style: use monospace for changeset messages

Similar to commits 9005c37dbe21 and 6e65c2153baa that make PR descriptions
display in a monospace font, apply monospace on changeset messages.

Note that in the files template, the reference to the non-existing 'commit'
class can be replaced instead of appending to it.

utils: move repo_name_slug to utils2 to prevent import cycle on setup_db

After commit 57a733313e4f, 'gearbox setup-db -c my.ini' fails with an import
cycle as follows:

Traceback (most recent call last):
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/bin/gearbox", line 11, in <module>
sys.exit(main())
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 199, in main
return gearbox.run(args)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 145, in run
return self._run_subcommand(remainder)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 149, in _run_subcommand
subcommand = self.command_manager.find_command(argv)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commandmanager.py", line 78, in find_command
cmd_factory = cmd_ep.resolve()
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/paster_commands/setup_db.py", line 27, in <module>
from kallithea.lib.db_manage import DbManage
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/db_manage.py", line 47, in <module>
from kallithea.model.repo_group import RepoGroupModel
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/model/repo_group.py", line 35, in <module>
import kallithea.lib.utils
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 48, in <module>
from kallithea.model.repo_group import RepoGroupModel
ImportError: cannot import name RepoGroupModel


i.e. kallithea.model.repo_group wants to import kallithea.lib.utils which
in turn wants to import kallithea.model.repo_group.

In fact there exists kallithea.lib.utils and kallithea.lib.utils2.
The current split is that 'utils2' contains 'simple' utilities, none of which
depend on kallithea models, controllers, ... In contrast, 'utils' does rely
on such kallithea classes.

As kallithea.model.repo_group was only include kallithea.lib.utils for its
repo_name_slug method, which has no dependency on other kallithea classes,
move that method (and its dependent recursive_replace) to
kallithea.lib.utils2 instead. This fixes the import cycle.

hg: set encoding to utf-8 by default to always show unicode characters correctly

Unicode characters would be shown as '?' if Kallithea was launched in a LANG=C
environment (or similar).

The problem could be solved by setting HGENCODING before launching Kallithea or
before importing Mercurial. These are often not good solutions.

Instead, introduce a hgencoding config setting that triggers monkey patching of
Mercurial.

pullrequests: don't show empty "additional changesets" (issue #280)

When opening a pullrequest on a revision range including the tipmost
revision, and then pushing a new revision on top of that, the PR page shows:
This pullrequest can be updated with changes on ...:
and then nothing.

This is because the available revisions looped over are in 'avail_cs' but
the guard around the loop is checking on 'avail_revs'. The former, while
derived from avail_revs, can become empty under circumstances like this one.

Fix the problem by changing the guard checking avail_cs rather than
avail_revs, and making sure the printed message is aligned to 'No additional
changes found'.