tests: finally block also needs to be executed if api_call itself fails

The cleanup code (finally block) should be executed as soon as something what
changes the db fails.

I don't remember which one but one of these tests failed and caused all the
others to fail as well because the database wasn't cleaned up after the first
failure. That made finding the root cause more difficult then it should have
been.

pytest migration: model: convert all tests to TestControllerPytest

The model tests were based on BaseTestCase which does not exist currently
for pytest-style tests. Nevertheless, there seems to be no advantage of
directly subclassing BaseTestCase over subclassing TestControllerPytest.
Thus, keep things simple and use TestControllerPytest.

pytest migration: add temporary assertRaises wrapper

Similar to the other temporary unittest-assert wrappers, add the missing
assertRaises and forward to pytest.raises.

pytest migration: simplify hg+git test class hierarchies

No need to use multiple inheritance when single inheritance can do it just
as well. Let the git/hg test cases just derive from _BaseTestCase which in
turn derives from TestControllerPytest.

Also remove empty setup_class/teardown_class methods.

pytest migration: convert functional tests with setup/teardown methods

pytest also supports setup/teardown methods like unittest (even though
pytest fixtures are more powerful and should be the end goal).
Only difference is the naming and signature of setUp (setup_method) and
tearDown (teardown_method).

test_admin_notifications: fix index test dependency

The index test only worked because another test had prepared some database
state it relied on.

More specifically, the index test creates a notification
as a newly created user, but that user had not been committed to the
database yet. When running the index test standalone, this causes the error:

IntegrityError: (IntegrityError) NOT NULL constraint failed:
notifications.created_by u'INSERT INTO notifications (subject, body,
created_by, created_on, type) VALUES (?, ?, ?, ?, ?)'
(u'test_notification_1', u'notification_1', None, '2016-03-11
21:15:19.066402', u'message')

By committing the database after the user is created, the problem is solved.

i18n: updated translation for Chinese (China)

Currently translated at 39.8% (452 of 1135 strings)

i18n: updated translation for German

Currently translated at 58.6% (666 of 1135 strings)

i18n: updated translation for French

Currently translated at 100.0% (1135 of 1135 strings)

i18n: updated translation for French

Currently translated at 98.5% (1119 of 1135 strings)

tests: fix Git on Windows sometimes failing on ' or ` in file:/// URLs

Some Git versions will fail when cloning a repository like:

git clone file:///path/to/strange-file-'`foobar myclone

Since '` (and some other characters) are filtered anyway (in
lib.utils.repo_name_slug) I think this test doesn't need to test those
characters.

tests: fix test_nodes.py and test_files.py which fail on windows due to mimetype differences

The mimetype of files may differ on different platforms (e.g. the mimetype of
.py files on windows is not text/x-python but text/plain). Therefore, compare
the mimetype to the value the platform uses.

compare: close Dulwich internals for Git repositories to avoid leaking open files

Fix failing test_compare.py on Windows by explicitly closing internal
object_store objects.

status: refactor internal dict for mapping status string to description

Slightly better performance ... but more importantly: do it "the right way".

docs: document Kallithea customization options

Create basic documentation for the various customization options in
Kallithea. Move the rcextensions section to this new page and reword.

settings: rescope 'Analytics HTML' option to generic HTML/JS/CSS customization

Update the setting name 'Analytics HTML block' to better cover the fact that
it can be used to make any HTML, JavaScript or CSS tweak.

vcs: fix processing of git commands that return output on stderr

The subprocessio module used for interfacing with Git is using threading. It
might not be fully deterministic.

The subprocessio module also had the "feature" that it stopped reading and
reported failure once it found output on stderr - even if the command completed
(or would complete) with success.

Sometimes (like https://drone.io/bitbucket.org/domruf/kallithea/24 ) tests on
linux could fail on totally normal informational output like:

Couldn't run git command (['git', '-c', 'core.quotepath=false', 'checkout', 'foobranch']).
Original error was: Subprocess exited due to an error:
Switched to branch 'foobranch'

On Windows it would fail even more often.

To fix that, ignore stderr while processing output. There is a risk that it in
some cases can make the process block on stderr and thus never finish ... but
there is no reports of that yet.

i18n: add translation for Spanish

Created new translation.

db: cache SCM instance short-term (tied to SQLAlchemy session lifetime)

Repeatedly checking whether SCM instances are invalidated is slow, and
we don't actually _want_ SCM instances to invalidate half-way through
a request either.

Therefore cache them in on the db.Repository object, the lifetime
of which is directly tied to the lifetime of the SQLAlchemy session,
the lifetime of which is tied directly to the individual HTTP request.
This way, we only check for invalidation the first time the SCM instance
is accessed in a request.

This will improve performance in cases where we have (by definition) badly
written code that retrieves repo objects several times.

db: get rid of vcs_full_cache - it should always be used

It might make tests less deterministic, but it makes sure that we test what we actually use.

tests: invalidate cache for new repositories - make tests more stable if garbage has been left behind in the db

It could be argued that this should be done elsewhere in the app ...

db: name the scm_instance_cached cache entries - reduce the risk of collisions

- and same with other cache regions

db: inline repo __get_instance to scm_instance_no_cache

Having two names for the same thing just made it look even more complicated
than it is.

There _should_ not be any reason for calling scm_instance_no_cache directly
anywhere ... but for now that is how it is.

helpers: cache user_or_none results in beaker long_term cache

Avoids repeated expensive 'LIKE' lookup on email addresses extracted from
changeset authors.

This information is only used for display of changelog information.

The cache will never expire - only when the server process expires. That is ok
for this very static information and the way it is used.

scm: simplify get_repos and get rid of RepoList overloads

The RepoList variants might have made sense before any repo info was stored in
the database. Now it has no value and doesn't help at all.

pullrequests: optimize iteration over reviewers - avoid fetching users one by one

.reviewers was mainly used for iteration and then dereferencing .user one by
one. That gave lots of queries and round trips to the database and was slow.

Instead, do something else. Either query directly or use a new method for
getting the list of reviewer users.

Reviewers will explicitly be shown in the order they have been added (assuming
database id's are monotonic).

setup.py: remove unused package_data

Per the setuptools documentation, package_data is ignored if
include_package_data is True, and indeed, we instead use MANIFEST.in.

tests: don't use the TESTS_TMP_PATH string

it differs each time you run the test
and therefore doesn't work if you use a separat kallithea test instance
(KALLITHEA_NO_TMP_PATH=1)

instead get the 'real' path from the DB

FIXME: breaks kallithea/tests/functional/test_admin_repos.py

paster: remove unused BasePasterCommand.notify_msg

This method is not referenced anywhere in the Kallithea or Paste code.
Additionally, calling it with any true-ish `log` argument would trigger
a TypeError exception.

vcs: the git backend requires string encoded repository names too

Shown by test.functional.test_admin_repos.test_delete_non_ascii on Windows.

tests: binary streams/files need to be opened in binary mode

Without binary, the stream may be closed on EOF character
http://stackoverflow.com/a/9069009/7029 .

windows: node paths should always use posixpath - avoid \ in repo internal paths on windows

Using \ on Windows would result in a wrong node path which could not be checked
out with Git on Windows.

localization: set language for formencode based on config file

Before, the language would be taken from the environment - such as LANG. Now it
will use exactly what has been configured as lang in .ini .

The formencode initialization could be done in many places and there is no
obvious place ... but this one seems ok.

tests: clarify user IP range test dependency on beaker caching of user objects

Requests with invalid request address would pass after configuring user IP
ranges because the IP range would not be validated as long as the user object
was found in the beaker cache.

Instead, wait until the beaker cache has expired and verify the user cannot log
in without a valid IP. Then provide a valid IP for later requests until the IP
range is removed again.

Based on original patch and research by Dominik Ruf.

i18n: updated translation for Greek

Currently translated at 10.9% (124 of 1132 strings)

graph: detect git branches and colourise them properly without rainbow effect (Issue #188)

This is based on research and patches by Andrew Shadura.

When using Git, only commits pointed to by branch references have their
branches set. In general, there's no way in Git to find out which branch
a commit belongs to, so we can try to deduce that from the merge topology.

In other words, if we know the branch name, we know it's a different colour
than any different branch. If we don't (it is None), we guesstimate the first
parent is probably on the same branch. The relevant part of the code before
2da0dc09 used a similar, yet simpler, algorithm.

graph: avoid collision between "heir has been found" and "child has no name" both using None as branch name

Instead of using None for both, introduce a new flag for searching for heir.

This will do that Git changesets without branch name will inherit colors from
each other instead of getting a new one every time. (Inheriting from from
changesets with branch name is still phony.)

tests: some test coverage of graph structure and coloring

This is a bad and unmaintainable test ... but better than nothing.

summary, changelog: add reviewer username to status change tooltips

Change the status change (circle) icon tooltips from "Changeset status:
NEW_STATE" to "Changeset status: NEWSTATE by REVIEWER_USERNAME".

This makes it much faster to check who is reviewing what and who approved or
rejected what.

summary: add tooltip and link to status change icons (same as on changelog page)

This change makes the changeset list on the summary page more similar to the one
on the changeset list. It adds the same tooltip ("Changeset status") to the
changeset status change icons that is already found on the changelog page. It
also adds a link to the first comment (as in the changelog page).

pullrequest: add URL changesets in txt notification email to reviewers

Similar to adding the URL for each changeset in the html notification email
(commit xxxxxxxxxxxx) do the same for txt emails. The e-mail client
presumably makes these URLs clickable.

vcs: better handling of invalid email addresses: don't consider them email addresses

13da89053853 was in principle right in always returning email adresses as
string ... but unfortunately the function also returned invalid email addresses
that didn't fit into strings.

To fix this, the function is refactored to always use regexp matching of valid
email addresses ... and to be simpler. The behaviour should be the same as
before for all valid email addresses.

tests: add test for author_email handling of invalid author strings

This shows that author_email can return invalid email addresses.

pytest migration: files: convert to TestControllerPytest and parametrize

Migrate the 'files' tests to pytest and its parametrize decorator.
The syntax of this decorator is slightly different: a first argument is a
comma-separated string with the method argument names for which the values
are provided as second argument.

pytest migration: make pytest's parametrize functionality available

To provide parameterized tests, a custom implementation is currently
provided at kallithea/tests/parameterized.py because nose does not provide
that out-of-the-box.
pytest, on the other hand, does have a built-in 'parametrize' (note:
different spelling) functionality. Therefore, once all tests have been
converted to pytest, we can get rid of the custom 'parameterized'
implementation. Also, the existing 'parameterized' implementation does not
seem to work under pytest-style tests.

This commit makes pytest's 'parametrize' decorator available alongside the
custom 'parameterized' decorator. The names are confusing but it is the
intention to kill the original 'parameterized' soon.