pullrequests: better handling of Mercurial pullrequests with missing revisions - don't crash
Trying to display a Mercurial PR with missing changesets could give a crash when trying to compute available updates after 3f646f7bac39 did that c.cs_ranges could be empty. That would normally not happen on Mercurial, but could happen when restoring an old filesystem backup ... or when using strip on the server.
hooks: always convert unicode to byte strings when passed to ui.status
Kallithea generally uses unicode strings internally, but ui.status follows the Mercurial convention and expects a byte string. Strings passed to ui.status should thus always by converted to byte strings. Do that explicitly with safe_str. (The alternative of using more byte strings internally seems less appealing.)
hooks: fix encoding problems of lock release ui messages
The vcs test test_push_unlocks_repository_hg would fail when waitress tried to append a unicode chunk to the byte stream. The unicode string came from the 'Released lock on repo' message passed to ui.status from the push hook.
ui.status do however follow the Mercurial convention and expects a byte string.
Changelog.rst is empty and just refer to the Mercurial logs. More readable release notes and other meta information is also available on the web site and in announcement mails.
templates: disable special mako error handler - ironically this gives better stack traces
Errors in templates could give truncated stack traces pointing at the mako error handler:
... File '.../kallithea/kallithea/controllers/files.py', line 202 in index return render('files/files.html') File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 244 in render_mako cache_type=cache_type, cache_expire=cache_expire) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 219 in cached_template return render_func() File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 241 in render_template return literal(template.render_unicode(**globs)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/template.py', line 452 in render_unicode as_unicode=True) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 803 in _render **_kwargs_for_callable(callable_, data)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 835 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 855 in _exec_template _render_error(template, context, compat.exception_as()) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 864 in _render_error result = template.error_handler(context, error) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/error.py', line 22 in handle_mako_error raise (exc, None, sys.exc_info()[2]) AttributeError: 'tuple' object has no attribute 'node'
Without the mako error handler we get a full and useful stack trace - including calls in generated but readable .html.py files.
File '.../kallithea/kallithea/controllers/files.py', line 202 in index return render('files/files.html') File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 244 in render_mako cache_type=cache_type, cache_expire=cache_expire) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 219 in cached_template return render_func() File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 241 in render_template return literal(template.render_unicode(**globs)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/template.py', line 452 in render_unicode as_unicode=True) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 803 in _render **_kwargs_for_callable(callable_, data)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 835 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 860 in _exec_template callable_(context, *args, **kwargs) File '.../data/templates/base/root.html.py', line 219 in render_body __M_writer(escape(next.body())) File '.../data/templates/base/base.html.py', line 57 in render_body __M_writer(escape(next.main())) File '.../data/templates/files/files.html.py', line 121 in render_main runtime._include_file(context, u'files_ypjax.html', _template_uri) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 730 in _include_file callable_(ctx, **_kwargs_for_include(callable_, context._data, **kwargs)) File '.../data/templates/files/files_ypjax.html.py', line 57 in render_body runtime._include_file(context, u'files_source.html', _template_uri) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 730 in _include_file callable_(ctx, **_kwargs_for_include(callable_, context._data, **kwargs)) File '.../data/templates/files/files_source.html.py', line 117 in render_body __M_writer(escape(h.pygmentize_annotation(c.repo_name,c.file,linenos=True,anchorlinenos=True,lineanchors='L',cssclass="code-highlight"))) File '.../kallithea/kallithea/lib/helpers.py', line 360 in pygmentize_annotation return literal(markup_whitespace(annotate_highlight(filenode, url_func(repo_name), **kwargs))) File '.../kallithea/kallithea/lib/annotate.py', line 57 in annotate_highlight highlighted = highlight(filenode.content, lexer, formatter) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/__init__.py', line 87 in highlight return format(lex(code, lexer), formatter, outfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/__init__.py', line 66 in format formatter.format(tokens, realoutfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/formatter.py', line 95 in format return self.format_unencoded(tokensource, outfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/formatters/html.py', line 850 in format_unencoded for t, piece in source: File '.../kallithea/kallithea/lib/annotate.py', line 168 in _wrap_tablelinenos for el in self.filenode.annotate)) File '.../kallithea/kallithea/lib/annotate.py', line 167 in <genexpr> annotate = ''.join((self.annotate_from_changeset(el[2]()) File '.../kallithea/kallithea/lib/vcs/backends/hg/changeset.py', line 273 in get_file_annotate sha = hex(annotate_data[0].node())
setup: move test dependencies to dev_requirements.txt to make them optional
Remove the need for having test tools on production systems. Installing test dependencies is made an extra explicit step.
pip is the future, but doesn't have the same tests_require features as setuptools kind of has.
I don't like this way of handling it without setup.py support and with explicit naming of the ugly dev_requirements.txt ... but that seems to be the way to do it.
dbmigrate-test executes Alembic upgrade scripts between two Kallithea versions in a clean environment. There is no automated testing of whether the upgraded database is functional, only that the upgrade proceeds without errors.
Use Alembic to migrate database away from SQLAlchemy Migrate. This eliminates the last vestiges of SQLAlchemy Migrate. Since we drop the Migrate table (and its contents), it is not possible to revert this database change; however, downgrading to this Alembic schema revision (9358dc3d6828) will in practice be enough to ensure compatibility with all previous Kallithea versions.
(As always, the Alembic migration script is committed in the same revision as the database schema changes.)
db: enable use of main Kallithea config as Alembic config
Newly generated Kallithea config .ini files will be valid Alembic config files, eliminating the need for a separate alembic.ini config redundantly specifying the database connection string.
We reference the Alembic migration environment using kallithea:alembic, which should work independently of how Kallithea is installed.
We also configure a default 'alembic' log level of WARNING, to reduce the amount of clutter in the config file, reduce the changes needed to upgrade existing config files for use with Alembic, and allowing us to change the default Alembic log level for all users down the road.
(It makes sense to define Alembic logging in code, while all other loggers are configured in the configuration file, because Alembic is special: it runs on the command line, not as part of the web app.)
The command is placed kallithea.lib.dbmigrate:UpgradeDb (which was the location of the old command, too), to ensure that "paster upgrade-db" continues to work, even if Kallithea is installed in "editable" mode (setup.py develop/pip install -e) and package metadata has not been updated (and also to prevent issues caused by stale .pyc files).
db: remove redundant unique constraint for repository groups
There's already a unique constraint on 'group_name' alone, no need for one on the combination of 'group_name' and 'group_parent_id'. (The extra constraint likely stems from confusion over what exactly goes into group_name; add comment to clarify that it is the full group path.)
db: remove redundant unique constraints from primary keys
Primary keys are always unique, both in the SQL standard and in SQLite, MySQL and PostgreSQL.
Setting unique=True cases SQLAlchemy to explicitly add additional, redundant UNIQUE indexes to the columns, which is at best needless metadata overhead, and at worst causes the database engine to waste time maintaining an extra index that serves no purpose.
As of the upgrade to version 1.0 in 3c4b6ddf6735, SQLAlchemy began to pass TEXT length limits to the database during table creation. Such limits are however not supported by SQLite, MySQL nor PostgreSQL, and while SQLite simply ignores it, it is a syntax error in PostgreSQL, breaking the creation of new PostgreSQL databases. With the lengths being unused and quite arbitrary, just drop them.
Hooks receive a line of the following format on standard input:
<old-value> SP <new-value> SP <ref-name> LF
where <old-value> is the old object name stored in the ref, <new-value> is the new object name to be stored in the ref and <ref-name> is the full name of the ref.
This means, we have to strip at least the LF in order to have a correct version of the ref name after the split. Also, when parsing the ref name itself, use all components but first instead of just second, as a ref name may have slashes in it.
Previously, failure to parse ref name correctly would lead to the following behaviour. A newly created repository with no commits pushed has HEAD set to refs/heads/master by default, even though there's no such ref in the repository yet. Upon first push, Kallithea rewrites this symbolic reference with a reference to a real branch.
However, due to a bug in ref name parsing, if a ref name had a slash, Kallithea would update HEAD to an invalid reference:
git push origin feature/branch
would rewrite HEAD to refs/heads/feature. All future attempts to work with this repository would fail because dulwich would complain it can't read HEAD as it is a directory.
db: ensure git hooks work when the repositories base path is a symlink
When a Git hook starts, it thinks its repo_path is its cwd. However, if the repositories base path is a symlink to a different location, base path won't match the location of the repository where the symlink will be resolved.
git: include an LF at the end of the service advertisement (Fixes #230)
This fixes hg-git/Dulwich and possibly other conservative Git clients, which do not ignore the absence of the LF.
The original comment was a guess based on reverse engineering the protocol not specified in the documentation yet at that moment. Now that the documentation exists and states this explicitly, just do as it says.
This function strips the repository name and any slashes from the URL path, leaving the protocol command, like info/refs or git-{receive,upload}-pack.
Using .split() and .strip() for this purpose isn't entirely reliable and is entirely unreadable, so it's better to write it out, even if it's a bit verbose.
Also, error out in the unlikely case the path doesn't start with a slash and the repository name; that shouldn't happen normally.
docs: use ., not source, as the user isn't guaranteed to have it
Even though bash is the default shell nearly everywhere, a restricted user running Kallithea doesn't necessarily have it as the default, so better use POSIX-compatible . command. Besided, it's also shorter.
setup: use modern bcrypt implementation instead of unsupported old one
py-bcrypt has been deprecated by bcrypt, and is no longer developed or supported.
bcrypt requires bytestrings instead of strings, use safe_str to ensure they're encoded before they're passed to bcrypt. Also, use check_pw to minimise the number of manual conversions and comparisons.
Installation of bcrypt will probably compile a C extension and require libffi-dev.
cleanup: get rid of dn as shortcut for os.path.dirname
We keep 'dirname' as shortcut despite having removed the 'join' shortcut: * the name is less ambiguous than 'join'. * dirname is often applied multiple times - spelling it out would be too verbose.
tests: introduce tests and reference dump for notification mails
The mails are dumped to a tracked html file: * changes shows up as diffs and are easy to spot and review * all mails can easily can be investigated in a browser and checked for content and consistency
The tests are mocking canonical_url because it has deep dependencies to pylons.url which requires (thread local?) environment setup that the tests doesn't have.
lock: fix for Mercurial 3.6+ - wrap hgweb to catch Locked exceptions from hooks
With Mercurial 3.6, the handling of WSGI responses changed. The hook exceptions are no longer raised directly when app(environ, start_response) is called so the 'except HTTPLockedRC as e' block in _handle_request (a few lines above ) does not work anymore because the exception happens later.
Therefore I created a wrapper class that can catch the exceptions.
This makes locking work again and fixes lock related tests like TestVCSOperations.test_clone_after_repo_was_locked_hg which expect certain output of the hg client in case of an HTTPLockedRC exception.
tests: set EMAIL for Git commit test_push_on_locked_repo_by_other_user_git - it _is_ necessary on some machines
7db1bcf1d95b too aggressively removed setting EMAIL (which was set in a way that didn't work on Windows).
On some machines the git commit in _add_files_and_push would fail with 'Please tell me who you are' and the actual test check of "Repository %s locked by user" would fail.
On other machines - also without any local git configuration - it works fine.
hooks: don't catch ImportError Exceptions in git hooks
These hooks are important. Being friendly and hiding errors is not the good kind of friendliness. It is better if they just fail if the Kallithea module can't be imported.
tests: Mercurial hooks must use ui.status for messages sent to the client
Mercurial changed so sys.stdout and sys.stderr now are intercepted instead of being sent to the client. That caused a regression that manual_test_vcs_operations.py test_push_new_file_hg caught - we no longer reported 'Repository size' to the user.
The issue was introduced by a Mercurial change, but the fix is backwards compatible with Mercurial 2.9.
The fix is to use ui.status everywhere. ui is a Mercurial thing, but handle_git_receive also creates a ui object for Git hooks so ui.status can be used there too.
hooks: if available, use sys.executable as executable for git hooks
Windows doesn't necessarily have "python2" available in $PATH, but we still want to make sure we don't end up invoking a python3. Using the absolute path seems more safe.
notifications: improve response time when number of notifications is large
NotificationsController always retrieved materialized list of all notifications in database, even to display only 10 of them. This is improved by feeding SQLAlchemy Query object directly to webhelpers.paginate.Page, avoiding eager load of all notifications.
(4.x seems like a more major upgrade - we stick to the 3 series for now.)
Note that select2-bootstrap.css *not* is upgraded. It is a separate thing (see LICENSE.md) and currently unused. (Select2 now has similar functionality built in - that should perhaps be used with Bootstrap instead.)
This creates a separate "Upgrading Kallithea" section, instead of trying (and failing) to repeat the same information for each installation page. Trying to maintain upgrade instructions for each installation method was perhaps overly ambitious. In practice, only the Linux instructions were up-to-date.
(Incidentally, this is kind of a throwback to Kallithea 0.1, which also had a separate upgrade section.)
templates: use a better class name for an enabled toggle button
"btn-success" is rather a poor choice for a class applied when the button is active, "active" is better for that. A side effect of the change is that "Enabled" is now shown in bold.