compare: ensure that repositories exist before proceeding
The index method on CompareController did not verify that other_repo existed, causing a rendering error if it wasn't.
Since neither controller method can proceed if either repository is non-existent, check existence and load Repository objects in __before__. Also perform type compatibility check up front while we're at it, remove redundant repository database lookups, and enable error message i18n.
paster: split paster specifics out of kallithea.lib.utils
BasePasterCommand and ask_ok are only useful in a Paster/command-line context, and can thus be removed from the already overly cluttered main utils module.
(The new common.py has been added to Mercurial as a copy of utils.py, preserving its file history, but creating a somewhat bewildering diff.)
First, find all calls to HasPermissionAll with only a single permission given, and convert to equivalent calls to HasPermissionAny.
Next, observe that it's hard to envision situations requiring multiple permissions (of the same scope: global/repo/repo group) to be satisfied. Sufficiently hard that there are actually no such examples in the code.
Finally, considering that (should it ever be needed) HasPermissionAll can be trivially built as a conjunction of HasPermissionAny calls (the decorators, too) with only a small performance impact, simply remove HasPermissionAll and related classes and functions.
A log level of WARNING meant that Alembic was entirely silent when nothing went wrong. With INFO, Alembic actually shows what migrations it is running.
kallithea/tests/models/test_dump_html_mails.ref.html is expected to contain trailing whitespace in the "-- " de-facto standard signature separator (as e.g. referenced in RFC 3676, section 4.3).
The subject line is used for mail threading in gmail and can thus not be changed without impacting users ... but now we do it.
* The tag '[Review]' is more spot-on than '[Added]'. * The subject should be short so it fits on one line, so abbreviate "pull request" to PR. * Add the PR owner - convenient for filtering comments on own PRs from comments on other PRs.
pullrequests: better handling of Mercurial pullrequests with missing revisions - don't crash
Trying to display a Mercurial PR with missing changesets could give a crash when trying to compute available updates after 3f646f7bac39 did that c.cs_ranges could be empty. That would normally not happen on Mercurial, but could happen when restoring an old filesystem backup ... or when using strip on the server.
hooks: always convert unicode to byte strings when passed to ui.status
Kallithea generally uses unicode strings internally, but ui.status follows the Mercurial convention and expects a byte string. Strings passed to ui.status should thus always by converted to byte strings. Do that explicitly with safe_str. (The alternative of using more byte strings internally seems less appealing.)
hooks: fix encoding problems of lock release ui messages
The vcs test test_push_unlocks_repository_hg would fail when waitress tried to append a unicode chunk to the byte stream. The unicode string came from the 'Released lock on repo' message passed to ui.status from the push hook.
ui.status do however follow the Mercurial convention and expects a byte string.
Changelog.rst is empty and just refer to the Mercurial logs. More readable release notes and other meta information is also available on the web site and in announcement mails.
templates: disable special mako error handler - ironically this gives better stack traces
Errors in templates could give truncated stack traces pointing at the mako error handler:
... File '.../kallithea/kallithea/controllers/files.py', line 202 in index return render('files/files.html') File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 244 in render_mako cache_type=cache_type, cache_expire=cache_expire) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 219 in cached_template return render_func() File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 241 in render_template return literal(template.render_unicode(**globs)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/template.py', line 452 in render_unicode as_unicode=True) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 803 in _render **_kwargs_for_callable(callable_, data)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 835 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 855 in _exec_template _render_error(template, context, compat.exception_as()) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 864 in _render_error result = template.error_handler(context, error) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/error.py', line 22 in handle_mako_error raise (exc, None, sys.exc_info()[2]) AttributeError: 'tuple' object has no attribute 'node'
Without the mako error handler we get a full and useful stack trace - including calls in generated but readable .html.py files.
File '.../kallithea/kallithea/controllers/files.py', line 202 in index return render('files/files.html') File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 244 in render_mako cache_type=cache_type, cache_expire=cache_expire) File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 219 in cached_template return render_func() File '.../kallithea-venv/lib/python2.7/site-packages/pylons/templating.py', line 241 in render_template return literal(template.render_unicode(**globs)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/template.py', line 452 in render_unicode as_unicode=True) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 803 in _render **_kwargs_for_callable(callable_, data)) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 835 in _render_context _exec_template(inherit, lclcontext, args=args, kwargs=kwargs) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 860 in _exec_template callable_(context, *args, **kwargs) File '.../data/templates/base/root.html.py', line 219 in render_body __M_writer(escape(next.body())) File '.../data/templates/base/base.html.py', line 57 in render_body __M_writer(escape(next.main())) File '.../data/templates/files/files.html.py', line 121 in render_main runtime._include_file(context, u'files_ypjax.html', _template_uri) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 730 in _include_file callable_(ctx, **_kwargs_for_include(callable_, context._data, **kwargs)) File '.../data/templates/files/files_ypjax.html.py', line 57 in render_body runtime._include_file(context, u'files_source.html', _template_uri) File '.../kallithea-venv/lib/python2.7/site-packages/mako/runtime.py', line 730 in _include_file callable_(ctx, **_kwargs_for_include(callable_, context._data, **kwargs)) File '.../data/templates/files/files_source.html.py', line 117 in render_body __M_writer(escape(h.pygmentize_annotation(c.repo_name,c.file,linenos=True,anchorlinenos=True,lineanchors='L',cssclass="code-highlight"))) File '.../kallithea/kallithea/lib/helpers.py', line 360 in pygmentize_annotation return literal(markup_whitespace(annotate_highlight(filenode, url_func(repo_name), **kwargs))) File '.../kallithea/kallithea/lib/annotate.py', line 57 in annotate_highlight highlighted = highlight(filenode.content, lexer, formatter) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/__init__.py', line 87 in highlight return format(lex(code, lexer), formatter, outfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/__init__.py', line 66 in format formatter.format(tokens, realoutfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/formatter.py', line 95 in format return self.format_unencoded(tokensource, outfile) File '.../kallithea-venv/lib/python2.7/site-packages/pygments/formatters/html.py', line 850 in format_unencoded for t, piece in source: File '.../kallithea/kallithea/lib/annotate.py', line 168 in _wrap_tablelinenos for el in self.filenode.annotate)) File '.../kallithea/kallithea/lib/annotate.py', line 167 in <genexpr> annotate = ''.join((self.annotate_from_changeset(el[2]()) File '.../kallithea/kallithea/lib/vcs/backends/hg/changeset.py', line 273 in get_file_annotate sha = hex(annotate_data[0].node())
setup: move test dependencies to dev_requirements.txt to make them optional
Remove the need for having test tools on production systems. Installing test dependencies is made an extra explicit step.
pip is the future, but doesn't have the same tests_require features as setuptools kind of has.
I don't like this way of handling it without setup.py support and with explicit naming of the ugly dev_requirements.txt ... but that seems to be the way to do it.
dbmigrate-test executes Alembic upgrade scripts between two Kallithea versions in a clean environment. There is no automated testing of whether the upgraded database is functional, only that the upgrade proceeds without errors.
Use Alembic to migrate database away from SQLAlchemy Migrate. This eliminates the last vestiges of SQLAlchemy Migrate. Since we drop the Migrate table (and its contents), it is not possible to revert this database change; however, downgrading to this Alembic schema revision (9358dc3d6828) will in practice be enough to ensure compatibility with all previous Kallithea versions.
(As always, the Alembic migration script is committed in the same revision as the database schema changes.)
db: enable use of main Kallithea config as Alembic config
Newly generated Kallithea config .ini files will be valid Alembic config files, eliminating the need for a separate alembic.ini config redundantly specifying the database connection string.
We reference the Alembic migration environment using kallithea:alembic, which should work independently of how Kallithea is installed.
We also configure a default 'alembic' log level of WARNING, to reduce the amount of clutter in the config file, reduce the changes needed to upgrade existing config files for use with Alembic, and allowing us to change the default Alembic log level for all users down the road.
(It makes sense to define Alembic logging in code, while all other loggers are configured in the configuration file, because Alembic is special: it runs on the command line, not as part of the web app.)
The command is placed kallithea.lib.dbmigrate:UpgradeDb (which was the location of the old command, too), to ensure that "paster upgrade-db" continues to work, even if Kallithea is installed in "editable" mode (setup.py develop/pip install -e) and package metadata has not been updated (and also to prevent issues caused by stale .pyc files).
db: remove redundant unique constraint for repository groups
There's already a unique constraint on 'group_name' alone, no need for one on the combination of 'group_name' and 'group_parent_id'. (The extra constraint likely stems from confusion over what exactly goes into group_name; add comment to clarify that it is the full group path.)
db: remove redundant unique constraints from primary keys
Primary keys are always unique, both in the SQL standard and in SQLite, MySQL and PostgreSQL.
Setting unique=True cases SQLAlchemy to explicitly add additional, redundant UNIQUE indexes to the columns, which is at best needless metadata overhead, and at worst causes the database engine to waste time maintaining an extra index that serves no purpose.
As of the upgrade to version 1.0 in 3c4b6ddf6735, SQLAlchemy began to pass TEXT length limits to the database during table creation. Such limits are however not supported by SQLite, MySQL nor PostgreSQL, and while SQLite simply ignores it, it is a syntax error in PostgreSQL, breaking the creation of new PostgreSQL databases. With the lengths being unused and quite arbitrary, just drop them.
Hooks receive a line of the following format on standard input:
<old-value> SP <new-value> SP <ref-name> LF
where <old-value> is the old object name stored in the ref, <new-value> is the new object name to be stored in the ref and <ref-name> is the full name of the ref.
This means, we have to strip at least the LF in order to have a correct version of the ref name after the split. Also, when parsing the ref name itself, use all components but first instead of just second, as a ref name may have slashes in it.
Previously, failure to parse ref name correctly would lead to the following behaviour. A newly created repository with no commits pushed has HEAD set to refs/heads/master by default, even though there's no such ref in the repository yet. Upon first push, Kallithea rewrites this symbolic reference with a reference to a real branch.
However, due to a bug in ref name parsing, if a ref name had a slash, Kallithea would update HEAD to an invalid reference:
git push origin feature/branch
would rewrite HEAD to refs/heads/feature. All future attempts to work with this repository would fail because dulwich would complain it can't read HEAD as it is a directory.
db: ensure git hooks work when the repositories base path is a symlink
When a Git hook starts, it thinks its repo_path is its cwd. However, if the repositories base path is a symlink to a different location, base path won't match the location of the repository where the symlink will be resolved.
git: include an LF at the end of the service advertisement (Fixes #230)
This fixes hg-git/Dulwich and possibly other conservative Git clients, which do not ignore the absence of the LF.
The original comment was a guess based on reverse engineering the protocol not specified in the documentation yet at that moment. Now that the documentation exists and states this explicitly, just do as it says.
This function strips the repository name and any slashes from the URL path, leaving the protocol command, like info/refs or git-{receive,upload}-pack.
Using .split() and .strip() for this purpose isn't entirely reliable and is entirely unreadable, so it's better to write it out, even if it's a bit verbose.
Also, error out in the unlikely case the path doesn't start with a slash and the repository name; that shouldn't happen normally.