Sometimes, when reusing cached data, the DOM part of select2 activation would be reused, but not the actual activation. Repeated select2 activation would thus give duplicated DOM entries.
Select2 doeesn't seem to have a good way to unload or redo, so instead just try to remove the old DOM parts.
settings: rework logic for flash message after repository scan
Make the code more readable by extracting added_msg and removed_msg away from the h.flash() call and reindenting the logic. There are no functional changes here.
These changes serve as preparatory work for a subsequent commit that will change the logic.
Commit d66201a7ce6e ("files: change "callbacks" function to the more descriptive name "post_load_state" and let it take an actual state data object") broke the 'Show Authors' button when visiting a file in the files browser. The button normally shows a count of authors and their avatars, but after the aforementioned commit it did nothing.
Following patch would have fixed the problems in commit d66201a7ce6e:
diff --git a/kallithea/templates/files/files_source.html b/kallithea/templates/files/files_source.html --- a/kallithea/templates/files/files_source.html +++ b/kallithea/templates/files/files_source.html @@ -81,11 +81,13 @@ <script> $(document).ready(function(){ var state = { + data: { node_list_url: node_list_url.replace('__REV__',${h.js(c.changeset.raw_id)}).replace('__FPATH__', ${h.js(h.safe_unicode(c.file.path))}), url_base: url_base.replace('__REV__',${h.js(c.changeset.raw_id)}), rev: ${h.js(c.changeset.raw_id)}, f_path: ${h.js(h.safe_unicode(c.file.path))} + } } - post_load_state(State.data); // defined in files.html + post_load_state(state.data); // defined in files.html }); </script>
But, later the code got refactored more, and commit 006d68c4d7b9 ("files: use the web browsers built-in js history instead of native.history.js") broke the feature further: the click handler for the button no longer got installed on the 'document-ready' event, but only when a new 'state' is loaded. And it seems there is never a situation where a new state preserves the button, so it makes no sense installing the click handler at that moment.
Instead, move the click handler back to the 'document-ready' event.
cli: fill in git_hook_interpreter at 'config-create' time to really fix potentially invalid interpreter in git hooks (Issue #333)
When generating a configuration file using 'kallithea-cli config-create', it is probably using the right Python interpreter, so fill in the current Python executable as 'git_hook_interpreter' in the generated file.
There should thus rarely be any need to configure this manually, and issue #333 will *really* be fixed.
As this causes an absolute path to be encoded inside the ini file, moving the virtualenv will require updating this path.
For development.ini we do not want to hardcode any path and are happy to leave it using the old heuristics at runtime.
hooks: make the Python interpreter for Git hooks configurable as 'git_hook_interpreter' (Issue #333)
Commit 5e501b6ee639 introduced the use of 'sys.executable' as interpreter for git hooks instead of 'python2' with the following argument:
"Windows doesn't necessarily have "python2" available in $PATH, but we still want to make sure we don't end up invoking a python3. Using the absolute path seems more safe."
But, sys.executable does not necessarily point to Python. When Kallithea is started under uWSGI, sys.executable points to the uwsgi executable. As a result, the interpreter encoded in the git hooks on the server repositories would be:
#!/path/to/uwsgi
And pushing to such repo would result in following client errors:
$ git push Password for 'http://user@localhost:5050': Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: unable to load configuration from hooks/pre-receive To http://localhost:5050/gitrepo-new ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'http://user@localhost:5050/gitrepo-new'
Fix this problem by introducing a configuration setting 'git_hook_interpreter' that allow administrators to specify which Python interpreter to use.
A subsequent commit will cause its value to be filled in automatically when generating a new ini file, but an administrator can always override it.
The use of /usr/bin/env is only needed for relative arguments (or to pass variables in the environment, which we don't do). It is thus not needed in case the Python interpreter for Git hooks is known with absolute path, as introduced in 5e501b6ee639.
hooks: add intermediate function _get_git_hook_interpreter
The logic to determine the right interpreter for Git hooks is about to change and possibly become more complex. Split it off in a separate function so such changes do not require extra code duplication and preserve the readability of the code.
In TurboGears2 2.3.12, the latest version in the 2.3.x range, the WebOb dependency requirement is [1] WebOb >= 1.2, < 1.8.0
In TurboGears2 2.4.0 (which is in pre-release state at the time of this commit), this becomes [2]: WebOb >= 1.8.0, < 1.10.0
In the Kallithea dependency list, we have matched our WebOb version requirements to that of TurboGears2 and use: WebOb >= 1.7, < 1.8
while our TurboGears2 requirement was liberal and accepted anything in the 2.x range: TurboGears2 >= 2.3.10, < 3
To avoid new Kallithea installations failing with conflicting WebOb version requirements after TurboGears2 2.4.0 is released, restrict the version of TurboGears2 to 2.3.x on the stable branch.
For the default branch, the update to TurboGears2 2.4.0 can be considered once it's released.
Commit f2f7a8c1281e changed the i18n-related ini settings to match TurboGears2. Even though the commit message correctly refers to 'i18n.enabled', the code incorrectly used 'i18n.enable'.
compare: correct display of special branch names in initial placeholder
When a branch name contains special characters like '<' or '>', and a 'compare' operation is performed with such branch as one of the two compare sides, then the special branch name will be part of the URL, e.g.
The encoded branch name is then used at page load as placeholders for the branch selection dropdowns. But, the special characters, were escaped too much, causing '<' to become < in the display of the dropdown.
The placeholder was escaped via the default mako escape filter, before being passed to make_revision_dropdown, thus too early. We want the raw value. h.js() (copied from the default branch) gives us that, while still formatting and escaping the string so it is safe inside the script tag.
compare: prevent XSS due to unescaped branch/tag/bookmark names
In the revision selection dropdown of the 'Compare' functionality, the branch/tag/bookmark names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
templates/summary: escape branch/tag/bookmark names in 'Download as zip' links to prevent XSS
On a repository summary page, in the 'Download' section where you can download an archive of the repository at a given revision, the branch/tag names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
lib: sanitize HTML for all types of README rendering, not only markdown
The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized.
Therefore, move the sanitizing one level up so it covers all renderers, for now and the future.
This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
cleanup: remove unnecessary (and potentially problematic) use of 'literal'
webhelpers.html.literal (kallithea.lib.helpers.literal) is only needed when the passed string may contain HTML that needs to be interpreted literally. It is unnecessary for plain strings.
Incorrect usage of literal can lead to XSS issues, via a malicious user controlling data which will be rendered in other users' browsers. The data could either be stored previously in the system or be part of a forged URL the victim clicks on.
For example, when a user browses to a forged URL where a repository changeset or branch name contains a javascript snippet, the snippet was executed when printed on the page using 'literal'.
Remaining uses of 'literal' have been reviewed with no apparent problems found.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
pullrequests: prevent XSS in 'Potential Reviewers' list when first and last names cannot be trusted
If a user first or last name contains javascript, these fields need proper escaping to avoid XSS attacks.
An example scenario is: - the malicious user creates a repository. This will cause this user to be listed automatically under 'Potential Reviewers' in pull requests. - another user creates a pull request on that repository and selects the suggested reviewer from the 'Potential Reviewers' list.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
Technical note: the other caller of addReviewMember in base.js itself does _not_ need to be adapted to escape the input values, because the input values (oData) are _already_ escaped (by the YUI framework).
db: drop constraint that started failing with MariaDB 10.2 / MySQL 5.7 (Issue #324)
The constraint was to prevent simple recursive parent references, but it only checked direct parents. We thus have to rely on the high level application maintaining the invariant anyway.
repos: introduce low level slug check of repo and group names
The high level web forms already slug-ify repo and repo group names. It might thus not create the exact repo that was created, but the name will be "safe".
For API, we would rather have it fail than not doing exactly what was requested.
Thus, always verify at low level that the provided name wouldn't be modified by slugification. This makes sure the API provide allow the same actual names as the web UI.
This will only influence creation and renaming of repositories and repo groups. Existing repositories will continue working as before.
This is a slight API change, but it makes the system more stable and can prevent some security issues - especially XSS attacks.
repos: only allow api repo creation in existing groups
Fix problem with '../something' paths being allowed; '..' will always exist and can't be created.
This also introduce a small API change: Repository groups must now exist before repositories can be created. This makes the API more explicit and simpler.
make-release: import version and copyright updates from default branch (dba4e770d4b6)
make-release has been improved on the default branch in commit dba4e770d4b6, and that version of the script also almost works on stable.
There are some parts we don't have in stable branch: dev_requirements.txt, run-all-cleanup, and a working dist file content check:
The MANIFEST.in file on the stable branch is not complete. Most of the missing entries could simply be added (.travis.yml, requirements.txt, tox.ini, and scripts/*) but there is one problematic entry: kallithea/i18n/en/LC_MESSAGES/kallithea.mo. In the check command, all .mo entries are filtered out, causing a remaining diff for the English .mo.
As that file has been removed on the default branch already, and the missing entries are not important for the release anyway, simply drop the check for now.
- Updated Git repository implementation to ensure context falls within 0 to 2**31-1 range (inclusive) when fetching a diff. - Added tests for Git repositories for checking passed-in negative and overflowing contexts (for the --unified option). - Updated Mercurial repository implementation to ensure context is not negative when fetching a diff. - Added tests for Mercurial repositories for checking passed-in negative context (for the --unified option).
setup: updated dependencies to accept latest Pylons release (1.0.3).
A couple of weeks ago the Pylons project has released version 1.0.3, removing the previously stable version 1.0.2 in the process. In turn, this would mean that new installation of Kallithea will end-up with older version of Pylons than what is actually available, and in particular a much older release than 1.0.2.
setup-db: print completion message to avoid confusion (issue #303)
There are cases where the last message of setup-db is a warning, giving the impression that the overall command failed which may not actually be the case.
For example, when git is not installed, warnings are given, but they are not an actual error. Kallithea will work fine for Mercurial repositories.
To avoid any confusion, print a completion message at the end. Any real errors will abort the command and not make it this far.
login: fix crash when entering non-ASCII password for login (Issue #300)
Avoid errors like UnicodeEncodeError: 'ascii' codec can't encode characters in position X: ordinal not in range(128) when the user enters non-ASCII passwords for existing internal accounts in the login prompt.
The password forms have "always" rejected non-ASCII passwords with Invalid characters (non-ASCII) in password
This script will automate how the current attribution and copyright listings are generated, thus also enabling it to be maintained from repository history in the future.
This doesn't change anything about who has or hasn't contributed or deserves credit, it just does a best effort attempt at maintaining existing attribution we inherited and automating it going forward. The automation might expose existing problems - they should be fixed by adding or removing exceptions in the script.
email: don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)
For example, on Linux, running `gearbox serve` with LANG=C, would crash in:
File ".../kallithea/lib/celerylib/tasks.py", line 307, in send_email % (' '.join(recipients), headers, subject, body, html_body)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 224: ordinal not in range(128)
Replacing render with render_unicode works in this case ... but there might be other problems elsewhere.
journal: make "repository:" filtering condition work as expected (Issue #261)
Before this revision, journal filtering conditions like as below never match against any entry, even if there are corresponded repositories.
- repository:foo/bar - repository:foo-bar
Whoosh library, which is used to parse filtering condition, does:
- treat almost all non-alphanumeric characters as delimiter at parsing condition - join each conditions at filtering by "AND", by default
For example, filtering condition "repository:foo/bar" is translated as "repository:foo AND repository:bar". This combined condition never matches against any entry, because it is impossible that "repository" field in DBMS table "user_logs" has both "foo" and "bar" values at same time.
Using TEXT for "repository" of JOURNAL_SCHEMA causes this issue, because TEXT assumes tokenization at parsing.
In addition to it, using TEXT also causes unintentional ignorance of "stop words" in filtering conditions. For example, "this", "a", "you", and so on are ignored at parsing, because these are too generic words (from point of view of generic "text search").
To make "repository:" filtering condition work as expected, this revision uses ID instead of TEST for "repository" of JOURNAL_COLUMN. ID avoids both tokenization and removing "stop words".
This replacement should be safe with already existing DBMS instance, because:
- JOURNAL_SCHEMA is used only to parse filtering condition - DBMS table "user_logs" itself is defined by UserLog class (in kallithea/model/db.py)
BTW, using ID also avoids normalization by lowercase-ing. But this doesn't violate current case-insensitive search policy, because LOWER-ing in actual SQL query is achieved by get_filterion() or so in kallithea/controllers/admin/admin.py.
To simplify email filtering, add a header indicating the type of email being sent. The 'type_' value corresponds to one of the types defined in class Notification in kallithea/model/db.py, e.g. 'cs_comment', 'pull_request', 'pull_request_comment', ...
git: include an LF at the end of the service advertisement (Fixes #230)
This fixes hg-git/Dulwich and possibly other conservative Git clients, which do not ignore the absence of the LF.
The original comment was a guess based on reverse engineering the protocol not specified in the documentation yet at that moment. Now that the documentation exists and states this explicitly, just do as it says.
lock: fix for Mercurial 3.6+ - wrap hgweb to catch Locked exceptions from hooks
With Mercurial 3.6, the handling of WSGI responses changed. The hook exceptions are no longer raised directly when app(environ, start_response) is called so the 'except HTTPLockedRC as e' block in _handle_request (a few lines above ) does not work anymore because the exception happens later.
Therefore I created a wrapper class that can catch the exceptions.
This makes locking work again and fixes lock related tests like TestVCSOperations.test_clone_after_repo_was_locked_hg which expect certain output of the hg client in case of an HTTPLockedRC exception.
setup: don't use setuptools 34 - it has indirect conflicts with the celery version supported on the stable branch (Issue #266)
Setuptools==34 requires packaging>=16.8 which has an unconstrained requirement of pyparsing ... but actually it doesn't work with pyparsing==1.5.7 ... which is required by celery<2.3 ... which this version of Kallithea requires.
Celery has been upgraded on the development branch but we don't want to do that on the stable branch.