Sometimes, when reusing cached data, the DOM part of select2 activation would be reused, but not the actual activation. Repeated select2 activation would thus give duplicated DOM entries.
Select2 doeesn't seem to have a good way to unload or redo, so instead just try to remove the old DOM parts.
settings: rework logic for flash message after repository scan
Make the code more readable by extracting added_msg and removed_msg away from the h.flash() call and reindenting the logic. There are no functional changes here.
These changes serve as preparatory work for a subsequent commit that will change the logic.
Commit d66201a7ce6e ("files: change "callbacks" function to the more descriptive name "post_load_state" and let it take an actual state data object") broke the 'Show Authors' button when visiting a file in the files browser. The button normally shows a count of authors and their avatars, but after the aforementioned commit it did nothing.
Following patch would have fixed the problems in commit d66201a7ce6e:
diff --git a/kallithea/templates/files/files_source.html b/kallithea/templates/files/files_source.html --- a/kallithea/templates/files/files_source.html +++ b/kallithea/templates/files/files_source.html @@ -81,11 +81,13 @@ <script> $(document).ready(function(){ var state = { + data: { node_list_url: node_list_url.replace('__REV__',${h.js(c.changeset.raw_id)}).replace('__FPATH__', ${h.js(h.safe_unicode(c.file.path))}), url_base: url_base.replace('__REV__',${h.js(c.changeset.raw_id)}), rev: ${h.js(c.changeset.raw_id)}, f_path: ${h.js(h.safe_unicode(c.file.path))} + } } - post_load_state(State.data); // defined in files.html + post_load_state(state.data); // defined in files.html }); </script>
But, later the code got refactored more, and commit 006d68c4d7b9 ("files: use the web browsers built-in js history instead of native.history.js") broke the feature further: the click handler for the button no longer got installed on the 'document-ready' event, but only when a new 'state' is loaded. And it seems there is never a situation where a new state preserves the button, so it makes no sense installing the click handler at that moment.
Instead, move the click handler back to the 'document-ready' event.
cli: fill in git_hook_interpreter at 'config-create' time to really fix potentially invalid interpreter in git hooks (Issue #333)
When generating a configuration file using 'kallithea-cli config-create', it is probably using the right Python interpreter, so fill in the current Python executable as 'git_hook_interpreter' in the generated file.
There should thus rarely be any need to configure this manually, and issue #333 will *really* be fixed.
As this causes an absolute path to be encoded inside the ini file, moving the virtualenv will require updating this path.
For development.ini we do not want to hardcode any path and are happy to leave it using the old heuristics at runtime.
hooks: make the Python interpreter for Git hooks configurable as 'git_hook_interpreter' (Issue #333)
Commit 5e501b6ee639 introduced the use of 'sys.executable' as interpreter for git hooks instead of 'python2' with the following argument:
"Windows doesn't necessarily have "python2" available in $PATH, but we still want to make sure we don't end up invoking a python3. Using the absolute path seems more safe."
But, sys.executable does not necessarily point to Python. When Kallithea is started under uWSGI, sys.executable points to the uwsgi executable. As a result, the interpreter encoded in the git hooks on the server repositories would be:
#!/path/to/uwsgi
And pushing to such repo would result in following client errors:
$ git push Password for 'http://user@localhost:5050': Enumerating objects: 3, done. Counting objects: 100% (3/3), done. Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: unable to load configuration from hooks/pre-receive To http://localhost:5050/gitrepo-new ! [remote rejected] master -> master (pre-receive hook declined) error: failed to push some refs to 'http://user@localhost:5050/gitrepo-new'
Fix this problem by introducing a configuration setting 'git_hook_interpreter' that allow administrators to specify which Python interpreter to use.
A subsequent commit will cause its value to be filled in automatically when generating a new ini file, but an administrator can always override it.
The use of /usr/bin/env is only needed for relative arguments (or to pass variables in the environment, which we don't do). It is thus not needed in case the Python interpreter for Git hooks is known with absolute path, as introduced in 5e501b6ee639.
hooks: add intermediate function _get_git_hook_interpreter
The logic to determine the right interpreter for Git hooks is about to change and possibly become more complex. Split it off in a separate function so such changes do not require extra code duplication and preserve the readability of the code.
In TurboGears2 2.3.12, the latest version in the 2.3.x range, the WebOb dependency requirement is [1] WebOb >= 1.2, < 1.8.0
In TurboGears2 2.4.0 (which is in pre-release state at the time of this commit), this becomes [2]: WebOb >= 1.8.0, < 1.10.0
In the Kallithea dependency list, we have matched our WebOb version requirements to that of TurboGears2 and use: WebOb >= 1.7, < 1.8
while our TurboGears2 requirement was liberal and accepted anything in the 2.x range: TurboGears2 >= 2.3.10, < 3
To avoid new Kallithea installations failing with conflicting WebOb version requirements after TurboGears2 2.4.0 is released, restrict the version of TurboGears2 to 2.3.x on the stable branch.
For the default branch, the update to TurboGears2 2.4.0 can be considered once it's released.
Commit f2f7a8c1281e changed the i18n-related ini settings to match TurboGears2. Even though the commit message correctly refers to 'i18n.enabled', the code incorrectly used 'i18n.enable'.
.hgtags: remove accidental unused double tag of 0.4.0rc1
Initially, when tagging 0.4.0rc1 I made a mistake locally, fixed it, then stripped the tagging commit, and retagged. However, it seems something went wrong in this procedure and the original commit also is shown in .hgtags.
This commit never got pushed, is hidden (obsolete) in my local repository, so remove the corresponding line in .hgtags to avoid any confusion.
git: fix handling of submodules that are not in the repo root (Issue #337)
GitChangeset.get_nodes() did not handle submodules correctly if they were not located in the repository root. The file .gitmodules was searched in the 'tree' object being handled, which would be a subdirectory, while the real .gitmodules file is in the root tree of the repository.
Instead of using 'name', the 'path' should be used.
This problem was noticed during indexing of such repositories.
sphinx 1.7.9 has requirement babel!=2.0,>=1.3, but you'll have babel 0.9.6 which is incompatible. sphinx 1.7.9 has requirement docutils>=0.11, but you'll have docutils 0.8.1 which is incompatible. sphinx 1.7.9 has requirement Pygments>=2.0, but you'll have pygments 1.5 which is incompatible.
Fix these by bumping the minimum versions of these dependencies.
this duplication replaced useful diagnostic message from pip with less useful ones. For example, the following message was displayed when the Babel dependency duplication is present:
kallithea 0.4.0rc1 has requirement Babel<2.7,==0.9.6, but you'll have babel 2.6.0 which is incompatible.
When removing the duplication in dev_requirements.txt, this becomes:
sphinx 1.7.9 has requirement babel!=2.0,>=1.3, but you'll have babel 0.9.6 which is incompatible.
which makes it clear that to solve this problem, we need to bump the minimum dependency for Babel in setup.py from 0.9.6 to 1.3.
Use less spacing after comment panels inside comment sections (10px from @kallithea-panel-margin instead of 20px) to hint that the comments are related ... and also to avoid using too much vertical space.
Also show Comment/Cancel buttoms closer to the comment form ("well") they are related to.
cli: fix 'front-end-build' on Windows (Issue #332)
On Windows, the command 'npm' is actually 'npm.cmd', a script and not a PE executable. 'subprocess' will not resolve 'npm' into 'npm.cmd', while it would resolve e.g. 'git' into 'git.exe', as the latter _is_ a PE executable.
One solution is to change all references to the problematic scripts by adding the '.cmd' extension explicitly, but this would not be compatible with UNIX systems and thus require special handling.
On Windows, the problem can be solved by passing 'shell=True' to subprocess. On UNIX, we don't need shell=True and prefer shell=False. A solution that fits both cases is 'shell=kallithea.is_windows'.
Note: on Windows, next to the 'npm.cmd' file (and same for license-checker etc.) there is also a file 'npm' (without extension). It is a shell script (interpreter /bin/sh) for use on Windows with mingw/msys/cygwin. This script will nevertheless never be used by the standard Windows command prompt and is not used by Kallithea.
docs: suggest creating a new virtual environment for major upgrades
While not strictly necessary, it may be interesting to start from a new virtualenv rather than upgrading inside the old one. This will remove old packages once installed manually but no longer necessary. At the same time, it makes a rollback of the upgrade process more easy.
scripts/make-release: fix PyPI upload by using twine
Upload via 'python2 setup.py sdist upload' is deprecated.
Worse, for unknown reasons it stopped working for 0.4, even though 0.3 did pretty much the same.
Following output was given:
WARNING: Uploading via this command is deprecated, use twine to upload instead (https://pypi.org/p/twine/) Traceback (most recent call last): File "setup.py", line 160, in <module> """, File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup return distutils.core.setup(**attrs) File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/command/upload.py", line 26, in run orig.upload.run(self) File "/usr/lib64/python2.7/distutils/command/upload.py", line 62, in run self.upload_file(command, pyversion, filename) File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/command/upload.py", line 136, in upload_file value = str(value).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 825: ordinal not in range(128)
The error is pointing to a unicode character in the README.rst file.
The proposed 'twine' command does not have this problem. As it seems that this is the future, we won't waste more time investigating the problem with 'sdist upload', and start using twine from now on.
Useful for release announcements, shortlog.py presents a list of committers corresponding to the specified revision set, along with their count of commits in that set.
Example usage: scripts/shortlog.py "only('.', branch('stable') & tagged() & public() & not '.')"
But in fact, there is no real reason to mention any one service. In fact, since the Kallithea project does not endorse any service, it is better to not list anything.
utils: let make_ui return an empty ui.ui() object if the requested hgrc file isn't present
89c30b145bb8 lost an "if repoui" in simplehg, and Kallithea would thus fail on serving repos without a .hg/hgrc :
File ".../kallithea/lib/middleware/simplehg.py", line 285, in _augment_hgrc for k, v in repoui.configitems(section): AttributeError: 'bool' object has no attribute 'configitems'
It is *only* simplehg that invoke make_ui with read_from 'file'. In this case it could return False if the requested file wasn't present.
It was odd and inconsistent that make_ui with read_from 'file' either returned an ui object or the value False.
To fix the problem, just let make_ui return the empty ui.ui() if no .hg/hgrc is present. An ui object that just doesn't have any config sections is much better than any alternative.
This causes 'python2 setup.py update_catalog' to fail with:
File "/usr/lib64/python2.7/site-packages/babel/core.py", line 1094, in parse_locale raise ValueError('expected only letters, got %r' % lang) ValueError: expected only letters, got u''
i18n: how_to: add instructions to regenerate translations
The wrap value of '76' is determined empirically by checking the total diff after regeneration using wrap values 70 - 90, using 'wc' to find the smallest delta. It also corresponds with the help text of Weblate that says that the default is to 'wrap at 77 characters'.
For the last problem, the original string could not easily be found. Google Translate says: "It is not allowed to change the status of the changeset associated with the closed pull request" But as this is just a list of old translations, we can remove them without real impact.
/home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/fr/LC_MESSAGES/kallithea.po:6252: duplicate message definition... /home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/fr/LC_MESSAGES/kallithea.po:7: ...this is the location of the first definition msgmerge: found 1 fatal error
/home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/zh_CN/LC_MESSAGES/kallithea.po:6927: duplicate message definition... /home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/zh_CN/LC_MESSAGES/kallithea.po:9: ...this is the location of the first definition msgmerge: found 1 fatal error
compare: correct display of special branch names in initial placeholder
When a branch name contains special characters like '<' or '>', and a 'compare' operation is performed with such branch as one of the two compare sides, then the special branch name will be part of the URL, e.g.
The encoded branch name is then used at page load as placeholders for the branch selection dropdowns. But, the special characters, were escaped too much, causing '<' to become < in the display of the dropdown.
The placeholder was escaped via the default mako escape filter, before being passed to make_revision_dropdown, thus too early. We want the raw value. h.js() (copied from the default branch) gives us that, while still formatting and escaping the string so it is safe inside the script tag.
compare: prevent XSS due to unescaped branch/tag/bookmark names
In the revision selection dropdown of the 'Compare' functionality, the branch/tag/bookmark names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
templates/summary: escape branch/tag/bookmark names in 'Download as zip' links to prevent XSS
On a repository summary page, in the 'Download' section where you can download an archive of the repository at a given revision, the branch/tag names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
lib: sanitize HTML for all types of README rendering, not only markdown
The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized.
Therefore, move the sanitizing one level up so it covers all renderers, for now and the future.
This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
cleanup: remove unnecessary (and potentially problematic) use of 'literal'
webhelpers.html.literal (kallithea.lib.helpers.literal) is only needed when the passed string may contain HTML that needs to be interpreted literally. It is unnecessary for plain strings.
Incorrect usage of literal can lead to XSS issues, via a malicious user controlling data which will be rendered in other users' browsers. The data could either be stored previously in the system or be part of a forged URL the victim clicks on.
For example, when a user browses to a forged URL where a repository changeset or branch name contains a javascript snippet, the snippet was executed when printed on the page using 'literal'.
Remaining uses of 'literal' have been reviewed with no apparent problems found.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
pullrequests: prevent XSS in 'Potential Reviewers' list when first and last names cannot be trusted
If a user first or last name contains javascript, these fields need proper escaping to avoid XSS attacks.
An example scenario is: - the malicious user creates a repository. This will cause this user to be listed automatically under 'Potential Reviewers' in pull requests. - another user creates a pull request on that repository and selects the suggested reviewer from the 'Potential Reviewers' list.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
Technical note: the other caller of addReviewMember in base.js itself does _not_ need to be adapted to escape the input values, because the input values (oData) are _already_ escaped (by the YUI framework).
compare: correct display of special branch names in initial placeholder
When a branch name contains special characters like '<' or '>', and a 'compare' operation is performed with such branch as one of the two compare sides, then the special branch name will be part of the URL, e.g.
The encoded branch name is then used at page load as placeholders for the branch selection dropdowns. But, the special characters, were escaped too much, causing '<' to become < in the display of the dropdown.
It was not correct to use h.jshtml() to escape in the template. That applied html formatting, too much and too early. We want the raw value. h.js() gives us that, while still formatting and escaping the string so it is safe inside the script tag.
compare: prevent XSS due to unescaped branch/tag/bookmark names
In the revision selection dropdown of the 'Compare' functionality, the branch/tag/bookmark names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
base: escape branch/tag/bookmark names in 'Switch To' menu to prevent XSS
On repository pages, the 'Switch To' did not escape branches correctly.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks with .html_escape() .
templates/summary: escape branch/tag/bookmark names in 'Download as zip' links to prevent XSS
On a repository summary page, in the 'Download' section where you can download an archive of the repository at a given revision, the branch/tag names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
lib: sanitize HTML for all types of README rendering, not only markdown
The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized.
Therefore, move the sanitizing one level up so it covers all renderers, for now and the future.
This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
cleanup: remove unnecessary (and potentially problematic) use of 'literal'
webhelpers.html.literal (kallithea.lib.helpers.literal) is only needed when the passed string may contain HTML that needs to be interpreted literally. It is unnecessary for plain strings.
Incorrect usage of literal can lead to XSS issues, via a malicious user controlling data which will be rendered in other users' browsers. The data could either be stored previously in the system or be part of a forged URL the victim clicks on.
For example, when a user browses to a forged URL where a repository changeset or branch name contains a javascript snippet, the snippet was executed when printed on the page using 'literal'.
Remaining uses of 'literal' have been reviewed with no apparent problems found.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
pullrequests: prevent XSS in 'Potential Reviewers' list when first and last names cannot be trusted
The user information passed to autocompleteFormatter from select2 is the raw data which might contain HTML markup controlled by the user.
That could cause XSS issues, already when loading a PR page.
To avoid that, make sure autocompleteHighlightMatch always escape user information. That makes the user safe as long as a rogue user isn't selected ...
hg: improve implementations of `successors` and `precursors` properties of Mercurial changesets
* On Mercurial versions supporting it, make the properties return the closest changesets in the obsolescence chain that are in the repository. * On older Mercurial versions, fall back to the previous implementations.