cli: fix 'front-end-build' on Windows (Issue #332)
On Windows, the command 'npm' is actually 'npm.cmd', a script and not a PE executable. 'subprocess' will not resolve 'npm' into 'npm.cmd', while it would resolve e.g. 'git' into 'git.exe', as the latter _is_ a PE executable.
One solution is to change all references to the problematic scripts by adding the '.cmd' extension explicitly, but this would not be compatible with UNIX systems and thus require special handling.
On Windows, the problem can be solved by passing 'shell=True' to subprocess. On UNIX, we don't need shell=True and prefer shell=False. A solution that fits both cases is 'shell=kallithea.is_windows'.
Note: on Windows, next to the 'npm.cmd' file (and same for license-checker etc.) there is also a file 'npm' (without extension). It is a shell script (interpreter /bin/sh) for use on Windows with mingw/msys/cygwin. This script will nevertheless never be used by the standard Windows command prompt and is not used by Kallithea.
docs: suggest creating a new virtual environment for major upgrades
While not strictly necessary, it may be interesting to start from a new virtualenv rather than upgrading inside the old one. This will remove old packages once installed manually but no longer necessary. At the same time, it makes a rollback of the upgrade process more easy.
scripts/make-release: fix PyPI upload by using twine
Upload via 'python2 setup.py sdist upload' is deprecated.
Worse, for unknown reasons it stopped working for 0.4, even though 0.3 did pretty much the same.
Following output was given:
WARNING: Uploading via this command is deprecated, use twine to upload instead (https://pypi.org/p/twine/) Traceback (most recent call last): File "setup.py", line 160, in <module> """, File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in setup return distutils.core.setup(**attrs) File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/command/upload.py", line 26, in run orig.upload.run(self) File "/usr/lib64/python2.7/distutils/command/upload.py", line 62, in run self.upload_file(command, pyversion, filename) File "/tmp/kallithea-release-JtnfD/lib/python2.7/site-packages/setuptools/command/upload.py", line 136, in upload_file value = str(value).encode('utf-8') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 825: ordinal not in range(128)
The error is pointing to a unicode character in the README.rst file.
The proposed 'twine' command does not have this problem. As it seems that this is the future, we won't waste more time investigating the problem with 'sdist upload', and start using twine from now on.
Useful for release announcements, shortlog.py presents a list of committers corresponding to the specified revision set, along with their count of commits in that set.
Example usage: scripts/shortlog.py "only('.', branch('stable') & tagged() & public() & not '.')"
But in fact, there is no real reason to mention any one service. In fact, since the Kallithea project does not endorse any service, it is better to not list anything.
utils: let make_ui return an empty ui.ui() object if the requested hgrc file isn't present
89c30b145bb8 lost an "if repoui" in simplehg, and Kallithea would thus fail on serving repos without a .hg/hgrc :
File ".../kallithea/lib/middleware/simplehg.py", line 285, in _augment_hgrc for k, v in repoui.configitems(section): AttributeError: 'bool' object has no attribute 'configitems'
It is *only* simplehg that invoke make_ui with read_from 'file'. In this case it could return False if the requested file wasn't present.
It was odd and inconsistent that make_ui with read_from 'file' either returned an ui object or the value False.
To fix the problem, just let make_ui return the empty ui.ui() if no .hg/hgrc is present. An ui object that just doesn't have any config sections is much better than any alternative.
This causes 'python2 setup.py update_catalog' to fail with:
File "/usr/lib64/python2.7/site-packages/babel/core.py", line 1094, in parse_locale raise ValueError('expected only letters, got %r' % lang) ValueError: expected only letters, got u''
i18n: how_to: add instructions to regenerate translations
The wrap value of '76' is determined empirically by checking the total diff after regeneration using wrap values 70 - 90, using 'wc' to find the smallest delta. It also corresponds with the help text of Weblate that says that the default is to 'wrap at 77 characters'.
For the last problem, the original string could not easily be found. Google Translate says: "It is not allowed to change the status of the changeset associated with the closed pull request" But as this is just a list of old translations, we can remove them without real impact.
/home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/fr/LC_MESSAGES/kallithea.po:6252: duplicate message definition... /home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/fr/LC_MESSAGES/kallithea.po:7: ...this is the location of the first definition msgmerge: found 1 fatal error
/home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/zh_CN/LC_MESSAGES/kallithea.po:6927: duplicate message definition... /home/www-data/weblate-hosted/vcs/kallithea/kallithea/kallithea/i18n/zh_CN/LC_MESSAGES/kallithea.po:9: ...this is the location of the first definition msgmerge: found 1 fatal error
compare: correct display of special branch names in initial placeholder
When a branch name contains special characters like '<' or '>', and a 'compare' operation is performed with such branch as one of the two compare sides, then the special branch name will be part of the URL, e.g.
The encoded branch name is then used at page load as placeholders for the branch selection dropdowns. But, the special characters, were escaped too much, causing '<' to become < in the display of the dropdown.
The placeholder was escaped via the default mako escape filter, before being passed to make_revision_dropdown, thus too early. We want the raw value. h.js() (copied from the default branch) gives us that, while still formatting and escaping the string so it is safe inside the script tag.
compare: prevent XSS due to unescaped branch/tag/bookmark names
In the revision selection dropdown of the 'Compare' functionality, the branch/tag/bookmark names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
templates/summary: escape branch/tag/bookmark names in 'Download as zip' links to prevent XSS
On a repository summary page, in the 'Download' section where you can download an archive of the repository at a given revision, the branch/tag names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
lib: sanitize HTML for all types of README rendering, not only markdown
The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized.
Therefore, move the sanitizing one level up so it covers all renderers, for now and the future.
This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
cleanup: remove unnecessary (and potentially problematic) use of 'literal'
webhelpers.html.literal (kallithea.lib.helpers.literal) is only needed when the passed string may contain HTML that needs to be interpreted literally. It is unnecessary for plain strings.
Incorrect usage of literal can lead to XSS issues, via a malicious user controlling data which will be rendered in other users' browsers. The data could either be stored previously in the system or be part of a forged URL the victim clicks on.
For example, when a user browses to a forged URL where a repository changeset or branch name contains a javascript snippet, the snippet was executed when printed on the page using 'literal'.
Remaining uses of 'literal' have been reviewed with no apparent problems found.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
pullrequests: prevent XSS in 'Potential Reviewers' list when first and last names cannot be trusted
If a user first or last name contains javascript, these fields need proper escaping to avoid XSS attacks.
An example scenario is: - the malicious user creates a repository. This will cause this user to be listed automatically under 'Potential Reviewers' in pull requests. - another user creates a pull request on that repository and selects the suggested reviewer from the 'Potential Reviewers' list.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
Technical note: the other caller of addReviewMember in base.js itself does _not_ need to be adapted to escape the input values, because the input values (oData) are _already_ escaped (by the YUI framework).
compare: correct display of special branch names in initial placeholder
When a branch name contains special characters like '<' or '>', and a 'compare' operation is performed with such branch as one of the two compare sides, then the special branch name will be part of the URL, e.g.
The encoded branch name is then used at page load as placeholders for the branch selection dropdowns. But, the special characters, were escaped too much, causing '<' to become < in the display of the dropdown.
It was not correct to use h.jshtml() to escape in the template. That applied html formatting, too much and too early. We want the raw value. h.js() gives us that, while still formatting and escaping the string so it is safe inside the script tag.
compare: prevent XSS due to unescaped branch/tag/bookmark names
In the revision selection dropdown of the 'Compare' functionality, the branch/tag/bookmark names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
base: escape branch/tag/bookmark names in 'Switch To' menu to prevent XSS
On repository pages, the 'Switch To' did not escape branches correctly.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks with .html_escape() .
templates/summary: escape branch/tag/bookmark names in 'Download as zip' links to prevent XSS
On a repository summary page, in the 'Download' section where you can download an archive of the repository at a given revision, the branch/tag names were not correctly escaped.
This means that if an attacker is able to push a branch/tag/bookmark containing HTML/JavaScript in its name, then that code would be evaluated. This is a cross-site scripting (XSS) vulnerability.
Fix the problem by correctly escaping the branch/tag/bookmarks.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
lib: sanitize HTML for all types of README rendering, not only markdown
The repository summary page will display a rendered version of the repository 'readme' based on its file extension. In commit 5746cc3b3fa5, the rendered output was already sanitized when the input was markdown. However, also readmes written in other formats, like ReStructuredText (RST) or plain text could have content that we want sanitized.
Therefore, move the sanitizing one level up so it covers all renderers, for now and the future.
This fixes an XSS issue when a repository readme contains javascript code, which would be executed when the repository summary page is visited by a user.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
cleanup: remove unnecessary (and potentially problematic) use of 'literal'
webhelpers.html.literal (kallithea.lib.helpers.literal) is only needed when the passed string may contain HTML that needs to be interpreted literally. It is unnecessary for plain strings.
Incorrect usage of literal can lead to XSS issues, via a malicious user controlling data which will be rendered in other users' browsers. The data could either be stored previously in the system or be part of a forged URL the victim clicks on.
For example, when a user browses to a forged URL where a repository changeset or branch name contains a javascript snippet, the snippet was executed when printed on the page using 'literal'.
Remaining uses of 'literal' have been reviewed with no apparent problems found.
Reported by Bob Hogg <wombat@rwhogg.site> (thanks!).
pullrequests: prevent XSS in 'Potential Reviewers' list when first and last names cannot be trusted
The user information passed to autocompleteFormatter from select2 is the raw data which might contain HTML markup controlled by the user.
That could cause XSS issues, already when loading a PR page.
To avoid that, make sure autocompleteHighlightMatch always escape user information. That makes the user safe as long as a rogue user isn't selected ...
hg: improve implementations of `successors` and `precursors` properties of Mercurial changesets
* On Mercurial versions supporting it, make the properties return the closest changesets in the obsolescence chain that are in the repository. * On older Mercurial versions, fall back to the previous implementations.
validate-commits can be used to verify a series of commits before submitting/pushing it. It will create a virtualenv and run run-all-cleanup and pytest.
tests: fix assert rewriting in non-test modules like api_base.py
pytest rewrites assert statements in tests so it can print details about the values involved when the assert fails.
Since pytest 3.0.0, this was no longer the case for files/modules that are not discovered as test modules, i.e. starting with 'test_'. Examples are: api/api_base.py models/common.py base.py fixture.py
.coveragerc: fix reporting of coverage to match what is run
Without this change, the HTML report will show all lines specified in run.omit. As they have 0% coverage, this will negatively impact the overall coverage percentage calculated in the report.
Note: at this moment, we have an overall 74% test coverage. Test by installing pytest-cov and running: py.test --cov=kallithea --cov-config=.coveragerc --cov-report html and opening htmlcov/index.html in a browser.
In a fresh virtualenv on the stable branch, pastescript 3.0.0 is installed which depends on paste 3.0.x. Using this virtualenv to upgrade to the default branch, using 'pip install --upgrade -e .' fails because on the default branch, the paste version is restricted with '>= 2.0.3, < 3'. Following error occurs:
pastescript 3.0.0 has requirement Paste>=3.0, but you'll have paste 2.0.3 which is incompatible. ... Traceback (most recent call last): File "<string>", line 1, in <module> File ".../kallithea/kallithea-release/setup.py", line 160, in <module> """, File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File ".../kallithea/venv/kallithea-release/lib/python2.7/site-packages/setuptools/command/develop.py", line 36, in run self.install_for_development() File ".../kallithea/venv/kallithea-release/lib/python2.7/site-packages/setuptools/command/develop.py", line 117, in install_for_development self.run_command('egg_info') File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File ".../kallithea/venv/kallithea-release/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 270, in run ep.require(installer=installer) File ".../kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2307, in require items = working_set.resolve(reqs, env, installer) File ".../kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 854, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.VersionConflict: (Paste 2.0.3 (.../kallithea/venv/kallithea-release/lib/python2.7/site-packages), Requirement.parse('Paste>=3.0'))
The '< 3' restriction is introduced with commit e1ab826131334150b1f003e26de3207c34fc6e67 in January 2017, at which point 2.0.3 was the latest version. Version 3.0.0 was introduced in October 2018.
lib: BaseRepoController: use webob.exc.HTTPNotFound if possible
In the entire code base, the use of 'paste' is very limited. In particular, 'paste.httpexceptions' is only still used in lib/base.py, in three occurrences: - two of them in class BasicAuth which derives from paste.auth.something. Here it probably makes sense to use paste.httpexceptions. - one in BaseRepoController, which has no specific relation to paste. This occurrence can be replaced with webob.exc like in the rest of the code base.