scm: select options should be tuples - not lists with 2 elements
We use tuples in all(?) other places, so we should do the same here.
Since lists can be unpacked as tuples, it currently works to use 2-lists. But in a following change for "correct" handling of webhelpers.select when using webhelpers2.select, we want to give special handling to tuples vs strings vs Options. We don't want to add list to that list.
Performing ssh actions towards Kallithea via an IPv6 connection gave the following error:
$ hg incoming ssh://kallithea@example.com/repo remote: Traceback (most recent call last): remote: File ".../bin/kallithea-cli", line 11, in <module> remote: load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')() remote: File ".../python2.7/site-packages/click/core.py", line 764, in __call__ remote: return self.main(*args, **kwargs) remote: File ".../python2.7/site-packages/click/core.py", line 717, in main remote: rv = self.invoke(ctx) remote: File ".../python2.7/site-packages/click/core.py", line 1137, in invoke remote: return _process_result(sub_ctx.command.invoke(sub_ctx)) remote: File ".../python2.7/site-packages/click/core.py", line 956, in invoke remote: return ctx.invoke(self.callback, **ctx.params) remote: File ".../python2.7/site-packages/click/core.py", line 555, in invoke remote: return callback(*args, **kwargs) remote: File ".../kallithea/bin/kallithea_cli_base.py", line 79, in runtime_wrapper remote: return annotated(*args, **kwargs) remote: File ".../kallithea/bin/kallithea_cli_ssh.py", line 74, in ssh_serve remote: vcs_handler.serve(user_id, key_id, client_ip) remote: File ".../kallithea/lib/vcs/backends/ssh.py", line 65, in serve remote: self.authuser = AuthUser.make(dbuser=dbuser, ip_addr=client_ip) remote: File ".../kallithea/lib/auth.py", line 407, in make remote: if not check_ip_access(source_ip=ip_addr, allowed_ips=allowed_ips): remote: File ".../kallithea/lib/auth.py", line 860, in check_ip_access remote: if ipaddr.IPAddress(source_ip) in ipaddr.IPNetwork(ip): remote: File ".../kallithea/lib/ipaddr.py", line 76, in IPAddress remote: remote: ValueError: '2' does not appear to be an IPv4 or IPv6 address abort: no suitable response from remote hg!
This was caused by IPv4-exclusive parsing of the SSH_CONNECTION variable. With an IPv6 address starting with '2a02:1810:', only the first '2' would survive.
According to 'man 1 ssh':
SSH_CONNECTION Identifies the client and server ends of the con‐ nection. The variable contains four space-sepa‐ rated values: client IP address, client port num‐ ber, server IP address, and server port number.
So, the client IP address will be the first space-separated word, regardless of IPv4 or IPv6. Use that knowledge without further parsing.
kallithea-cli: use null handler to mute all console logging for ssh-serve
Augment logger_root to use the null handler when running ssh-serve. Set the log level to CRITICAL to filter early and minimize the amount of logging sent to the null handler.
It is however recommended to configure the system to use some logging facility that can handle logging for SSH access from many different processes.
kallithea-cli: introduce generic custom logging configuration for each cli command
Command line commands have different needs for logging than the server process.
To enable that use case, allow config sections to be enabled (and thus augment existing sections), depending on which kallithea-cli command is running.
kallithea-cli: set ssh_locale when creating .ini file so it doesn't have to be set manually before using ssh
The Python standard library does not seem to provide a simple way to determine the current locale without calling locale.setlocale() and thus potentially influencing the rest of the program.
Instead, try to determine the locale based on environment variables present when 'kallithea-cli config-create' is run, and use it to prepopulate the ini file.
In the SSH client configuration, the setting 'SendEnv' could contain variables like 'LANG', 'LC_ALL', and others. This causes these environment variables (with their values at the client-side) to be set in the server. However, not every locale setting valid in the client, is also valid on the server.
This could lead to the error: 'locale.Error: unsupported locale setting' when 'from mercurial import archival, merge as hg_merge, patch, ui' is called.
Fix this problem by providing an ini setting 'ssh_locale' that the user can set correctly, and which will be used to set LC_ALL and LANGUAGE in the 'kallithea-cli ssh-serve' process.
If an environment variable LC_ALL is set, it takes precedence over all other 'LC_xxx' variables, as well as over LANG. So, setting LC_ALL ensures that no user setting of 'LC_xxx' or 'LANG' could influence ssh-serve badly.
There is one environment variable that might overrule LC_ALL, specifically for showing messages: 'LANGUAGE'. GNU gettext lets it take precedence over LC_ALL [1]: "GNU gettext gives preference to LANGUAGE over LC_ALL and LANG for the purpose of message handling"
So, also set LANGUAGE to the same value as we set LC_ALL to.
The principle of setting a specific locale in the server process to fix this error, was first proposed by Dominik Ruf.
Text related to the case where the parent directory of 'authorized_keys' does not exist or is not writable, was contributed by Bradley M. Kuhn <bkuhn@ebb.org>.
ssh: add clone_ssh_tmpl setting when migration database for SSH
It is slightly dangerous to use the db model in the alembic migration script ... but in this simple case we do it anyway. It is also a bit dirty to modify an existing migration script, but in this case the changes are landing at once and it is only about making the migration process slightly more smooth.
This commit also incorporates a fix for Windows by Dominik Ruf, and better handling of the case where the parent dir of 'authorized_keys' does not exist or is not writable, by Bradley M. Kuhn <bkuhn@ebb.org>.
Error reporting when using SSH is different from when using HTTP. SSH doesn't use HTTP error codes but can write anything to stdout/stderr. Some tests thus need to have their expectations tweaked.
Based on work by Ilya Beda <ir4y.ix@gmail.com> on https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support . Bootstrap support, updates for POST methods, and tests by Anton Schur <tonich.sh@gmail.com>. Additional Bootstrap fixes by Dominik Ruf. Also heavily modified by Mads Kiilerich.
ini: create separate log handlers for color and null, and add comments hinting how they can be used
Let development.ini use color for the root logger as before. The special effect of color_sql was not visible with the default sqlalchemy log level of WARN, so just use color there as well.
clone_url: simplify the logic - move summary handling of different URLs with/without id to db
Drop the half-baked concept of a separate "clone_uri_by_id" - just always use a single template and change back and forth between {repo} and _{repoid} as necessary.
Make it clear that clone_uri_tmpl always is passed as argument.
clone_url: always pass a clone_uri_tmpl, with Repository.DEFAULT_CLONE_URI as last resort
clone_url() had a layering violation of using c.clone_uri_tmpl . This refactoring now makes it clear that this only was used from PullRequest.__json__(), so move the hack there and simplify it.
tests: introduce doctest_mock_ugettext to allow doctests of localized code
Future doctests will require some extra mocking, as the code-under-test uses translation (ugettext aka '_') and its provider TurboGears2 needs a context.
Avoid this complexity by mocking ugettext as the identity function. This is done by providing a pytest fixture 'doctest_mock_ugettext' that will mock uggettext in the module that uses the fixture.
locale: fix environment checks: LC_ALL has precedence over LC_CTYPE
'man 7 locale' describes following precedence:
If the second argument to setlocale(3) is an empty string, "", for the default locale, it is determined using the following steps:
1. If there is a non-null environment variable LC_ALL, the value of LC_ALL is used.
2. If an environment variable with the same name as one of the categories above exists and is non-null, its value is used for that category.
3. If there is a non-null environment variable LANG, the value of LANG is used.
So, if LC_ALL is set, it is used regardless of LC_CTYPE or LANG. Hence, when suggesting users what to change when their locale is invalid, we should also first check LC_ALL instead of LC_CTYPE.
The formencode version range included both 1.2.x and 1.3.x releases. However, since 1.3.0, _to_python and validate_python are deprecated and renamed to _convert_to_python and _validate_python, respectively.
With current pytest, these (long) deprecation warnings are shown in the test logs.
There are two options: - restrict maximum version to 1.2.x - bump minimum version to 1.3.x
In this commit we choose the latter approach, going towards the future rather than the past.
Upgrade notes for these libraries have not been investigated thoroughly, but testing seems to show that it works. We are also early in the development phase, so big problems will be caught by general testing before going wide.
Note: TurboGears2 is not upgraded to 2.4 yet. That upgrade would require us to first move away from using the Pylons compatibility layer.