gist: make it a bit more clear how gist_access_id is used ... and how it is different from gist_id

A gist has a gist_access_id which gives access to it. For private Gists, it is
a multi-letter secure random string.

gist_id is the primary key in the database and thus an automatically
incrementing integer. It is also used as the not-so-secret gist_access_id for
public gists.

This gets rid of one odd safe_unicode applied to an int.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

lib: refactor _get_access_path as get_path_info

We will need it later when it gets more tricky to get the path from environ ...

lib: avoid comparing ordering with None

Fails in py3.

lib: mktime needs a tuple - not a list

Py3 is more strict.

lib: base64 followup from 82b1eaec25f5

Needed for py3.

lib: avoid import string.replace

Odd way to do it ... and gone in py3.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

vcs: make hg get_changesets compatible with py3

If start was 0, we would end up with start_pos None and getting a compare with
None which would fail in py3.

Instead, make sure start_pos is None iff start is None.

Also, make the actual check more robust.

vcs: change check_integrity to work on paths

Py3 failed because nodes can't be hashed.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

vcs: change get_diff to use _run_git_command directly

This will allow run_git_command to return strings while get_diff still return
raw bytes.

lib: let get_git_version invoke git as all other commands do, without special options

There is no need for _bare or _safe. It is fine to have '-c
core.quotepath=false' before '--version', and it is perfectly fine to get a
RepositoryError if things go terribly wrong.

vcs: use more correct git revision identifiers

It will have the right types and make a difference for py3.

vcs: replace __getslice__ with slice handling in __getitem__

Needed for py3.

vcs: fix __eq__ to return false if types are different

Follow up to ae12fabba699.

vcs: drop Repository.name_unicode

It wouldn't make much sense in py3 ... and is barely used.

vcs: replace "has_key" with "in" in decorator skip criteria

Needed for py3.

tests: fix ordering of get_inline_comments output

Py3 would give different ordering.

tests: fix test_files.py json comparison to load json to struct so it is independent of serialization order

Py3 would fail the test because different ordering.

tests: sort groups in test_enforce_groups check to make it independent of ordering

Py3 would give another ordering.

tests: fix ordering of options of "ret_type must be one of ..." returned from api

Py3 would fail becauase it use another ordering.

tests: avoid unicode u'' repr in test_diff_markup output

It will go away with py3 anyway.

tests: refactor test_redirection_to_login_form_preserves_get_args to test more correctly

Preparing for py3.

lib: introduce string conversion functions for ASCII without further encoding concerns

Avoid the trial-and-error and vagueness of the "safe" functions.

This should replace some use of safe_unicode and safe_str. It will mostly be a
noop in py2 but will be crucial in py3.

lib: establish py3 compatible strategy for string handling: introducing safe_bytes and deprecating safe_str

The meaning of safe_str will change when moving to py3. All use of safe_str is
thus tech debt that we have to chop off, mostly by moving to either
safe_unicode or safe_bytes ... or dropping because we know what we are doing
and rely on the improved type safety in py3.

vcs: drop unused _diff_name_status

It popped up as something to do for py3. Don't waste time on it.

lib: some cleanup of utils.py imports

Utils really shouldn't rely on model ...

hg: bump minimum version to 5.1

We will soon move to Python 3 which only will support 5.1 or later.

Remove old hacks and tech debt.

Also avoids future warning:
DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

py3: only use safe_str for string conversion - not for arbitrary __str__ invocation

Usually, we actually just want unicode string, and it is better to just unicode().

lib: only maintain one copy of safe_str / safe_unicode

The standalone-ish nature of vcs gets a bit in the way. It already depends on
some very generic Kallithea functionality. But for now, avoid code duplication,
and let Kallithea use vcs functionality instead of duplicating it.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

feeds: replace webhelpers.feedgenerator with simple mako templating for rendering RSS/Atom XML

Most of the complexity in RSS libraries is in dynamically supporting all kinds
of attributes. For our use, we have a small static set of attributes we use,
and it is simpler to just use mako.

Also, webhelpers is dead, and the alternatives seem quite heavy.

repo: don't just report user name and email in one field - separate things properly

In the repo RSS feed, report author as
<author>name@example.com (User Name)</author>
instead of using
<dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">User Name &lt;name@example.com&gt;</dc:creator>

And in the ATOM feed with name and email separate:
<author>
<name>User Name</name>
<email>name@example.com</email>
</author>
Instead of
<name>User Name &lt;name@example.com&gt;</name>

journal: don't include email in author name - avoid double data

In journal RSS feed, report author as:
<author>name@example.com (User Name)</author>
instead of double email due to:
<author>name@example.com (User Name &lt;name@example.com&gt;)</author>

In the journal ATOM feed, report author as:
<author>
<name>user Name</name>
<email>test_admin@example.com</email>
</author>
instead of using double email due to:
<name>name@example.com (User Name &lt;name@example.com&gt;)</name>

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

py3: prepare for types.NotImplementedType going away

From 2to3 types.

py3: use explicit octal literals

From 2to3 numliterals.

model: use unicode for internal rendering of repo snippets

Preparing for py3.

cleanup: use isinstance instead of comparing types

From 2to3 idioms.

tests: fix some odd source formatting / spacing around ','

From 2to3 -f ws_comma.

changeset: store hexified source hash in context so it is readily available in template

Avoid hex as "encoding" - it is not supported in py3.

Mercurial store source and transplant_source differently ... but that will now
be handled in the controller instead of in the template.

home: don't send _changeset_cache in repo_switcher_data result

It is unused, and it is problematic to serialize bytes in json ... and it is
especially obvious in py3.

lib: drop redundant json.dumps encoding=utf8

Not supported in py3 - json is by definition(?) always UTF-8.

app: drop finally handling in BaseVCSController.__call__

Our optional 'wrapper' middleware is logging response times more correctly
after b42ee1bdf082 - there is no point in logging the timing of the __call__ in
the main app.

Similarly, the session was removed too early. But TurboGears is already picking
up our db session and using DBSessionRemoverMiddleware to .remove() it ... and
at the right time. So just stop trying.

utils: drop make_ui parameter clear_session - this utility function should never mess with sessions

clear_session seems to have been based on the idea that utility functions that
happened to create a session also should clean up. But instead, we now
consistently take care of session removal at a high level.

app: let app instance creation remove its SA Session when done

The app instance is usually created early or on demand before serving the first
request, and the app instance might sit idle for a while before receiving the
first request. The app creation will touch the database, and we have to
consider the lifetime of the database session it is using. The cbf524e4c1a3
commit message gives some reasons we should remove the session when done.

But instead of handle it inside utility functions, we would rather handle
sessions at a high level as here.

home: drop disabled caching of repo_switcher_data

The caching condition has been False since ffd45b185016 . That makes sense -
caching as implemented would leak the repo list of other users without
considering permissions. The repo list is however only loaded on demand, and
caching is not that important.

routing: fix files_annotate_home annotate value to be compatible with Routes >= 2

The routing entry for files_annotate_home had annotate=True. That primarily
served to let the controller files.index differentiate files_annotate_home from
files_home and files_home_nopath . Anything true can work.

test_files.py is creating files_annotate_home URLs in an odd way: Instead of
explicitly specifying it is a files_annotate_home URL, it passes expected
controller parameters and expects routing to find a URL that would provide
these parameters. It thus also has to specify the otherwise "invisible"
annotate value. For Routes < 2, True works just fine. For Routes >= 2, it seems
to expect values that actually can be encoded in URLs.

Thus, instead of True, use '1'.

routing: drop default f_path for changelog_file_home

There is no point in creating a changelog_file_home URL without specifying
f_path. The default value of None is thus never any good. And especially, it is
unclear what a value of None really means.

routing: drop action='index' - that is already the default

These default actions were already left out in some places - now be consistent
about it.

pygrack: address webob AcceptValidHeader DeprecationWarning

Got warnings like:
kallithea/tests/other/test_vcs_operations.py::TestVCSOperations::test_yada_yada
.../site-packages/webob/acceptparse.py:649: DeprecationWarning: The behavior of AcceptValidHeader.__contains__ is currently being maintained for backward compatibility, but it will change in the future to better conform to the RFC.
DeprecationWarning,

There *must* be a better alternative to pygrack ... for now, let's keep it alive.

page: drop most paginate customizations - the bare implementation seems good enough

The main difference from this seems to be that redundant controls (such as
"previous" on first page) no longer will be shown.

default_link_tag is based on the base class implementation with minimal
changes, wrapping links in <li> as before, and with different handling of
current_page to emit markup as before.

page: minimal change to move from webhelpers.paginate to paginate

webhelpers is dead and doesn't work with py3. paginate is not very actively
maintained, but it is the natural successor to webhelpers.paginate, it seems
stable, and it works with py3.

This is a minimal change that seems to work. It preserves existing tech debt
... and adds a little bit more. It will be cleaned up next.

webhelpers.paginate had built-in SqlAlchemy support - now we have to handle it
explicitly.

page: consistently invoke Page with kwargs instead of explicitly providing a custom url generator

Drop the option of providing an url maker as 'url', and thus also hardcode the
assumption that the page parameter is called page.

page: pass url query params to Page instead of passing request.GET.mixed() to .pager

Standardize on one transparent way to use Pager:

Avoid passing random parameters to .pager() ... or any parameters at all.

page: replace RepoPage with Page given the reverse collection

That seems to be all RepoPage did ...

We must still take care to make sure the collection works correctly, even when
filtered so indices might be higher than repo length. vcs module takes care of
that by internally creating a list of hashes (which it can reverse), while the
Changeset instances are only created on demand. We can save some resources by
not retrieving the whole list of Changesets just to reverse it so we can use a
few entries.

summary: handle repo like changelog does

Show "There are no changesets yet" instead of relying on pager to silently eat
EmptyRepositoryError. Use .get_changesets() instead of using
c.db_repo_scm_instance directly.

vcs: fix get_changesets to .reverse() list of revision hashes in place instead of using reversed()

This is slightly more performant, and will also make CollectionGenerator work
with reverse=True. Before, CollectionGenerator was used on get_changesets, but
never with reverse option. Trying to use reverse, it would fail when applying
len():

TypeError: object of type 'listreverseiterator' has no len()

db: migration step after 95c01895c006 failed to add usk_public_key_idx in alembic step b74907136bc1

Use meta reflection to check if the indices exists before running the
upgrade step. The upgrade step only has to be run if it is an old database -
not if it has been created after the schema changes were introduced.

db: introduce migration step after 93834966ae01 dropped non-nullable inherit_default_permissions

The database migration step was lazily and naively skipped ... but that turns
out to be a problem when new users are added.
In the database, the original column 'inherit_default_permissions' was
marked as non-nullable without default value. In the Kallithea code after
commit 93834966ae01, the column 'inherit_default_permissions' was no longer
known, and thus not given a value when new users are added. As a result, the
database complained:

IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "inherit_default_permissions" violates not-null constraint

Fix that now by adding an appropriate db migration step to actually remove
the columns.

Use meta reflection to check if columns exist before running the upgrade
step. The upgrade step only has to be run if it is an old database - not if
it has been created after the schema changes were introduced.

For the downgrade step, make sure to set a default value for non-nullable
columns.