config: tweak template http_server conditionals - don't leave an empty section when using UWSGI
The web servers run by 'gearbox serve' share the same basic configuration:
[server:main] host = ... port = ... use = egg:... ...
UWSGI doesn't use gearbox and doesn't use a [server:main] section but will read it's own section:
[uwsgi] http = ...:... ...
Before, 'make-config my.ini http_server=uwsgi' would create an empty [server:main] section, and only after 6a5fb5070765 it stopped putting unused host/port information there.
Now, let everything but UWSGI share the [server:main] template section and put host/port first. Everything UWSGI is handled in its own conditional template section.
- use jQuery instead of YUI in js code - since the old library used seconds and the new library uses ms we have to make timestamps 1000 times bigger - the new library uses a slightly different data format
auth: consume request body before responding 401 or 403 during authentication
In order to work correctly with reverse proxies like Apache, the application needs to consume the whole body before returning and closing the connection. Otherwise the reverse proxy may complain about a broken pipe.
For example, if the client sends a lot of data and kallithea doesn't read all that data before sending 401, the connection will be closed before the reverse proxy has sent all the data. In this case an apache reverse proxy will fail with a broken pipe error.
This is not necessary for all wsgi servers. Waitress automatically buffers (and therefore reads) all the data and uwsgi has a 'post-buffering' option to do the same. But AFAIK there is no way to push to a password protected hg repository when using gunicorn without this changeset.
commands: remove custom (re-)initializing of database model
The model has already beeen initialized through TurboGears after make_app_without_logging has been called.
Since kallithea.lib.utils2.engine_from_config no longer is used, remove it. All remaining references to engine_from_config are directly to the one from sqlalchemy.
diffs: move as_html and _safe_id from method to a pure function - avoid calling the method as function
as_html was sometimes used in a way where we actually don't want to use the whole DiffProcessor - we just created a dummy instance and passed custom input as parameter to the instance method.
Instead, make it the function we apparently want.
Make it clear that as_html not just returns a "diff" but that it is a html diff.
diffs: do check of cut_off_limit on the raw diff before any parsing and allocation
This redefines the exact meaning of cut_off_limit but it will still be approximately the same thing. It makes it more a limitation of what amount of work should be done, more than how much html should be outputted.
It could make sense to push the limit further back to vcs to also prevent computing or allocating memory for huge diffs.
diffs: inline prepare() into __init__ and make the result available as .parsed
Make it more clear what the DiffProcessor is: Something that works on a raw diff as input, mainly compute when initialized, and returns an object where the result is available in different ways.
diffs: avoid extra copy of diff when splitting into files
Instead of allocating memory for an extra copy of the whole raw diff, just use buffer() to give read-only string views of the relevant sections of the big diff string given as input.
diffs: cleanup of variable naming around cut_off_limit
A brief summary of this area:
The base controller sets self.cut_off_limit from config and is used for diffs, unless controllers are given a fulldiff query parameter. In a few cases, these are passed to templates as c.cut_off_limit or c.fulldiff . Also, if the diff function returns a LimitedDiffContainer, c.limited_diff is set so the UI can report the data set is partial.
templates: back out the DataTables paging disabling from c5bff92d5084
Lessons learned: * It was not sufficiently tested with gravatars. * It was not sufficiently tested on Chrome - Firefox seems to handle some big pages better. * It is unclear how big amounts of data we want to optimize for, but this regressed too much for some cases.
ini: drop insertion of header comments in generated ini files
The header comments were kind of redundant and could easily get out of sync. Also, we are moving towards just generating files and don't need this and don't want to maintain it.
ini: add support for adding extra entries to sections
When adding entries to an existing section, assume that comments and empty lines *before* a section header belongs to that section. That is an approximation; not always entirely correct, but the best we can come up with.
ini: drop redundant template update of [handler_console] level=DEBUG
After 05d09cec7b26, the handlers in the default template no longer had a 'level' setting the ini generator could override with DEBUG. We are apparently fine without it, so just remove the last traces.
templates: disable paging and page size controls in DataTables
Since all data is sent to the client side, it can just as well be rendered. PageDown or scrolling is more convenient for paging than paging buttons, so short of "infinite scrolling", showing all entries is an acceptable solution ... especially when it very rarely is hundres of entries.
There could perhaps be further changes to how DataTables is shown - this is one step.
index: show repositories and repository groups in the same table
Having two different tables with their own paging and search gave a bad UI. Instead, do like all other UIs that show directory content and show both "folders" and "files" in the same list.
The rendering of repo groups is changed to use js data instead of a taking data from an html table.
Repository groups are shoe-horned into the repository DataTable. The columns are no perfect match - some of the existing columns are thus given an empty default value.
repogroups: fix display of links to parent repo groups
Links inside panel-primary were shown with the same color as the panel background. Arguably, that's a bug in Bootstrap, but marking it as panel-title fixes that.
pygments: reimplement get_lem under the name get_extension_descriptions
The old implementation was cryptic and over-engineered. And reduce() is not a builtin in Python 3.
This function works on static input, and it was verified that this implementation returns exactly the same as the old one. And is simpler and slightly more readable. In my opinion.
py3: replace list comprehension with for-loop due to scope
The scope of the list comprehension variable 'x' will be limited to the list comprehension in python3. Thus switching to a full loop (without this scope restriction) in preparation for python3.
utils: remove Session after we are done using it in set_app_settings
When the Kallithea WSGI application or celeryd is started, TurboGears app_config.make_base_app calls Kallithea app_cfg.setup_configuration which runs utils.set_app_settings. That function will read settings from the database and store them in the global config. It uses a database session which is created on demand, but this session was not dismissed but left around for the next thing that asked for a database session. MySQL will by default close connections after 1 hour, so when celery tried to run a task after 1 hour of inactivity, it could fail because of the closed connection.
utils.set_app_settings must thus remove the Session after use, just like auth.set_available_permissions do.
This will thus fix for example some MySQL connection problems seen with Celery.
tests: use temporary copy of test.ini, possibly customized for TEST_DB
It was an undocumented feature that if setting the environment variable TEST_DB, that would be used for tests instead of the default kallithea/tests/test.ini sqlalchemy.url . That did however not work for Git hooks and tests would fail.
Instead, create a copy of test.ini in the temp folder and use that for testing. If TEST_DB is set, edit the file so the specified DB URL is used. This fixes Git hook related tests if TEST_DB is used.
Since this also changes the path of %(here)s to the temporary location, just use the default paths.
This also has the advantage that the data folders are now in the temp folder as well. Therefore a broken data folder, from a past run, can no longer influence a test.
templates: tweak file edit page markup to be slightly more correct and make it look slightly better
- use <label> for the commit message label - use .form-group - use form-inline only for the heading - use form-group-sm to make the <select> less bloated
templates: jQuery DataTables need explicit width="100%" for resizing to work correctly
With Firefox, the resizing worked when making the browser window smaller, but not when when making the browser window larger. With Chrome and other browsers, neither worked.
summary: use shared changelog_table implementation on summary page
The main purpose of this change is to simplify code and avoid duplication.
Using the shared implementation also happens to give a slightly different view of the changesets. There will be no table headers, the columns are shown in different order, author gravatars are shown, commit messages can be expanded, and there is slightly different styling.
Small remaining amounts of template code is moved from changelog_summary_data.html to summary.html.
changelog: make the table a re-usable template element
This moves the changelog table to a separate template file, without further changes, except the necessary renaming to make it fully parameterized without accessing c directly.
controllers: consistently use c.cs_comments and cs_statuses
c.comments and c.statuses were also used for lists of comments and statuses. To get more consistency and avoid confusion and conflicts, use different for names for mappings from changeset hashes.
Split commit message into 2 <div> elements: one with just the first line, and one with the full message. This makes it easier to handle the toggling of showing the whole commit message. This way, we no longer have to mess with the line height, and we avoid problems with cutting off parts of the text below the text baseline.
Also stop cutting off the first line if it is too long for the table column.
It adds a bit of overhead to store the first line twice, but having them separate make things simpler and is worth it.
Under some circumstances, the floating of the elements gets altered, if the tooltip elements gets put next to the source element. Therefore use body as the container.