Release Notes: Mitto Version 2.9.0¶
Breaking Changes¶
Before upgrading an existing Mitto system with
mitto upgrade,javacmust first be installed. This can be done via:sudo apt install openjdk-8-jdk-headless
This step is not necessary when running Mitto in a cloud provider image built with
packerusing Mitto release2.9.0-rc3or later.When running the
apt installcommand, if errors are encountered reconfiguring themittopackage, it may mean that Mitto was not fully installed prior to runningapt install. If such errors occur, runningmitto upgrade, despite theapt installerrors, may resolve the errors.
Issues Resolved in this Release¶
New Feature¶
Issue |
Summary |
Resolution |
|---|---|---|
Regex - show matching file info in logs at start of job. |
||
MongoDB input |
||
System Monitoring Job Type |
||
Create IncludeTransform |
Done |
|
Create ExcludeTransform |
Done |
|
Update styles for json-schema Drag and Drop element |
Done |
|
tojson, tojsonl code |
||
files - add pagination + search |
||
Move Create jobs wizard from modal to page like Sequence editor |
Done |
|
stretch the modals to give more room for editing (e.g. SQL) |
Done |
|
Hide Credentials for dbo |
Done |
|
set default output database / database when using SQL connector |
||
Duplicate button for sequences |
Done |
|
Allow comments in JSON configuration files |
||
Test amCharts FDG in Mitto |
||
Change control record in Mitto |
Done |
|
potentially create AWS IAM user for each new Mitto instance - to use the AWS api (client could whitelist IPs via Mitto instance for example) |
Done |
|
automatically altering column data types (string lengths) makes snapshots more difficult with current methods |
Done |
|
check source endpoint periodically (daily?) for endpoints that don’t have jobs in Mitto - alert the user and allow them to add those jobs’ |
Done |
|
UI for filtering history by date range |
Done |
|
based on the schedules, show in the UI (on the job manager page for example) the next time a job is supposed to run and the delta between when it actually runs |
Done |
|
Monitoring job can return results of SQL query |
Done |
|
ability to define default database output (localhost postgres is default) |
Done |
|
need a way to apply new messaging/notification settings to all existing jobs |
Done |
Improvement¶
Issue |
Summary |
Resolution |
|---|---|---|
Allow creation of report of a release’s Jira tickets without overwriting remainder of release notes. |
Done |
|
sql job - add SQL statement of job to job log on job run |
||
IMAP plugin icon |
||
Add Debian postinstall step to remove existing dataeditor plugin |
||
job edit - hjson - pretty format on save |
Done |
|
Change default sender for notifications |
||
Remove unused frontend code from the dataeditor. |
Done |
|
Links for “prod” version of documentation |
||
command job wizard - increase size of “command” box |
||
naming and sort order consistency |
Done |
|
update documentation to show ticket status |
||
xsv plugin wizard - remove “define the columns” screen and remove SDL from resulting job config |
||
query plugin wizard - remove SDL from resulting job |
||
tag filter - order alphabetically |
Done |
|
Mitto and TLS < 1.2 |
||
Modify frontend to call /metrics endpoints on websocket messages only. |
Done |
|
Use mitto.settings.TMP_PATH instead of /tmp |
||
add created_at and updated_at information to job page |
Done |
|
Add support for multiple versions of Mitto documentation |
||
Unify set of filter across pages |
Done |
|
wrap schedule text on jobs lists pages |
Done |
|
Add metrics for mitto instance |
||
File upload: modify version string |
Done |
|
monitor job type - add SQL tab |
Done |
|
Changes to generic job builtin to support comments in job configurations |
||
add credential id to credentials page |
Done |
|
JDBC input |
||
Swimlanes - add # of failed jobs and decimal precision |
Done |
|
Support removal of Schedules for Extracts in Tableau Online |
||
Add ability to prevent writing header for |
||
io - Unicode encode/decode handling |
||
Tableau Extract Refresh job type - list Extract Refresh schedules in alphabetical order |
||
Query - Add “create_engine” arguments |
||
mitto cli - add version |
||
add /api/key documentation to Swagger /docs |
||
ExtraColumnsTransform - Ignore columns not just objects |
||
Benchmark picke/json, JSON/UJSON |
Done |
|
add “custom” option to job timeouts |
Done |
|
Copy create_engine to apitoolbox |
||
graph - Add ability to filter by time range |
Done |
|
Graph - add filter by status, time range filters |
Done |
|
Job exited with returncode 143 - Nested sequence/job error |
Done |
|
Select all (multi-select) jobs |
Done |
|
Ace worker not found |
Done |
|
Command plugin - tag jobs created via wizard with “cmd” |
Done |
|
Excel plugin - wizard - output - move Schema above Table |
||
file manager - indicate the user can drag and drop in files (like csv plugin wizard) |
Done |
|
Named Credentials - Store Google Credentials as named credentials |
||
New release mechanism: |
||
Graph - Filter by tag (API support) |
Done |
|
wizard icons - Excel, Imap |
||
IO - Do not fetch table info on every step |
||
allow ReflectTransform to take extra SDL in job - remove lines 27-29 in existing code |
Done |
|
job timeout - add 3 hours and custom number of minutes |
Done |
|
add support for “end of” schedules - end of month, end of quarter, etc |
||
command job - add AceEditor for “cmd” |
Done |
|
Show line number in message when error is found in Job |
Done |
|
add Mitto job STEP to remove JSON store |
||
Feature Request: Mitto - Job Listing Page - Open RHS Job Panel |
Done |
|
Remove the “upgrade” action from the /api/pkg Mitto API endpoint |
Done |
|
snowflake - add QUERY_TAG for Mitto connections to snowflake |
Done |
|
named credentials - add databases as an option |
Done |
|
Consolidate pagination response format |
Done |
|
Cleanup mitto package.json scripts |
Done |
|
Graph - Filter by tag |
Done |
|
swimlane - currently running job should be at the top of the list and have a width for the duration it has been running |
Done |
|
add search to history page |
Done |
|
Improve SSL on nginx |
||
table viewer crosstab - fit to width of screen |
Done |
|
Add the :embed query string parameter |
Done |
|
add regex search to file manager files |
||
Implement simple pagination for the dataeditor |
Done |
|
show which table in the database the data editor is tied to |
Done |
|
set a default timeout for jobs? - SQL jobs for example most of the time complete within seconds |
Done |
|
add “last run” time taken column? |
Done |
|
Allow store = True/False |
||
[Mit 2110] Need a way to change the admin password in UX |
Done |
|
Mitto Sales Demo - Initial Password Setup instructions |
Done |
|
message notifications should probably only be turned on by default for sequences |
||
with pagination we can’t use CTRL+F to find specific jobs - will need wildcard job filtering at the same time as pagination |
Done |
|
fit Navigation menu wording |
Done |
|
Install haveged |
Done |
|
auto generated tags |
Done |
|
Allow Save & Start from the Save component |
Done |
|
need a way to add and then enable/disable all jobs in a sequence based on something (tag, etc) |
Done |
|
Add locking to dpkg.json |
Done |
Sub-task¶
Issue |
Summary |
Resolution |
|---|---|---|
Fix cron migration script. |
||
Add API endpoint to change password |
||
Mitto UI is not working on the fresh installation. |
||
Migrate GET api/v2/jobs to POST api/jobs/search |
Done |
|
Fix adding same job to sequence on front-end |
Done |
|
Time filter update |
||
Add new query parameter to the GET api/v2/jobs?exclude= |
||
GET /api/v2/jobs - pages: Stage, Jobs - need to add ability to filter by time ranges |
||
GET /api/graph - pages: Graph - need to add ability to filter by job’s type and by search term |
||
GET /api/v2/history - pages: Swimlanes, History - need to add ability to filter by job’s tags |
||
Check if websockets are receiving notify messages on the job edit. |
||
Fix navs endpoint |
||
Fix Jobs created without conf |
||
Fix GET about/config endpoint |
||
Migrate xsv bultin to plugin V2 |
||
Migrate dataeditor plugin to use APIv2 |
||
Migrate query plugin to use APIv2 |
||
Delete unused API v1 endpoints |
||
Switch the usage of old endpoint |
Done |
Task¶
Issue |
Summary |
Resolution |
|---|---|---|
upgrade from 2.8 to 2.9 fails due to library dependency |
||
Fix failed CircleCI builds - delete package-lock.json |
||
Investigate SyncStore memory consumption |
||
List API endpoints that the front end uses which are still being served by uwsgi |
||
Investigate python-rocksdb==0.6.9.1 differences. |
||
Bug¶
Issue |
Summary |
Resolution |
|---|---|---|
Upgrade fails if gcloud credentials are present and unencrypted. |
Cannot Reproduce |
|
remove ssh from jpype in requirements.txt |
||
job config - hjson - commenting last line of object moves that line to line above |
||
CircleCI jpype segfault |
||
sequence - jobs last completed all increase as jobs run |
Duplicate |
|
Job completion times for sequence are incorrect |
||
job logs - out of order by timestamp |
||
XSV plugin: Unclear error message if no XLS files present |
||
Credentials with invalid JSON are created |
||
jobs - run duration much longer than it should be |
||
database page - schemas don’t appear to be sorted |
||
jobs - tag filter - long tag text overflows |
||
webhooks - overflows it’s UI section on smaller screen |
||
jobs - multi select action throws js error - vendor-ba42ebbbb53f5017be2e.bundle.js:336 TypeError: ctrl.status is not a function |
||
Google sheets wizard - 404 Not Found on 2nd screen |
||
loading indicator flashes and interrupts user action while jobs starting |
||
sequence is created with “Invalid Value” for schedule. |
Duplicate |
|
ui and cli performance issues |
||
Excel wizard fails when invalid start_row specified. |
||
Credentials list on first screen doesn’t show up when clicking previous from the next screen |
||
job’s updated at should not update when a job is started |
||
system job - fails on 2.9 - KeyError: ‘jobs’ |
||
xlsx file with non-data sheets (data on sheet #5) causes wizard to fail |
||
sampling - 70% samples as integer - error = psycopg2.errors.InvalidTextRepresentation: invalid input syntax for integer: “0.7” |
||
global filters - multi selected values text outside button |
||
upgrade from 2.8.12 to 2.9.0-rc0 fails |
||
Snowflake - Object ‘TEST’ already exists. |
||
“builtin” test failures |
||
Fix sequence schedule ‘Invalid value’ |
||
job edit - saving returns this console error - vendor-4f8b5ddafc7b7d4a44e4.bundle.js:336 TypeError: Cannot read property ‘length’ of undefined |
Done |
|
json/jsonl inputs - json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1) |
Cannot Reproduce |
|
job edit - save & start - add indicator that something is happening |
Done |
|
query plugin wizard - add credentials to first screen (input) |
||
“Save” and “Save & Start” buttons intermittently fail to work |
Done |
|
data editor - bottom scroll bar |
||
database page - PostgreSQL listed 2x |
||
swimlanes - global type filter reverts on quick changes |
Done |
|
swimlanes - order of jobs should be by completed date desc, running at the top |
||
job history - status filter - make multi select and/or add “all” option |
Done |
|
change password - false error appears |
Done |
|
tableau extract refresh wizard - 500 error using existing credential |
||
files - uploading same file should overwrite existing file (or at least prompt) |
||
job schedules not firing |
||
jobs page - selecting job (gray background) and keyboard arrow keys |
Done |
|
The Dataeditor updates the wrong record |
Done |
|
Fix dataeditor pagination |
Done |
|
query job - output to snowflake - table names with mixed capitalization - KeyError: ‘test_snowflake_naming_MIXED’ |
||
sql job transaction parameter doesn’t do anything? |
Won’t Do |
|
HJSON - Job created with (working) hjson fails on creation |
||
sort order of a single job’s history is in ascending status id instead of descending status id |
||
data editor - change generic “database” to the name of the database of the data editor |
||
builtin.query.QueryInput uses incorrect comparison for upsert |
||
test issue - ignore |
Cannot Reproduce |
|
repeated GET /v2/metrics calls while idle in UI |
Done |
|
add /store endpoints to swagger /docs |
||
informatory message logged as WARNING |
||
job - schedule is blank on job creation (should say manual) |
Done |
|
job edit - pressing escape key throws error - Error: Transition Rejection($id: 39 type: 4, message: This transition is invalid, detail: The following parameter values are not valid for state ‘edit’: [id:undefined]) |
Done |
|
Excel plugin wizard - credentials should be directly under database on last screen |
||
Excel plugin wizard - unable to specify index for value for “sheet” |
||
command plugin wizard - first screen includes blank key and value |
||
sql - editing job - Uncaught TypeError: Cannot assign to read only property ‘length’ of string ‘ select 1;’ |
||
job edit - unable to save job - Uncaught TypeError: ctrl.conf.input.query.join is not a function |
Done |
|
job edit - command/ctrl + s no longer saves the job config |
Done |
|
Databases - Page load is very slow on Stage |
Done |
|
global filters - ran during - jobs text with relative date needs to be reworded |
Done |
|
global filters - search - needs minimum width |
Done |
|
global filters - ran during - value and text of start date incorrect |
Done |
|
jobs pages - filters in url parameter and on page, but not actually applying on page swap |
Done |
|
files page - typo on upload note |
Done |
|
swimlanes - metrics missing styling |
Done |
|
Fix jobs filtering |
Done |
|
test_redshift_upsert fails after PR#676 |
||
Fix mitto 0020 py migration error |
||
Query jobs - wizard should add stream_results input parameter with a reasonable default value |
||
Pickup SDL fixes made in 2.8.x |
||
Fix job conf validation - “init() takes exactly 3 positional arguments (2 given)”, |
||
Fix multiple Custom sequence schedules |
||
adding same job to sequence 2x causes 500 error |
||
update PostgreSQL version in Mitto UI |
||
Frontend does not react on socket messages |
Done |
|
Sequence ui bug when paginate jobs not currently in seq |
Done |
|
Specify CPU architecture used when building rocksdb. |
||
RegEx - uppercase vs lower case column name out of order - missing values |
||
io - tojson fails when empty data is given |
||
/var/mitto/tmp - files not being cleaned up |
||
subscriptions - issue when subscription id exists in subscription status table but not subscription table - TypeError: ‘NoneType’ object is not iterable |
||
Column name with comma or single quote - jsonpath parse error |
||
ReflectTransform - Snowflake as input, output table empty |
||
Confirm consistent upsert logic across all implementations. |
||
subscriptions - 1 subscription failed out of many - job errors with returncode 138 |
||
Files - Slow page load on stage |
||
correct special case of float sampling |
||
tests/test_job_subscriptions_process.py fails |
||
Snowflake error: Authentication token has expired |
||
Regex job failing on second run - snowflake |
||
Incorrect error message when killing the job manually. |
Done |
|
CircleCI build failures |
Done |
|
Query builtin: ‘sqlalchemy.sql.sqltypes’ has no attribute ‘TIMESTAMP_TZ’ |
||
Test failures for BigQuery |
||
Imap job requires output format field. |
||
Jobs search does not find jobs from the previous/next page |
Done |
|
Pagination widget shows wrong numbers. |
Done |
|
BigQuery: address PendingDeprecationWarning |
||
Invalid CRON schedule (0 5 1 0 0) - Re: Mitto Fixed. |
||
Database page 500 error when table starts with space |
||
Database page - non analytics database shows wrong tables in schema |
Done |
|
Command line plugin wizard - timeout doesn’t get added to job config |
||
Regex single file with modified date today the mtime is not a datetime |
||
|
||
Job doesn’t finish with broken webhook JSON |
||
Sampling - sampling large integer values can raise an exception |
||
Sampling - some values which should sample as SDL Integer are sampled as SDL Float |
||
job returns 500 error - Exactly 5 or 6 columns has to be specified for iteratorexpression. (type=value_error.croniterbadcron) |
||
Existing credential can be edited and saved without filling in the entire form |
Done |
|
Fix broken mariadb tests |
||
MySQL & MSSQL - Cannot upgrade from Boolean to Integer |
||
|
||
‘SnowflakeDialect’ object has no attribute ‘default_schema_name’ |
||
Database Manager - job with output with invalid schema/table name - 502 bad gateway error |
||
PostgreSQL - 20 digit integer - sycopg2.errors.NumericValueOutOfRange: numeric field overflow DETAIL: A field with precision 18, scale 8 must round to an absolute value less than 10^10. |
||
logs pagination happens on mouse wheel scroll up and down |
Done |
|
ReflectTransform - job config step without schema key |
||
SQL job with – comments - psycopg2.ProgrammingError: can’t execute an empty query |
||
Query - UPSERT STEPS CountTable (should be CollectMeta) out of order |
||
MySQL: Cannot insert more than 65535 chars |
||
Update tests to reflect the quoted SQL created as a result of MIT-1726 |
||
Fix unittests |
Done |
|
File Manager Load time |
||
MySQL - error when increasing size of varchar column |
Done |
|
Redshift - Invalid argument(s) ‘s3_bucket’,’s3_access_key’,’s3_secret_key’ sent to create_engine() |
||
MySQL - Text columns altered on every run |
||
iov2.utils.json_encoder can cause reduced precision for decimal numbers |
||
update DEBIAN/opt/mitto/etc/nginx/nginx.conf |
||
Snowflake - rollback calls count was reduced but still high |
Done |
|
Store to table - investigate memory issue |
||
Sequences can be duplicated from pane view, but duplicating certain sequences results in error |
Done |
|
Mitto 2.8 - jobs in sequence removed by dragging do not stay removed |
Done |
|
CSV wizard io job fails on added columns |
Done |
|
Random pytest DB assertion error |
Done |
|
Redshift - error converting BOOLEAN to STRING |
||
Redirect after login not working. |
Done |
|
Very long job name - able to scroll down page way beyond content |
Done |
|
Backticks not working in mysql queries. |
Declined |
|
Tableau Extract Refresh - credentials are created for every job instead of one Tableau Server/Online credential |
||
job notifications modal - text overflowing out of modal |
Done |
|
Some Mitto scheduler log statements going to /var/log/syslog |
||
Job schedules - bogus value in “custom” field with “daily” schedule causes job to be uneditable/unrunnable |
||
Job Runner - Handle job kill |
||
Understand, and possibly correct, incorrect pylint warnings |
Done |
|
Renaming a job doesn’t rename the store directory |
Done |
|
closing add job modal sends user to /stage instead of returning user to current page |
Done |
|
Mitto Bug - XSVRe /CSV Jobs - Monetary Columns with “(” import as varchar |
||
Tests - Fix warnings |
||
Blank Job page when using a direct link for nonexistent job |
Done |
|
PostgreSQL reserved words - ALTER COLUMN from TYPE VARC… |
Done |
|
swimlanes - large number of job runs > 10k doesn’t match pagination number |
Done |
|
extra dots when adding long named jobs to sequence |
Done |
|
column name with single quote (CSV, Google Sheets) fails |
||
Query - io input error on upsert |
||
file list sorting - B is listed before a |
Done |
|
stage page missing pagination |
||
kill of cmd jobs leaves processes running |
||
[CUSTOMER] psycopg2.errors.InvalidCursorState: opening multiple cursors from within the same client connection is not allowed. |
||
Redshift - concurrent jobs into meta table - Serializable isolation violation on table |
||
Change CountTable to CollectMeta in Query job |
Done |
|
Data Editor Column Headings Pushed to the right |
Done |
|
Any mitto CLI command creates an auth app instance |
||
Credentials Page - /credentials_type request is made on every Add/Edit modal opening |
Done |
|
INVALID - Store - upsert doesn’t work without “updated_at” set |
Done |
|
date being converted to timestamp with time zone instead of date |
||
“short” date being converted to string instead of date |
||
CUSTOMER BUG - SQL Server - table upsert incorrect (datetime2 data type) - store upsert correct however |
Done |
|
JSON job consuming 4x memory compared to size of file (2.2GB) |
Done |
|
Delete Same-name job in different tab, can’t recreate job until you exit job-creation window |
Done |
|
MSSQL - job should create schema if it doesn’t exist (like PostgreSQL currently does) |
||
500 error when editing existing data editor row, and saving with a different ID value than original |
||
Invalid boolean for Redshift |
||
Sampling bug when converting Float to String |
Done |