Issues Resolved in this Release¶
New Feature¶
Issue |
Summary |
Resolution |
|---|---|---|
Regex - show matching file info in logs at start of job. |
||
MongoDB input |
||
System Monitoring Job Type |
||
Create IncludeTransform |
Done |
|
Create ExcludeTransform |
Done |
|
Update styles for json-schema Drag and Drop element |
Done |
|
tojson, tojsonl code |
||
files - add pagination + search |
||
Move Create jobs wizard from modal to page like Sequence editor |
Done |
|
stretch the modals to give more room for editing (e.g. SQL) |
Done |
|
Hide Credentials for dbo |
Done |
|
set default output database / database when using SQL connector |
||
Duplicate button for sequences |
Done |
|
Allow comments in JSON configuration files |
||
Test amCharts FDG in Mitto |
||
Change control record in Mitto |
Done |
|
potentially create AWS IAM user for each new Mitto instance - to use the AWS api (client could whitelist IPs via Mitto instance for example) |
Done |
|
automatically altering column data types (string lengths) makes snapshots more difficult with current methods |
Done |
|
check source endpoint periodically (daily?) for endpoints that don’t have jobs in Mitto - alert the user and allow them to add those jobs’ |
Done |
|
UI for filtering history by date range |
Done |
|
based on the schedules, show in the UI (on the job manager page for example) the next time a job is supposed to run and the delta between when it actually runs |
Done |
|
Monitoring job can return results of SQL query |
Done |
|
ability to define default database output (localhost postgres is default) |
Done |
|
need a way to apply new messaging/notification settings to all existing jobs |
Done |
Improvement¶
Issue |
Summary |
Resolution |
|---|---|---|
Allow creation of report of a release’s Jira tickets without overwriting remainder of release notes. |
Done |
|
sql job - add SQL statement of job to job log on job run |
||
IMAP plugin icon |
||
Add Debian postinstall step to remove existing dataeditor plugin |
||
job edit - hjson - pretty format on save |
Done |
|
Change default sender for notifications |
||
Remove unused frontend code from the dataeditor. |
Done |
|
Links for “prod” version of documentation |
||
command job wizard - increase size of “command” box |
||
naming and sort order consistency |
Done |
|
update documentation to show ticket status |
||
xsv plugin wizard - remove “define the columns” screen and remove SDL from resulting job config |
||
query plugin wizard - remove SDL from resulting job |
||
tag filter - order alphabetically |
Done |
|
Mitto and TLS < 1.2 |
||
Modify frontend to call /metrics endpoints on websocket messages only. |
Done |
|
Use mitto.settings.TMP_PATH instead of /tmp |
||
add created_at and updated_at information to job page |
Done |
|
Add support for multiple versions of Mitto documentation |
||
Unify set of filter across pages |
Done |
|
wrap schedule text on jobs lists pages |
Done |
|
Add metrics for mitto instance |
||
File upload: modify version string |
Done |
|
monitor job type - add SQL tab |
Done |
|
Changes to generic job builtin to support comments in job configurations |
||
add credential id to credentials page |
Done |
|
JDBC input |
||
Swimlanes - add # of failed jobs and decimal precision |
Done |
|
Support removal of Schedules for Extracts in Tableau Online |
||
Add ability to prevent writing header for |
||
io - Unicode encode/decode handling |
||
Tableau Extract Refresh job type - list Extract Refresh schedules in alphabetical order |
||
Query - Add “create_engine” arguments |
||
mitto cli - add version |
||
add /api/key documentation to Swagger /docs |
||
ExtraColumnsTransform - Ignore columns not just objects |
||
Benchmark picke/json, JSON/UJSON |
Done |
|
add “custom” option to job timeouts |
Done |
|
Copy create_engine to apitoolbox |
||
graph - Add ability to filter by time range |
Done |
|
Graph - add filter by status, time range filters |
Done |
|
Job exited with returncode 143 - Nested sequence/job error |
Done |
|
Select all (multi-select) jobs |
Done |
|
Ace worker not found |
Done |
|
Command plugin - tag jobs created via wizard with “cmd” |
Done |
|
Excel plugin - wizard - output - move Schema above Table |
||
file manager - indicate the user can drag and drop in files (like csv plugin wizard) |
Done |
|
Named Credentials - Store Google Credentials as named credentials |
||
New release mechanism: |
||
Graph - Filter by tag (API support) |
Done |
|
wizard icons - Excel, Imap |
||
IO - Do not fetch table info on every step |
||
allow ReflectTransform to take extra SDL in job - remove lines 27-29 in existing code |
Done |
|
job timeout - add 3 hours and custom number of minutes |
Done |
|
add support for “end of” schedules - end of month, end of quarter, etc |
||
command job - add AceEditor for “cmd” |
Done |
|
Show line number in message when error is found in Job |
Done |
|
add Mitto job STEP to remove JSON store |
||
Feature Request: Mitto - Job Listing Page - Open RHS Job Panel |
Done |
|
Remove the “upgrade” action from the /api/pkg Mitto API endpoint |
Done |
|
snowflake - add QUERY_TAG for Mitto connections to snowflake |
Done |
|
named credentials - add databases as an option |
Done |
|
Consolidate pagination response format |
Done |
|
Cleanup mitto package.json scripts |
Done |
|
Graph - Filter by tag |
Done |
|
swimlane - currently running job should be at the top of the list and have a width for the duration it has been running |
Done |
|
add search to history page |
Done |
|
Improve SSL on nginx |
||
table viewer crosstab - fit to width of screen |
Done |
|
Add the :embed query string parameter |
Done |
|
add regex search to file manager files |
||
Implement simple pagination for the dataeditor |
Done |
|
show which table in the database the data editor is tied to |
Done |
|
set a default timeout for jobs? - SQL jobs for example most of the time complete within seconds |
Done |
|
add “last run” time taken column? |
Done |
|
Allow store = True/False |
||
[Mit 2110] Need a way to change the admin password in UX |
Done |
|
Mitto Sales Demo - Initial Password Setup instructions |
Done |
|
message notifications should probably only be turned on by default for sequences |
||
with pagination we can’t use CTRL+F to find specific jobs - will need wildcard job filtering at the same time as pagination |
Done |
|
fit Navigation menu wording |
Done |
|
Install haveged |
Done |
|
auto generated tags |
Done |
|
Allow Save & Start from the Save component |
Done |
|
need a way to add and then enable/disable all jobs in a sequence based on something (tag, etc) |
Done |
|
Add locking to dpkg.json |
Done |
Sub-task¶
Issue |
Summary |
Resolution |
|---|---|---|
Fix cron migration script. |
||
Add API endpoint to change password |
||
Mitto UI is not working on the fresh installation. |
||
Migrate GET api/v2/jobs to POST api/jobs/search |
Done |
|
Fix adding same job to sequence on front-end |
Done |
|
Time filter update |
||
Add new query parameter to the GET api/v2/jobs?exclude= |
||
GET /api/v2/jobs - pages: Stage, Jobs - need to add ability to filter by time ranges |
||
GET /api/graph - pages: Graph - need to add ability to filter by job’s type and by search term |
||
GET /api/v2/history - pages: Swimlanes, History - need to add ability to filter by job’s tags |
||
Check if websockets are receiving notify messages on the job edit. |
||
Fix navs endpoint |
||
Fix Jobs created without conf |
||
Fix GET about/config endpoint |
||
Migrate xsv bultin to plugin V2 |
||
Migrate dataeditor plugin to use APIv2 |
||
Migrate query plugin to use APIv2 |
||
Delete unused API v1 endpoints |
||
Switch the usage of old endpoint |
Done |
Task¶
Issue |
Summary |
Resolution |
|---|---|---|
upgrade from 2.8 to 2.9 fails due to library dependency |
||
Fix failed CircleCI builds - delete package-lock.json |
||
Investigate SyncStore memory consumption |
||
List API endpoints that the front end uses which are still being served by uwsgi |
||
Investigate python-rocksdb==0.6.9.1 differences. |
||
Bug¶
Issue |
Summary |
Resolution |
|---|---|---|
Upgrade fails if gcloud credentials are present and unencrypted. |
Cannot Reproduce |
|
remove ssh from jpype in requirements.txt |
||
job config - hjson - commenting last line of object moves that line to line above |
||
CircleCI jpype segfault |
||
sequence - jobs last completed all increase as jobs run |
Duplicate |
|
Job completion times for sequence are incorrect |
||
job logs - out of order by timestamp |
||
XSV plugin: Unclear error message if no XLS files present |
||
Credentials with invalid JSON are created |
||
jobs - run duration much longer than it should be |
||
database page - schemas don’t appear to be sorted |
||
jobs - tag filter - long tag text overflows |
||
webhooks - overflows it’s UI section on smaller screen |
||
jobs - multi select action throws js error - vendor-ba42ebbbb53f5017be2e.bundle.js:336 TypeError: ctrl.status is not a function |
||
Google sheets wizard - 404 Not Found on 2nd screen |
||
loading indicator flashes and interrupts user action while jobs starting |
||
sequence is created with “Invalid Value” for schedule. |
Duplicate |
|
ui and cli performance issues |
||
Excel wizard fails when invalid start_row specified. |
||
Credentials list on first screen doesn’t show up when clicking previous from the next screen |
||
job’s updated at should not update when a job is started |
||
system job - fails on 2.9 - KeyError: ‘jobs’ |
||
xlsx file with non-data sheets (data on sheet #5) causes wizard to fail |
||
sampling - 70% samples as integer - error = psycopg2.errors.InvalidTextRepresentation: invalid input syntax for integer: “0.7” |
||
global filters - multi selected values text outside button |
||
upgrade from 2.8.12 to 2.9.0-rc0 fails |
||
Snowflake - Object ‘TEST’ already exists. |
||
“builtin” test failures |
||
Fix sequence schedule ‘Invalid value’ |
||
job edit - saving returns this console error - vendor-4f8b5ddafc7b7d4a44e4.bundle.js:336 TypeError: Cannot read property ‘length’ of undefined |
Done |
|
json/jsonl inputs - json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1) |
Cannot Reproduce |
|
job edit - save & start - add indicator that something is happening |
Done |
|
query plugin wizard - add credentials to first screen (input) |
||
“Save” and “Save & Start” buttons intermittently fail to work |
Done |
|
data editor - bottom scroll bar |
||
database page - PostgreSQL listed 2x |
||
swimlanes - global type filter reverts on quick changes |
Done |
|
swimlanes - order of jobs should be by completed date desc, running at the top |
||
job history - status filter - make multi select and/or add “all” option |
Done |
|
change password - false error appears |
Done |
|
tableau extract refresh wizard - 500 error using existing credential |
||
files - uploading same file should overwrite existing file (or at least prompt) |
||
job schedules not firing |
||
jobs page - selecting job (gray background) and keyboard arrow keys |
Done |
|
The Dataeditor updates the wrong record |
Done |
|
Fix dataeditor pagination |
Done |
|
query job - output to snowflake - table names with mixed capitalization - KeyError: ‘test_snowflake_naming_MIXED’ |
||
sql job transaction parameter doesn’t do anything? |
Won’t Do |
|
HJSON - Job created with (working) hjson fails on creation |
||
sort order of a single job’s history is in ascending status id instead of descending status id |
||
data editor - change generic “database” to the name of the database of the data editor |
||
builtin.query.QueryInput uses incorrect comparison for upsert |
||
test issue - ignore |
Cannot Reproduce |
|
repeated GET /v2/metrics calls while idle in UI |
Done |
|
add /store endpoints to swagger /docs |
||
informatory message logged as WARNING |
||
job - schedule is blank on job creation (should say manual) |
Done |
|
job edit - pressing escape key throws error - Error: Transition Rejection($id: 39 type: 4, message: This transition is invalid, detail: The following parameter values are not valid for state ‘edit’: [id:undefined]) |
Done |
|
Excel plugin wizard - credentials should be directly under database on last screen |
||
Excel plugin wizard - unable to specify index for value for “sheet” |
||
command plugin wizard - first screen includes blank key and value |
||
sql - editing job - Uncaught TypeError: Cannot assign to read only property ‘length’ of string ‘ select 1;’ |
||
job edit - unable to save job - Uncaught TypeError: ctrl.conf.input.query.join is not a function |
Done |
|
job edit - command/ctrl + s no longer saves the job config |
Done |
|
Databases - Page load is very slow on Stage |
Done |
|
global filters - ran during - jobs text with relative date needs to be reworded |
Done |
|
global filters - search - needs minimum width |
Done |
|
global filters - ran during - value and text of start date incorrect |
Done |
|
jobs pages - filters in url parameter and on page, but not actually applying on page swap |
Done |
|
files page - typo on upload note |
Done |
|
swimlanes - metrics missing styling |
Done |
|
Fix jobs filtering |
Done |
|
test_redshift_upsert fails after PR#676 |
||
Fix mitto 0020 py migration error |
||
Query jobs - wizard should add stream_results input parameter with a reasonable default value |
||
Pickup SDL fixes made in 2.8.x |
||
Fix job conf validation - “init() takes exactly 3 positional arguments (2 given)”, |
||
Fix multiple Custom sequence schedules |
||
adding same job to sequence 2x causes 500 error |
||
update PostgreSQL version in Mitto UI |
||
Frontend does not react on socket messages |
Done |
|
Sequence ui bug when paginate jobs not currently in seq |
Done |
|
Specify CPU architecture used when building rocksdb. |
||
RegEx - uppercase vs lower case column name out of order - missing values |
||
io - tojson fails when empty data is given |
||
/var/mitto/tmp - files not being cleaned up |
||
subscriptions - issue when subscription id exists in subscription status table but not subscription table - TypeError: ‘NoneType’ object is not iterable |
||
Column name with comma or single quote - jsonpath parse error |
||
ReflectTransform - Snowflake as input, output table empty |
||
Confirm consistent upsert logic across all implementations. |
||
subscriptions - 1 subscription failed out of many - job errors with returncode 138 |
||
Files - Slow page load on stage |
||
correct special case of float sampling |
||
tests/test_job_subscriptions_process.py fails |
||
Snowflake error: Authentication token has expired |
||
Regex job failing on second run - snowflake |
||
Incorrect error message when killing the job manually. |
Done |
|
CircleCI build failures |
Done |
|
Query builtin: ‘sqlalchemy.sql.sqltypes’ has no attribute ‘TIMESTAMP_TZ’ |
||
Test failures for BigQuery |
||
Imap job requires output format field. |
||
Jobs search does not find jobs from the previous/next page |
Done |
|
Pagination widget shows wrong numbers. |
Done |
|
BigQuery: address PendingDeprecationWarning |
||
Invalid CRON schedule (0 5 1 0 0) - Re: Mitto Fixed. |
||
Database page 500 error when table starts with space |
||
Database page - non analytics database shows wrong tables in schema |
Done |
|
Command line plugin wizard - timeout doesn’t get added to job config |
||
Regex single file with modified date today the mtime is not a datetime |
||
|
||
Job doesn’t finish with broken webhook JSON |
||
Sampling - sampling large integer values can raise an exception |
||
Sampling - some values which should sample as SDL Integer are sampled as SDL Float |
||
job returns 500 error - Exactly 5 or 6 columns has to be specified for iteratorexpression. (type=value_error.croniterbadcron) |
||
Existing credential can be edited and saved without filling in the entire form |
Done |
|
Fix broken mariadb tests |
||
MySQL & MSSQL - Cannot upgrade from Boolean to Integer |
||
|
||
‘SnowflakeDialect’ object has no attribute ‘default_schema_name’ |
||
Database Manager - job with output with invalid schema/table name - 502 bad gateway error |
||
PostgreSQL - 20 digit integer - sycopg2.errors.NumericValueOutOfRange: numeric field overflow DETAIL: A field with precision 18, scale 8 must round to an absolute value less than 10^10. |
||
logs pagination happens on mouse wheel scroll up and down |
Done |
|
ReflectTransform - job config step without schema key |
||
SQL job with – comments - psycopg2.ProgrammingError: can’t execute an empty query |
||
Query - UPSERT STEPS CountTable (should be CollectMeta) out of order |
||
MySQL: Cannot insert more than 65535 chars |
||
Update tests to reflect the quoted SQL created as a result of MIT-1726 |
||
Fix unittests |
Done |
|
File Manager Load time |
||
MySQL - error when increasing size of varchar column |
Done |
|
Redshift - Invalid argument(s) ‘s3_bucket’,’s3_access_key’,’s3_secret_key’ sent to create_engine() |
||
MySQL - Text columns altered on every run |
||
iov2.utils.json_encoder can cause reduced precision for decimal numbers |
||
update DEBIAN/opt/mitto/etc/nginx/nginx.conf |
||
Snowflake - rollback calls count was reduced but still high |
Done |
|
Store to table - investigate memory issue |
||
Sequences can be duplicated from pane view, but duplicating certain sequences results in error |
Done |
|
Mitto 2.8 - jobs in sequence removed by dragging do not stay removed |
Done |
|
CSV wizard io job fails on added columns |
Done |
|
Random pytest DB assertion error |
Done |
|
Redshift - error converting BOOLEAN to STRING |
||
Redirect after login not working. |
Done |
|
Very long job name - able to scroll down page way beyond content |
Done |
|
Backticks not working in mysql queries. |
Declined |
|
Tableau Extract Refresh - credentials are created for every job instead of one Tableau Server/Online credential |
||
job notifications modal - text overflowing out of modal |
Done |
|
Some Mitto scheduler log statements going to /var/log/syslog |
||
Job schedules - bogus value in “custom” field with “daily” schedule causes job to be uneditable/unrunnable |
||
Job Runner - Handle job kill |
||
Understand, and possibly correct, incorrect pylint warnings |
Done |
|
Renaming a job doesn’t rename the store directory |
Done |
|
closing add job modal sends user to /stage instead of returning user to current page |
Done |
|
Mitto Bug - XSVRe /CSV Jobs - Monetary Columns with “(” import as varchar |
||
Tests - Fix warnings |
||
Blank Job page when using a direct link for nonexistent job |
Done |
|
PostgreSQL reserved words - ALTER COLUMN from TYPE VARC… |
Done |
|
swimlanes - large number of job runs > 10k doesn’t match pagination number |
Done |
|
extra dots when adding long named jobs to sequence |
Done |
|
column name with single quote (CSV, Google Sheets) fails |
||
Query - io input error on upsert |
||
file list sorting - B is listed before a |
Done |
|
stage page missing pagination |
||
kill of cmd jobs leaves processes running |
||
[CUSTOMER] psycopg2.errors.InvalidCursorState: opening multiple cursors from within the same client connection is not allowed. |
||
Redshift - concurrent jobs into meta table - Serializable isolation violation on table |
||
Change CountTable to CollectMeta in Query job |
Done |
|
Data Editor Column Headings Pushed to the right |
Done |
|
Any mitto CLI command creates an auth app instance |
||
Credentials Page - /credentials_type request is made on every Add/Edit modal opening |
Done |
|
INVALID - Store - upsert doesn’t work without “updated_at” set |
Done |
|
date being converted to timestamp with time zone instead of date |
||
“short” date being converted to string instead of date |
||
CUSTOMER BUG - SQL Server - table upsert incorrect (datetime2 data type) - store upsert correct however |
Done |
|
JSON job consuming 4x memory compared to size of file (2.2GB) |
Done |
|
Delete Same-name job in different tab, can’t recreate job until you exit job-creation window |
Done |
|
MSSQL - job should create schema if it doesn’t exist (like PostgreSQL currently does) |
||
500 error when editing existing data editor row, and saving with a different ID value than original |
||
Invalid boolean for Redshift |
||
Sampling bug when converting Float to String |
Done |