Release Notes - Mitto [MIT] - 2.8.0¶
Epic¶
issue |
summary |
|---|---|
Make using Mitto with Snowflake more efficient |
|
Allow non-admin users |
|
Flat File Connector |
Improvement¶
issue |
summary |
|---|---|
Add support to Subscriptions for CSV/Excel attachments |
|
Subscriptions - add support for relative dates in subscription value filters |
|
Subscriptions - add support for concurrent Tableau exports |
|
Add SMTP timeout parameter for Subscriptions / Tableau Request |
|
TRIAL RELATED - SQL Server TIMESTAMP data type - binary data - upsert in query job |
|
add multi select on files in file manager to download/delete |
|
Add ability to create a sitemap to mitto/docs |
|
Users API |
|
Modify job_tableau_export.py to eliminate the use of ‘tableau login’ |
|
implement COPY INTO for Snowflake |
|
Change publish destination for Mitto documentation |
|
Wizard - update css for dynamic json-editor elements |
|
Wizard - add ability to select all/none items in scrolled list |
|
Wizard - add ability to link to external help from wizard pages |
|
MariaDB support in Mitto |
|
Add enable/disable support to the web hooks API |
|
Extract job improvements |
|
MaxTimestamp updates from LeafLogix connector |
|
Mitto Feature Request : Delete File Dialog should list file name to be deleted |
|
Ability to HIDE or OBSCURE Credentials in Tableau Extract Job |
|
apply documentation updates to the cookicutter for plugins |
|
named credentials - add “generic” named credential |
|
Webhooks - Extend DB with last_triggered info |
|
Additional frontend functionality for v2 wizards |
|
generic job - add Type dropdown |
|
jobs in sequence show their job name rather than their job title - show job title |
|
sorting in a job’s JSON config - does this have to be alphabetical or can it be user sorted? |
|
Improvements to dependency specification for Mitto plugins |
|
job history - add status filter |
|
command job - create wizard |
|
Move share/bin/* cmds to cli |
|
Create an IMAP wizard |
|
Convert builtins to use new wizard |
|
webhooks - add enable/disable button |
|
Do not return credentials password or have a “View password” icon |
|
Remove the XSV sampling page |
|
Mitto Feature Request - Browser Title Bar |
|
Mitto Feature Request: Sequence Job Names |
|
“bulk” delete of files from the Mitto file manager |
|
add /inventory/delta endpoint |
|
In Swimlane graphic tooltip include total run time |
|
Swimlanes - add “search” |
|
Swimlanes - add “status” filter |
|
show Title instead of Name for jobs in sequence list |
|
add “Type” filter to all jobs and history pages |
|
Swimlanes - add status filter |
|
Create a missing property filter |
|
add filters (status, time range) to the Graph page - reduce data size and improve performance |
|
Excel wizard |
|
add wizard for other flat file types beyond just CSV (Excel, JSON, etc) |
|
Improvements for Job API |
|
Use TOKEN_EXPIRY and create test cases |
|
Validate SQL queries for graph API |
|
sort the crosstab columns in the database -> schema -> table page |
|
show the same “job information” that is on the jobs/history page when looking at the jobs in a sequence on the sequence page |
|
make all crosstab columns sortable |
|
add “type” as a parameter to /jobs endpoint |
|
Convert notifier package to FastAPI |
|
Create an updated_at mixin in mitto.iov2.mixins |
|
Add basic Job stats to Job list View in Sequence Details Screens |
|
database controls - table viewer - make “drop” button gray/red. |
|
database viz - let the user know they can click to drill in |
|
add indexes on Mitto tables |
|
autotag jobs with their connector for builtins |
|
add pagination anywhere we show a list of jobs (sequence editor for example) |
|
Ensure valid table names |
|
add Redshift section (S3 etc) to output of query job |
|
when downloading a file from the file manager put the name of the file in the dialog box |
|
Job Search - case sensitive? |
|
Response time on SCHEDULE on job/sequence detail screen |
|
multi select “status filter” |
|
[CUSTOMER REQUEST] notifications use case via bash script |
|
Implement the hamburger menu on small/medium screens |
|
Test for changes on the job page. |
Sub-task¶
issue |
summary |
|---|---|
Drag and Drop UI element for json-schema |
Task¶
issue |
summary |
|---|---|
Tests - Fix tests that use CREATE USER |
|
Upgrade to pydantic 1.0 |
|
500 error trying to remove a sequence created via a python script |
Bug¶
issue |
summary |
|---|---|
Microsoft Access connector jobs failing because of column header case |
|
MSSQL - Query with Reflect - RuntimeError: Invalid reflection class: sqlalchemy.dialects.mssql.base.UNIQUEIDENTIFIER{noformat} |
|
credentials - credential (basic and generic) is empty after creating it |
|
Excel wizard - add indicator that Mitto is doing something when clicking next - long pauses |
|
sequence - when adding jobs to sequence, jobs aren’t enabled by default |
|
sequence - creating new sequence, jobs end up in job id order (ascending) instead of user defined order |
|
Ownership of named credentials in /var/mitto/credentials prevents editing |
|
Job sequence - info about jobs in sequence missing after updating |
|
Job sequence - disabled jobs sometimes disappear |
|
icon colors for jobs in sequence on sequence page |
|
Sequence with no jobs hangs. |
|
running empty sequence - sequence runs indefinitely |
|
Google BigQuery - Create dataset if it doesn’t exist |
|
Google BigQuery - google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS |
|
Incorrect ‘showing’ message on stage. |
|
Wizard - hidden fields no longer hidden after cutover to Pydantic 1.4 |
|
query - wizard - 400 Bad Request 400 Bad Request 'tables_meta' |
|
CircleCI build deprecation warnings |
|
Missing X-REMOTE-USER header |
|
Snowflake column naming convention doesn’t match PostgreSQL naming convention |
|
Logs Page - Large exceptions are not shown |
|
database page - meta schema does not show up on the list of schemas |
|
MySQL - job fails if database isn’t already created |
|
Snowflake - Fails to adapt from boolean to number |
|
long job names in sequence editor - can’t see full name or hover to see text |
|
Cred.io - SQL Syntax error (ALTER TABLE plt_wealth_user_plans_view ALTER COLUMN amount TYPE NUMERIC) |
|
Google BigQuery (as output) - 400 DELETE must have a WHERE clause at [1:1] |
|
Google BigQuery (as output) - 404 Not found: Dataset mitto-183418:meta was not found in location US |
|
Google BigQuery (as job output) - 400 Name meta not found inside meta at [3:14] |
|
Mitto wizards (CSV, Query, ??) when using Redshift as output are not adding extra Redshift info to the resulting job config |
|
file download - title and name don’t sync when using wizard |
|
csv with no headers - front-end support |
|
Cannot Duplicate Job in Stage |
|
Wizard fails on csv |
|
job edit (name, title, etc) button doesn’t save when user clicks edit again - user has to press enter to save a change |
|
PostgreSQL to Snowflake - query job with upsert and ExtraColumnsTransform rename_columns FALSE |
|
ReflectTransform with upsert on Snowflake - fails on first upsert attempt - KeyError: ‘id’ |
|
ReflectTransform - Snowflake to Snowflake - table DDL is correct, but all values are NULL |
|
logs - copy paste isn’t possible with the way the lines are selected |
|
ReflectTransform - PostgreSQL to Snowflake REFLECT job fails on the second run |
|
ReflectTransform - SQL Server error - TypeError: ‘>’ not supported between instances of ‘NoneType’ and ‘int’ |
|
column of zeros creates bool on first run and converts to numeric on second run (fails) |
|
data editor primary key is string with spaces |
|
UI not showing schemas starting with “pg” (no underscore!) |
|
Snowflake - can’t convert data type from BOOLEAN to INT/FLOAT |
|
Snowflake - KeyError: ‘check’ |
|
Excel plugin - int and boolean being converted to float |
|
running 24 concurrent sequences - FATAL: sorry, too many clients already |
|
Migration script fails |
|
SQL wizard is blank (2.7.5) |
|
GTI Mitto Duplication Error |
|
When duplicating Job unusual error |
|
csv with no headers - wizard, sdl, and resulting table bugs |
|
500 /api/config |
|
“Upload Timed Out - Please Try Again” |
|
history - able to click next page when there is no next page |
|
pip install fails on fresh virtualenv |
|
jscs errors |
|
multiselect jobs -> actions -> create sequence - no jobs enabled |
|
/key/me 500 error |
|
MySQL - MySQL ALTER TABLE test ALTER COLUMN stuff TYPE VARCHAR(19) |
|
XSV plugin - bad file format (XLSX extension converted to CSV) returns 500 error in wizard |
|
CSV plugin - can’t upload TSV file - You can’t upload files of this type |
|
[CUSTOMER] Excel - how to deal with invalid date values? xlrd.xldate.XLDateNegative: -621438.0 |
|
searching for “site” in sequence editor shows failed jobs list instead of just jobs that match “site” |
|
sequence - searching after selecting a job removes the selection from the job |
|
“Logs” button shows no log info for failed job |
|
search input - searches too fast for user to type in box |
|
Handle websocket connections after server shutdown/page close |
|
clean up SQL wizard text |
|
errors on upgrade to 2.7 |
|
Exception info missing from log |
|
error in validation of config provided to generic job wizard |
|
Mitto Left Sidebar Inaccessible when browser is too thin |
|
Job Details page - Redundant API calls |
|
Monitor Job returncode set to query result |
|
Database API offset and limit error |
|
rename CSV plugin to Flat File |
|
Credential manager UI bug |
|
columns missing (that have data in source table) from query job SDL |
|
mariadb BOOL(EAN) error |
|
PostgreSQL has a limit of 63 characters for column name length - how do we deal with this? |
|
seems to be a Mitto job filename character length maximum (63 chars?) - some jobs require longer names |
|
XSVre fails due to runtime table errors |
|
CSV job creates table with all nulls except for index column |
|
Query - MSSQL to MSSQL error - Progea |
|
MSSQL - varchar > 8000 characters needs to be VARCHAR(MAX) |
|
datetime shows up as time if it only has values from today |
|
meta.meta table isn’t updating based on subsequent job runs |
|
Query job converted “timestamp with time zone” to “time” |
|
Downloading twbx from file manager tries to render in browser instead of actually downloading |
|
handle new columns being added to source object and backfilling all rows |
|
Missing error message after sampling |
|
[Xcel] RuntimeError: Job ‘xsvre’ returned: 1 |
|
SDL-based jobs have null for name |
|
xsv not working on blackline |
|
New CSV File job fails |
|
Creating CSV job - data type is Date/Datetime but database is actually varchar |
|
samples dates as datetime always |
|
time field being converted to datetime - should be string? |
|
Tableau Data Extract Refresh - 500 error when creating job on playworks.zuarbase.net |
|
MySQL - double precision column being converted to string |
|
Mitto Sales Demo - TypeError: Decimal(‘106.42’) is not JSON serializable |
|
Rehrig - 42000 error when trying to use Query connector |
|
Query job (with upsert) fails on 2nd run - looks like is mixing up the primary key and last updated columns |
|
multiple primary keys does not work with query connector even after fixing the config |
|
add an option to NOT use a transaction for SQL job |
|
TypeError: Object of type ‘UUID’ is not JSON serializable |
|
500 error when GET’ing database endpoints too often |
|
UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use “pip install psycopg2-binary” instead. |
|
psycopg2.ProgrammingError: syntax error at or near “user” LINE 1: ALTER TABLE salesforce.cttmp__user RENAME TO user |
|
Netsuite Mitto Demo credentials error |
|
meta table counts not being recorded? |
|
[Redshift] Timestamp format is invalid. See DATEFORMAT and TIMEFORMAT Strings for valid formats. |
|
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: sorry, too many clients already |
|
timestamp of file manager is off |
|
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected |
|
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected |
|
weird conversion of SQL on toughmudder.zuarbase.net after upgrade |
|
[Redshift] psycopg2.NotSupportedError: ALTER COLUMN TYPE is not supported |
|
[Marketo] AttributeError: ‘Sampler’ object has no attribute ‘sample_text’ |
|
[Marketo] AttributeError: ‘Sampler’ object has no attribute ‘sample_text’ |
|
location of uploaded files (CSVs, etc) has changed with new verison which breaks old jobs |
|
TypeError: ‘dict’ object does not support indexing |
|
Invalid JWT -> 500 Error |
|
email alerts come from mitto@akiri.co even though jobs on separate mitto instance |
|
Password save : Google chrome |
New Feature¶
issue |
summary |
|---|---|
Add “Previous” button to first page of job create wizards |
|
Ace editor is shown above footer |
|
File list UX |
|
Better css for rounded buttons on environment variable editor |
|
Better card multi-selected state |
|
Show ‘running’ on job card when running |
|
preinstall R in Mitto |
|
Show successful output of cmd jobs in logs. |
|
system alerts - high RAM usage |
|
Duplicate button for sequences |
|
Merge the session pytest fixtures |
|
Test amCharts FDG in Mitto |
|
Add parameters to the MaxTimestamp step |
|
Amazon Vendor Central Meta XSVre Job |
|
Rclone job |
|
Change control record in Mitto |
|
API for filtering history by date range |
|
ability to define indexes on tables in the UI/job |
|
Allow arbitrary job ordering |