Release Notes: Mitto Version 3.2.0¶
Releases 3.0.0 and 3.1.0 were unpublished internal releases for which no release notes are available. Release notes for 3.2.0 contain a description of all changes since Mitto 2.11.0.
Overview¶
The Mitto 3.2.0 release is focused on improving Mitto as a platform. This work enhances currently available features, and lays stable foundation for future improvements.
New Features¶
Docker¶
Mitto 2.11.0 ran as a monolithic application on Ubuntu 18.04. With the introduction of 3.2.0, Mitto now runs as eight Docker services.
Zuar-hosted Mitto instances deploy Docker on Ubuntu 22.04. Customer-hosted Mitto instances have the option of deploying Mitto 3.2.0 via Docker on other operating systems.
Mitto’s Docker services:
mitto_app_nginx
mitto_webapp
mitto_auth
mitto_db
mitto_nginx
mitto_certbot
mitto_scheduler
mitto_vsftpd
Packaging¶
Previous to Mitto 3.2.0, Mitto and Mitto Connectors were distributed as Debian packages.
Starting with Mitto 3.2.0, Mitto and Mitto connectors are distributed as Python packages.
Postrgres¶
Mitto 3.2.0 includes Postgres 14.6.
OpenResty¶
Mitto 3.2.0 includes OpenResty 1.21.
Licensing¶
Mitto now uses a new licensing system that makes it easier to evaluate new features on a trial basis.
Mitto and all Mitto Connectors are all now individually licensed each license with an individual, distinct expiration date.
If a license is close to expiring, notices are displayed in the user interface.
If a feature’s license is not renewed and is allowed to expire, the licensed feature will no longer function.
Any configuration and data associated with an exprired feature remains on the system after expiry - re-licensing the feature will restore the configuration.
User Interface¶
The user interface has been re-themed with a simplified, light appearance.
Webhooks¶
Jobs can be started via webhooks.
Performance¶
Optimizations in how Mitto “learns” data types can result in 20-40% shorter job run times, depending upon the input data.
Logging¶
Logs - Jobs¶
Logs for Mitto jobs are now stored in JSON format.
The presentation of logs the UI has not changed.
Logs - Services¶
Logs for all Mitto services running in Docker containers are (?)
Python¶
Mitto now uses Python 3.9.10.
Python 3.9.10 is the version of Python made available in cmd jobs.
Other Improvementes¶
Job history supports per-page navigation
Output from
cmdjobs is now loggedNamed credentials are now sorted alphabetically
Wizards only show named credentials suitable for use with the job type being created.
The running of scheduled jobs can be paused in the UI as well as on the command-line.
Breaking Changes¶
To reduce image build times and sizes, Pandas and NumPy are no longer present in Mitto’s Python virtualenv.
Custom Python Versions and virtualenvironments¶
Installation of a custom Python executable and custom Python virtual environments is not currently supported.
Plugin Version Requirements for Mitto 3.2.0¶
Plugin |
Version Required |
|---|---|
15five |
|
Access |
|
ADP |
2023.02.01 |
2023.01.16 |
|
Amazon AWS |
|
Amazon Selling Partner |
2023.01.19 |
Amazon Vendor Central |
|
Centro Basis |
|
Clarizen |
2023.01.16 |
Coupa |
2022.11.24 |
Digital Ocean |
|
Domo |
2020.12.7 |
Dynamics |
2023.1.12 |
Dynamics 365 |
2023.01.19 |
Enquire CRM |
|
FifteenFive |
2022.01.16 |
Gainsight NXT |
|
ghost |
|
GitHub |
|
Gong |
2023.01.16 |
Google Adwords |
2023.02.03 |
Google Analytics |
2022.10.27 |
Google Sheets |
2023.02.03 |
Hubspot |
2022.11.21 |
IMAP |
|
Jira |
2023.02.01 |
2023.02.01 |
|
2023.1.19 |
|
Marketo |
2023.02.01 |
Matomo |
|
mavenlink |
2023.02.01 |
metrc |
2022.11.28 |
Mitto API |
|
mj freeway |
2023.1.19 |
MP-Access |
2023.1.30 |
2022.10.27 |
|
Paycom |
2023.02.01 |
Pipedrive |
2022.01.16 |
Piwik |
|
QuickBooks Enterprise |
|
QuickBooks Online |
2023.02.03 |
2023.1.23 |
|
2023.01.17 |
|
Salesforce |
2023.1.18 |
ServiceNow |
2023.01.16 |
Shopify |
2023.02.01 |
SkyBox |
|
sugarcrm |
2023.01.16 |
Tableau REST API |
2023.1.20 |
Tamarac |
2023.1.18 |
2023.01.16 |
|
Treez |
|
Version One |
2023.01.16 |
Wurk |
2021.7.16 |
2022.10.27 |
Issues Resolved in this Release¶
Issue |
Type |
Summary |
Resolution |
|---|---|---|---|
Improvement |
text field in which user must type store key isn’t readily visible |
Done |
|
Bug |
Fix job config path in databse |
Done |
|
Bug |
incorrect default directory for command jobs |
Done |
|
Bug |
directory structure, ownership, and permissions problems |
Done |
|
Bug |
Bulk job tagging issue |
Done |
|
Bug |
Error after create rclone job with “use existing named credentials” |
Done |
|
Bug |
‘/auth/docs’ are missing |
Done |
|
Bug |
system monitor job failure |
Done |
|
Bug |
‘/v2/store’ endpoint appears to be broken |
Done |
|
Improvement |
“select all” on jobs page causes all jobs to disappear |
Done |
|
Improvement |
unexpected behavior when clicking active toggle for job in squence |
Done |
|
New Feature |
Named Credentials for Gmail and Outlook IMAP |
Done |
|
Bug |
Settings page says mitto-plugin-sfdc is installed, but salesforce job gets ModuleNotFound |
Done |
|
Bug |
Selected tags not applied |
Done |
|
Improvement |
display additional information about license / license server |
Done |
|
Bug |
incorrect hostname displayed in browser tab |
Done |
|
Bug |
first login is slow |
Done |
|
Bug |
Dropdowns (Status, Type, Tag, Ran During) don’t do anything on Mitto 3.1.8 Stage page |
Done |
|
Bug |
[Salesforce] connection to server at “localhost” (127.0.0.1), port 5432 failed: Connection refused |
Done |
|
Bug |
Buttons on Job Details Page Don’t Resize when Window is Resized |
Done |
|
Task |
develop process/instructions for migrating customer from 2.11.x to 3.2.x |
Done |
|
New Feature |
Log CMD job output |
Done |
|
Improvement |
User preference doesn’t persist across pages on jobs page |
Done |
|
Bug |
must delete default env variables before a CMD job can be created |
Done |
|
Improvement |
Zuar logo should be orange |
Done |
|
Improvement |
Add list of plugins on settings page |
Done |
|
Bug |
Fix Loader class to support licensing expiration |
Done |
|
Improvement |
Implement tow-way mitto integration with license-service-v2 |
Done |
|
Bug |
upgrade OpenResty |
Done |
|
Bug |
mitto tableu extract job fails from web interface |
Done |
|
Improvement |
Remove |
Done |
|
Bug |
Jobs page doesn’t save layout option |
Done |
|
Bug |
Table extract credentials. Extra field |
Done |
|
Improvement |
Improve time of job start |
Done |
|
Bug |
CSV Job. Empty Verify the information step |
Done |
|
Improvement |
Add navigation item state editor |
Done |
|
Bug |
Can not download txt file |
Done |
|
Bug |
Errors in the console after dragging elements |
Done |
|
Bug |
Error in the console after check checkbox |
Done |
|
Improvement |
Change default login page path |
Done |
|
Bug |
Handle jobs print to stdout |
Done |
|
Bug |
Error creating generic_job |
Done |
|
Bug |
Error in runtime |
Done |
|
Bug |
ExcelInput consumes all available memory when processing xlsx files with pivot tables |
Done |
|
Task |
Regression testing v 3.0.0 |
Done |
|
Bug |
Timeout modal always shows Custom option |
Done |
|
New Feature |
Update select menu UI for multiselect |
Done |
|
Improvement |
Drag n drop is not obvious |
Done |
|
Improvement |
HIde login error when submit button is clicked |
Done |
|
Bug |
UI timezone error |
Done |
|
Bug |
Filters changed on Stage and Jobs pages |
Done |
|
Bug |
Labels from Swimlanes stay in the window |
Done |
|
Bug |
Jobs without tab “stage” in Stage |
Done |
|
Bug |
Settings error |
Done |
|
Improvement |
Include Job Runner status in websocket message |
Done |
|
Bug |
Graph page job dots not clickable |
Done |
|
Improvement |
Update Job edit buttons |
Done |
|
Bug |
Job notes not displayed |
Done |
|
Bug |
Title sort doesn’t account for case |
Done |
|
Improvement |
Update UI for job actions button |
Done |
|
Improvement |
Licensing in job scheduler |
Done |
|
Improvement |
rclone plugin - pre-commit integration |
Done |
|
Bug |
Tableau extract - credentials type issue |
Done |
|
Improvement |
Fix pylint_diff for mitto and plugins |
Done |
|
Bug |
Fields with default value is not vissible on frontend |
Done |
|
Improvement |
Setup plugins builds on CI |
Done |
|
Bug |
Incorrect local timezone |
Done |
|
Improvement |
Develop a common log format for mitto and plugins |
Done |
|
Improvement |
Collect logs from dockerized mitto instance to ElasticSearch |
Done |
|
Improvement |
Implement json log formatter |
Done |
|
Bug |
traceback no longer logged to logfile |
Done |
|
Improvement |
ExcelInput - improve error handling when one or more column headers are missing |
Done |
|
Bug |
Tableau Extract - “the selected credentials are invalid” |
Done |
|
Bug |
Builtin Extract plugin > error getting named credentials |
Done |
|
Bug |
revisit the implementation of |
Done |
|
Bug |
Missing package in mitto-dev-runtime |
Done |
|
Bug |
500 error selecting from system table |
Done |
|
Bug |
BaseInputV3 instances must allow extra fields to be added |
Done |
|
Improvement |
Use parallel jobs on CircleCI for lint and tests |
Done |
|
New Feature |
In app messaging |
Done |
|
Bug |
Uppercase in tags won’t return in filtered results |
Done |
|
Bug |
“make static” fails with vite error on local mitto |
Done |
|
Bug |
problems with Job ui updates |
Done |
|
Bug |
‘dc run –rm mitto setup’ fails when installing ‘turbodbc’ |
Done |
|
Bug |
Docker build issue |
Done |
|
Bug |
‘static_builtin’ Makefile target missing |
Done |
|
Bug |
Subscription date range = yesterday loads dashboard with data through current day |
Done |
|
Bug |
reflect transform converts tinyint to bit |
Done |
|
New Feature |
Refactor Mitto Job supervisor |
Done |
|
New Feature |
Implement License Handler in Mitto |
Done |
|
Epic |
Implement licensing flow for Mitto ver. 3.0+ |
Done |
|
Bug |
increase version of ‘responses’ from requirements.txt |
Done |
|
Bug |
missing Makefile targets for tests, lint, flake8 |
Done |
|
Improvement |
Support for “pausing” scheduled jobs in the UI |
Done |
|
Bug |
Add jobs sections adds sequence item every time being accessed |
Done |
|
Bug |
Logs page doesn’t show logs |
Done |
|
Improvement |
CircleCI - add support for running plugin tests that require credentials |
Done |
|
Improvement |
CircleCI - add ability to run tests with all database engines |
Done |
|
New Feature |
Create docker-compose based development environment |
Done |
|
Bug |
snowflake error related to Python 3.9.10 upgrade |
Done |
|
Bug |
zeep retry error - related to Python 3.9.10 migration |
Done |
|
Bug |
Mitto 3.0 / master should use CircleCI Python 3.9 image |
Done |
|
Bug |
mitto.Dockerfile in current master branch specifies wrong librocksdb |
Done |
|
Bug |
Notes Section of Mitto Job display/ edit issue |
Done |
|
Improvement |
explore eliminating Pandas and NumPy dependencies |
Done |
|
Improvement |
replace ‘xlrd’ Python package |
Done |
|
Improvement |
upgrade to more recent version of Postgres |
Done |
|
Improvement |
Convert BaseTransform to Pydantic class |
Done |
|
Improvement |
Convert BaseStep to Pydantic class |
Done |
|
Bug |
Angular strict mode and legacy plugins wizards issue |
Done |
|
Bug |
condition evaluates to constant |
Done |
|
Task |
remove aggressive filtering of Pylint error messages |
Done |
|
Bug |
incorrect Tableau Export view filter encoding |
Done |
|
Bug |
static files for plugins apparently not handled correctly in 3.0 |
Done |
|
Bug |
Job editor too small |
Done |
|
Bug |
vizql export ignores personal access token creds |
Done |
|
Improvement |
documentation: remove mention of Tableau Export Job from job pag |
Done |
|
Improvement |
Final changes for 2.9.8 |
Done |
|
New Feature |
Auto reload on front end development code change |
Done |
|
Improvement |
Update Tableau Export documentation to include Personal Access Token credentials |
Done |
|
Improvement |
Update Readme with link describing how to authenticate with Zuar’s ECR |
Done |
|
Bug |
Redundant query params on jobs search |
Done |
|
Bug |
CI: JS Tests error |
Done |
|
Bug |
Disappeared edit buttons |
Done |
|
Bug |
Tags apply to Type filter |
Done |
|
Bug |
vsftpd is broken |
Done |
|
Bug |
‘make docker_jstest’ not working |
Done |
|
Improvement |
resolve TODOs |
Done |
|
Improvement |
Makefile + documentation tweaks |
Done |
|
Improvement |
add link to documentation for custom crontab spec |
Done |
|
New Feature |
Pagination: Go directly to page number. |
Done |
|
New Feature |
Add pagination info to querystring |
Done |
|
Improvement |
investigate CircleCI caching of flake8 and lint results |
Done |
|
New Feature |
Enable Tag filter dropdown on all job pages |
Done |
|
Bug |
Can’t backspace after typing in search |
Done |
|
Bug |
Error adding Sequence (on dev) |
Done |
|
Improvement |
Update Settings > Password UI |
Done |
|
Bug |
Missing selection styling on Files list |
Done |
|
Improvement |
Styling job title as link |
Done |
|
Bug |
Cannot unselect Job using checkbox |
Done |
|
Bug |
Missing info when zero jobs in tag |
Done |
|
Bug |
Capitalize “Ran during” selected values |
Done |
|
Bug |
No way to select all jobs in card view |
Done |
|
Bug |
Tags in lists and cards |
Done |
|
Improvement |
adjust wizard layout |
Done |
|
Improvement |
Style login page to match new design |
Done |
|
Bug |
Cannot click Logout button when error banner is shown |
Done |
|
Improvement |
add snapshot job |
Done |
|
Improvement |
Upgrade Python to 3.8+ |
Done |
|
Improvement |
Migrate from Webpack to Vite |
Done |
|
Improvement |
Log all error information when job returns non-zero exit code |
Done |
|
Improvement |
Investigate parameter to skip |
Done |
|
Improvement |
Support Tableau Personal Access Tokens for authentication |
Done |
|
New Feature |
Allow suspension of job running and re-enablement to make upgrades more convenient. |
Done |
|
New Feature |
Test automation for acceptance testing |
Done |
|
Bug |
handle command job with no command |
Done |
|
Bug |
Dockerized mitto: backup stopped working |
Done |
|
Improvement |
Documentation updates |
Done |
|
Improvement |
Deployment of dockerized Mitto |
Done |
|
Improvement |
New licensing scheme |
Done |
|
Improvement |
XSV wizard: reduce the total numbers of sniffing the CSV file |
Done |
|
Improvement |
StoreInput - don’t create a store if one does not exist |
Done |
|
Epic |
Standalone Mitto package |
Done |
|
Epic |
Docker support |
Done |
|
Improvement |
CreateTable / CreateTempTable steps - add logging of table DDL |
Done |
|
Improvement |
Slow S3 uploads |
Done |
|
Bug |
Stage sorting isn’t working |
Done |
|
Bug |
Export - AttributeError: ‘CSVExport’ object has no attribute ‘log_requests’ |
Done |
|
Improvement |
JSON-Editor add custom editor for jobs select view |
Done |
|
Bug |
io job with sdl but no columns key - TypeError: ‘Dict’ object is not callable |
Done |
|
New Feature |
Create $MITTO_DATA/store directory |
Done |
|
Improvement |
file search - add case insensitive flag / checkbox |
Done |
|
Bug |
exception in builtin.CreateTable |
Done |
|
Bug |
Invalid crontab string saved for schedule |
Done |
|
Bug |
Google Adwords wizard breakage |
Done |
|
Bug |
stale browser data causes 422 on jobs page after mitto upgrade |
Done |
|
Improvement |
Fix Tableau wizard validation error. |
Done |
|
Improvement |
Job Actions - Improve wait logic and process handling |
Done |
|
Task |
Tests - Try to get rid of session fixture (again) |
Done |
|
Bug |
job logs - missing data |
Done |
|
Bug |
update frontend for v2 jobs api |
Done |
|
Bug |
Add back functionality to allow check for job existence by name |
Done |
|
Improvement |
add encoding as a parameter to the Excel input (and other file inputs) |
Done |
|
Improvement |
Allow creation of report of a release’s Jira tickets without overwriting remainder of release notes. |
Done |
|
Improvement |
Revist need for Jpype fork |
Done |
|
Bug |
job config - hjson - commenting last line of object moves that line to line above |
Done |
|
Improvement |
job duration value - more granularity |
Done |
|
Bug |
Job completion times for sequence are incorrect |
Done |
|
Improvement |
Extend file download popup to include “Open” and “Download” buttons |
Done |
|
Bug |
global filters - multi selected values text outside button |
Done |
|
Improvement |
IMAP plugin icon |
Done |
|
Improvement |
[Blocked] wizards - sort credentials alphabetically |
Done |
|
Bug |
Snowflake - Object ‘TEST’ already exists. |
Done |
|
Improvement |
Tableau Extract Wizard - only display credentials valid with Tableau |
Done |
|
Improvement |
Tableau Extract Wizard - detect absence of extracts on site |
Done |
|
Improvement |
DropStore step - add store/name parameter to drop a specific store |
Done |
|
Bug |
job edit - job types with more than one tab - prompted to save when no changes made |
Done |
|
Bug |
query job - output to snowflake - table names with mixed capitalization - KeyError: ‘test_snowflake_naming_MIXED’ |
Done |
|
New Feature |
start/kill button on logs page |
Done |
|
Improvement |
IMAP - add job parameter to specify file types that will be downloaded |
Done |
|
Improvement |
Disk usage - more specifics (/var/mitto/data, /var/mitto/store, database, etc) |
Done |
|
Improvement |
Webhooks - add notes section (like notes on job page) |
Done |
|
Improvement |
Subscriptions - ensure info about each failed subscription appears in job’s error output |
Done |
|
Improvement |
IO - Handle max column length on type upgrades |
Done |
|
Improvement |
use site_id instead of “site name” with TableauAuth() |
Done |
|
Improvement |
handle job warning messages differently than error messages |
Done |
|
Improvement |
output to JSON (equivalent of mittio.iov2.tocsv) |
Done |
|
Improvement |
wizard improvement |
Done |
|
Improvement |
add “Store” button to jobs to easily see store data in the UI |
Done |
|
Improvement |
add “Data” button to jobs to easily see database data |
Done |
|
Improvement |
add save button when editing sequences |
Done |
|
Improvement |
Use apt_auth.conf |
Done |
|
Improvement |
save output database credentials (connection string) and allow as an option when creating a job |
Done |
|
Improvement |
ability to find jobs without tags |
Done |
|
New Feature |
DATA EDITOR: drop down menu for enum data types |
Done |
|
Improvement |
SQLAlchemy does not recognize data type ‘point’ |
Done |