System Monitor

Starting in Mitto 2.9, users can add a system job that monitors system resources including memory, disk space and services.

To create this job, click on the Add Job button on the bottom left of the Zuar Runner UI, and select “Generic Job”.

Enter a title like: [system] monitor memory and disk, under type select “System”.

Use the following config:

{
    // job type should be `system`
    // writes to a file named `system_status.jsonl`
    "json_file": "system_status",
    "max_disk": 90,
    "max_memory": 90,
    "write_json": "true"
}

The system job will fail if the resources on the machine are above the set thresholds, or if any of the system’s services are down.

If write_json is set to true, every time the system job runs it will write JSON data to a JSON lines file defined by json_file. This file will include the amount of disk and memory used, as well as any jobs that were running at the time.

max_memory and max_disk set the thresholds at which the system job will fail for memory and disk usage.

A common practice for using system jobs is to set up a webhook on job failure to receive notifications when the resources on the box are failing.

To create a webhook, click the little orange edit pencil next to Webhooks. Under Event select “JOB_FAILURE”. Enter the URL for your webhook and the JSON you want to send.

Below is an example config that works for webhooks sent to a slack channel, including the current memory and disk usage and the URL of Zuar Runner:

{
    "text": "*SYSTEM MONITOR WARNING:* - *memory:* ${job['status']['kvp']['current memory usage']} *disk:* ${job['status']['kvp']['current disk usage']} - https://${system['fqdn']}"
}

For example job configs including configs for pulling the data out of the JSON lines file, see the Zuar Runner job templates repo.