System Monitor

Starting in Mitto 2.9, users can add a system job that monitors system resources including memory, disk space and services.

To create this job, click on the Add Job button on the bottom left of the Zuar Runner UI, and select “Generic Job”.

Enter a title like: [system] monitor memory and disk, under type select “System”.

Use the following config:

{
    // job type should be `system`
    // writes to a file named `system_status.jsonl`
    "json_file": "system_status",
    "max_disk": 90,
    "max_memory": 90,
    "write_json": "true"
}

The system job will fail if the resources on the machine are above the set thresholds, or if any of the system’s services are down.

If write_json is set to true, every time the system job runs it will write JSON data to a JSON lines file defined by json_file. This file will include the amount of disk and memory used, as well as any jobs that were running at the time.

max_memory and max_disk set the thresholds at which the system job will fail for memory and disk usage. The default setting for each is 90 %.

A common practice for using system jobs is to set up a webhook on job failure to receive notifications when the resources on the box are failing.

To create a webhook, click the little orange edit pencil next to Webhooks. Under Event select “JOB_FAILURE”. Enter the URL for the webhook and the JSON to send to it.

To test the webhook, set the “max_disk” to a number below the current disk usage found in the UI on the Settings Tab. Ensure that the Event selected is “JOB_FAILURE” and click the Test link in the Edit Webhook modal. Remember to reset the “max_disk” to 90 after the test.

Below is an example config that works for webhooks sent to a slack channel, including the current memory and disk usage and the URL of Zuar Runner:

{
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": ":rotating_light: Mitto Job Failed!"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "Mitto URL: https://${system['fqdn']}\nJob Title: ${job['title']}\nJob URL: https://${system['fqdn']}/#!/job/${job['id']}"
      }
    }
  ]
}

The json above will send a notification similar to this to the specified Slack channel upon job failure:

For more example job configs including configs for pulling the data out of the JSON lines file, see the Zuar Runner job templates repo .