# System Monitor

Starting in Mitto 2.9, users can add a `system` job that monitors
system resources including memory, disk space and services.

To create this job, click on the `Add Job` button on the bottom left
of the Zuar Runner UI, and select "Generic Job".

Enter a title like: `[system] monitor memory and disk`, under `type`
select "System".


Use the following config:

```json
{
    // job type should be `system`
    // writes to a file named `system_status.jsonl`
    "json_file": "system_status",
    "max_disk": 90,
    "max_memory": 90,
    "write_json": "true"
}
```

The system job will fail if the resources on the machine are above the
set thresholds, or if any of the system's services are down.

If `write_json` is set to true, every time the system job runs it will
write JSON data to a JSON lines file defined by `json_file`. This file
will include the amount of disk and memory used, as well as any jobs
that were running at the time.

`max_memory` and `max_disk` set the thresholds at which the system job
will fail for memory and disk usage.

A common practice for using system jobs is to set up a webhook on job
failure to receive notifications when the resources on the box are
failing.

To create a webhook, click the little orange edit pencil next to
`Webhooks`. Under `Event` select "JOB_FAILURE". Enter the URL for your
webhook and the JSON you want to send.

Below is an example config that works for webhooks sent to a slack
channel, including the current memory and disk usage and the URL of
Zuar Runner:

```json
{
    "text": "*SYSTEM MONITOR WARNING:* - *memory:* ${job['status']['kvp']['current memory usage']} *disk:* ${job['status']['kvp']['current disk usage']} - https://${system['fqdn']}"
}
```

For example job configs including configs for pulling the data out of
the JSON lines file, see the [Zuar Runner job templates
repo](https://github.com/zuarbase/mitto-job-templates/tree/master/system).