Automate Bash, R, and Python Scripts With Zuar Runner
In this blog post we will discuss the Zuar Runner command line job, and then walk through examples of automating scripts from various languages:
- Bash
- R
- Python
With Zuar's Runner platform, you can automate your ELT/ETL processes and have data flowing from hundreds of potential sources into a single destination. Transport, warehouse, transform, model, report, and monitor: it's all managed by Runner.
Related how-to video:
Runner Command Line Jobs
A powerful feature in Runner is the ability to automate commands and scripts with command line jobs.
These "cmd" jobs allow Runner users to run any command on the shell of their Runner instance.
An example might be a job that looks like the following:
{
"cmd": "echo 'hello world' > /var/runner/data/hello_world.txt",
"cmd_env": {},
"exec": false,
"shell": true
}Instructions for creating a command line job in Runner can be found here.
In our example case, title the job [cmd] Hello World!, add the JSON configuration above, and click "Done". On the newly created job's page, click the little pencil to edit the job type. Select cmd from the select menu.
Running this job will create a text file at: /var/runner/data/hello_world.txt.
You can verify it's there by looking on the Files page in Runner (clicking on Files in the menu on the left).
Here are the contents of the hello_world.txt:
Bash Scripts With Runner
Alternatively, you can upload and run a bash script in your Runner command jobs.
Bash, the most commonly available unix shell, is
the shell, or command language interpreter, for the GNU operating system.
Bash commands can be saved in a file and executed as a script.
The following Bash script (test_bash_script.sh) creates a CSV file: written_by_bash.csv
#!/bin/bash
echo "id, name" > /var/runner/data/written_by_bash.csv
echo "1, Allen" >> /var/runner/data/written_by_bash.csv
echo "2, Steve" >> /var/runner/data/written_by_bash.csvUpload the Bash script to Runner via the file manager.
Then create and run the following command job. Once again, the job type needs to be set to cmd.
{
"cmd": "bash /var/runner/data/test_bash_script.sh",
"cmd_env": {},
"exec": false,
"shell": true
}The resulting file should look like this:
R Scripts With Runner
You can also automate R scripts with Runner.
R is a language and environment for statistical computing and graphics.
The following script (test_r_script) will create a tab separated file (written_by_r.tsv) with the numbers 1-5 on the first line and the numbers 6-10 on the second line.
x <- matrix(1:10, ncol = 5)
fil <- tempfile("data")
write(t(x), fil)
if(interactive()) file.show(fil)
unlink(fil)
write(x, file = "/var/runner/data/written_by_r.tsv", sep = "\t")Upload the R script to Runner via the file manager.
Create another cmd job (HINT: you can duplicate an existing job) and edit the job's config:
{
"cmd": "Rscript /var/runner/data/test_r_script.R",
"cmd_env": {},
"exec": false,
"shell": true
}Finally, on the main job page, change the job type from io to cmd.
Running the job will create a file at: /var/runner/data/written_by_r.tsv. You can verify this file exists in the file editor. The resulting file should look like this:
Python Scripts With Runner
Automating a Python script is a similar process in Runner with a command job.
"Python is an interpreted, high-level, general-purpose programming language."
The following script (test_python_script.py) will create a CSV: written_by_python.csv.
#!/usr/bin/env python3
"""NOTE: You must use python3"""
f = open("/var/runner/data/written_by_python.csv", "w")
f.write("id, name\n1, Allen\n2, Steve\n")
f.close()One thing to note is that #!/usr/bin/env python won't work, as Runner has python3 installed.
Upload the Python script to Runner via the file manager.
Again, we'll create a simple job with the following JSON config, and we'll be sure to change the job type from io to cmd.
{
"cmd": "python3 /var/runner/data/test_python_script.py",
"cmd_env": {},
"exec": false,
"shell": true
}The resulting file should look like this:
Learn more about automating Python scripts with Runner.
Conclusion
The script examples covered above were extremely simple, however they show the power of automating scripts in Runner with the command line job.
While the examples above cover a lot of use cases, they only touch the surface of the Runner command line job.
Some other command line interface utilities Runner users use include:
While Runner has an extensive (and growing!) plugin library to facilitate many automation tasks out of the box, command line jobs enable many more advanced automation use cases.
To close, here are a few helpful Runner-related links:
Team Zuar
Greg Rossi
