Ideas for using scripting in a developer’s daily life
This article aims to present the importance of scripting and automation in the “daily life” of a developer, that is, to demonstrate that scripting can prove to be a very important tool for automating many repetitive tasks.
The plus, in the writer’s opinion, is that scripting allows one to automate many tasks by actively exercising one’s problem solving skills.
There are many tools that allow one to “bypass” scripting, but two advantages of simple scripting may be:
- it allows you to actively learn and practice your programming skills;
- it does not require you to install any new software on your system.
We are not claiming here that scripting is the solution to every problem; instead, we are suggesting that through scripting one can solve many problems to one’s satisfaction by understanding the underlying principles.
Scripting Languages
There are nowadays a great many scripting languages. It is not possible to mention even the main ones here. We limit ourselves to briefly presenting three languages:
- Bash,
- PowerShell,
- Python.
Bash can be used primarily on Unix systems, but there are a few programs available that allow it to be used on Windows as well. PowerShell, conversely, is built into Windows systems, but is also downloadable to many Unix systems. Finally, Python: a true programming language, complete and multifunctional, which, due to its interpreted nature, lends itself well to scripting.
Bash
Bash (acronym for Bourne Again SHell, cf. Wikipedia), is a GNU shell. The syntax that is used to issue commands to the shell is also referred to by the same name. Bash is referred to in this sense as a language.
The advantage of Bash is that it is now found on almost every Unix system. If you are dealing with a Linux server, for example, you are likely to find Bash installed there (or some other similar shell, such as Zsh, which has a similar and compatible syntax in many ways).
What can Bash be used for? For almost any automatable task. Let’s look at a concrete example.
Bash: Example
Bash can be used, for example, to automate the backup of a database.
Let us imagine, for example, that we have a MySQL database running on a Linux server (we can imagine Debian or Ubuntu). Let us further imagine that the backup is to take place at the beginning of each month.
The Linux system provides us with cron jobs, tasks that run automatically at a certain deadline or cyclically on the system.
We can then schedule cron a task that executes a database backup script.
The command that backs up a MySQL database is
mysqldump -u [username] –p [password] -h [hostname] [database_name] > [dumpfilename .sql]
Let us further imagine that we need to copy the dump to another server, such as a file server used for storing historical data. We can perform the dump copy by means of the command scp:
scp -i [private_key] [username]@[hostname/IP]:[location_in_filesystem]
Let us begin to integrate the commands seen so far into a script:
#!/bin/bash
time=$(date +%Y%m%d%H%M%S)
db_pass=$(cat /home/user/.secret_db_pw_file)
mysqldump -u user –p "${db_pass}" my_database > "/tmp/my_db_dump_${time}.sql"
scp -i /home/user/.ssh/server_key "/tmp/my_db_dump_${time}.sql" user@remote.backup.server:/var/backups
Finally, we save the script in /usr/local/bin, let’s make it executable and add to the database server crontab something like the following:
0 0 1 * * /usr/local/bin/backup.sh
Of course, the script can be modified to save the backup in a different way, for example to an AWS S3 bucket: just know the commands to execute and add them to the script!
PowerShell
PowerShell is a shell available for various systems, developed by Microsoft and widely used for managing Windows servers. More information is available in Microsoft’s documentation and on Wikipedia.
PowerShell, unlike other shells, works using objects called cmdlets (“command lets”): we can therefore imagine PowerShell as an “object-oriented” shell. From Wikipedia:
La differenza fondamentale tra l'approccio Unix e quello di PowerShell risiede nel fatto che piuttosto che creare una "pipeline" (lett. tubo) basata su input ed output testuali, PowerShell fa passare i dati da una cmdlet all'altra come oggetti (dati dotati di una struttura ben precisa).
We can create a PowerShell script by simply naming a file with the extension .ps1; execution is launched by simply calling the script name with relative or absolute path relative to the execution directory, e.g:
.\my_script.ps1
PowerShell: Example
We present an example of a PowerShell script that allows a series of files to be downloaded from a remote server using the scp command.
Let us imagine that we have a server (let us say, for ease, Linux) that stores a long series of files, the names of which are predictable. We can, specifying more, imagine that we have a list of ZIP files called split.zXX, where XX denotes a number between 01 and 99. Performing the manual copy of all the files requires launching 99 scp commands; while, on the one hand, there are tools that allow us to use a graphical interface to perform this operation, a few lines of a PowerShell script allow us to achieve the same goal, learn the syntax of the language and avoid installing new tools.
PowerShell (as well as Bash and other languages) allows the execution of for loops. So we write a for loop in PowerShell that formats the filename and downloads it:
for ($i=1; $i -le 99; $i++)
{
if ($i -le 9)
{
$j = "0$i"
}
else
{
$j = "$i"
}
scp -i C:\Users\User\.ssh\server_key user@${SERVER_IP}:/home/user/split.z$j .
}
Let’s save this loop to a file called download.ps1 and run it:
.\download.ps1
At the end of the execution, we will have the files saved in the local system.
Python
More than a scripting language, Python is a comprehensive and multi-functional programming language. This is not the space to present the language itself; rather, here we want to focus on its potential for scripting.
Python has, in our opinion, some essential advantages in the landscape of scripting languages:
- it integrates a great deal of functionality through its many libraries;
- it is found installed by default on many UNIX systems;
- it has a simple, essential and clean syntax.
Python can be used for many different purposes in scripting: system administration, network management, data processing, file processing (both text and images). In addition, Python is much used as a language for Artificial Intelligence: thus, one has many possibilities to integrate AI tools into a script in Python.
Python: Example
Let’s look at a couple of simple examples of scripting in Python.
The first example allows us to query an API endpoint for a reference value, sending an alert when a certain maximum threshold or minimum threshold is exceeded.
Let us imagine that the data returned by the API has a JSON format of the type:
{
'time': Timestamp,
'data' : {
'key1': Number,
'key2': Number,
'key3': Number
}
}
We can then monitor the range of values returned with the following script:
import requests
url = '...'
data = requests.get(url)
data_json = data.json()
rate = data_json["data"]["key3"]
if rate > 50_000:
print("More than 5_000!")
elif rate < 20_000:
print("Less than 2_000!")
else:
print("The value is in the expected range...")
In the second case, let’s see how to edit files using Python. Suppose we want to generate a set of files in JSON format. Suppose we are working with the same format as in the previous example. We want to generate files with the current timestamp and key1, key2, key3 values calculated following a specific procedure.
import os
import json
from datetime import datetime
def get_val1():
...
def get_val2():
...
def get_val3():
...
if not os.path.exists("files"):
os.mkdir("files")
for i in range(10):
val1 = get_val1()
val2 = get_val2()
val3 = get_val3()
my_json = {
"time": datetime.now(),
"data": {
"key1": val1,
"key2": val2,
"key3": val3
}
}
file_number = f"0{i}"
file_path = f"files/file{file_number}.json"
with open(file_path, "w") as f:
f.write(json.dumps(my_json, default=str))
We hope to be able to give with this example an idea of the complexity of the tasks that can be accomplished by means of a Python script. Here we are using:
- the os module to manage directories on the filesystem;
- the json module to work with the JSON format;
- the datetime module for handling dates and times.
The functions get_val1, get_val2, and get_val3 allow us to calculate the values to be saved in the JSON object; the script also checks that the files directory is present in the path from which it is executed and, if it does not find it, creates it. The for loop then represents a cycle of iterations of working on different files.
Conclusions
This short article is just a taste of what can be done by means of scripting with a little perseverance, some Web research, and a little inventiveness: few problems resist a solution that combines such ingredients.
Continue reading more tech-pills in our tech-blog!
