Improved Scripting Practices — Python
As we all know that Python usage has increased lately and it has reached a point in time where we use Python scripting to fulfill every small automation need. Python is very simple language to learn and anyone can start scripting in it within a day of learning. I personally would not recommend Python as a primary programming language to start with for any of the starters out there, but if your requirement is to finish something so quick, then you can go ahead and implement using Python.
So when we write these Python scripts there are ways to improve our scripting practice by many means. Following are the certain set of things which I improved in my scripting over the last few months. Most of you may already know these things. But this article is for the some of you who just started your scripting life and want to improve the same. The following things are applicable for both Python 3 and Python 2 versions.
1. Understanding Absolute Path vs Relative Path:
Ever since I started scripting, I always found issues with the path usage inside my script. Let’s take a simple scenario where we might face this issue. We all use scripting to automate/schedule something. Scheduling leads to cron usage in Linux systems. So when we use crons, we need to specify the full path to where our script is placed. Just like below,
0 * * * * /usr/bin/python /opt/some/path/blahBlah.pySo, in the above case, inside blahBlah.py if we have to read/write a file, then we may need to specify absolute path to the file. Relative paths won’t work in this case. Using absolute paths may be an overhead if you’re gonna keep all the required files in the same path as your script.
Lately, I found a new way and I started using it.
I started to assign a variable in all my scripts to get the path of current working directory from which I’ll use relative path. This seems to work for most of the cases, when you know the path of the file that is gonna be used relative to the path of your script. Let’s take the following directory structure
/
|
-> opt/
|
-> myScript/
|
-> blahBlah.py
fileToRead.txtHere in this case fileToRead.txt is always gonna be in the same folder as my script. So we can use the following script to improve our path usage.
Note: This works in both Linux based and Windows platforms.
import os# Define Script Directory.
# In our case scriptPath will contain '/opt/myScript'
scriptPath = os.path.dirname(os.path.abspath(__file__))...# Usage
fileData = open(scriptPath + '/fileToRead.txt', 'r').read()
2. Logging Improvements:
This is one of the most basic cross cutting concern which is essential for troubleshooting our script at anytime. Many of us while starting our scripting life will probably append our script’s output to a file for logging, by starting the script like this,nohup python blahBlah.py >>blahBlah.log 2>>&1 & .
But as time goes on, this file blahBlah.log size may increase and will become difficult when we try to debug something.
If you guys started with logging when you started scripting then it’s a great thing. Else, you guys can use TimedRotatingFileHandler under Python’s logging.handlers. We have many other logging handlers under python’s logging library but I personally prefer this one. Check the references below in case you want to check other handlers. TimedRotatingFileHandler supports rotation of log files at certain timed intervals. I’ll use the following method to initialize the logging in any of my scripts and in places of using print method/statement, I’ll use logging.
import logging
from logging.handlers import TimedRotatingFileHandler# Initialize logger function
def initLogger():
logger = logging.getLogger("Rotating Log")
formatter = logging.Formatter(
"%(asctime)s | %(levelname)s | %(module)s | %(message)s"
)
logger.setLevel(logging.DEBUG)
log_handler = TimedRotatingFileHandler(
"/var/log/some/path/blahBlah.log",
when = "d",
interval = 1,
backupCount = 5
)
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)
return logger...# Call Initialize method
logger = initLogger()...# Usage
logger.info("Some Information Here")
logger.debug("Some Debugging Level Information Here")
logger.warning("Some Warning Message Here")
logger.error("Some Error Log Here")
The configuration that I’ve used here will be rotating the logs each day at 12:00 AM and will delete the backed up logs which are older than 5 days. Formatter that I’ve used is for basic formatting, you can check references to get to know formatting in detail.
3. Configurable scripting:
Configuration is one of the key elements which reduces code changes for each and every simple changes like path modifications/environment based modifications in our scripts. My personal preference would be configparser which is a python library that lets you parse configurations similar to Microsoft’s ini files. We have other config parsing libraries like config and configs which can also be used. You can check the references for understanding each of the libraries.
Here is an example on how this can be done.
Let’s take the previous scenario of logging. We can configure the log externally by creating a file named config.ini and by fetching the configuration whenever the script runs. A configuration file for logging scenario may contain following configurable items.
[Logger]
type=Rotating Log
path=/var/log/some/path/blahBlah.log
rotateInterval=1
backupCount=5
when=d
formatter=%(asctime)s | %(levelname)s | %(module)s | %(message)sHere, for fetching first 5 configured items, we can use configparser.ConfigParser() method. But for 6th configuration item, we have special characters. So we need to use configparser.RawConfigParser() to fetch the raw data without any issue. Take a look at the blahBlah.py script below for understanding the same.
import os
import logging
from logging import TimedRotatingFileHandlerimport configparserscriptPath = os.path.dirname(os.path.abspath(__file__))# Initialize config
config = configparser.ConfigParser()
config.read(scriptPath + '/config.ini')# Initialize raw config
rawConfig = configparser.RawConfigParser()
rawConfig.read(scriptPath + '/config.ini')# Improved initialize logger method
def initLogger():
logger = logging.getLogger(config['Logger']['type'])
formatter = logging.Formatter(
rawConfig['Logger']['formatter']
)
logger.setLevel(logging.DEBUG)
log_handler = TimedRotatingFileHandler(
config['Logger']['path'],
when = config['Logger']['when'],
interval = int(config['Logger']['rotateInterval']),
backupCount = int(config['Logger']['backupCount'])
)
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)
return logger
References:
