How I converted my bash scripts into Python?

Python is interpreted language and  can be used instead of bash scripts. For new automations and DevOps technologies Python is getting used because of it's simplicity, error handling and large number of libraries. Many organizations are planning to convert their existing shell scripts to Python.

Python and Shell scripts are similar in nature as both are interpreted i.e. executed statement by statement.

First of all you need to figure out what all things are used in existing shell script and figure out if any existing library is there for the same:

• Passing arguments
• CURL commands to call an API and Keycloak auth
• Kubernetes Client for submitting Job
• String manipulation for pattern matching

Below are some useful libraries for above functionalities and in general for converting bash to Python.

• argparse/click - for parsing arguments passed to a script (in detail) you can parse command line arguments which needs to be passed to the script.
• requests - for sending HTTP requests and get response like CURL
• pyyaml - for parsing YAML file, converting data to yaml and from yaml
• json -  for parsing json file, converting data to json and from json
• kubernetes - for interacting with kubernetes cluster
• python-keycloak - for creating clients, authentication, administrator activities over Keycloak

Let's see each library and use case in detail.

Parsing arguments passed to Script(argparse/click)

Passing parameters to bash script is quite straight forward we can use $1 for first parameter, $2 for second and so on. In case of Python we have different libraries like argparse which makes it easy to read and process arguments.

Install argparse library using pip tool:

pip install argparse

# OR

pip3 install argparse

Below is the example how argparse can be used to parse arguments and use them inside Python scripts, more info can be found here.

import argparse

parser = argparse.ArgumentParser(description="Description of your script")
parser.add_argument('--para1', help='This is parameter1', required=True)

print(parser.parse_args().para1)

• You can add multiple parameters and define help parameter as description for the particular parameter
• When running script with option "-h" this help will be printed

Sending HTTP request (requests/urllib3)

CURL commands are used to send HTTP request and get response as output. In Python we have requests library which is simple, yet powerful. You can use online CURL to python converter.

You can install requests library using pip tool:

pip install requests

# OR

pip3 install requests

Below is example of converting CURL command to Python requests, more info can be found here.

# Conversion of below CURL command to requests
# curl -k -X POST "http://keycloak-server.bootvar.com/token" -d "grant_type=client_credentials&client_id=bootvar&client_secret=secret"

import requests

data = {
  'grant_type': 'client_credentials',
  'client_id': 'bootvar',
  'client_secret': 'secret'
}

response = requests.post('http://keycloak-server.bootvar.com/token', data=data, verify=False)
print(response.text)
print(response.status_code)

Reading and parsing yaml files (pyyaml)

YAML is a data representation which can be used to pass and receive well structured data to various applications. In DevOps tools(like k8s, Jenkins) mostly YAML data representation is used.

It is challenging to read and parse YAML file in shell script, we need to parse file with some basic commands like sed and grep. For parsing YAML data we have library in Python called pyyaml.

You can install pyyaml using pip tool, more info can be found here.

pip install pyyaml

# OR

pip3 install pyyaml

PyYaml can be used as below:

import yaml

# Open and read yaml file
with open('bootvar_struct.yaml', "r") as stream:
  content = yaml.safe_load(stream)

for key, value in content['key'].items():
  print(str(key) + "=>" + str(value))

• import yaml - Pay attention it's yaml not pyyaml
• yaml.safe_load - can be used to load and convert raw data to YAML and can be iterated over

Reading and parsing JSON files (json)

Similar to YAML, JSON is also a data representation format. YAML and JSON are interconvertible. For parsing JSON we have binary called jq in linux(bash/shell).

For reading JSON we have json library in Python, you can install it using pip, more info can be found here.

pip install json

# OR

pip3 install json

Library can be used in following way(similar to pyyaml):

import json

# Open and read yaml file
with open('bootvar_struct.json', "r") as stream:
  content = json.safe_load(stream)

for key, value in content['key'].items():
  print(str(key) + "=>" + str(value))

Using Kubernetes Python Library

In shell we can use kubectl binary or curl commands to interact with kubernetes cluster, but it gets complicated if we want to do complete cluster orchestration via curl.

In python we have kubernetes library/package to interact with Kubernetes cluster, in similar way we can install kubernetes package using pip.

pip install kubernetes

# OR

pip3 install kubernetes

Below is the simple example for reading namespaced config-map, more information about Python client can be found here.

import kubernetes

kconf = k8sclient.Configuration()
kconf.host = "https://kube-server.bootvar.com:6443"
kconf.verify_ssl = False
kconf.api_key = {"authorization": "Bearer <TOKEN>" }
kClient = k8sclient.ApiClient(kconf)
v1 = k8sclient.CoreV1Api(kClient)

ret = v1.read_namespaced_config_map('configmap-name', 'namespace-name')

print(ret.data['config-key'])

Conclusion:

Shell scripts are sometimes difficult to debug, complex and unstructured to maintain.

Both python and shell scripts are interpreted, and we can easily convert shell scripts to python using ready to use libraries in Python.