Bonus Content - TextFSM CLI Parsing
We aren't always afforded the privilege of having YANG models for every possible device or traditional CLI based data gathering exercise. In this module we'll show you a practical way of scaling CLI parsing for scenarios where MDP (Model Driven Programmability) is not an available option.
What is TextFSM?
TextFSM (Text Finite State Machine) is a Python module that helps with parsing CLI and text based output in a reliable and consistant fashion. Many times as part of network test and orchestration automation, one encounters situationas where there is absolutely no choice other that parsing CLI output. This is where TextFSM can shine. In this bonus module, we use Paramiko and TextFSM to get structured Python data from unstructured CLI data using a deterministic finite state machine. Additional Links:
Install TextFSM
Install TextFSM into your container environment:
pip3.6 install textfsm
[root@cfa2c7b31323 workspace]# pip3.6 install textfsm
Collecting textfsm
Downloading https://files.pythonhosted.org/packages/a1/0d/a1b490503545b3b4600b965eae5d44cc2b6ce27cfb44f4debc563dbb56d3/textfsm-0.4.1.tar.gz
Installing collected packages: textfsm
Running setup.py install for textfsm ... done
Successfully installed textfsm-0.4.1
[root@cfa2c7b31323 workspace]#
Install Paramiko
We're going to use Paramiko to interact with the CLI over SSH. Go ahead and pip install it.
pip3.6 install paramiko
Create Script
Create a new script called textfsm_cli_parse.py in your workspace:
textfsm_cli_parse.py
Imports
With pip installs done, you now need the following imports. Also go ahead and define the access information to your XRv device. Conceptually the CLI parsing concepts you learn in this module can be applied to any device that sends text-based CLI output.
import textfsm
import paramiko
import tempfile
import time
# Connection info
ip = '10.2.100.12' # XRv
user = 'admin'
pwd = 'cisco.123'
Declare the Template
In this exercise, we want to take unstructured CLI output from the "show clock detail" command, and have that converted into a Python datastructure more useful for programming and network automation. TextFSM operates as a state machine, taking raw CLI output and a template as the inputs to the state machine, and producing structured data in the output of the state machine. Below is one example of using a TextFSM template for parsing "show clock detail" on XRv.
template_show_clock_detail = r"""Value month (\w+)
Value year (\d+)
Value day (\d+)
Value hour (\d+)
Value minute (\d+)
Value sec (\d+)
Value millisec (\d+)
Value timesrc ([a-zA-Z ]+)
Value timezone (\w+)
Value wday (\w+)
Start
^${hour}:${minute}:${sec}\.${millisec}\s\w+\s${wday}\s${month}\s${day}\s${year}
^Timezone:\s${timezone}\sTimesource:\s${timesrc}
"""
The Parse Function
We need to define a function that takes a TextFSM template, and raw CLI output, and returns structured data using TextFSM. The comments in the function describe what's happening in more detail.
def parse_textfsm(template, output):
# Define the text processing template for show clock detail output.
# Create temp file to hold template.
tmp = tempfile.NamedTemporaryFile(delete=False)
# Write template to file for textfsm.
with open(tmp.name, 'w') as f:
f.write(template)
# Get read handle for textfsm.
with open(tmp.name, 'r') as f:
# Instantiate a new TextFSM wrapper.
fsm = textfsm.TextFSM(f)
# Parse the output text according to template rules.
fsm_results = fsm.ParseText(output)
# Convert to list of dictionaries since TextFSM may return multiple
# 'row' results.
parsed = [dict(zip(fsm.header, row)) for row in fsm_results]
# Return the parsed data.
return parsed
Define the Command Executor
We also need to define a function to execute commands on a device using Paramiko. The function below takes a given command, and executes it over SSH using Paramiko on the device you supplied in your globals at the beginning.
# Define a function that we'll use to execute and return CLI output.
def cmd(command):
"""
Runs a CLI command on a given SSH session, returning output.
"""
# Create new paramiko instance
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ip, username=user, password=pwd)
# Run command and get stdin, stdout, std err streams.
stdin, stdout, stderr = ssh.exec_command(command)
# Close this handle, not using it.
stdin.close()
# Read regular stdout and std err
output_ok = stdout.read()
output_err = stderr.read()
# Return error text if exist else regular std out.
# Convert to utf-8 string since python3 paramiko gives byte object
if output_err:
return output_err.decode('utf-8')
else:
return output_ok.decode('utf-8')
Putting it All Together
With all the prerequisite functions defined, you can now execute the "show clock detail" command on the XRv device and observe the structured output. In this example we call "show clock detail" twice, parsing and printing the structured list-of-dictionaries output to demoonstrate that each parsed data structure contains the key-value pairs corresponding to the template we provided the state machine.
# Get show version output.
output_t0 = cmd(command="show clock detail")
# Parse the show clock details.
parsed_t0 = parse_textfsm(template=template_show_clock_detail, output=output_t0)
# Wait for a second to let clock advance.
time.sleep(5)
# Get show version output.
output_t1 = cmd(command="show clock detail")
# Parse the show clock details.
parsed_t1 = parse_textfsm(template=template_show_clock_detail, output=output_t1)
# Print the clock details to console.
# Print out the clock details
print(parsed_t0)
print(parsed_t1)
The Full Script
At this point your script should look as follows:
import textfsm
import paramiko
import tempfile
import time
# Connection info
ip = '10.2.100.12' # XRv
user = 'admin'
pwd = 'cisco.123'
template_show_clock_detail = r"""Value month (\w+)
Value year (\d+)
Value day (\d+)
Value hour (\d+)
Value minute (\d+)
Value sec (\d+)
Value millisec (\d+)
Value timesrc ([a-zA-Z ]+)
Value timezone (\w+)
Value wday (\w+)
Start
^${hour}:${minute}:${sec}\.${millisec}\s\w+\s${wday}\s${month}\s${day}\s${year}
^Timezone:\s${timezone}\sTimesource:\s${timesrc}
"""
def parse_textfsm(template, output):
# Define the text processing template for show clock detail output.
# Create temp file to hold template.
tmp = tempfile.NamedTemporaryFile(delete=False)
# Write template to file for textfsm.
with open(tmp.name, 'w') as f:
f.write(template)
# Get read handle for textfsm.
with open(tmp.name, 'r') as f:
# Instantiate a new TextFSM wrapper.
fsm = textfsm.TextFSM(f)
# Parse the output text according to template rules.
fsm_results = fsm.ParseText(output)
# Convert to list of dictionaries since TextFSM may return multiple
# 'row' results.
parsed = [dict(zip(fsm.header, row)) for row in fsm_results]
# Return the parsed data.
return parsed
# Define a function that we'll use to execute and return CLI output.
def cmd(command):
"""
Runs a CLI command on a given SSH session, returning output.
"""
# Create new paramiko instance
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ip, username=user, password=pwd)
# Run command and get stdin, stdout, std err streams.
stdin, stdout, stderr = ssh.exec_command(command)
# Close this handle, not using it.
stdin.close()
# Read regular stdout and std err
output_ok = stdout.read()
output_err = stderr.read()
# Return error text if exist else regular std out.
# Convert to utf-8 string since python3 paramiko gives byte object
if output_err:
return output_err.decode('utf-8')
else:
return output_ok.decode('utf-8')
# Get show version output.
output_t0 = cmd(command="show clock detail")
# Parse the show clock details.
parsed_t0 = parse_textfsm(template=template_show_clock_detail, output=output_t0)
# Wait for a second to let clock advance.
time.sleep(5)
# Get show version output.
output_t1 = cmd(command="show clock detail")
# Parse the show clock details.
parsed_t1 = parse_textfsm(template=template_show_clock_detail, output=output_t1)
# Print the clock details to console.
# Print out the clock details
print(parsed_t0)
print(parsed_t1)
Execute Python Script
python3.6 textfsm_cli_parse.py
The prints should yield something along the lines of:
[{'month': 'May', 'year': '2018', 'day': '25', 'hour': '15', 'minute': '32', 'sec': '54', 'millisec': '083', 'timesrc': 'Hardware calendar', 'timezone': 'UTC', 'wday': 'Fri'}]
[{'month': 'May', 'year': '2018', 'day': '25', 'hour': '15', 'minute': '32', 'sec': '59', 'millisec': '964', 'timesrc': 'Hardware calendar', 'timezone' 'UTC', 'wday': 'Fri'}]
Each print statement's output corresponds to a call to "show process details" structured parsing of the raw
XRv CLI output for that command.
mt-csrv#show clock detail
*16:33:56.880 UTC Fri May 25 2018
No time source
mt-csrv#
Summary
This was just a simple example showing how one can take a few foundational Python constructs, namely, a function for getting the CLI output, a function for taking CLI Output + Template = Structured output, and a template, and then creating very succinct Python call patterns for repeatable & deterministic CLI parsing. Many show commands already have open source templates available, for example, the Network to Code templates on GitHub. If you find yourself stuck parsing CLI, remember that using consistant templates are more efficient than having regular expresions sprinkled all around a large library of scripts; rather; doing the raw-to-structured translation through centralized tempaltes is good from a re-useability perspective.
