How to run Fabric using AWS Lambda and Paramiko?
We usually come across few scenarios where we need something to run on the remote servers for the sake of automation. I came across such a use case in AWS where I need to run fabric on a server to be triggered from AWS Lambda. This fabric script further does few tasks on other servers.
Background/Scenario: While working on a spot instance, my architecture needs few files to be updated when a spot instance comes up or goes down. AWS Lambda gets triggered by the spot instance according to the events. Since AWS Lambda supports few language/platforms including Python, I thought using Fabric would be a good idea to trigger from Lambda but Fabric needs “fab” binary to get executed and AWS Lambda does allow us to install anything on it. So, incorporated one EC2 instance on which few DevOps tools such as Chef and Jenkins were already running and installed Fabric on it.
Now the next task was to execute the fabfile present on the server. Since Python on AWS Lambda does not have Paramiko module, I had build a package which contains the Paramiko module along with my pPthon code and uploaded the same on AWS Lambda. This would trigger Fabric configured on the server.
Here are the steps which can help you to implement the above scenario.
- Install Fabric on a server and create a fabfile at any desired location containing your logic (say /opt/fabfile.py). This could be done on any existing EC2 instance running any utility service as Fabric would need few resource to run.a. Install a virtualenv on the system:
[js]sudo pip install virtualenv
#Or
sudo apt-get install python-virtualenv[/js]b. Setup/use virtualenv.
[js]#create a folder for your app
virtualenv ~/my_app
#Activate the virtualenv
cd ~/my_app/bin
source activate[/js]c. Install the Paramiko module inside the virtual environment:
[js]pip install paramiko[/js]
d. Add your Python source file:
[js]cd ~/my_app
vi my_script.py
import sys
import time
import select
import paramiko
def functionName(event, context):
host = ‘x.x.x.x’
i = 1
while True:
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host,username="user",password="Pasword",port=22)
print "Connected to %s" % host
Break
except paramiko.AuthenticationException:
print "Authentication failed when connecting to %s" % host
sys.exit(1)
Except:
print "Could not SSH to %s, waiting for it to start" % host
i += 1
time.sleep(2)
# If we could not connect within time limit
if i == 30:
print "Could not connect to %s. Giving up" % host sys.exit(1)
# Send the command (non-blocking)
stdin, stdout, stderr = ssh.exec_command("sudo /usr/bin/fab -f /home/username/fabfile.py copy")
print "Start Execuitng Fab"
"""
while stdout.readline():
print stdout.readline()
ssh.close()
"""
while not stdout.channel.exit_status_ready():
# Only print data if there is data to read in the channel
if stdout.channel.recv_ready():
rl, wl, xl = select.select([stdout.channel], [], [], 0.0)
if len(rl) > 0:
# Print data from stdout
print stdout.channel.recv(512),
#
# Disconnect from the host
#
print "Command done, closing SSH connection"
ssh.close()
functionName("asdfsafd","Asdfasdf")[/js]e. Copy all content from lib/python2.7/site-packages and lib64/python2.7/site-packages directories into ~/myapp and compress the whole package into a zip file.
- Create a zip package to upload on an AWS Lambda function.
- Upload the zip on the AWS Lambda function. Against “Handler” field in AWS Lambda function write my_script.functionName. After running it, errors could come which are to the unavailability of few “.so”. Add the required “.so” file under “my_app” and update the zip.
- After all required “.so” file is available, the Lambda function would do the desired task.
This use case is typically helpful for the scenario where we want to perform few tasks from a server which we have used as our utility server for which we have few scripts ready and needs to be triggered after an event.
