I’m calling this a Remote Service Monitor because that what I’m using it for, but really it can pass any command to the linux shell and watch for any specific keyword in the response. This script first pings the remote machine, if the ping fails it sends out an email, if the email fails, it writes to the log. However, if the ping is successful, it then attempts to open a SSH connection with the remote machine, if that fails, it both writes to the log and sends an email. If the SSH connection is successful, then it sends whatever command you define and then reads the response from the server in. Finally it scans the response for the defined keyword, if it finds that keyword, it writes to the log and sends an email. If it doesn’t find the keyword, it just writes to the log.
Example: So lets say you want to monitor a particular service like ntpd on a remote linux box, or like in my case, just across the network. If the process is stopped, or if the server doesn’t respond, you want to get an email telling you so. I set the remoteHost to the proper IP, set the user/pass for the remote server, set the commandToSend to “service ntpd status” which if the process is running it returns;
Then, I set the keyword to “stopped” because if the service is stopped, the response from the server is:
Then, you fill in the rest of the variables, define where you want the log to be written. Set your SMTP server (look in your email client) and set the sender and receiver addresses. Most SMTP server’s won’t accept SMTP requests from public (residential) IP’s, but many mail servers will accept SMTP from business grade IP’s, if your using this at work, give it a shot, it has a decent chance of working.
Then I set it on a cron job for once every 10 mins, and wait for the process to stop. :: Click here for a walk through on cron ::
If the ping fails = You get an email, or it writes to the log
If the ping is successful = Write to log, and Continue
If the SSH connection fails = You get an email, and it writes to the log
If the SSH connection is successful = Write to log, and Continue
If the keyword is found in the response = You get an email, and it writes the log
If the keyword is not found in the response = Write to log, end.
To run this script on windows change the flag “-c” to “-n” on line 46.
For my purposes, the ping was as important as verifying the service was running.
Its ‘over-commented’ on purpose.
(if the code doesn’t display properly, try a shift-refresh)
#!/usr/bin/python
# You'll probably need to install paramiko, and maybe subprocess
import subprocess
import os
import paramiko
import logging
import logging.handlers
import smtplib
# Set your Variables
# Requires SMTP email server, no authentication
remoteHost = "IP Address or hostname" # host to ping and connect to
user = "username" # username to login to remoteHost
passwd = "password" # password for the above username
commandToSend = 'command to send to shell' # the command to send to remoteHost
failedKeyword = 'keyword to watch for in response' # Keyword to watch for in the response
logName = '/path/to/log' # path to where you want the log file to be
smtpHost = "smtp.domain.com" # SMTP server ip/hostname
sender = 'email@domain.com' # email sender
receiver = 'email@domain.com' # email receiver
# logging configuration, set to 500 MG worth of logs in 5 files, roller works
logging.basicConfig(level=logging.INFO, filename=logName)
logger = logging.getLogger('myLogger')
formatter = logging.Formatter("##### {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}(asctime)s\t")
handler = logging.handlers.RotatingFileHandler(logName, maxBytes=1024*1024*10, backupCount=5)
handler.setFormatter(formatter)
logger.addHandler(handler)
# email messages, fill these out with the information you want emailed at
# each email point; ping failure, ssh failure, keyword found
msgPing = """From: Display Name <email@domain.com>
To: ToWhom <email@domain.com>
Subject: Text Goes Here """
msgSSHlogin = """From: Display Name <email@domain.com>
To: ToWhom <email@domain.com>
Subject: Text Goes Here """
msgProccesses = """From: Display Name <email@domain.com>
To: ToWhom <email@domain.com>
Subject: Text Goes Here """
# opens subprocess and pings remoteHost
# if running on Windows, change flag from -c to -n
# this watches for one ping response, up 1 to 3 for three responses
res = subprocess.call(['ping', '-c', '1', remoteHost])
if res == 0: # 0 = successful
# writes successful ping to log
logger.info(' Ping to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s Successful ' {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost))
try:
# preps ssh connection
ssh = paramiko.SSHClient()
# sets paramiko to accept all server keys
ssh.set_missing_smtpHost_key_policy(paramiko.AutoAddPolicy())
# opens connection to remoteHost
ssh.connect(remoteHost, username=user, password=passwd)
# sets paramiko to watch for standard input, output,
# and errors, and sends command
stdin, stdout, stderr = ssh.exec_command(commandToSend)
# grabs response and stores it in result
result = stdout.read()
# if it finds the failedKeyword in result,
if failedKeyword in result:
# send failed message email
email = smtplib.SMTP(smtpHost)
email.sendmail(sender, receiver, msgProccesses)
# write failed message to log
logger.exception("failed message")
else:
# if failedKeyword isn't in the result
# write success message in log
logger.info("success message")
# closes ssh connection
ssh.close()
# if the connection is unsuccessful
except Exception,e:
# close the connection
ssh.close()
# write to log failed connection
logger.exception("SSH Connection to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s unsuccessful" {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost))
# send email with failed ssh connection message
email = smtplib.SMTP(smtpHost)
email.sendmail(sender, receiver, msgSSHlogin)
# if ping is unsuccessful
else:
try:
# send failed ping email message
email = smtplib.SMTP(smtpHost)
email.sendmail(sender, receiver, msgPing)
except Exception, email:
# write failed ping message to log
logger.exception("Ping Attempt to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s unsuccessful" {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost))
