I’m calling this a Remote Service Monitor because that what I’m using it for, but really it can pass any command to the linux shell and watch for any specific keyword in the response. This script first pings the remote machine, if the ping fails it sends out an email, if the email fails, it writes to the log. However, if the ping is successful, it then attempts to open a SSH connection with the remote machine, if that fails, it both writes to the log and sends an email. If the SSH connection is successful, then it sends whatever command you define and then reads the response from the server in. Finally it scans the response for the defined keyword, if it finds that keyword, it writes to the log and sends an email. If it doesn’t find the keyword, it just writes to the log.
Example: So lets say you want to monitor a particular service like ntpd on a remote linux box, or like in my case, just across the network. If the process is stopped, or if the server doesn’t respond, you want to get an email telling you so. I set the remoteHost to the proper IP, set the user/pass for the remote server, set the commandToSend to “service ntpd status” which if the process is running it returns;
Then, I set the keyword to “stopped” because if the service is stopped, the response from the server is:
Then, you fill in the rest of the variables, define where you want the log to be written. Set your SMTP server (look in your email client) and set the sender and receiver addresses. Most SMTP server’s won’t accept SMTP requests from public (residential) IP’s, but many mail servers will accept SMTP from business grade IP’s, if your using this at work, give it a shot, it has a decent chance of working.
Then I set it on a cron job for once every 10 mins, and wait for the process to stop. :: Click here for a walk through on cron ::
If the ping fails = You get an email, or it writes to the log
If the ping is successful = Write to log, and Continue
If the SSH connection fails = You get an email, and it writes to the log
If the SSH connection is successful = Write to log, and Continue
If the keyword is found in the response = You get an email, and it writes the log
If the keyword is not found in the response = Write to log, end.
To run this script on windows change the flag “-c” to “-n” on line 46.
For my purposes, the ping was as important as verifying the service was running.
Its ‘over-commented’ on purpose.
(if the code doesn’t display properly, try a shift-refresh)
#!/usr/bin/python # You'll probably need to install paramiko, and maybe subprocess import subprocess import os import paramiko import logging import logging.handlers import smtplib # Set your Variables # Requires SMTP email server, no authentication remoteHost = "IP Address or hostname" # host to ping and connect to user = "username" # username to login to remoteHost passwd = "password" # password for the above username commandToSend = 'command to send to shell' # the command to send to remoteHost failedKeyword = 'keyword to watch for in response' # Keyword to watch for in the response logName = '/path/to/log' # path to where you want the log file to be smtpHost = "smtp.domain.com" # SMTP server ip/hostname sender = 'email@domain.com' # email sender receiver = 'email@domain.com' # email receiver # logging configuration, set to 500 MG worth of logs in 5 files, roller works logging.basicConfig(level=logging.INFO, filename=logName) logger = logging.getLogger('myLogger') formatter = logging.Formatter("##### {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}(asctime)s\t") handler = logging.handlers.RotatingFileHandler(logName, maxBytes=1024*1024*10, backupCount=5) handler.setFormatter(formatter) logger.addHandler(handler) # email messages, fill these out with the information you want emailed at # each email point; ping failure, ssh failure, keyword found msgPing = """From: Display Name <email@domain.com> To: ToWhom <email@domain.com> Subject: Text Goes Here """ msgSSHlogin = """From: Display Name <email@domain.com> To: ToWhom <email@domain.com> Subject: Text Goes Here """ msgProccesses = """From: Display Name <email@domain.com> To: ToWhom <email@domain.com> Subject: Text Goes Here """ # opens subprocess and pings remoteHost # if running on Windows, change flag from -c to -n # this watches for one ping response, up 1 to 3 for three responses res = subprocess.call(['ping', '-c', '1', remoteHost]) if res == 0: # 0 = successful # writes successful ping to log logger.info(' Ping to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s Successful ' {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost)) try: # preps ssh connection ssh = paramiko.SSHClient() # sets paramiko to accept all server keys ssh.set_missing_smtpHost_key_policy(paramiko.AutoAddPolicy()) # opens connection to remoteHost ssh.connect(remoteHost, username=user, password=passwd) # sets paramiko to watch for standard input, output, # and errors, and sends command stdin, stdout, stderr = ssh.exec_command(commandToSend) # grabs response and stores it in result result = stdout.read() # if it finds the failedKeyword in result, if failedKeyword in result: # send failed message email email = smtplib.SMTP(smtpHost) email.sendmail(sender, receiver, msgProccesses) # write failed message to log logger.exception("failed message") else: # if failedKeyword isn't in the result # write success message in log logger.info("success message") # closes ssh connection ssh.close() # if the connection is unsuccessful except Exception,e: # close the connection ssh.close() # write to log failed connection logger.exception("SSH Connection to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s unsuccessful" {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost)) # send email with failed ssh connection message email = smtplib.SMTP(smtpHost) email.sendmail(sender, receiver, msgSSHlogin) # if ping is unsuccessful else: try: # send failed ping email message email = smtplib.SMTP(smtpHost) email.sendmail(sender, receiver, msgPing) except Exception, email: # write failed ping message to log logger.exception("Ping Attempt to {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304}s unsuccessful" {80e463235c561985fcb9d065cb7af58becf1df7010d7a45bb4eb7315e5a8b304} (remoteHost))