Infrastructure Monitoring with Nagios

Image Credits : xmodulo

Server management is a real pain and the pain keeps getting worse with more and more server getting added to the infrastructure. So how do organizations sustain with huge server farms, datacenters in place? How can super admins promise an SLA of 99.99% uptime with a very low response and resolution time? Quiet obviously the answer is server monitoring solutions. It could have been so tedious for a human to monitor servers 24x7 especially when most of the systems are stable and its only once in a while some manual intervention is needed.

So what is it that needs to be really monitored? It really depends from one organization to other. For a  web development platform, response time of the page may matter a lot. The kind of traffic, 4xx's 5xx's could be a concern too. Disk Space, CPU, Memory, Swap space, particular processes and services running, DB server replication, read writes, no. of connections, query execution time and many more parameters together. Most of these checks are required by all organizations. Out of the many monitoring tools out there, one of the most used is Nagios.

Nagios is an open source software application that helps in monitoring systems, network and Infrastructure. Nagios is on top of the Linux and hence, whatever you could do with Linux could also be done with Nagios. The best part of using Nagios is the plugin based architecture and 100's and 1000's of plugins that it supports to literally allow you to monitor anything.

Nagios comes with multiple notable features that makes it distinguishing. It uses the standard protocols i.e TCP, UDP, ICMP for monitoring servers across network. You can perform multiple resource checks on any host using the NRPE addon, the checks varies from CPU, Disk RAM and many more. Not just resource checks, you could also add event handlers that perform certain actions when certain events are noticed. Checks are performed at the specified intervals, by default the interval is 5 minutes. There are 2 types of checks, Active - The one that are nags initiated. Passive - The one that are initiated externally.

Nagios consists of various objects that needs to be defined and used.

1. Hosts : Hosts are the systems/ servers that need to be monitored in the infrastructure. Nagios also provides the facility to group set of hosts together to give a better monitoring experience. Say you can group all web servers together in a "WebServers" host group. Typically a host definition may look like : "define host{
   use                             linux-box 
   host_name                       test_host 
   alias                           CentOS 6 
   address                         5.175.142.66 
   }"
2. Services : Services are the checks that needs to be performed. There are a wide range of service checks that can be performed on any host. Just like host group, service checks can also be grouped together. E.g you may need to check the CPU utilization of all servers together, you may group it that way. A service definition may look like : "define service{
           use                     generic-service
           host_name               test_host
           service_description     CPU Load
           check_command           check_nrpe!check_load 
           }"
3. Contacts : Contacts are the people who need to be contacted if a notification needs to be sent for any event that occurs. You can configure contacts to send emails, samosas, or even custom messages to any service that allows messaging. Contacts can also be grouped together into a contact group. E.g there is a notification about come process getting shut down on QA server that the Admin may not necessarily be bothered about, in such a case the notification can only be sent to QA group. A contact definition will look like : "[define contact{
           name                            generic-contact
           service_notification_period     24x7
           host_notification_period        24x7
           service_notification_options    w,u,c,r,f,s
           host_notification_options       d,u,r,f,s
           service_notification_commands   notify-service-by-email
           host_notification_commands      notify-host-by-email
           register                        0   
   
           }"
4. Commands : Commands define the exact command that will be executed on the remote hosts while executing a particular check. These are the simplest way to get particular check executed, you may also pass bash commands to perform any particular check. A command definition may look like : "define command{
           command_name check_nrpe
           command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
           }"
5. Time Period : If a downtime is scheduled at a particular time regularly and you don't want Nagios to send you any alert at these hours, you can achieve this by adding a time period definition. This looks like : "define timeperiod{
           timeperiod_name 24x7-except-night-12-2
           alias           24x7 Except 00:00 - 02:00
           sunday          02:00-23:59
           monday          02:00-23:59
           tuesday         02:00-23:59
           wednesday       02:00-23:59
           thursday        02:00-23:59
           friday          02:00-23:59
           saturday        02:00-23:59
   }"

You can also set a monitoring schedule for a particular object if you do not want to add it to the existing service/hosts check. This allows you to explicitly look at a particular check.

Sometimes writing the definition can become a real pain using the same definition for all services and hosts can be a real pain even if you decide to copy-paste the definitions. Templates come for help here. You can define a template with all the necessary details of definition and simply use the same template everywhere in the configs. A typical template definition look like :

define host{

        name                            generic-host    

        notifications_enabled           1               

        event_handler_enabled           1               

        flap_detection_enabled          1               

        process_perf_data               1               

        retain_status_information       1               

        retain_nonstatus_information    1               

        notification_period             24x7            

        register                        0               

        }

define contact{

        name                            generic-contact         

        service_notification_period     24x7                    

        host_notification_period        24x7                    

        service_notification_options    w,u,c,r,f,s             

        host_notification_options       d,u,r,f,s               

        service_notification_commands   notify-service-by-email 

        host_notification_commands      notify-host-by-email    

        register                        0                        

        }

Monitoring in Nagios is parallel, i.e a number of hosts and service checks will go simultaneously in parallel. This could be resource consuming but this is always better than sequential monitoring as you can be sure that all your servers are doing well and don't have to wait too long for any kind of update. The add ons for Nagios are simple to make and add to the Nagios community. The configs are all split and simple to understand too. Nagios has a huge documentation and help examples for quickly getting started. 

Happy Monitoring!!

Learning Chef - Part - II

some rights reserved by Matt Ray

...continued from  Learning Chef - Part - I

Consider you have to install an application. You 1st install and configure the application on a single server. It could be a developers laptop/workstation. In order to setup the application you have to perform various installation procedures i.e install packages, start services, manage database etc.
After sometime you are going to make the Application available to a larger number of public than your laptop/workstation can handle and so will need to add a database server and will make this a multi tier application. So now we have one server handling the Application request and a separate server for database. To avoid data loss we will add another App server and Database server to keep the data redundant so that data loss is avoided.
As time passes the load increases on the server with more number of people trying to access the server so we may need to add a scaling solution like a load-balancer to the server.
As in how the application usage increase and the amount of users increases we will need to add more and more app servers and add more load balancers for them.
As the database is not able to cope up with the high demand, we need to add a DB cache to the existing solution, Making the infra even more complex.

Chef is Infrastructure as a code. Using Chef you can programmatically provision and configure components. Chef ensures that each node complies to the policy. Policies are determined by the configurations included in each Node's run list. You can define the policy in your Chef configuration. Your policy states what state each resource should be in, but not how to get there. Chef-client will pull the policy from the Chef-server and enforce the policy on the Node. Policy will state what needs to installed but not how it needs to install. Chef is intelligent enough to figure that out. Chef will enforce the policy based on the resource that you specified.

Setup:

Setup a Chef Environment first by setting up Chef workstation, use the following command on Ubuntu,

curl -L https://www.opscode.com/chef/install.sh | sudo bash

 
 Login to manage.opscode.com and download the starter-kit there. Extract the chef-repo to your home directory. It should show the following contents.

cd chef-repo
ls
  .berkshelf
  .chef
  cookbooks
  roles
  .gitignore
  Berksfile
  chefignore
  README.md
  Vagrantfile

Check the .chef file present in the directory, it should show the following content.

cd .chef
  knife.rb
  org-validator.pem
  user.pem

Knife.rb will show your chef-server configuration for the workstation to be identified by the chef-server.

vim knife.rb
  # See http://docs.opscode.com/config_rb_knife.html for more information on knife configuration options

  current_dir = File.dirname(__FILE__)
  log_level                :info
  log_location             STDOUT
  node_name                "user"
  client_key               "#{current_dir}/user.pem"
  validation_client_name   "org-validator"
  validation_key           "#{current_dir}/org-validator.pem"
  chef_server_url          "https://api.opscode.com/organizations/user"
  cache_type               'BasicFile'
  cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
  cookbook_path            ["#{current_dir}/../chef-repo/cookbooks"]

Writing Recipes:

package "apache2" do
  action :install
end

template "/etc/apache2/apache.conf" do
  source "apache2.conf.erb"
  owner "root"
  group "root"
  mode "0644"
  variable(:allow_override => "All")
  notifies :reload, "service[apache2]

service "apache2" do
  action [:enable,:start]
  supports :reload => true
end

Lets consider the above recipe and understand it.
Each recipe has resources in it. The resources have :
- types -> package, template, service are the types of resources in the code
- names -> apache2, /etc/apache2/apache.conf, apache2(service) are the names of the resources in the code
- parameters ->   source "apache2.conf.erb"   owner "root"   group "root"   mode "0644"   supports :reload => true
- action to put the resource on desired state -> action :install, action [:enable,:start]
- send notification to other resources -> notifies :reload, "service[apache2]


A cookbook can be created by the command
knife cookbook create cookbookname
This will automatically create the cookbook along with all the necessary files inside it. You can delete, check the available cookbooks, delete a cookbook, upload and download a cookbook using different knife commands. You can check all the options by
knife cookbook --help

To bootstrap a new node you need to use the following command:

knife bootstrap hostname --sudo -x username -P password --ssh-port 2222 -N nodename

Creating Environments
Many a times you will want development environment and production environment to have little different configurations. e.g xdebug to be installed on dev but not on production. PayPal enabled on prod but not dev etc. Chef allows you to define different environments and also allows you to assign different nodes to a particular environment.

create a directory called environments in the chef-repo. Add the following content to a file dev.rb in it.

name "dev"
description "The dev environment"

create another file called prod.rb there and add the following content to it.

name "prod"
description "The prod environment"

Upload the environment to the chef server.
You can verify whether the environments are created in the chef server by logging in there.

Creating roles:
Roles provide a way to apply a group of recipes and attributes to all the nodes performing a particular function. e.g all the nodes which would work as db server can be assigned a db server roles and accordingly all the db server specific recipes can be applied to those nodes.

Roles can be created in the following manner. Create a roles directory in the chef-repo if it does not exist. add a file base.rb there with the following content.

name "base"
description "Base role applied to all nodes."
run_list(
  "recipe[users::sysadmins]",
  "recipe[sudo]",
  "recipe[apt]",
  "recipe[git]",
  "recipe[build-essential]",
  "recipe[vim]"
)
override_attributes(
  :authorization => {
    :sudo => {
      :users => ["ubuntu", "vagrant"],
      :passwordless => true
    }
  }
)

Here the runlist method defines a list of recipes to be applied to all the nodes that have base role. The override_attributes method tells lets us override the default attributes used by the recipes in the list. Here we are overriding attributes used by the sudo cookbook so that "vagrant" and "ubuntu" users can run sudo without entering password.

Next create another role Webserver by creating a file webserver.rb in the roles directory with the following content.

name "webserver"
description "Web server role"
all_env = [
  "role[base]",
  "recipe[php]",
  "recipe[php::module_mysql]",
  "recipe[apache2]",
  "recipe[apache2::mod_php5]",
  "recipe[apache2::mod_rewrite]",
]

run_list(all_env)

env_run_lists(
  "_default" => all_env,
  "prod" => all_env,
  #"dev" => all_env + ["recipe[php:module_xdebug]"],
  "dev" => all_env,
)

Here it shows that a method env_run_lists method in a role to define different run lists for different environments. To simplify things we create an all_env array to define the common run list for all environments, and then merge in any additional run list items unique to each environment.

Next create another role db_master.rb file with following contents:

name "db_master"
description "Master database server"

all_env = [
  "role[base]",
  "recipe[mysql::server]"
]

run_list(all_env)

env_run_lists(
  "_default" => all_env,
  "prod" => all_env,
  "dev" => all_env,
)

upload the created roles to chef-server and also verify the same. Upload roles by:
knife role from file roles/base.rb
knife role from file roles/webserver.rb
knife role from file roles/db_master.rbenv


Setting up a user account for sys-admin
Define a user account for yourself on all the nodes with admin privileges. This can be done by defining a data bag for the users cookbook, with attributes that describe the user account to create.

mkdir -p data_bags/users
vim data_bags/users/$USER.json

Add the following to the $USER.json
{
  "id": "jkg",
  "ssh_keys": "ssh-rsa ...SecretKey... roshan4074@gmail.com",
  "groups": [ "sysadmin", "dba", "devops" ],
  "uid": 2001,
  "shell": "\/bin\/bash"
}

Upload the data bag as well to the chef-server and verify
knife data bag create users
knife data bag from file users $USER.json

Learning Chef - Part - I

some rights reserved by Matt Ray

The information here has been collected from Nathan Harvey's Video tutorials on Chef's website and from Chef's official documentation. Before starting with the tutorial, I thought it would be better to understand common jargons used in chef.

Three primary entities: workstation, chef-server, node

• Chef-Work Station : System from where the configuration management professional / devops / sys admin will be working.
• Chef-Server : System/Server where all the infrastructure as a code will be stored. Also the Chef-Server will have many other features that we will seee later.
• Nodes : Servers in your infrastructure that will be managed by chef, They may represent a physical server or a virtual server. They may represent hardware you own or multiple compute instances in a public or a private cloud. Each node will belong to one organization and and other organization may not have access to it. Each node will belong to one environment. Either in staging or Production etc. Each node will have zero or more roles. An application called chef-client run on each of the node. The chef-client will gather the current system configuration. It will download the desired system configuration from the Chef-server and configure that node such that it adheres to the policy defined.
• Knife : Command line utility that acts as an interface between local chef-repo(on work station) and server. Knife lets you manage nodes, cookbooks, recipes, roles, stores of json data, including encrypted data, environments, cloud resources including provisioning. The installation of chef on management workstations, Searching of indexed data on chef server. You can even extend knife to use plugins for managing cloud resources. E.g knife-ec2, knife-rackspace, knife-vcloud plugins.
• Cookbooks : A cookbook is a container to describe or to contain our configuration data. It contain the recipes. It can also include templates, files, custom resources etc.
• Recipes : A Recipe is a configuration file that describes resources and their desired state. A recipe can install and configure software components, Manage files, Deploy Applications, Execute other recipes, etc.
• Sample Recipe

package "apache2" // 1st resource is package and chef knows that it should be installed on the server. If the package doesn’t exist it will install it.

template "/etc/apache2/apache.conf" do //Next resource is a template. The template will manage a file at /etc/apache2/apache.conf

source "apache2.conf.erb"

owner "root"

group "root"

mode "0644"

variable(:allow_override => "All")

notifies :reload, "service[apache2] //state of the apache2 if apache2.conf exist it knows that it doesn’t need to create that file. However chef needs to make sure that the file has proper contents. So it will generate a temporary file. It will use the source that we specified above apache2.conf.erb and then it will also use any variable content that we specified, i.e AllowOverride All. Once the temporary file is created, Chef will then compare the two files. If they are the same, chef will discard the temporary file and then move on to the next resource, then notifies line will be ignored. However if the two files are different. The chef-client will discard the version on the disk and place the temporary file into the proper location, i.e overwrite existing file. Whenever the overwrite happens a notification will be sent. Then it will tell Apache to reload with new configs.

end

service "apache2" do //service should be enabled and start automatically

action [:enable,:start]

supports :reload => true
end

• Roles : A way of identfying different types of servers. e.g Load-balancer, app server, DB cache, DB , monitoring etc. Roles may include list of configs to be applied called as runlist. May include data attributes for configuring infra, i.e ports to listen on, list of apps to be deployed.
• Data bags : Stores of json data
• Attributes : Attributes are mentioned in cookbooks/recipes. An attribute gives the detail about the node. It tells about the state of the node; before the chef-client run, present state and state after the chef-client run
• Resources : Items that we sysadmins manipulate to manage complexity. i.e Networking, Files, Directories, Symlinks, Mounts, Registry key, Scripts, Users, Groups, Packages, Services File-systems etc. Resource represent a piece of the system and its desired state. e.g a package to be installed, A service to be running, A file to be generated, a cronjob to be configured, a user to be managed, etc.
• Ohai : Ohai is a tool used to detect the attributes on the node. These attributes are then passed to the chef client at the beginning of the chef-client run. Ohai is installed on a node as a part of chef-client installation. Ohai has the following types of attributes: Platform details, Network usage, Memory usage, Processor usage, Kernel data, Hostnames, FQDNs, etc. So, Ohai is a utility that will give you all the information of your system level data.
• Shef : Chef-Shell was earlier known as Shef. Its a recipe debugging tool that allows breakpoints within recipes. Its runs as an irb session.
• Environments : Environments can be development, test, staging and production. They may contain data attributes specific to an environment. Starts with single environment e.g default is 1st. Different names/URLs for payment services, location for package repository, version of chef configs etc.
• Run List : The joining of a node to a set of policies is called as a run-list. The chef-client will download all the necessary components that make up the run-list.e.g recipe[npt::client], recipe[users], role[webserver]. Run List is a collection of policies that a node should follow. Chef-client obtains the run-list from chef-server. chef-client ensures the node complies with the policy in the run-list.
• Search : You can search for nodes with Roles, Find topology data, i.e IP addresses, hostnames, FQDNs. Searchable index about your infrastructure. e.g load balancer needs to know which application should I be sending requests to? Chef-client can ask Chef-server which application servers are available and which application server should I be sending load to. And in return the chef server can send a list of nodes and then the load balancer can figure out which one based on the hostname or IP address or Fully Qualified Domain Name.
• Organization : Everyone has their own infra and wont manage anyone else's infra. Organizations are independent tenants on Enterprise chef. So this could be different companies, business units or departments for managing.

Checking Open Ports on a Remote Computer using PortQry

Some rights reserved by Ryan Franklin

 Today for one of the projects the SFTP connection kept failing for some reason. The user-id password used for connecting to the host was correct the hostname was as well correct. There was no way to find out what went wrong. Thankfully command-line gives a good log to verify what goes wrong.

I tried connecting to the SFTP host with various tools like FileZilla, WinSCP but could not get good enough logs. Finally i tried connecting the server using ssh on command-line using my Ubuntu machine. The connection used to time out. That is what i see in the logs as well. I assumed that probably the SFTP port number 22 was closed for the host.

I googled for if i could find a tool to check if a particular port on a machine is accessible or not. I finally found something called as PortQry taht could be used on Windows machine using Commandline.

Its a very small 140 KB command-line based software tool that you can use to check if a port on some machine is accessible or not.

After using this tool i got to know that the machine had a Firewall  kind of protection which wasn't allowing me to access the SFTP port on it.

Here's how you PortQry on Windows:

• Download the software using the link : http://www.microsoft.com/en-in/download/details.aspx?id=17148

• Double click an unzip the files to any location say C:/

• Hit Windows+R in the run box enter "cmd"

• Go to the directory where the PortQry was extracted.

• Execute the program PortQry by entering PortQry<enter>

• This will display a list of help information and the correct usage of the command

The following is the syntax to check the port status :
portqry -n myhostname.net -e 80

PortQry can inform the status of a port as "Listening", "Not Listening", or "Filtered"
Listening : There is some service active on that port
Not Listening : Port is closed
Filtered : No response, Presumably its behind some kind of firewall.

Syntax
portqry -n name_to_query [-p protocol] [-e || -r || -o endpoint(s)]

Common command line switches:
-n : IP address or name of system to query
-p : TCP or UDP or BOTH (default is TCP)
-e : single port to query (valid range: 1-65535)
-r : range of ports to query (start:end)

For single port use
portqry -n 127.0.0.1 -e 80

For a Range of ports, use the -r switch:
portqry -n 127.0.0.1 -r 80:85

Note:
- PortQry also displays extended information for known services, such as SMTP, POP3, IMAP4, FTP, and is capable of performing LDAP queries.
- A GUI based alternative is also available now called PortQryUI

Sample Output:
C:\PortQryV2>portqry -n 127.0.0.1 -e 40
Querying target system called:
 127.0.0.1
Attempting to resolve name to IP address...
Name resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): FILTERED

C:\PortQryV2>portqry -n 127.0.0.1 -e 80
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 80 (http service): LISTENING

C:\PortQryV2>portqry -n 127.0.0.1 -e 22
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): NOT LISTENING

Second Place Story : Server ghouls haunt bulk ingestion

Some rights reserved by Julie Rybarczyk

I recently shared one of my stories while monitoring an Ingestion Server with AppFirst for one of the Halloween contest they conducted. Thankfully, won 2nd prize in the same. Sharing the same story here along with the link to visit.

Second Place Story

Server ghouls haunt bulk ingestion
Company: Roshvert
I was suppose to monitor an Ingestion Server that was performing a bulk ingestion through an EC2 instance with around 200 GB of data to be ingested to another server.
Since it was a huge amount of data and the ingestion would take another day to complete, I kept the ingestion going and the logs were performing well. I decided then that I’d log in early tomorrow morning to check the ingestion status. During this time, the log files were supposed to be created automatically through the ingestion and the name of the log file for any particular day should be log_dd-mm-yyyy.txt with date of that day mentioned. It was a staging server and the code was supposed to be supplied for UAT in a day or two.
I logged in early the next morning to check the ingestion status. I was totally puzzled as I couldn’t make out what was happening:

• The log file for the previous day log_27-08-2013.txt was showing everything went well until 11pm midnight and no logs thereafter.
• The log file for today log_28-08-2013.txt got created with no data in it.
• The ingestion process was running with no errors.
• The server logs showed no errors.
• The system never went down.
• Nearly 150 GB of data was still to be ingested and was not progressing at all.
• None of the logs showed any updates as to why the ingestion was not progressing.

Since the delivery was urgent, I stopped the ingestion on the instance and restarted it. To my horror, the ingestion was not progressing at all. I tried running ingestion on other instances, and it worked fine.
Then something hit me, and I went back to check the logs of ingestion. The ingestion logs still showed nothing with 0 kb space used by the logs. Wait!!! Space? 0kb? 150 GB data still remaining?
I immediately checked the disk space and found zero space available. Whoaa!!!
What actually happened is while performing the ingestion, the server created a duplicate copy of the data on the same instance, and until the entire ingestion completes, this data used to remain there. Around 250 GB of disk space was used by ingestion by midnight and the disk was full. I immediately attached a bigger volume to the instance and restarted the ingestion. Thankfully it was complete in a few hours and that saved me from a big trouble!!!