Software Configuration Management System

Picture credits : Paul Downey

Any application would generally consist of Web servers, Application Servers, Memcache systems, SQL and NoSQL Database servers, Load Balancers, Messaging queues, etc. Although this is pretty much enough, however as a precaution/privilege we also ensure proper redundancies so that whenever there is a failure we have a back plan in place to handle the failure. In order to keep a track of server performances we also have logging servers, Analytics servers and Monitoring servers in place. All these servers need to available again within no time in case something goes wrong(which does go wrong).

In traditional systems the admin guy managed all these by managing the wiring of the server and SSHing the servers and maintaining them throughout. There was nothing wrong with the idea except of time taken to get the process done. When something goes wrong get into that machine and spend hours finding out what went wrong and correct it my defining a good downtime. With a configuration management(CM) system in place now, we describe a state of a server and use some tool that just ensures that the server resides in that state throughout. The CM system ensures that right packages are installed, config files have correct values and permissions set and that the expected services are running on the host system and many more.

Software Deployment is another concern that a Devops person has to take care of which is at times addressed by CM tools too, although may not be considered a good practice always. Deployment is the process where the software that is written/developed by a company is built/compiled/processed and the required binaries and static files and other necessary files are copied to the server. The expected services are started as well. This is done mostly by using some scripting language and now we have some deployment specific tools that have their own advantages over scripting languages rollback being an important one. Capistrano and Fabric are famous ones.

Many a times the deployment process involves multiple remote servers. In complex environments the deployment process, the order of execution of tasks play an important role. A deployment may fail if an expected event occurs before another. E.g the database server needs to be up and running before the web server is brought up. Or in a high availability environment servers needs to be 1st taken out of the load balancer one by one before deployment and later added back to the load balancer post successful deployment. This automated arrangement, coordination and management of complex systems is called orchestration.

With a bunch of IAAS providers in the cloud market, virtualization has taken up huge pace. The evaluation of any new CM tool that comes to the IT world is largely done based on the number of cloud providers it supports. An important feature of a CM tool is provisioning. Provisioning is the process of spinning up of server for that cloud provider automatically. Many CM tools providers have plugins written to communicate with many cloud providers. Chef, Ansible, Puppet, CFEnginer, Salt have already become favorite for many out there.

I have personally used Ansible and Chef as of now. Cloud is fun indeed .. :)

s3cmd to push large files greater than 5GB to Amazon S3

image credits:  Stefano Bertolo

Use command line utility to push s3cmd files on Amazon S3.

Install s3cmd from s3tools.org or
apt-get install yum install s3cmd OR s3cmd

Configure s3cmd by
vim ~ / .s3cfg

<Paste những info add to it and you access-key and secret-key>

[Default]
access_key = TUOWAAA99023990001
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = US 
cache_file = 
cloudfront_host = cloudfront.amazonaws.com 
default_mime_type = binary / octet-stream 
delay_updates = False 
delete_after = False 
delete_after_fetch = False 
delete_removed = False 
dry_run = False 
enable_multipart = True 
encoding = UTF-8 
encrypt = False 
EXPIRY_DATE = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False 
force = False 
get_continue = False 
gpg_command = / usr / bin / gpg 
gpg_decrypt =% (gpg_command) s -d --verbose --no-use-agent --batch --yes --passphrase-fd% (passphrase_fd) s -o% (output_file) s% (input_file) s 
gpg_encrypt =% (gpg_command) s -c --verbose --no-use-agent --batch --yes --passphrase-fd% (passphrase_fd) s -o% (output_file) s% (input_file) s 
gpg_passphrase = 
guess_mime_type = True 
host_base = s3.amazonaws.com 
host_bucket =% (bucket) s.s3.amazonaws.com 
human_readable_sizes = False 
ignore_failed_copy = False 
invalidate_default_index_on_cf = False 
invalidate_default_index_root_on_cf = True 
invalidate_on_cf = False 
list_md5 = False 
log_target_prefix = 
max_delete = -1 
mime_type = 
multipart_chunk_size_mb = 15 
preserve_attrs = True 
progress_meter = True 
proxy_host = 
proxy_port = 0 
put_continue = False 
recursive = False 
recv_chunk = 4096 
reduced_redundancy = False 
restore_days = 1 
secret_key = sd / ceP_vbb # eDDDK 
send_chunk = 4096 
server_side_encryption = False 
simpledb_host = sdb.amazonaws.com 
skip_existing = False 
socket_timeout = 300 
urlencoding_mode = normal 
use_https = True 
use_mime_magic = True 
verbosity = WARNING 
website_endpoint = http: //% (bucket) s.s3-their Website% (location) s.amazonaws.com/ 
website_error = 
website_index = index.html

access_key = YOUR-ACCESS-KEY-HERE
You can see how to use s3cmd at: http://s3tools.org/usage

Here I came across a typical scenario where I could not upload files greater than 5GB. You could do this to print two Ways:

1. Using the --multipart-chunk-size-mb flag: s3cmd put --multipart-chunk-size-mb = 4096 201412.tar.gz s3: // apache-logs / I could not do this since I Had an older version of s3cmd installed and I did not really have time to download and install những version.
2. Splitting Into the large files using small files and then uploading it split command.

• Original file

-rw-r - r--. 1 root root 5.4G Jan 20 06:54 201412.tar.gz

• Split Command

split -b 3G 2014backup.tar.gz "201 412"

• Post Split

-rw-r - r--. 1 root root 23 Apr 06:41 3.0G 201412aa
-rw-r - r--. 1 root root 23 Apr 06:43 2.4G 201412ab

• Upload files những

201 412 * s3cmd put s3: // apache-logs /

Saved some time :)

Hadoop 1.0

some rights reserved by intelfreepress

Got opportunity to work with Hadoop with a recent project. Understanding the architecture conceptually and then learn using it is fun. I am still learning it though. This part explains Hadoop 1.0 which I believe helped me to understand 2.0 well.

Before understanding Hadoop it would make more sense if we get rid of a few jargon that would come our way.

Oozie is like a cronjob. In addition to running jobs time based, it also allows running jobs on particular statuses, like:
i. Run job B when Job A completes
ii. Run Job B when a particular file is present.
Oozie is a kind of job coordinator. When you write Oozie, you need to specify in an xml the kind of workflow you want it to follow.

Hive is a dataware housing system for hadoop.

Pig Latin is a kind of Data Analysis. Those who are not java programmers and may not be able to write complex java code can use pig to run the MapReduce job on Hadoop. Syntax is similar to erlang programming language where you could say load this file , group by etc. A simple program of 2 -3 lines which in java may take a some effort could be written easily here. The limitation could be that you may not be able to write very complex logic in Pig. If you have something simple as a log file that you just want to read data from that is separated by tabs, you could use Pig.

Mahout is a machine learning. It provides all the machine learning algorithms which are in place. Mahout just writes a wrapper around existing algos so that you can use it along with Hadoop.

HBase is hadoop database. Its a NoSQL kind of DB which is inside Hadoop and a lower layer of file system that is HDFS.

HDFS Hadoop distributed file system, will sit on top of your existing filesystem. In the native OS the file system could be ext3, ext4 etc. The HDFS sits as a pseudo filesystem on top of your existing filesystem so that you can visualize all the different filesystems on differnt nodes and clusters as one.

Flume and Scoop: Flume handles unstructured data like tweets, logs or semistructured data like xmls. Scoop handles structured data. We can configure flume to read data form apache log directory : /var/log/apache2/error.log

Hadoop has two very important parts.
1. Storage unit
2. Processing unit

Storage unit is where you store, read and write data onto i.e HDFS. This HDFS is distributed across nodes. They are natively redundant, that means it is by default redundant and you do not need to configure RAID or any other software or hardware for this. It would by default make 3 copies of each block. This number is configurable to any number of copies you need.

Hadoop has a Master Slave architecture. It has a master node and a few slave nodes that talk to the master. The master node in Hadoop is known as Name Node. All the nodes that have Data Nodes running could be slave nodes. Different instances can have different or same type of filesystems, but HDFS will be laid across all these nodes. So when you run the command "hadoop fs -ls /" it doesn't bother as to which node the file is present, It will bring files from all nodes data node1 data node2 data node3... and show up. The NameNode keeps track of all the data blocks.

Processing Unit is provided by the MapReduce framework. This can be called as the brain of Hadoop. When you supply a job to Hadoop, it will split the jobs according to the processors you have. If you have a Hadoop cluster with two nodes, each node with one processor. You can run two processes in one cluster parallely.
Locality of reference: Earlier, when you wanted to process a data you would move data close to the processing unit. This required you to 1st copy the data.  It would pull the data in the RAM and then it would process it there. This is fine when dealing with small amount of data. But in case of Hadoop you will be dealing with data in terabytes and petabytes. Here copying or moving of huge data is time consuming. In order to not invlove any data movement, now Hadoop instead moves the processing unit itself to where the data is present. This is called locality of reference.

(To be contd..)

Mumbai Technology Meetup - DevOps Special

On July 27th 10 am a DevOps special meetup was conducted at Directiplex, Mumbai. Its very rare to see a meetup given importance as much as any other technology conference. Speakers from different organizations were present and shared their knowledge. Tremendous knowledge and experience shared free of cost. The meetup went on from 10 am (Started a little late) and went around till 5.30 pm. The agenda itself was too appealing.

No entry fees. Its a free event. Just ensure you learn and make use of that learniing :-)

======================================

11.00 am - 12.00 pm : SaltStack [incl. LXC basic]: by Rigved Rakshit - Directi

Rigved introduced to LXC its setup, its concepts and how its similar/different than Docker. SOme commands and configs. Due to lack of time he could not cover Saltstack though

=====================================

12.00 pm - 1.00 pm : Configuration Management at Rackspace by Shaunak Kashyap - Rackspace

Shaunak conducted this on Hangout while he was at a 12 hr difference. Shaunak showed how rackspace uses Ansible for getting provisioning and other CM task automated.

======================================

1.00 pm - 1.30 pm : Chef Fundamentals and DevOps by Sanju Burkule - OpexSoftware

Sanju took a brief introduction to Chef and how OpexSoftware who are partners of Chef and conduct professional Chef training with certifications. Sanju also shared his knowledge on how Chef is different from Puppet as he has used both.

======================================

1.30 pm - 2.00 pm : Lunch - Lets not talk about this. Blame the rain.

======================================

2.00 pm - 3.00 pm: Puppet [incl. preparatory VirtualBox fundamentals] by Ashish Chandra. - Reliance Jio

Ashish took introduction to Puppet some basics , how easy is it to setup a Puppet master and get going. He also shared some of his scripts that he uses to provision 500 instances in 6 - 7 minutes.

=======================================

3.00 pm - 4.00 pm: Ansible by Aditya Patawari - BrowserStack

This was the 2nd time I met Aditya, we met earlier at RootConf in Bangalore. Aditya shared introduction to Ansible and how is it better/different than Chef//Puppet.

=======================================

4.00 pm - 5.00 pm : Capistrano by Mayur Rokade - Directi

Mayur conducted a live demo of how to use Cap for deploys and a a little intro and setup for Capistrano.

=======================================

5.00 pm - 6.00 pm : Docker Fundamentals by Augustine Correa - Organizer of the event

Bugzilla Mail Sending Issue

Lately an issue was assigned to me where Bugzilla Email notification failed with an 504 gateway timed out error. We use gmail service for sending mails.

After checking the configuration everything seemed to be just fine, except the email was not getting sent and while updating any issue in Bugzilla a 504 error was sure to come.

After a little debugging we got rid of 504 by disabling Email service, but this was not quiet what we wanted. After googling a bit I got to know Bugzilla did not support Gmail as SMTP earlier, but now it did and that we need to install a few external packages for this. I found a tonne of articles with some misleading information or I don't know if I was doing something wrong there.

I applied this patch 1st in the Bugzilla setup directory.

patch < mypatchfile

I first installed Net-SMTP-SSL package after reading a few blogs with CPAN Shell 

perl -MCPAN -e shell

cpan> install Net::SMTP::SSL

./checksetup.pl

Check for Net-SMTP-SSL (v1.01)     ok: found v1.01

Later I tried sending notifications with SMTP in Administration > Parameters > Email (Many articles say that you will see a Gmail option or TLS option, however after installing many packages I didn't see any of those. I was unable to send a mail with SMTP even after many trails.

I finally switched to Sendmail, there was a delay but the mail was getting sent now. But the old problem was still there. Every time you update an issue, you get a 504. I suspect that because there was a delay in the mail being sent, the page used to wait for the mail being sent and then show up, but since it was too long, there could be a time-out value either in Apache or in Bugzilla config that showed up the 504 page.

Next I chose the option use_mailer_queue to be ON, and started the jobqueue.pl deamon. Now the mails are getting sent with no 504. I still suspect that it might have worked without the Net-SMTP-SSL package too.

RootConf 2014

I attended conference and workshop on Devops and Cloud Infrastructure - RootConf 2014 at Bangalore from 14th May to 17th May 2014.

Go Continuous Integration Tool developed by Thoughtworks

CI and Release Management tool developed by Thoughtworks. Helps to manage the build, test it and finally release. It allows you to distribute your build across many systems. So you can run your software on different platforms and make sure that it runs over all of them. You can even divide your tests and parallely run them on different systems, that way you get faster results. All the environments can be managed centrally, so you can promote builds from one environment to the next one.

Simple Steps:

1. Install Go Agent software on all the machines that are part of you system/cloud. 

2. Next configure all the agents to connect to the Go Server

3. Finally Approve every build from Management Dashboard.

4. Associate relevant resources tags for appropriate build tasks with the compatible agents (e.g resource linux, etc)

SELinux : Security Enhanced Linux

This session typically discussed about how SELinux is useful and how people do not understand its importance. Some security policies were discussed. Three important modes of SELinux; enforcing, permissive and disable. Also the hands on session discussed the behavior of these 3 modes of SELinux.

Docker : Light weight linux container

The dry-run of this session was conducted in the Docker-meetup we attended. It was a nice revision as well as some additional parts of docker were known in the session. Docker is a containerization tool that helps you make a light weight Linux container to pack, ship and run you application anywhere. It is an easy to learn tool with very few commands to be learnt and make your own Dockerfile. The containers can be shipped by making an image registry(or by using docker's image registry) by pushing the container from a dev environment and then pulling it from stage, test or prod. You are sure to see the similar environment as that of dev. So no more, "It works on my machine" reason to be heard.

Ansible : Configuration Management Tool

This is just another configuration management tool just like chef. The pros about the tools are :

1. No client server architecture

2. Very easy to install and almost no configuration needed.

3. Very simple to write playbooks, a non programmer can as well understand and write the code within no time since its just a yaml file.

Cons are:

1. Doesn't work on windows.

2. Not much support available since its relatively new.

Jenkins : Continuous Integration

Continuous Integration workshop was pretty much the way we do it traditionally. The workshop covered basics of Jenkins how a job is made and a build is tested an automated.

Conference Update 

Most of the conference talks focused on Docker LXC, Puppet/Ansible. Self healing techniques was introduced and sounded something that we could try and introduce wherein the servers would first intelligently check and see if the problem could be solved by its own with various scripts based on the type of issue. Technique to on board a new team member without merely asking him to read documentation and rather practice hands on was discussed in brief. Tsuru is a new tool that we heard of that may help us to simplify and automate docker as well. A demo on how to testing your infrastructure with Kitchen was shown and that is exactly how we learned it also a part of it showed integration with Docker which is something we need to implement. Introduction to Microsoft Azure cloud gave us an idea of how it is different than other cloud providers. Heartbleed was discussed in brief along with few other security threats and a demo on how Heartbleed actually affected.

Continuous Integration with Jenkins

Continuous Integration:

What's the deal? As per a tutorial that I came across a few months ago, its a process in which all the development work is integrated at some predefined time interval and the resulting work is tested and built automatically.

So the basic functionality of Jenkins is to monitor a version control system and if changes are encountered in it, it will start the build system and monitor it. Build system could be e.g Apache ant or Maven. Jenkins can monitor entire build process and it has functionality to report the monitoring and send out notifications as per the results encountered. Plugins could be added to Jenkins for building and testing many kind of applications.

To use Jenkins, you will need,
An accessible source code repository, e.g github, with your code checked in.
A working build script e.g maven script checked into the repository. Jenkins can be started from a command line or also from a web application server.

Installation of Jenkins:
Jenkins provides debian/ubuntu packages that installs it. Post installation, you can start jenkins with:
Download Jenkins.war file and install using the following command:
java -jar jenkins.war --httpPort=9000

sudo service jenkins start or
/etc/init.d/jenkins start

Running jenkins locally will start in on http://localhost:8080

At times you may also want to create a virtual host for Jenkins to run. You can create a virtual host on ubuntu by adding a file in /etc/apache2/sites-available/jenkins

Following details need to be added in this file:

<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName jenkins.local
ServerAlias ci
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPreserveHost on
ProxyPass / http://localhost:8080/
</VirtualHost>

You will then need to enable this virtual host by the following command
sudo a2ensite jenkins

To disable this site at any instance you can use the command
sudo dissite jenkins

Once the site is accessible on your localhost, you will need to



--------------------------------------------------------------------------------
meetup
for installation goto jenkins-ci.org
latest versus LTS try to go for LTS and not latest. latest might give you good plugins and fast processing etc. but it may break
installation can also be done using the war file java -jar jenkins.war

Configure Jenkins Server

Install a Patched Ruby Interpreter With Rbenv and Ruby-build for 2.0.0-p247

Installation of Ruby 2.0.0-p247 recently had some issues with Openssl package for Centos 6.5. I had to patch the version to get it running. Following script was written later that worked to automate the patch later.

 
#!/bin/sh
rm ~/.rbenv/cache/* -rf
mkdir /tmp/build
wget https://raw.github.com/sstephenson/ruby-build/master/share/ruby-build/2.0.0-p247 
cp 2.0.0-p247 /tmp/build/ 
# download and patch the ruby sources
wget http://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p247.tar.gz
tar xvzf ruby-2.0.0-p247.tar.gz
cd ruby-2.0.0-p247
curl https://gist.githubusercontent.com/spkane/8059362/raw/01585dcf6b33254124566f4521a3946e6f26e0a9/ruby-2.0.0-p247-openssl-el65.patch | patch -p1
cd ..
tar -cvzf ruby-2.0.0-p247-openssl.tar.gz ruby-2.0.0-p247
# download and patch the ruby-build version definition
sed 's|"2.0.0-p247.*|"2.0.0-p247-openssl.tar.gz" "file:///tmp/ruby-build/2.0.0-p247"|' < 2.0.0-p247 > 2.0.0-p247
#install the patched version
rbenv install /tmp/build/2.0.0-p247
rbenv rehash

Integrating Docker with Chef

Ever since Docker was introduced to me, the first thing that came to my mind was, Docker is a replacement for Chef. Docker does almost everything that Chef did and provides me with a light weight solution. While Chef just configures the system. Well I was wrong. I read many articles, heard many speeches and came to a conclusion that, it was not Chef vs Docker, it was Chef with Docker. When they work together, you get the most powerful tool to deliver fast, light and efficiently.

I conducted a demo at Docker-Bangalore meetup #4 to show how integration with Chef and Docker works. More information on the meetup page.

What is Chef?

• Configuration Management Tool.
• Helps write Infrastructure as a code.
• Users write code called ‘recipes’ to help manage and configure servers and applications.
• Runs in client/server as well as stand-alone configuration called chef-solo.
• Ensures each resource is properly configured and is in the desired state.
• More on my blog Learning Chef.

What is Docker?

• Light weight Linux container that packages an application and all its dependencies in a virtual container.
• The container can be packed, shipped and the application can run on any Linux machine on public cloud, private cloud or bare metal.
• Does not include OS.
• Layers the changes made in the container just like a version control system, which allows you to reach any previous state within no time.
• More on my blog Docker.

Why use Chef+Docker, What are the issues with Vagrant+Chef?

• If each project is in its own VM, resource usage is prohibited. 
• All the projects in a Single VM, management will be difficult, different versions for same software for different projects. 
• Building and Rebuilding takes long time, partial builds or rebuilds are difficult. Unless explicitly snapshot is created at various stages. 
• If you rely on external dependencies(which we do), full rebuilds can fail  due to one or more broken dependencies. 
• If your stack has multiple components (web, db, cache, etc) and they are installed in 1 Vagrant VM, the resulting setup differs from production. Or you could use multiple VMs… but point # 1 holds.

Chef Pros:

• Great at provisioning (knife bootstrap and plugins)
• Configuring services – Writing your infrastructure as a code.
• Testing your Infrastructure

Chef Cons:

• Packaging applications
• Deployments and rollbacks
• Dynamic service delivery

Docker Pros:

• Packaging applications
• Deployments and rollbacks
• Dynamic service delivery

Docker Cons:

• Managing persistent storage
• Complex networking
• Monolithic services

How to get Chef and Docker work together?

The key is understanding the Developers and Operations workflow and making them work together.

Developers own:

• Code and Libraries
• Build and Test Automation
• System Packages
• Runtime Configuration
• Release Management
• Logs monitoring
• Horizontal Scaling

Operations own the Platform:

• Hosts
• Routers
• Monitoring
• Logs
• Security
• Backing Services

Basically,

Developers take the control of the containers
&
Everything outside containers for Operations

Setting up Chef and Docker to work together:

• Install Chef workstation with Omnibus Installer.
• Create an account on manage.opscode.com or create your own chef-server.
• Create a vagrant node or a cloud instance.
• Site install docker community cookbook with site install.
• Create your own cookbook which depends on docker.
• Upload all cookbooks.
• Bootstrap the node.
• Login and verify of the node has the required images.

References:

• Brain Flad's chef-docker cookbook
• Gabriel Monroy's ChefConf presentation
• StackOverflow, Google groups and Quora