Integrating Docker with Chef

Ever since Docker was introduced to me, the first thing that came to my mind was, Docker is a replacement for Chef. Docker does almost everything that Chef did and provides me with a light weight solution. While Chef just configures the system. Well I was wrong. I read many articles, heard many speeches and came to a conclusion that, it was not Chef vs Docker, it was Chef with Docker. When they work together, you get the most powerful tool to deliver fast, light and efficiently.

I conducted a demo at Docker-Bangalore meetup #4 to show how integration with Chef and Docker works. More information on the meetup page.

What is Chef?

• Configuration Management Tool.
• Helps write Infrastructure as a code.
• Users write code called ‘recipes’ to help manage and configure servers and applications.
• Runs in client/server as well as stand-alone configuration called chef-solo.
• Ensures each resource is properly configured and is in the desired state.
• More on my blog Learning Chef.

What is Docker?

• Light weight Linux container that packages an application and all its dependencies in a virtual container.
• The container can be packed, shipped and the application can run on any Linux machine on public cloud, private cloud or bare metal.
• Does not include OS.
• Layers the changes made in the container just like a version control system, which allows you to reach any previous state within no time.
• More on my blog Docker.

Why use Chef+Docker, What are the issues with Vagrant+Chef?

• If each project is in its own VM, resource usage is prohibited. 
• All the projects in a Single VM, management will be difficult, different versions for same software for different projects. 
• Building and Rebuilding takes long time, partial builds or rebuilds are difficult. Unless explicitly snapshot is created at various stages. 
• If you rely on external dependencies(which we do), full rebuilds can fail  due to one or more broken dependencies. 
• If your stack has multiple components (web, db, cache, etc) and they are installed in 1 Vagrant VM, the resulting setup differs from production. Or you could use multiple VMs… but point # 1 holds.

Chef Pros:

• Great at provisioning (knife bootstrap and plugins)
• Configuring services – Writing your infrastructure as a code.
• Testing your Infrastructure

Chef Cons:

• Packaging applications
• Deployments and rollbacks
• Dynamic service delivery

Docker Pros:

• Packaging applications
• Deployments and rollbacks
• Dynamic service delivery

Docker Cons:

• Managing persistent storage
• Complex networking
• Monolithic services

How to get Chef and Docker work together?

The key is understanding the Developers and Operations workflow and making them work together.

Developers own:

• Code and Libraries
• Build and Test Automation
• System Packages
• Runtime Configuration
• Release Management
• Logs monitoring
• Horizontal Scaling

Operations own the Platform:

• Hosts
• Routers
• Monitoring
• Logs
• Security
• Backing Services

Basically,

Developers take the control of the containers
&
Everything outside containers for Operations

Setting up Chef and Docker to work together:

• Install Chef workstation with Omnibus Installer.
• Create an account on manage.opscode.com or create your own chef-server.
• Create a vagrant node or a cloud instance.
• Site install docker community cookbook with site install.
• Create your own cookbook which depends on docker.
• Upload all cookbooks.
• Bootstrap the node.
• Login and verify of the node has the required images.

References:

• Brain Flad's chef-docker cookbook
• Gabriel Monroy's ChefConf presentation
• StackOverflow, Google groups and Quora

Docker - Lightweight Linux Container

Docker: Its a tool that helps you to pack, ship and run any application as a light-weight Linux container. More on https://www.docker.io/

Works best on Linux kernel 3.8 Ubuntu 12.04 precise has 3.2 and needs to be upgraded. 

Install Docker with on Ubuntu 12.04:

sudo apt-get update

sudo apt-get install linux-image-generic-lts-raring linux-headers-generic-lts-raring

sudo reboot

To check docker version:

sudo docker version

Client version: 0.11.1

Client API version: 1.11

Go version (client): go1.2.1

Git commit (client): fb99f99

Server version: 0.11.1

Server API version: 1.11

Git commit (server): fb99f99

Go version (server): go1.2.1

Last stable version: 0.11.1

To check info about docker installed:

sudo docker info

Containers: 1

Images: 9

Storage Driver: aufs

 Root Dir: /var/lib/docker/aufs

 Dirs: 11

Execution Driver: native-0.2

Kernel Version: 3.11.0-20-generic

WARNING: No swap limit support

To pull an existing docker image:

sudo docker pull <imagename>

sudo docker pull busybox

HelloWorld in docker:

sudo docker run busybox echo HelloWorld

Search for an existing image in the index:

docker search <image-name>

sudo docker search stackbrew/ubuntu

NAME                       DESCRIPTION                                     STARS     OFFICIAL   TRUSTED

stackbrew/ubuntu           Barebone ubuntu images                          36                   

jprjr/stackbrew-node       A stackbrew/ubuntu-based image for Docker,...   2                    [OK]

hcvst/erlang               Erlang R14B04 based on stackbrew/ubuntu         0                    [OK]

stackbrew/ubuntu-upstart                                                   0                    

Pull an existing image:

sudo docker pull ubuntu

Pulling repository ubuntu

a7cf8ae4e998: Pulling dependent layers 

3db9c44f4520: Downloading [=================>                                 ] 22.18 MB/63.51 MB 2m19s

74fe38d11401: Pulling dependent layers 

316b678ddf48: Pulling dependent layers 

99ec81b80c55: Pulling dependent layers 

5e019ab7bf6d: Pulling dependent layers 

511136ea3c5a: Download complete 

6cfa4d1f33fb: Download complete 

To the check the available images:

sudo docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

ubuntu              13.10               5e019ab7bf6d        2 weeks ago         180 MB

ubuntu              saucy               5e019ab7bf6d        2 weeks ago         180 MB

ubuntu              12.04               74fe38d11401        2 weeks ago         209.6 MB

ubuntu              precise             74fe38d11401        2 weeks ago         209.6 MB

ubuntu              12.10               a7cf8ae4e998        2 weeks ago         171.3 MB

ubuntu              quantal             a7cf8ae4e998        2 weeks ago         171.3 MB

ubuntu              14.04               99ec81b80c55        2 weeks ago         266 MB

ubuntu              latest              99ec81b80c55        2 weeks ago         266 MB

ubuntu              trusty              99ec81b80c55        2 weeks ago         266 MB

ubuntu              raring              316b678ddf48        2 weeks ago         169.4 MB

ubuntu              13.04               316b678ddf48        2 weeks ago         169.4 MB

busybox             latest              2d8e5b282c81        2 weeks ago         2.489 MB

ubuntu              10.04               3db9c44f4520        2 weeks ago         183 MB

ubuntu              lucid               3db9c44f4520        2 weeks ago         183 MB

To run a command within an image:

docker run<image> command

sudo docker run ubuntu echo HelloWorld

HelloWorld

To install something on an ubuntu image

sudo docker run apt-get install <package>

find ID of the container 

sudo docker ps -l

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES

0dac167b178d        ubuntu:14.04        ps aux              12 minutes ago      Exited (0) 12 minutes ago                       goofy_bell

committing changes made to the images:

docker commit 0da firstcommit

723aa6ead77a14ff05cd2c640163345ec5a36fa9a4c757a6872a1ec919ab9345

To get log of the present container:

sudo docker logs 0dac167b178d

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

root         1  0.0  0.0   7132   644 ?        Rs   09:10   0:00 ps aux

To inpect the details of an image

sudo docker inspect <id: 3-4 characters of id will work too>

sudo docker inspect 0da

<json output>

Push container image to the index

sudo docker push ubuntu

Creating Dockerfile:

All instructions in Dokerfile are in the form of 

INSTRUCTION arguments

Instruction are not case sensitive but CAPS are recommended. The first instruction in any Dockerfile is the FROM instruction. The syntax is:

FROM <image>

FROM ubuntu

This will look for the image in Docker index. You can also search docker index by the command docker search

Next is the RUN instruction. The RUN instruction will execute any commands on the current image. After executing, it will also commit the changes. The committed image can be used for the next instructions from the Dockerfile. This way the committed changes form a layer of changes just like any other source code control system. Syntax of RUN command:

RUN <command>

RUN apt get install -y apache2

Here the RUN command is equivalent to docker run image command + docker commit container_id. Here image will be automatically replaced with the current image and container_id is the result of the previous commit.

Once you have created your Dockerfile you can use docker build to create you image from it. You can use the command in this way.

Create a Dockerfile with the content

FROM ubuntu

RUN apt-get install -y memcached

Save and close the file. If the file is in you r present directory:

docker build .

If the file in in some other location 

docker build path/to/file

If passing through STDIN

docker build - < Dockerfile

If passing through github URL

docker build github.com/roshan4074

you can check the container with the command:

sudo docker images

To apply a tag to an image you the command: docker tag 

sudo docker tag <container_id>

To comment a code use the "#' symbol followed by the text

To specify the contact info of the Maintainer of the Dockerfile:

MAINTAINER Name contact@email

To trigger a command as soon as a container starts, use ENTRYPOINT instruction

ENTRYPOINT echo "Hello, Container Started"

Another Format to use ENTRYPOINT Instrcution is 

ENTRYPOINT ["echo", "Hello, Container Started"]

This is the preferred format

e.g

ENTRYPOINT ["wc", "-l"]

To execute a certain command by a particular user use the command USER

USER roshan

T open a particular port for a process use EXPOSE instruction

EXPOSE 8080