Pages

Saturday 26 April 2014

Learning Chef - Part - I

some rights reserved by Matt Ray
The information here has been collected from Nathan Harvey's Video tutorials on Chef's website and from Chef's official documentation. Before starting with the tutorial, I thought it would be better to understand common jargons used in chef.
Three primary entities: workstation, chef-server, node
  • Chef-Work Station : System from where the configuration management professional / devops / sys admin will be working.
  • Chef-Server : System/Server where all the infrastructure as a code will be stored. Also the Chef-Server will have many other features that we will seee later.
  • Nodes : Servers in your infrastructure that will be managed by chef, They may represent a physical server or a virtual server. They may represent hardware you own or multiple compute instances in a public or a private cloud. Each node will belong to one organization and and other organization may not have access to it. Each node will belong to one environment. Either in staging or Production etc. Each node will have zero or more roles. An application called chef-client run on each of the node. The chef-client will gather the current system configuration. It will download the desired system configuration from the Chef-server and configure that node such that it adheres to the policy defined.
  • Knife : Command line utility that acts as an interface between local chef-repo(on work station) and server. Knife lets you manage nodes, cookbooks, recipes, roles, stores of json data, including encrypted data, environments, cloud resources including provisioning. The installation of chef on management workstations, Searching of indexed data on chef server. You can even extend knife to use plugins for managing cloud resources. E.g knife-ec2, knife-rackspace, knife-vcloud plugins.
  • Cookbooks : A cookbook is a container to describe or to contain our configuration data. It contain the recipes. It can also include templates, files, custom resources etc.
  • Recipes : A Recipe is a configuration file that describes resources and their desired state. A recipe can install and configure software components, Manage files, Deploy Applications, Execute other recipes, etc.
    • Sample Recipe
package "apache2" // 1st resource is package and chef knows that it should be installed on the server. If the package doesn’t exist it will install it.

template "/etc/apache2/apache.conf" do //Next resource is a template. The template will manage a file at /etc/apache2/apache.conf
source "apache2.conf.erb"
owner "root"
group "root"
mode "0644"
variable(:allow_override => "All")
notifies :reload, "service[apache2] //state of the apache2 if apache2.conf exist it knows that it doesn’t need to create that file. However chef needs to make sure that the file has proper contents. So it will generate a temporary file. It will use the source that we specified above apache2.conf.erb and then it will also use any variable content that we specified, i.e AllowOverride All. Once the temporary file is created, Chef will then compare the two files. If they are the same, chef will discard the temporary file and then move on to the next resource, then notifies line will be ignored. However if the two files are different. The chef-client will discard the version on the disk and place the temporary file into the proper location, i.e overwrite existing file. Whenever the overwrite happens a notification will be sent. Then it will tell Apache to reload with new configs.
end

service "apache2" do //service should be enabled and start automatically
action [:enable,:start]
supports :reload => true
end
  • Roles : A way of identfying different types of servers. e.g Load-balancer, app server, DB cache, DB , monitoring etc. Roles may include list of configs to be applied called as runlist. May include data attributes for configuring infra, i.e ports to listen on, list of apps to be deployed.
  • Data bags : Stores of json data
  • Attributes : Attributes are mentioned in cookbooks/recipes. An attribute gives the detail about the node. It tells about the state of the node; before the chef-client run, present state and state after the chef-client run
  • Resources : Items that we sysadmins manipulate to manage complexity. i.e Networking, Files, Directories, Symlinks, Mounts, Registry key, Scripts, Users, Groups, Packages, Services File-systems etc. Resource represent a piece of the system and its desired state. e.g a package to be installed, A service to be running, A file to be generated, a cronjob to be configured, a user to be managed, etc.
  • Ohai : Ohai is a tool used to detect the attributes on the node. These attributes are then passed to the chef client at the beginning of the chef-client run. Ohai is installed on a node as a part of chef-client installation. Ohai has the following types of attributes: Platform details, Network usage, Memory usage, Processor usage, Kernel data, Hostnames, FQDNs, etc. So, Ohai is a utility that will give you all the information of your system level data.
  • Shef : Chef-Shell was earlier known as Shef. Its a recipe debugging tool that allows breakpoints within recipes. Its runs as an irb session.
  • Environments : Environments can be development, test, staging and production. They may contain data attributes specific to an environment. Starts with single environment e.g default is 1st. Different names/URLs for payment services, location for package repository, version of chef configs etc.
  • Run List : The joining of a node to a set of policies is called as a run-list. The chef-client will download all the necessary components that make up the run-list.e.g recipe[npt::client], recipe[users], role[webserver]. Run List is a collection of policies that a node should follow. Chef-client obtains the run-list from chef-server. chef-client ensures the node complies with the policy in the run-list.
  • Search : You can search for nodes with Roles, Find topology data, i.e IP addresses, hostnames, FQDNs. Searchable index about your infrastructure. e.g load balancer needs to know which application should I be sending requests to? Chef-client can ask Chef-server which application servers are available and which application server should I be sending load to. And in return the chef server can send a list of nodes and then the load balancer can figure out which one based on the hostname or IP address or Fully Qualified Domain Name.
  • Organization : Everyone has their own infra and wont manage anyone else's infra. Organizations are independent tenants on Enterprise chef. So this could be different companies, business units or departments for managing.

Wednesday 23 April 2014

Automation for VMware vCloud Director using Chef's knife-vcloud - Part-II

Version 1.2.0


 Some right reserved by Phil Wiffen

For some reason with the previous repo I could not see the list of all vApps. Only some of it (a mixture of both chef node and non chef nodes) were seen. So I went ahead with another version of knife-vcloud plugin available which solved my problem to a large extent.
Plugin is available at https://github.com/astratto/knife-vcloud

Configuration used:
  • CentOS 6.5
  • Chef 11.8.2
  • knife-vcloud 1.2.0
Following steps were used to complete the automation process:
Installation is fairly simple
gem install knife-vcloud
gem list | grep vcloud
- See if after entering the above command you see the gem knife-cloud. If yes the setup was successful. If no something went wrong.

cd ~./chef
vim knife.rb
Configuration is almost automated:
knife vc configure

You will be prompted for vcloud_url, login and password. After entering the details check that the details you entered are reflected in the knife.rb file.

knife[:vcloud_url] = 'https://vcloud.server.org'
knife[:vcloud_org_login] = 'vcloud_organization'
knife[:vcloud_user_login] = 'vcloud_user'
knife[:vcloud_password] =

Note: The organization was not updated for me, and it kept giving authorization failure for quite sometime. If you see that the organization is not updated automatically, please update it manually in the knife.rb file.

The subsequent commands would also change for the detailed listing. Although the documentation at many instances says that the name of VM or vApp should suffice to pull up the required details, note that at many instances you will be required to enter the ID and not just the name.

To see the list of catalog items

[root@chefworkstation ~]# knife vc catalog show All_ISOs
Description: All ISO Dumps
Name                                           ID                                          
CentOS-6.3                                          WhAtEvEr-Id-tO-bE-SeEn1       
CentOS-6.4_x64                                   WhAtEvEr-Id-tO-bE-SeEn2        
Ubuntu-copy                                           WhAtEvEr-Id-tO-bE-SeEn3        

To see details of the organization

[root@chefworkstation ~]# knife vc org show MYORG
CATALOGS                                                                 
Name                                  ID                                 
All_ISOs                                  WhAtEvEr-Id-tO-bE-SeEn4
Master Catalog                        WhAtEvEr-Id-tO-bE-SeEn5
                                                                         
VDCs                                                                     
Name                                  ID                                 
MyorgVDC-Tier1     WhAtEvEr-Id-tO-bE-SeEn6
MyorgVDC-Tier2        WhAtEvEr-Id-tO-bE-SeEn7
MyorgVDC-Tier3        WhAtEvEr-Id-tO-bE-SeEn8

NETWORKS                                                                 
Name                                  ID                                 
MyorgNet-Router                   WhAtEvEr-Id-tO-bE-SeEn9

TASKLISTS                                                                
Name                                  ID                                 
                        WhAtEvEr-Id-tO-bE-SeEn10
To create a new vApp:

[root@chefworkstation ~]# knife vc vapp create MyorgVDC-Tier1 chefnode2 "Just Created node2" WhAtEvEr-Id-tO-bE-SeEn
vApp creation...
Summary: Status: error - time elapsed: 52.012 seconds
WARNING: ATTENTION: Error code 400 - The following IP/MAC addresses have already been used by running virtual machines: MAC addresses: 10:20:30:40:50:0f IP addresses: 192.168.0.20 Use the Fence vApp option to use same MAC/IP. Fencing allows identical virtual machines in different vApps to be powered on without conflict, by isolating the MAC and IP addresses of the virtual machines.
vApp created with ID: WhAtEvEr-Id-tO-bE-SeEn

Note: that there are certain problems that were corrected later.
To show the deatils of created vApp:
[root@chefworkstation ~]# knife vc vapp show WhAtEvEr-Id-tO-bE-SeEn1
Note: --vdc not specified, assuming VAPP is an ID
Name: chefnode2
Description: Just Created node2
Status: stopped
IP: 192.168.0.12
Networks
MyorgNet-Router
   Gateway      Netmask        Fence Mode  Parent Network       Retain Network
      192.168.0.1  255.255.255.0  bridged     MyorgNet-Router  false        
      VMs
      Name    Status   IPs           ID                                    Scoped ID                          
      centos  stopped  192.168.0.12  WhAtEvEr-Id-tO-bE-SeEn  WhAtEvEr-Id-tO-bE-SeEn

To show the vm specific details:

[root@chefworkstation ~]# knife vc vm show WhAtEvEr-Id-tO-bE-SeEn --vapp MyvApp_Chef
Note: --vapp and --vdc not specified, assuming VM is an ID
VM Name: centos
OS Name: CentOS 4/5/6 (64-bit)
Status: stopped
Cpu                                          
Number of Virtual CPUs  1 virtual CPU(s)     

Memory                                       
Memory Size             2048 MB of memory    

Disks                                        
Hard disk 1             16384 MB             
Hard disk 2             16384 MB             

Networks                                     
MyorgNet-Router                          
Index                 0                    
Ip                    192.168.0.12         
External ip                                
Is connected          true                 
Mac address           10:20:30:40:50:0f    
Ip allocation mode    MANUAL               

Guest Customizations                         
Enabled                 false                
Admin passwd enabled    true                 
Admin passwd auto       false                
Admin passwd                                 
Reset passwd required   false                
Computer name           centos
  

To set new info to the vm:

[root@chefworkstation ~]# knife vc vm set info --name ChefNewNode WhAtEvEr-Id-tO-bE-SeEn --vapp MyvApp_Chef centos
Note: --vapp and --vdc not specified, assuming VM is an ID
Renaming VM from centos to ChefNewNode
Summary: Status: success - time elapsed: 7.09 seconds

To update other info:


[root@chefworkstation ~]# knife vc vm set info --ram 512 WhAtEvEr-Id-tO-bE-SeEn --vapp MyvApp_Chef
Note: --vapp and --vdc not specified, assuming VM is an ID
VM setting RAM info...
Summary: Status: success - time elapsed: 9.843 seconds

To edit network info:


[root@chefworkstation ~]# knife vc vm network edit WhAtEvEr-Id-tO-bE-SeEn MyorgNet-Router --net-ip 192.168.0.117 --ip-allocation-mode MANUAL
Note: --vapp and --vdc not specified, assuming VM is an ID
Forcing parent network to itself
VM network configuration...
Guest customizations must be applied to a stopped VM, but it's running. Can I STOP it? (Y/N) Y
Stopping VM...
Summary: Status: success - time elapsed: 7.092 seconds
VM network configuration for MyorgNet-Router...
Summary: Status: success - time elapsed: 6.783 seconds
Forcing Guest Customization to apply changes...
Summary: Status: success - time elapsed: 22.639 seconds

To show the changes made:

[root@chefworkstation ~]# knife vc vm show WhAtEvEr-Id-tO-bE-SeEn
Note: --vapp and --vdc not specified, assuming VM is an ID
VM Name: ChefNewNode
OS Name: CentOS 4/5/6 (64-bit)
Status: running

Cpu                                          
Number of Virtual CPUs  1 virtual CPU(s)     

Memory                                       
Memory Size             512 MB of memory     

Disks                                        
Hard disk 1             16384 MB             
Hard disk 2             16384 MB             

Networks                                     
MyorgNet-Router                          

Index                 0                    
Ip                    192.168.0.117        
External ip                                
Is connected          true                 
Mac address           10:20:30:40:50:0f    
Ip allocation mode    MANUAL               

Guest Customizations                         
Enabled                 true                 
Admin passwd enabled    true                 
Admin passwd auto       false                
Admin passwd                                 
Reset passwd required   false                

Computer name           centos

Reference Links: