Automation for VMware vCloud Director using Chef's knife-vcloud

 Some right reserved by jdhancock

Plugin is available at https://github.com/opscode/knife-vcloud

Configuration used:

• CentOS 6.5
• Chef 11.8.2
• knife-vcloud 1.0.0

Following steps were used to complete the automation process:

cd ~
git clone https://github.com/opscode/knife-vcloud.git
cd knife-vcloud/
bundle install
gem build knife-vcloud.gemspec
gem install knife-vcloud-1.0.0.gem
gem list | grep vcloud
- See if after entering the above command you see the gem knife-cloud. If yes the setup was successful. If no something went wrong.

cd ~./chef
vim knife.rb

- add the following details to the last line of this file (Note: username is @ i.e organisation name supplied) :
knife[:vcloud_username] = "username@orgname"
knife[:vcloud_password] = "##########"
knife[:vcloud_host] = "xxx.xxxxxxxxxxxxx.com"

[ESC]:wq

knife vcloud server list
- Should list all the existing servers

You can also create your own server using "knife vcloud server create" with additional parameters with caution.

e.g

knife vcloud server create --vcpus 2 -m 1024 -I TestServer -A 'roshan' -K "MyPassword" -r 'role[webserver]' --network myNetword-id

Good Luck!!

Reference Links:

• https://github.com/opscode/knife-vcloud
• http://blogs.clogeny.com/category/tutorials/
• http://www.getchef.com/blog/chefconf-talks/writing-a-knife-plugin-for-your-shiny-new-vmware-vcloud-director-based-cloud-chirag-jog/

Public, Private and Hybrid Cloud

A lot has been said, heard and read about Cloud. There so many ways that the cloud gets filtered further. In my previous blog we discussed about Saas, Paas and Iaas. More and more companies are looking for cloud as the solution for their business needs. We shall further discuss 3 important types of cloud.

1. Public Cloud
2. Private Cloud
3. Hybrid Cloud

Some rights reserved by FutUndBeidl

 Public Cloud : Public cloud is considered to be a standard cloud computing model where there is a direct interaction with the users of cloud. It also called as 'shared cloud'. All applications, infrastructures or storage are made directly available to the users. It could be a Pay as You Go service or free as well. Types of public clouds include all Saas, Pass and Iaas platforms. The primary benefit is that it is accessible from anywhere anytime. Public cloud is an ultimate choice when you have lots of users for you application. For e.g an email application like Google, a social network like Facebook. A collaboration is needed among developers over a Paas or for employees to work remotely public cloud is the best choice. This cloud may or may not be managed by the providers but usually it is. Also, it can be scaled very easily as per our needs.
For e.g Iaas based service like Dropbox you can add and remove space dynamically as per your choice.

Some rights reserved by FutUndBeidl

Private Cloud : Here the services and Applications are not exposed to general public and are instead kept private. Highest level of security and control is maintained in these kind of architectures. These services often run behind a firewall and are also called as 'Enterprise clouds'. Advantage is security and resources can be shared among groups. There's a choice to make for H/W and S/W with private clouds and the ability is greatly dependent on what is being used. Its used mostly by companies dealing with high level confidential data.
Many companies are now opting for Enterprise cloud

Hybrid Cloud : Even though many organizations make the use of private as well as public cloud as per their need, there could be vendors looking for functionality of both a private as well as a public cloud. This is achieved with a Hybrid Cloud. At times there are companies that want their data to be secure as well as still are required to communicate to the customers over the network. Many of such companies choose a Hybrid Cloud. Here basically you can set access permissions for which applications need to be publicly accessible and which of them should not be and needs to be in private cloud. 

(To be contd..)

Virtualization with Vagrant

 some rights reserved by MedithIT

While going ahead with Chef configuration management tutorials i came across "Vagrant". I heard this term many a times. For quite sometime i thought it was just another VirtualBox like application. until i finally started using it.

What is Vagrant?

Open Source application for creating and configuring virtual dev environment. Vagrant manages VMs hosted in VirtualBox. Basically its a commandline utility that allows you to communicate with VirtualBox (or any virtualization software) with an easy set of commands. Many describe it as a wrapper around the virtualization software too.

How is it used?

Fist of all download Vagrant for your OS from this link. Install it. As mentioned its a commandline utility tool, you need commandline to access this.

1. You will need to initialize vagrant box using the init command. This will initialize a vagrant environment in the present directory you are in. The second argument will set the name for your box and the third will set the URL to access in the Vagrant file.
   
• Syntax: vagrant init [box-name] [url-to-access]
• Example: vagrant init precise32 http://files.vagrantup.com/precise32.box
2. Next you need to create and configure the vagrant box as per your vagrant file. Use the up command for this. This command will be used frequently as this is how you start you machine as well.
   
• vagrant up
3. Now that the machine has started you still have not logged in to the machine. You can use ssh to login. You don't need to use the traditional long ssh command to login to your box a simple vagrant ssh suffices.
   
• vagrant ssh
• Note: Please check the documentation on the vagrant website as there is a list of optional parameters that you may need in case you run into any errors. Fatal can be expected.
4. To check valid configurations to ssh into a running vagrant box use vagrant ssh-config
• vagrant ssh-config
5. As you now know how to setup and start using your vagrant box, we also need to know how to shut it down. This is simple too by just a "vagrant halt" similar to the halt command in linux machine.
• vagrant halt
6. These were the few basics of vagrant. You can do more with vagrant as well. The vagrant box command gives you other alternatives that you could try out.
• vagrant box add [box-name] [url-path]
• This adds a box with the specified name using the local file path or url specified to access it.
• vagrant box list
• Lists all the boxes installed and available
• vagrant box remove [box-name] [provider]
• Removes the box with the specified box name for the specific provider. Providers are VirtualBox or VMWare or any other utility.
• vagrant box repackage [box-name] [provider]
• Repackages the given box and puts it to present directory for redistribution purpose. When a box is added, vagrant unpacks and stores it internally and the original box is not preserved.
7. Restarting a vagrant machine can be done using vagrant reload. Its equivalent to a vagrant halt and vagrant up.
• vagrant reload
8. To check the current state of the machine, i.e to verify if the machine is running, stopped or not created etc a status command helps
• vagrant status
9. To save the status of the machine and suspend it so that you can resume it at a later instance and not completely shut it down, you can use suspend command.
• vagrant suspend
10. To resume a suspended machine use vagrant resume
• vagrant resume
11. Finally, to stop and delete/destroy and existing machine use the destroy command. All the resources allocated are destroyed as if the machine was never there. This command asks for confirmation before destroying.
• vagrant destroy

Checking Open Ports on a Remote Computer using PortQry

Some rights reserved by Ryan Franklin

 Today for one of the projects the SFTP connection kept failing for some reason. The user-id password used for connecting to the host was correct the hostname was as well correct. There was no way to find out what went wrong. Thankfully command-line gives a good log to verify what goes wrong.

I tried connecting to the SFTP host with various tools like FileZilla, WinSCP but could not get good enough logs. Finally i tried connecting the server using ssh on command-line using my Ubuntu machine. The connection used to time out. That is what i see in the logs as well. I assumed that probably the SFTP port number 22 was closed for the host.

I googled for if i could find a tool to check if a particular port on a machine is accessible or not. I finally found something called as PortQry taht could be used on Windows machine using Commandline.

Its a very small 140 KB command-line based software tool that you can use to check if a port on some machine is accessible or not.

After using this tool i got to know that the machine had a Firewall  kind of protection which wasn't allowing me to access the SFTP port on it.

Here's how you PortQry on Windows:

• Download the software using the link : http://www.microsoft.com/en-in/download/details.aspx?id=17148

• Double click an unzip the files to any location say C:/

• Hit Windows+R in the run box enter "cmd"

• Go to the directory where the PortQry was extracted.

• Execute the program PortQry by entering PortQry<enter>

• This will display a list of help information and the correct usage of the command

The following is the syntax to check the port status :
portqry -n myhostname.net -e 80

PortQry can inform the status of a port as "Listening", "Not Listening", or "Filtered"
Listening : There is some service active on that port
Not Listening : Port is closed
Filtered : No response, Presumably its behind some kind of firewall.

Syntax
portqry -n name_to_query [-p protocol] [-e || -r || -o endpoint(s)]

Common command line switches:
-n : IP address or name of system to query
-p : TCP or UDP or BOTH (default is TCP)
-e : single port to query (valid range: 1-65535)
-r : range of ports to query (start:end)

For single port use
portqry -n 127.0.0.1 -e 80

For a Range of ports, use the -r switch:
portqry -n 127.0.0.1 -r 80:85

Note:
- PortQry also displays extended information for known services, such as SMTP, POP3, IMAP4, FTP, and is capable of performing LDAP queries.
- A GUI based alternative is also available now called PortQryUI

Sample Output:
C:\PortQryV2>portqry -n 127.0.0.1 -e 40
Querying target system called:
 127.0.0.1
Attempting to resolve name to IP address...
Name resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): FILTERED

C:\PortQryV2>portqry -n 127.0.0.1 -e 80
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 80 (http service): LISTENING

C:\PortQryV2>portqry -n 127.0.0.1 -e 22
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): NOT LISTENING

Data Recovery with TestDisk

some rights reserved by epSos .de

 You buy a portable hard disk drive to keep a back up of all your important data, Or you are carrying an extremely important doc in a USB drive, or you have some data on your internal hard disk drive. You try to get the data and ... boom!!! No Data!!! You have absolutely no clue how to get back the important data you had in the drives. Finally you just consider it lost and curse your fate and carry on.

This happened to me too. 1 TeraBytes of important data in my external Hard disk drive got corrupt in a second and was lost. I tried all OS's i could right from MAC to Windows to Linux but no luck. I was about to give up when i came across Data Recovery softwares on Internet.

Data Recovery softwares are those softwares that help you read corrupt/lost/deleted data from your drives. There were so many softwares out there. I started trying out my luck on all the softwares i could with a windows machine(i should not have done that, but i have no option!!). Unfortunately most of them just gives a list of deleted items on the disk that i never needed. One or two of them did get me the data however since they were trial versions, i could only view the data and not copy them. Anyways it just gave me the hope that the data was still there in the disk.

Finally i searched for Open Source alternatives and there i got my answer. TestDisk!!!

What is TestDisk?
As per their website wiki: TestDisk is a free data recovery software designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software, certain types of viruses, or human error. It can also be used to repair some filesystem errors.

Installation:
You don't need to install it. Its a very small 1.5 MB software, where you just need to run the executable, it was really easy to use command line based software with most of the information self explanatory.

Steps to Use:

1. Logs Creation : Like every other Linux based software, this one as well makes a log of every session carried out. when you execute the TestDisk.exe You are 1st asked for if, you would like to CREATE a new log, APPEND  the existing log, or want NO LOG.
2. Media/Disk Detection :Next you need to select the disk you want to recover the lost data from. It shows a list of all volumes connected to you computer and you can select using keyboard arrow keys.
3. Disk/Partition Table Type Selection : The next screen prompts you to select the partition table type. In most of the cases it detects all by itself the type. Otherwise it keeps it to none for you to select. Mine was a windows machine that i was running this program on and i used this with quiet a many volumes and every time it detected Intel. I am assuming windows users could probably select Intel in case they are unsure of this option. But selection "None"is not recommended as its very rare that a drive is non partitioned.
4. Next Screen Gives the following Options :

Analyse Analyse current partition structure and search for lost partitions to restore them
Advanced Filesystem Utils

FAT: Boot and FAT repair

NTFS: Boot and MFT repair

ext2/ext3: Find Backup SuperBlock
FAT file undelete
NTFS file undelete
ext2 file undelete
Image Creation

Geometry Change disk geometry

Options Modify options

MBR Code Write TestDisk MBR code to first sector

Delete Delete all data in the partition table

Analyzing of disk is done to look for lost partitions. This takes long time based on the size of the disk. Most of the times for windows if you get an error "The type of the file system is RAW." or "The disk in drive D is not formatted. Do you want to format it now?" then that means the Boot sector is damaged. You can click on Advanced file system Utils and and use the FAT: Boot and FAT repair/NTFS: Boot and MFT repair based on the type of partitioin you have. This was the problem with my disk too and got corrected in no time.



To be contd ...

Monitoring in Linux/Unix Environment using TOP

Some rights reserved by Steve Jurvetson

Top

Top is the Linux performance monitoring program. For windows user, its analogous to the Task Manager. This command displays the active processes at real-time and updates the list regularly. Other system details like CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands and much more.
The 1st line of the command mentions the following:

• current time in hh:mm:ss format seconds keep updating
• uptime of the machine, how long has the machine been running
• no. of users logged in with running sessions
• average load on the system, 3 values mentioned are load in last one minute, 5 minutes and 15 minutes

The 2nd line mentions the following :

• Total number of processes running
• Present number of running processes
• Total sleeping processes
• Total stopped processes
• Total Zombie processes(waiting for parent process to stop)

The 3rd row mentions the following :

• % of CPU for user processes
• % of CPU for system/kernel processes
• % of CPU for prioritized or priority upgraded processes nice
• % of CPU not used
• % of CPU awaiting i/o operation
• % of CPU serving h/w interrupts
• % of CPU for s/w interrupts
• % of CPU stolen from virtual memory(steal time) this will be zero if no virtual machine running.

The 4th and 5th row mentions the following:

• The use of Physical memory
• The use of swap memory
• Both free buffer and cached

The details of the processes are given  with the following details:

• PID: Process ID
• USER: The owner user of the process
• PR: Priority of the process
• NI: Nice value of the process
• VIRT: Amount of Virtual Memory used by the process
• RES: Amount of physical memory used by the process
• SHR: Shared memory of the process
• S: Status of the process Sleep Running Zombie
• %CPU: % of CPU used
• %MEM: % of RAM used
• COMMAND: name of the process

The default sorting of the list displayed is based on CPU usage. you can change the sorting of the list as per your convenience.
Changing the sorting:
Press Shift+o. A list will be displayed giving all possible options using which you may sort the list, a letter corresponding to the sort criteria. Select that particular letter and hit 'return/enter' and see the new sorted list.
Display Processes for a specific User:
top -u username
This command will show the details of all the processes under the specific username mentioned in the command.
Highlight any Running Process :
Press Z after running top to highlight the running process to identify them easily.
Show absolute Path of the process:
To see the path from where the processes are being invoked press 'c' after running top
Change screen refresh interval:
To change the screen refresh interval of the processes running press 'd' and enter any number in seconds to set the time interval for refresh.
Kill Running Process:
to kill any of the running processes press 'k' and enter the process id of the process to be killed. After this you will be required to enter the signal (15) to kill the process.
To sort by CPU Utilization: Shift+p
Save Results of Top Command: press Ctrl+w
for help: press h
Exit top after specific Repetition: top -n <number>
Manual Page for top : man top

Looking for Free Windows OS? ReactOS is here.

Some rights reserved by Plum Lady

I keep looking for free stuff available online. In the early days when my searches were only limited to torrents, keygens, patches etc, didn't get much of a exposure to free legal stuff. FOSS came for rescue and then many different things lined up. Thanks to Ubuntu!!! Its been more than 3 years that i quit windows for daily use... But what haunts me at times is the when there is some software that i need to use desperately and its setup is only available for Windows. Wine helps many a times but not always. At times i felt, if Windows could be free too ... :(

Then on one fine day i wanted to make a doc. I use Libre generally. But this one needed to be highly compatible since i was suppose to send it to some windows user. I got Office 2007 installed in Wine. I used it for quite sometime and then it started throwing some weird error and got closed time and again. At this instance the first thing is googled is Free and open source windows like OS. While there were many linux pages pulled up that i already knew about, this one caught my attention. "Free and Open Source Windows like OS - ReactOS ".

So, what is ReactOS? Its an effort to provide Free and Open Source alternative to Windows. The term "React" also comes from a reaction of dissatisfaction of Microsoft's monopolistic position(source Wikipedia).

ReactOS is an Open Source implementation of NT family of windows. A part of the implementation has been assisted from Wine. The OS is still in its alpha stage and has been selected as a best project from Forge Magazine.

Many alternatives for the set up are available i.e VM images, iso images etc for everyone to have maximum access. The site has all kind of download options available. If in case you happen to find some issue in the installation, you can always report it to the team in their JIRA. The installations are both automated and manual for testing purpose.

You can even contribute to the community as a Dev, QA, Release or Tech Writer. Wishing to get this one released soon. :) 

Free and Open Source Alternatives for most of the paid softwares on Windows Machine

 Some rights reserved by opensource.com

I have recently been using windows more than usual for official use. Being an Ubuntu user i am not much used to either use a trial version of any paid software or pay for any software. I googled a lot for my daily use softwares to make my life comfortable and found that almost for every paid software there was an alternative in the Free and Open Source World. Here is what i found:

• Operating System : Widows (Paid) -> ReactOS (Alternative) I have a different blog written for this.
• Office Utility : Microsoft Office (Paid) -> Libre Office, Open Office(Alternative). For years i only knew about Open Office as an open source alternative. However after going through the threads on Ubuntu forums as to why was Libre Office considered over Open Office as a default Office utility for the OS i came to know that Libre Office development was far too faster than that of Open Office. I even saw the results myself to verify this fact. I would recommend Libre Office for the same reason.
• Video/Music Player : Windows Media Player (Most Probably Paid - I have hardly heard of Microsoft ever releasing anything free of cost :D) -> VLC media Player, Media Player Classic(Alternative).
• Remote File Sharing : WinSCP, FileZilla, Putty
• IDE for C# Programming : Microsoft Visual Studio (Paid) -> Sharp Develop (Alternative) (Although i was looking for Command line utility as i am only used to VIM as such, but found this one and made a note of it.
• File Compression : WinRAR, WinZIP(Paid) -> 7zip(Alternative)
• PDF Reader : Adobe Acrobat Reader(Freeware+Commercial) -> Foxit(Alternative)
• Email Client : Outlook(Paid) -> Thunderbird(Alternative)
• VM : VMWare(Paid) -> VirtualBox(Alternative)
• Reference Management : Zotero (Zotero is another good add-on cum software for Reference Management. You can save your websites and URL/PDF preferences online so that whenever you clean your cache on browsers and you desperately needed some link and you cleared your cache and reinstalled new browser. You can check your Zotero and get URL/PDF back as all these preferences are stored online in your separate account.)
• Cloud Storage : GoogleDrive/Dropbox

Second Place Story : Server ghouls haunt bulk ingestion

Some rights reserved by Julie Rybarczyk

I recently shared one of my stories while monitoring an Ingestion Server with AppFirst for one of the Halloween contest they conducted. Thankfully, won 2nd prize in the same. Sharing the same story here along with the link to visit.

Second Place Story

Server ghouls haunt bulk ingestion
Company: Roshvert
I was suppose to monitor an Ingestion Server that was performing a bulk ingestion through an EC2 instance with around 200 GB of data to be ingested to another server.
Since it was a huge amount of data and the ingestion would take another day to complete, I kept the ingestion going and the logs were performing well. I decided then that I’d log in early tomorrow morning to check the ingestion status. During this time, the log files were supposed to be created automatically through the ingestion and the name of the log file for any particular day should be log_dd-mm-yyyy.txt with date of that day mentioned. It was a staging server and the code was supposed to be supplied for UAT in a day or two.
I logged in early the next morning to check the ingestion status. I was totally puzzled as I couldn’t make out what was happening:

• The log file for the previous day log_27-08-2013.txt was showing everything went well until 11pm midnight and no logs thereafter.
• The log file for today log_28-08-2013.txt got created with no data in it.
• The ingestion process was running with no errors.
• The server logs showed no errors.
• The system never went down.
• Nearly 150 GB of data was still to be ingested and was not progressing at all.
• None of the logs showed any updates as to why the ingestion was not progressing.

Since the delivery was urgent, I stopped the ingestion on the instance and restarted it. To my horror, the ingestion was not progressing at all. I tried running ingestion on other instances, and it worked fine.
Then something hit me, and I went back to check the logs of ingestion. The ingestion logs still showed nothing with 0 kb space used by the logs. Wait!!! Space? 0kb? 150 GB data still remaining?
I immediately checked the disk space and found zero space available. Whoaa!!!
What actually happened is while performing the ingestion, the server created a duplicate copy of the data on the same instance, and until the entire ingestion completes, this data used to remain there. Around 250 GB of disk space was used by ingestion by midnight and the disk was full. I immediately attached a bigger volume to the instance and restarted the ingestion. Thankfully it was complete in a few hours and that saved me from a big trouble!!!