Virtualization with Vagrant

 some rights reserved by MedithIT

While going ahead with Chef configuration management tutorials i came across "Vagrant". I heard this term many a times. For quite sometime i thought it was just another VirtualBox like application. until i finally started using it.

What is Vagrant?

Open Source application for creating and configuring virtual dev environment. Vagrant manages VMs hosted in VirtualBox. Basically its a commandline utility that allows you to communicate with VirtualBox (or any virtualization software) with an easy set of commands. Many describe it as a wrapper around the virtualization software too.

How is it used?

Fist of all download Vagrant for your OS from this link. Install it. As mentioned its a commandline utility tool, you need commandline to access this.

1. You will need to initialize vagrant box using the init command. This will initialize a vagrant environment in the present directory you are in. The second argument will set the name for your box and the third will set the URL to access in the Vagrant file.
   
• Syntax: vagrant init [box-name] [url-to-access]
• Example: vagrant init precise32 http://files.vagrantup.com/precise32.box
2. Next you need to create and configure the vagrant box as per your vagrant file. Use the up command for this. This command will be used frequently as this is how you start you machine as well.
   
• vagrant up
3. Now that the machine has started you still have not logged in to the machine. You can use ssh to login. You don't need to use the traditional long ssh command to login to your box a simple vagrant ssh suffices.
   
• vagrant ssh
• Note: Please check the documentation on the vagrant website as there is a list of optional parameters that you may need in case you run into any errors. Fatal can be expected.
4. To check valid configurations to ssh into a running vagrant box use vagrant ssh-config
• vagrant ssh-config
5. As you now know how to setup and start using your vagrant box, we also need to know how to shut it down. This is simple too by just a "vagrant halt" similar to the halt command in linux machine.
• vagrant halt
6. These were the few basics of vagrant. You can do more with vagrant as well. The vagrant box command gives you other alternatives that you could try out.
• vagrant box add [box-name] [url-path]
• This adds a box with the specified name using the local file path or url specified to access it.
• vagrant box list
• Lists all the boxes installed and available
• vagrant box remove [box-name] [provider]
• Removes the box with the specified box name for the specific provider. Providers are VirtualBox or VMWare or any other utility.
• vagrant box repackage [box-name] [provider]
• Repackages the given box and puts it to present directory for redistribution purpose. When a box is added, vagrant unpacks and stores it internally and the original box is not preserved.
7. Restarting a vagrant machine can be done using vagrant reload. Its equivalent to a vagrant halt and vagrant up.
• vagrant reload
8. To check the current state of the machine, i.e to verify if the machine is running, stopped or not created etc a status command helps
• vagrant status
9. To save the status of the machine and suspend it so that you can resume it at a later instance and not completely shut it down, you can use suspend command.
• vagrant suspend
10. To resume a suspended machine use vagrant resume
• vagrant resume
11. Finally, to stop and delete/destroy and existing machine use the destroy command. All the resources allocated are destroyed as if the machine was never there. This command asks for confirmation before destroying.
• vagrant destroy

Checking Open Ports on a Remote Computer using PortQry

Some rights reserved by Ryan Franklin

 Today for one of the projects the SFTP connection kept failing for some reason. The user-id password used for connecting to the host was correct the hostname was as well correct. There was no way to find out what went wrong. Thankfully command-line gives a good log to verify what goes wrong.

I tried connecting to the SFTP host with various tools like FileZilla, WinSCP but could not get good enough logs. Finally i tried connecting the server using ssh on command-line using my Ubuntu machine. The connection used to time out. That is what i see in the logs as well. I assumed that probably the SFTP port number 22 was closed for the host.

I googled for if i could find a tool to check if a particular port on a machine is accessible or not. I finally found something called as PortQry taht could be used on Windows machine using Commandline.

Its a very small 140 KB command-line based software tool that you can use to check if a port on some machine is accessible or not.

After using this tool i got to know that the machine had a Firewall  kind of protection which wasn't allowing me to access the SFTP port on it.

Here's how you PortQry on Windows:

• Download the software using the link : http://www.microsoft.com/en-in/download/details.aspx?id=17148

• Double click an unzip the files to any location say C:/

• Hit Windows+R in the run box enter "cmd"

• Go to the directory where the PortQry was extracted.

• Execute the program PortQry by entering PortQry<enter>

• This will display a list of help information and the correct usage of the command

The following is the syntax to check the port status :
portqry -n myhostname.net -e 80

PortQry can inform the status of a port as "Listening", "Not Listening", or "Filtered"
Listening : There is some service active on that port
Not Listening : Port is closed
Filtered : No response, Presumably its behind some kind of firewall.

Syntax
portqry -n name_to_query [-p protocol] [-e || -r || -o endpoint(s)]

Common command line switches:
-n : IP address or name of system to query
-p : TCP or UDP or BOTH (default is TCP)
-e : single port to query (valid range: 1-65535)
-r : range of ports to query (start:end)

For single port use
portqry -n 127.0.0.1 -e 80

For a Range of ports, use the -r switch:
portqry -n 127.0.0.1 -r 80:85

Note:
- PortQry also displays extended information for known services, such as SMTP, POP3, IMAP4, FTP, and is capable of performing LDAP queries.
- A GUI based alternative is also available now called PortQryUI

Sample Output:
C:\PortQryV2>portqry -n 127.0.0.1 -e 40
Querying target system called:
 127.0.0.1
Attempting to resolve name to IP address...
Name resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): FILTERED

C:\PortQryV2>portqry -n 127.0.0.1 -e 80
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 80 (http service): LISTENING

C:\PortQryV2>portqry -n 127.0.0.1 -e 22
Querying target system called:
 127.0.0.1
Attempting to resolve IP address to a name...
IP address resolved to xx.xx.xx.xx
querying...
TCP port 22 (ssh service): NOT LISTENING

Data Recovery with TestDisk

some rights reserved by epSos .de

 You buy a portable hard disk drive to keep a back up of all your important data, Or you are carrying an extremely important doc in a USB drive, or you have some data on your internal hard disk drive. You try to get the data and ... boom!!! No Data!!! You have absolutely no clue how to get back the important data you had in the drives. Finally you just consider it lost and curse your fate and carry on.

This happened to me too. 1 TeraBytes of important data in my external Hard disk drive got corrupt in a second and was lost. I tried all OS's i could right from MAC to Windows to Linux but no luck. I was about to give up when i came across Data Recovery softwares on Internet.

Data Recovery softwares are those softwares that help you read corrupt/lost/deleted data from your drives. There were so many softwares out there. I started trying out my luck on all the softwares i could with a windows machine(i should not have done that, but i have no option!!). Unfortunately most of them just gives a list of deleted items on the disk that i never needed. One or two of them did get me the data however since they were trial versions, i could only view the data and not copy them. Anyways it just gave me the hope that the data was still there in the disk.

Finally i searched for Open Source alternatives and there i got my answer. TestDisk!!!

What is TestDisk?
As per their website wiki: TestDisk is a free data recovery software designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software, certain types of viruses, or human error. It can also be used to repair some filesystem errors.

Installation:
You don't need to install it. Its a very small 1.5 MB software, where you just need to run the executable, it was really easy to use command line based software with most of the information self explanatory.

Steps to Use:

1. Logs Creation : Like every other Linux based software, this one as well makes a log of every session carried out. when you execute the TestDisk.exe You are 1st asked for if, you would like to CREATE a new log, APPEND  the existing log, or want NO LOG.
2. Media/Disk Detection :Next you need to select the disk you want to recover the lost data from. It shows a list of all volumes connected to you computer and you can select using keyboard arrow keys.
3. Disk/Partition Table Type Selection : The next screen prompts you to select the partition table type. In most of the cases it detects all by itself the type. Otherwise it keeps it to none for you to select. Mine was a windows machine that i was running this program on and i used this with quiet a many volumes and every time it detected Intel. I am assuming windows users could probably select Intel in case they are unsure of this option. But selection "None"is not recommended as its very rare that a drive is non partitioned.
4. Next Screen Gives the following Options :

Analyse Analyse current partition structure and search for lost partitions to restore them
Advanced Filesystem Utils

FAT: Boot and FAT repair

NTFS: Boot and MFT repair

ext2/ext3: Find Backup SuperBlock
FAT file undelete
NTFS file undelete
ext2 file undelete
Image Creation

Geometry Change disk geometry

Options Modify options

MBR Code Write TestDisk MBR code to first sector

Delete Delete all data in the partition table

Analyzing of disk is done to look for lost partitions. This takes long time based on the size of the disk. Most of the times for windows if you get an error "The type of the file system is RAW." or "The disk in drive D is not formatted. Do you want to format it now?" then that means the Boot sector is damaged. You can click on Advanced file system Utils and and use the FAT: Boot and FAT repair/NTFS: Boot and MFT repair based on the type of partitioin you have. This was the problem with my disk too and got corrected in no time.



To be contd ...

Monitoring in Linux/Unix Environment using TOP

Some rights reserved by Steve Jurvetson

Top

Top is the Linux performance monitoring program. For windows user, its analogous to the Task Manager. This command displays the active processes at real-time and updates the list regularly. Other system details like CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands and much more.
The 1st line of the command mentions the following:

• current time in hh:mm:ss format seconds keep updating
• uptime of the machine, how long has the machine been running
• no. of users logged in with running sessions
• average load on the system, 3 values mentioned are load in last one minute, 5 minutes and 15 minutes

The 2nd line mentions the following :

• Total number of processes running
• Present number of running processes
• Total sleeping processes
• Total stopped processes
• Total Zombie processes(waiting for parent process to stop)

The 3rd row mentions the following :

• % of CPU for user processes
• % of CPU for system/kernel processes
• % of CPU for prioritized or priority upgraded processes nice
• % of CPU not used
• % of CPU awaiting i/o operation
• % of CPU serving h/w interrupts
• % of CPU for s/w interrupts
• % of CPU stolen from virtual memory(steal time) this will be zero if no virtual machine running.

The 4th and 5th row mentions the following:

• The use of Physical memory
• The use of swap memory
• Both free buffer and cached

The details of the processes are given  with the following details:

• PID: Process ID
• USER: The owner user of the process
• PR: Priority of the process
• NI: Nice value of the process
• VIRT: Amount of Virtual Memory used by the process
• RES: Amount of physical memory used by the process
• SHR: Shared memory of the process
• S: Status of the process Sleep Running Zombie
• %CPU: % of CPU used
• %MEM: % of RAM used
• COMMAND: name of the process

The default sorting of the list displayed is based on CPU usage. you can change the sorting of the list as per your convenience.
Changing the sorting:
Press Shift+o. A list will be displayed giving all possible options using which you may sort the list, a letter corresponding to the sort criteria. Select that particular letter and hit 'return/enter' and see the new sorted list.
Display Processes for a specific User:
top -u username
This command will show the details of all the processes under the specific username mentioned in the command.
Highlight any Running Process :
Press Z after running top to highlight the running process to identify them easily.
Show absolute Path of the process:
To see the path from where the processes are being invoked press 'c' after running top
Change screen refresh interval:
To change the screen refresh interval of the processes running press 'd' and enter any number in seconds to set the time interval for refresh.
Kill Running Process:
to kill any of the running processes press 'k' and enter the process id of the process to be killed. After this you will be required to enter the signal (15) to kill the process.
To sort by CPU Utilization: Shift+p
Save Results of Top Command: press Ctrl+w
for help: press h
Exit top after specific Repetition: top -n <number>
Manual Page for top : man top

Looking for Free Windows OS? ReactOS is here.

Some rights reserved by Plum Lady

I keep looking for free stuff available online. In the early days when my searches were only limited to torrents, keygens, patches etc, didn't get much of a exposure to free legal stuff. FOSS came for rescue and then many different things lined up. Thanks to Ubuntu!!! Its been more than 3 years that i quit windows for daily use... But what haunts me at times is the when there is some software that i need to use desperately and its setup is only available for Windows. Wine helps many a times but not always. At times i felt, if Windows could be free too ... :(

Then on one fine day i wanted to make a doc. I use Libre generally. But this one needed to be highly compatible since i was suppose to send it to some windows user. I got Office 2007 installed in Wine. I used it for quite sometime and then it started throwing some weird error and got closed time and again. At this instance the first thing is googled is Free and open source windows like OS. While there were many linux pages pulled up that i already knew about, this one caught my attention. "Free and Open Source Windows like OS - ReactOS ".

So, what is ReactOS? Its an effort to provide Free and Open Source alternative to Windows. The term "React" also comes from a reaction of dissatisfaction of Microsoft's monopolistic position(source Wikipedia).

ReactOS is an Open Source implementation of NT family of windows. A part of the implementation has been assisted from Wine. The OS is still in its alpha stage and has been selected as a best project from Forge Magazine.

Many alternatives for the set up are available i.e VM images, iso images etc for everyone to have maximum access. The site has all kind of download options available. If in case you happen to find some issue in the installation, you can always report it to the team in their JIRA. The installations are both automated and manual for testing purpose.

You can even contribute to the community as a Dev, QA, Release or Tech Writer. Wishing to get this one released soon. :) 

Free and Open Source Alternatives for most of the paid softwares on Windows Machine

 Some rights reserved by opensource.com

I have recently been using windows more than usual for official use. Being an Ubuntu user i am not much used to either use a trial version of any paid software or pay for any software. I googled a lot for my daily use softwares to make my life comfortable and found that almost for every paid software there was an alternative in the Free and Open Source World. Here is what i found:

• Operating System : Widows (Paid) -> ReactOS (Alternative) I have a different blog written for this.
• Office Utility : Microsoft Office (Paid) -> Libre Office, Open Office(Alternative). For years i only knew about Open Office as an open source alternative. However after going through the threads on Ubuntu forums as to why was Libre Office considered over Open Office as a default Office utility for the OS i came to know that Libre Office development was far too faster than that of Open Office. I even saw the results myself to verify this fact. I would recommend Libre Office for the same reason.
• Video/Music Player : Windows Media Player (Most Probably Paid - I have hardly heard of Microsoft ever releasing anything free of cost :D) -> VLC media Player, Media Player Classic(Alternative).
• Remote File Sharing : WinSCP, FileZilla, Putty
• IDE for C# Programming : Microsoft Visual Studio (Paid) -> Sharp Develop (Alternative) (Although i was looking for Command line utility as i am only used to VIM as such, but found this one and made a note of it.
• File Compression : WinRAR, WinZIP(Paid) -> 7zip(Alternative)
• PDF Reader : Adobe Acrobat Reader(Freeware+Commercial) -> Foxit(Alternative)
• Email Client : Outlook(Paid) -> Thunderbird(Alternative)
• VM : VMWare(Paid) -> VirtualBox(Alternative)
• Reference Management : Zotero (Zotero is another good add-on cum software for Reference Management. You can save your websites and URL/PDF preferences online so that whenever you clean your cache on browsers and you desperately needed some link and you cleared your cache and reinstalled new browser. You can check your Zotero and get URL/PDF back as all these preferences are stored online in your separate account.)
• Cloud Storage : GoogleDrive/Dropbox

Second Place Story : Server ghouls haunt bulk ingestion

Some rights reserved by Julie Rybarczyk

I recently shared one of my stories while monitoring an Ingestion Server with AppFirst for one of the Halloween contest they conducted. Thankfully, won 2nd prize in the same. Sharing the same story here along with the link to visit.

Second Place Story

Server ghouls haunt bulk ingestion
Company: Roshvert
I was suppose to monitor an Ingestion Server that was performing a bulk ingestion through an EC2 instance with around 200 GB of data to be ingested to another server.
Since it was a huge amount of data and the ingestion would take another day to complete, I kept the ingestion going and the logs were performing well. I decided then that I’d log in early tomorrow morning to check the ingestion status. During this time, the log files were supposed to be created automatically through the ingestion and the name of the log file for any particular day should be log_dd-mm-yyyy.txt with date of that day mentioned. It was a staging server and the code was supposed to be supplied for UAT in a day or two.
I logged in early the next morning to check the ingestion status. I was totally puzzled as I couldn’t make out what was happening:

• The log file for the previous day log_27-08-2013.txt was showing everything went well until 11pm midnight and no logs thereafter.
• The log file for today log_28-08-2013.txt got created with no data in it.
• The ingestion process was running with no errors.
• The server logs showed no errors.
• The system never went down.
• Nearly 150 GB of data was still to be ingested and was not progressing at all.
• None of the logs showed any updates as to why the ingestion was not progressing.

Since the delivery was urgent, I stopped the ingestion on the instance and restarted it. To my horror, the ingestion was not progressing at all. I tried running ingestion on other instances, and it worked fine.
Then something hit me, and I went back to check the logs of ingestion. The ingestion logs still showed nothing with 0 kb space used by the logs. Wait!!! Space? 0kb? 150 GB data still remaining?
I immediately checked the disk space and found zero space available. Whoaa!!!
What actually happened is while performing the ingestion, the server created a duplicate copy of the data on the same instance, and until the entire ingestion completes, this data used to remain there. Around 250 GB of disk space was used by ingestion by midnight and the disk was full. I immediately attached a bigger volume to the instance and restarted the ingestion. Thankfully it was complete in a few hours and that saved me from a big trouble!!!

Custom ROM on LG Optimus 2x

Some rights reserved by Johan Larsson

Ever since I started using LG Optimus 2X P990, i noticed some or the other issue with its Battery. The OS was as well not upgradable as it supported updates only to Android 2.3. I tried almost all battery saving apps from the App store. Well, battery was not the only problem, the phone used to freeze after sometime after using it. After browsing a lot through the XDA Developers forum i found that the default firmware provided by LG wasn't good enough for the hardware. Only a new custom ROM would help. There were numerous mentions on CynogenMod being the best of all. However getting a custom ROM was not a one click solution.
Here are some steps how you could get your own custom ROM on LG Optimus 2X although these are common for most of the phones.

1. Install LG Optimus P990 USB driver on your windows PC
2. Experts suggest that your phone should have a good battery life before going ahead with this as it take time for the installation to finish.
3. You will first need to root your phone before going ahead with Custom ROM. I used OneClick root software. It took a while, but did the job.
4. Install Titanium Backup to backup all your apps and other data
5. Enable USB debugging mode from settings
6. To back up your contacts you can use www.phonecopy.com
7. Download any version of you choice form Cyanogens' website. I chose a stable one, although you can go for any of the nightly release too.
8. Install Clock work mode recovery software on your phone.
9. Copy the Cynogenmod zip to your SD CARD
10. Power off the phone
11. Restart the phone by pressing the start button and holding the Volume button down.
12. You will be in Clock Work Recovery mode screen
13. Click on Wipe data
14. Click on Wipe Cache
15. Click on Install Zip from sdcard
16. Select the zip and wait for the installation to finish
17. After the installation restart you phone to find the new ROM installed.
18. You may find that Google Apps are missing here. You can look for CyanogenMod gapps and install them using Steps 11, 12, 15, 16 above.
19. Restart the phone and you can see google apps as well.

There are many other custom ROMs available. Some famous ones are CynogenMod, Paranoid, AOKP, Liquid Smooth etc.

New Relic

I received a mail some months ago to register to New Relic, they had some cool offers then like free t-shirts etc. I didn't check it then as i was really busy with other stuff. However since my new DevOps learning experience and course material had mentions of monitoring tools, after googling a bit i came across New Relic too.

So New Relic is a software analytics company. It delivers a Saas)Software as a service) model, that monitors Web and Mobile application real-time. Monitoring using New Relic seemed easy as i tried it on one or two systems and the logs generated looked really accurate and helpful too.

While there are may other useful ways as well for any organization for monitoring using New Relic, i only tried it for a systems performance. You can monitor your app, by simply supplying a a config file to the project directory and compiling and deploying the project. New Relic will provide real-time updates of the resources used by the app. You can add many plugins to your New Relic dashboard and enhance your monitoring experience.

Installation

• Installation on windows was simple as you just have to download New Relic executable from their website. free trial is available. 
• After installing New Relic Server monitor. Log in to your New Relic account (Create one if you did not create.) You can see you dashboard.
• Click on Dashboard and click on Add more. It will pull up the list of compatible Operating systems along with instructions to go ahead with monitoring.
• It is important to note that their is a unique key supplied on this page for the monitoring to show any progress or logs in the system.

Configuring

• The configuration only involves copying the key to the New Relic Server monitor on your windows machine
• Go to Start > Programs > New Relic Server Monitor > Configure copy the key to the space provided.
• If you are using any proxy then you can add more details in the proxy tab

Start Service

• Once the required configuration has been done, just click on start service and start monitoring in you Dashboard in the web interface.

You can see Pie Charts, Graphs of total memory used by process, total memory free, disk space, network usage etc. Screenshots are attached. Happy Monitoring.