Pages

Thursday, 2 January 2014

Monitoring in Linux/Unix Environment using TOP

Some rights reserved by Steve Jurvetson

Top

Top is the Linux performance monitoring program. For windows user, its analogous to the Task Manager. This command displays the active processes at real-time and updates the list regularly. Other system details like CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands and much more.
The 1st line of the command mentions the following:
  • current time in hh:mm:ss format seconds keep updating
  • uptime of the machine, how long has the machine been running
  • no. of users logged in with running sessions
  • average load on the system, 3 values mentioned are load in last one minute, 5 minutes and 15 minutes
The 2nd line mentions the following :
  • Total number of processes running
  • Present number of running processes
  • Total sleeping processes
  • Total stopped processes
  • Total Zombie processes(waiting for parent process to stop)
The 3rd row mentions the following :
  • % of CPU for user processes
  • % of CPU for system/kernel processes
  • % of CPU for prioritized or priority upgraded processes nice
  • % of CPU not used
  • % of CPU awaiting i/o operation
  • % of CPU serving h/w interrupts
  • % of CPU for s/w interrupts
  • % of CPU stolen from virtual memory(steal time) this will be zero if no virtual machine running.
The 4th and 5th row mentions the following:
  • The use of Physical memory
  • The use of swap memory
  • Both free buffer and cached
The details of the processes are given  with the following details:
  • PID: Process ID
  • USER: The owner user of the process
  • PR: Priority of the process
  • NI: Nice value of the process
  • VIRT: Amount of Virtual Memory used by the process
  • RES: Amount of physical memory used by the process
  • SHR: Shared memory of the process
  • S: Status of the process Sleep Running Zombie
  • %CPU: % of CPU used
  • %MEM: % of RAM used
  • COMMAND: name of the process

The default sorting of the list displayed is based on CPU usage. you can change the sorting of the list as per your convenience.
Changing the sorting:
Press Shift+o. A list will be displayed giving all possible options using which you may sort the list, a letter corresponding to the sort criteria. Select that particular letter and hit 'return/enter' and see the new sorted list.
Display Processes for a specific User:
top -u username
This command will show the details of all the processes under the specific username mentioned in the command.
Highlight any Running Process :
Press Z after running top to highlight the running process to identify them easily.
Show absolute Path of the process:
To see the path from where the processes are being invoked press 'c' after running top
Change screen refresh interval:
To change the screen refresh interval of the processes running press 'd' and enter any number in seconds to set the time interval for refresh.
Kill Running Process:
to kill any of the running processes press 'k' and enter the process id of the process to be killed. After this you will be required to enter the signal (15) to kill the process.
To sort by CPU Utilization: Shift+p
Save Results of Top Command: press Ctrl+w
for help: press h
Exit top after specific Repetition: top -n <number>
Manual Page for top : man top


Friday, 22 November 2013

Looking for Free Windows OS? ReactOS is here.

Some rights reserved by Plum Lady


I keep looking for free stuff available online. In the early days when my searches were only limited to torrents, keygens, patches etc, didn't get much of a exposure to free legal stuff. FOSS came for rescue and then many different things lined up. Thanks to Ubuntu!!! Its been more than 3 years that i quit windows for daily use... But what haunts me at times is the when there is some software that i need to use desperately and its setup is only available for Windows. Wine helps many a times but not always. At times i felt, if Windows could be free too ... :(

Then on one fine day i wanted to make a doc. I use Libre generally. But this one needed to be highly compatible since i was suppose to send it to some windows user. I got Office 2007 installed in Wine. I used it for quite sometime and then it started throwing some weird error and got closed time and again. At this instance the first thing is googled is Free and open source windows like OS. While there were many linux pages pulled up that i already knew about, this one caught my attention. "Free and Open Source Windows like OS - ReactOS ".

So, what is ReactOS? Its an effort to provide Free and Open Source alternative to Windows. The term "React" also comes from a reaction of dissatisfaction of Microsoft's monopolistic position(source Wikipedia).

ReactOS is an Open Source implementation of NT family of windows. A part of the implementation has been assisted from Wine. The OS is still in its alpha stage and has been selected as a best project from Forge Magazine.

Many alternatives for the set up are available i.e VM images, iso images etc for everyone to have maximum access. The site has all kind of download options available. If in case you happen to find some issue in the installation, you can always report it to the team in their JIRA. The installations are both automated and manual for testing purpose.

You can even contribute to the community as a Dev, QA, Release or Tech Writer. Wishing to get this one released soon. :) 

Thursday, 7 November 2013

Free and Open Source Alternatives for most of the paid softwares on Windows Machine

 Some rights reserved by opensource.com

I have recently been using windows more than usual for official use. Being an Ubuntu user i am not much used to either use a trial version of any paid software or pay for any software. I googled a lot for my daily use softwares to make my life comfortable and found that almost for every paid software there was an alternative in the Free and Open Source World. Here is what i found:

  • Operating System : Widows (Paid) -> ReactOS (Alternative) I have a different blog written for this.
  • Office Utility : Microsoft Office (Paid) -> Libre Office, Open Office(Alternative). For years i only knew about Open Office as an open source alternative. However after going through the threads on Ubuntu forums as to why was Libre Office considered over Open Office as a default Office utility for the OS i came to know that Libre Office development was far too faster than that of Open Office. I even saw the results myself to verify this fact. I would recommend Libre Office for the same reason.
  • Video/Music Player : Windows Media Player (Most Probably Paid - I have hardly heard of Microsoft ever releasing anything free of cost :D) -> VLC media Player, Media Player Classic(Alternative).
  • Remote File Sharing : WinSCP, FileZilla, Putty
  • IDE for C# Programming : Microsoft Visual Studio (Paid) -> Sharp Develop (Alternative) (Although i was looking for Command line utility as i am only used to VIM as such, but found this one and made a note of it.
  • File Compression : WinRAR, WinZIP(Paid) -> 7zip(Alternative)
  • PDF Reader : Adobe Acrobat Reader(Freeware+Commercial) -> Foxit(Alternative)
  • Email Client : Outlook(Paid) -> Thunderbird(Alternative)
  • VM : VMWare(Paid) -> VirtualBox(Alternative)
  • Reference Management : Zotero (Zotero is another good add-on cum software for Reference Management. You can save your websites and URL/PDF preferences online so that whenever you clean your cache on browsers and you desperately needed some link and you cleared your cache and reinstalled new browser. You can check your Zotero and get URL/PDF back as all these preferences are stored online in your separate account.)
  • Cloud Storage : GoogleDrive/Dropbox

Sunday, 3 November 2013

Second Place Story : Server ghouls haunt bulk ingestion

Some rights reserved by Julie Rybarczyk


I recently shared one of my stories while monitoring an Ingestion Server with AppFirst for one of the Halloween contest they conducted. Thankfully, won 2nd prize in the same. Sharing the same story here along with the link to visit.

Second Place Story
Server ghouls haunt bulk ingestion
Company: Roshvert
I was suppose to monitor an Ingestion Server that was performing a bulk ingestion through an EC2 instance with around 200 GB of data to be ingested to another server.
Since it was a huge amount of data and the ingestion would take another day to complete, I kept the ingestion going and the logs were performing well. I decided then that I’d log in early tomorrow morning to check the ingestion status. During this time, the log files were supposed to be created automatically through the ingestion and the name of the log file for any particular day should be log_dd-mm-yyyy.txt with date of that day mentioned. It was a staging server and the code was supposed to be supplied for UAT in a day or two.
I logged in early the next morning to check the ingestion status. I was totally puzzled as I couldn’t make out what was happening:
  • The log file for the previous day log_27-08-2013.txt was showing everything went well until 11pm midnight and no logs thereafter.
  • The log file for today log_28-08-2013.txt got created with no data in it.
  • The ingestion process was running with no errors.
  • The server logs showed no errors.
  • The system never went down.
  • Nearly 150 GB of data was still to be ingested and was not progressing at all.
  • None of the logs showed any updates as to why the ingestion was not progressing.
Since the delivery was urgent, I stopped the ingestion on the instance and restarted it. To my horror, the ingestion was not progressing at all. I tried running ingestion on other instances, and it worked fine.
Then something hit me, and I went back to check the logs of ingestion. The ingestion logs still showed nothing with 0 kb space used by the logs. Wait!!! Space? 0kb? 150 GB data still remaining?
I immediately checked the disk space and found zero space available. Whoaa!!!
What actually happened is while performing the ingestion, the server created a duplicate copy of the data on the same instance, and until the entire ingestion completes, this data used to remain there. Around 250 GB of disk space was used by ingestion by midnight and the disk was full. I immediately attached a bigger volume to the instance and restarted the ingestion. Thankfully it was complete in a few hours and that saved me from a big trouble!!!

Monday, 21 October 2013

Custom ROM on LG Optimus 2x

Some rights reserved by Johan Larsson


Ever since I started using LG Optimus 2X P990, i noticed some or the other issue with its Battery. The OS was as well not upgradable as it supported updates only to Android 2.3. I tried almost all battery saving apps from the App store. Well, battery was not the only problem, the phone used to freeze after sometime after using it. After browsing a lot through the XDA Developers forum i found that the default firmware provided by LG wasn't good enough for the hardware. Only a new custom ROM would help. There were numerous mentions on CynogenMod being the best of all. However getting a custom ROM was not a one click solution.
Here are some steps how you could get your own custom ROM on LG Optimus 2X although these are common for most of the phones.
  1. Install LG Optimus P990 USB driver on your windows PC
  2. Experts suggest that your phone should have a good battery life before going ahead with this as it take time for the installation to finish.
  3. You will first need to root your phone before going ahead with Custom ROM. I used OneClick root software. It took a while, but did the job.
  4. Install Titanium Backup to backup all your apps and other data
  5. Enable USB debugging mode from settings
  6. To back up your contacts you can use www.phonecopy.com
  7. Download any version of you choice form Cyanogens' website. I chose a stable one, although you can go for any of the nightly release too.
  8. Install Clock work mode recovery software on your phone.
  9. Copy the Cynogenmod zip to your SD CARD
  10. Power off the phone
  11. Restart the phone by pressing the start button and holding the Volume button down.
  12. You will be in Clock Work Recovery mode screen
  13. Click on Wipe data
  14. Click on Wipe Cache
  15. Click on Install Zip from sdcard
  16. Select the zip and wait for the installation to finish
  17. After the installation restart you phone to find the new ROM installed.
  18. You may find that Google Apps are missing here. You can look for CyanogenMod gapps and install them using Steps 11, 12, 15, 16 above.
  19. Restart the phone and you can see google apps as well.
There are many other custom ROMs available. Some famous ones are CynogenMod, Paranoid, AOKP, Liquid Smooth etc.

New Relic

I received a mail some months ago to register to New Relic, they had some cool offers then like free t-shirts etc. I didn't check it then as i was really busy with other stuff. However since my new DevOps learning experience and course material had mentions of monitoring tools, after googling a bit i came across New Relic too.

So New Relic is a software analytics company. It delivers a Saas)Software as a service) model, that monitors Web and Mobile application real-time. Monitoring using New Relic seemed easy as i tried it on one or two systems and the logs generated looked really accurate and helpful too.

While there are may other useful ways as well for any organization for monitoring using New Relic, i only tried it for a systems performance. You can monitor your app, by simply supplying a a config file to the project directory and compiling and deploying the project. New Relic will provide real-time updates of the resources used by the app. You can add many plugins to your New Relic dashboard and enhance your monitoring experience.

Installation

  • Installation on windows was simple as you just have to download New Relic executable from their website. free trial is available. 
  • After installing New Relic Server monitor. Log in to your New Relic account (Create one if you did not create.) You can see you dashboard.
  • Click on Dashboard and click on Add more. It will pull up the list of compatible Operating systems along with instructions to go ahead with monitoring.
  • It is important to note that their is a unique key supplied on this page for the monitoring to show any progress or logs in the system.
Configuring
  • The configuration only involves copying the key to the New Relic Server monitor on your windows machine
  • Go to Start > Programs > New Relic Server Monitor > Configure copy the key to the space provided.
  • If you are using any proxy then you can add more details in the proxy tab
Start Service
  • Once the required configuration has been done, just click on start service and start monitoring in you Dashboard in the web interface.
You can see Pie Charts, Graphs of total memory used by process, total memory free, disk space, network usage etc. Screenshots are attached. Happy Monitoring.





Wednesday, 7 August 2013

Shell Scripting Tuts

Some rights reserved by Linux Screenshots


VIM shortcuts
:tabfind open new tab
:tabn and tabp for next and previous tabs
:tabedit to open a new tab with a new filename
esc v for marking, d to cut, y to copy, p to paste
:wrap to enable wordwrap

Shell Scripting
1. Use any scripting editor, i will use VIM

2. After the shell script is complete set execute permission to the script for the script to execute. The permissions can be changed by the command chmod.
    Syntax is
    chmod permission your-script name
    e.g
    $ chmod +x script
    $ chmod 755 script
    755 sets permission for user (read 4 write 2 execute 1) group and others (read 4 execute 1)

3. Executing your script : There are three ways of executing a script:
    Syntax:
        bash script-name
        sh script-name
        ./script-name
    E.g
        $ bash script
        $ sh script
        $ ./script
    Note: ./means current directory. But only .(dot) means execute a given command in the current shell without starting a new copy of the shell. The syntax for .(dot) command is . command-name e.g $ . foo
# This is my 1st shell program
clear
echo "Knowledge is Power"
Anything after "#" is a comment
clear - this will clear the screen
echo - prints the message

4. The computer memory is divided into small locations and each location is called as a memory address. These memory addresses hold data in them. A programmer can give a name to these memory locations. This is called as variable. These variable can take values.
Linux has two types of variables:
i. System Variables (Maintained by Linux system, seen in caps case)
ii. User defined varibales (User defined, small case)
Variable declaration:
n=10
Dont put spaces on either side of the equals sign that should give an error. Variables are case sensitive. While printing the values of the variables use the '$' symbol, e.g echo $n

5. Shell Arithmetic
The term expr is used to perform arithmetic operations in shell.
Syntax : expr op1 operator op2
e.g
$ expr 1 + 3
$ expr 2 - 1
$ expr 10 / 2
$ expr 20 % 3
$ expr 10 \* 3
$ echo `expr 6 + 3`

Note : Backticks ` is called backtick and this is mean to execute any command in the script e.g echo "today is `date`"

6. Exit status
By default in Linux if any command is executed, it would return two types of values, 1. 0 (ZERO) 2. Non ZERO
zero is success while non zero is not
This is called as the exit status of the command. By default to find the exist status of any command. Just do a $? will give you the exit status
e.g In the terminal try removing a file that does not exist rm newfile, Now do an echo $? it should give you a non zero value. Now do an ls and again echo $? it should give you a 0

7. User input
to accept an input from the user, use the command 'read'
read var1, var2, var3

8. WildCards
There are three wildcards in shell
?, *, [...]
? - will Match a single character
* - will Match a group of characters
[...]* - will match a group of characters in the mentioned set
e.g
[abc]* will look for all the matches starting with a,b and c
[!abcd] or [!a-d] everything except a,b,c,d

Note: If you want to execute two or more commands in one line, use a semicolon
e.g ls; who; date; cal

9. Command line arguments
Some commands need arguments to be passed. e.g if you do an rm, you would need to specify the file you want to remove. The specifying of the file is the command line argument to that command. For every shell script there is a command line argument that cane be specified by $<number>. $0 is the name of the script. $1 is the 1st argument. $2 is the 2nd argument and so on...

10. Redirection of Standard input output
Three symbols are used for redirection'
> , < , >>
> : command > file : this will write to the file provided
< : command < file : take input from the file provided
>> : command >> file : will append input to the file if file already exist

11. Pipes
pipe is a utility to connect the output of one program to the input of another program without any temporary file. Pipe is a temporary storage place, where the output of one command is stored, and then passed to the input of another command. Pipes are used to run more than two commands for the same command line.
$ ls | more
$ who | sort
$ who | sort > user_list
$ who | wc -l
$ ls -l | wc  -l
$ who | grep raju

12. Filters
If a Linux command accepts its input from the standard input and produces its output on standard output is know as a filter. A filter performs some kind of process on the input and gives output. For e.g.. Suppose you have file called 'hotel.txt' with 100 lines data, And from 'hotel.txt' you would like to print contains from line number 20 to line number 30 and store this result to file called 'hlist' then give command:
$ tail +20 < hotel.txt | head -n30 >hlist

13. If Condition
if is used for decision making. If a particular condition is true command1 is executed.
Syntax:
if condition
then
  command1
fi

condition is comparison between values, comparison can be done using the expr operator or the by cheking the exit status.
#!/bin/bash
if cat $1
then
  echo "$1 file found on this location"
fi

if [ $2 -gt 0 ]
then
  echo "$2 positive number"
fi

if [ -f $3 ]
then
  echo "$3 file exists"
fi

if...else...if
#!/bin/bash
if [ -f $1 ]
then
  echo "$1 exist"
else
  if [ -f $2 ]
  then
    echo "$2 exist"
  else
    if [ -f $3 ]
    then
      echo "$3 exist"
    fi 
  fi 
fi
execution
$ sh -x shell3.sh asdff adfdag shell1.sh
+ [ -f asdff ]
+ [ -f adfdag ]
+ [ -f shell1.sh ]
+ echo shell1.sh exist
shell1.sh exist
$ sh -x shell3.sh asdff shell2.sh shell1.sh
+ [ -f asdff ]
+ [ -f shell2.sh ]
+ echo shell2.sh exist
shell2.sh exist
$ sh -x shell3.sh shell3.sh shell2.sh shell1.sh
+ [ -f shell3.sh ]
+ echo shell3.sh exist
shell3.sh exist

#!/bin/bash
echo "1. Linux \n 2. Windows \n 3. None"
read ch
if [ $ch -eq 1 ]
then
  echo "You seem like a geek"
else
  echo "You are good"
  if [ $ch -eq 3 ]
  then
    echo " But, I dont know, what you do"
  fi 
fi

#!/bin/bash
echo "1. student\n2.work\n3.none"
read ch
if [ $ch -eq 1 ]
then
  echo "Happy Studying"
elif [ $ch -eq 2 ]
then
  echo "You should study too"
elif [ $ch -eq 3 ]
then
  echo "Study an get a job"
fi



14. Looping in Shell
Looping is execution of a particular expression time and again until a particular condition is satisfied.
bash supports for loops and while loops
looping needs 1. initializing of a variable, 2. test condition in beginning, 3. body of loop at the end modifies the initialized variable

15. For Loop

Syntax:

for { variable name } in { list }
do
execute one for each item in the list until the list is
not finished (And repeat all statement between do and done)
done

  e.g

for i in 1 2 3 4 5
do
echo "Welcome $i times"
done

#!/bin/bash
for i in `seq 1 10`
do
  echo "Welcome $i times"
done


16. While loops
 Syntax:

while [ condition ]
  do
     command1
     command2
     command3
  done

To be contd...

Friday, 19 July 2013

Build your own private cloud with Ubuntu - II

Some rights reserved by ErinOfEarth


Continuing from...

Build your own private cloud with Ubuntu - I

  Create node for your MAAS server. You must have atleast 2 nodes before going ahead with juju setup. Go to the webpage for maas server and Click on Add nodes.
We left all the other information fields there blank except for the MAC-ADDRESS. You can get the mac address of your computer by ipconfig on windows and ifconfig on Ubuntu - Ubuntu users can do: 
ifconfig | grep hwadrs

Then we went to
http://maas.ubuntu.com/docs/juju-quick-start.html
Thereafter we created ~/.juju/environments.yaml
vim ~/.juju/environments.yaml
Added the following details to this file

juju: environments
environments:
  maas:
    type: maas
    maas-server: 'http://localhost:5240'
    maas-oauth: '${maas-api-key}'
    admin-secret: 'nothing'

Got an error default-series: missing

Added this line to the above file
    default-series: 'precise'
Error Gone

juju status
New error
Juju INFO Connecting to environment... Unhandled error in Deferred: Unhandled Error

Investigated, result:
Juju packages needs to be installed from ppa:juju/pkg

Went to
https://juju.ubuntu.com/docs/
removed existing juju
sudo apt-get update
            sudo apt-get install juju-core (error no package)


sudo add-apt-repository ppa:juju/devel
            sudo apt-get update && sudo apt-get install juju-core

Configuring Juju
juju generate-config -w   
previously written environment.yml was moved and the changes were added to the new yml file.

error: environment has no access-key or secret-key

changed default: amazon to default: maas
juju bootsrap
error:no tools
sudo juju -v sync-tools

roshan@ubuntu-cloud:~$ sudo juju bootstrap
error: cannot start bootstrap instance: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT

CONFLICT from maas means that it doesn't have any nodes available for juju to acquire (juju bootstrap needs to boot a node).
It's not the best error unfortunately, juju needs fixing to show all the text. You need the server edition of 12.04

Nodes were disconnected for sometime...
Next article should have the addition of nodes...

Creating an exactly similar snapshot of your existing running EC2 ami instance

Some rights reserved by Martin Fisch


Everything was going pretty well on the existing EC2 instance, when the developers offshore came up with another version of the existing CMS, which needed to be tested along with the existing CMS. One way was to have another virtual host created for the new CMS but there were some existing resources that could not be shared between the two CMSs.

The only way out of this was to have another instance with the new CMS version. Installing the same applications in the new instance was surely time consuming, this would take another day for sure. So we came up with the idea of creating a snapshot of the existing instance.

Caution: If you only try to create a copy of the existing instance you will only get the Instance along with the OS not the applications along with it. To create the exact similar copy of the instance the steps are slightly different.

Steps to create an exact similar copy are as follows:

  1. Create an EBS volume using the web interface. You may use Amazon Documentation for this.
  2. Attach this volume to your existing instance. note down the volume and the device id. (Vol -123456, /dev/sdg) Documentation
  3. Next you would need is a file system to work on, on the attached volume. Log in to your instance, and use the commands below
    • mkfs.ext3 /dev/sdg
    • tune2fs -c 0 /dev/sdg
  4. Now mount this disk
    • mkdir /mnt/data
    • mount /dev/sdg /mnt/ebs
  5. Now you can copy all the data to your EBS volume but before you do that, shutdown the running services that may harm the data, I stopped MySQL Apache, ElasticSearch etc on my instance. To copy all data use rsync : This will sync all the data in your present volume to the new volume.
    • rsync -avx --exclude /mnt / /mnt/ebs //you may use --exclude <directory name> to exclude any directories
  6. This should take long time depending on how big the data is. Once the copying is done, Unmount the volume by:
    • umount /dev/sdg
    • You may want to check the volume space by df -h
  7. Detach the volume for the present instance from the console. Documentation
  8. Right click on the volume and click on create AMI image. When the image is created. Right click on it and click on Launch Instance.

You now have an exact copy of your old instance running.