HADOOP INSTALLATION ON SINGLE NODE UBUNTU 12.04

    INSTALLATION GUIDE FOR HADOOP IN UBUNTU 12.04 (SINGLE NODE)

 

Installing Java-7-oracle

 

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java7-installer

 

 

Create a separate user for hadoop

 

$ sudo addgroup hadoop

$ sudo adduser –ingroup hadoop hduser

 

Configure SSH

 

su – hduser

ssh-keygen -t rsa -P “”

 

To be sure that SSH installation is went well, you can open a new terminal and try to create ssh session using hduser by the following command:

$ssh localhost

 

if localhost is not connected reinstalltion of ssh is needed

 

sudo apt-get install openssh-server

 

Edit Sudoers

pkexec visudo

 

Add below line to add hduser into sudoers

hduser (ALL)=(ALL) ALL

 

Ctrl + O to save nano 

 

and exit from the editor  

 

Disable IPv6

 

 

$sudo gedit /etc/sysctl.conf

This command will open sysctl.conf in text editor, you can copy the following lines at the end of the file:

 

#disable ipv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

 

If you faced a problem telling you don’t have permissions, just remember to run the previous commands by your root account.

These steps required you to reboot your system, but alternatively, you can run the following command to re-initialize the configurations again.

 

$sudo sysctl -p

To make sure that IPV6 is disabled, you can run the following command:

$cat /proc/sys/net/ipv6/conf/all/disable_ipv6

 

Configuration of Hadoop

Installing Hadoop

 

Now we can download Hadoop to begin installation. Go to Apache Downloads and download Hadoop version 1.0.4.(current stable version)

Then you need to extract the tar file and rename the extracted folder to ‘hadoop’. Open a new terminal and run the following command:

 

$ cd /home/hduser

$ sudo tar xzf hadoop-0.20.2.tar.gz

$ sudo mv hadoop-0.20.2 hadoop

 

Update $HOME/.bashrc

You will need to update the .bachrc for hduser (and for every user you need to administer Hadoop). To open .bachrc file, you will need to open it as root:

 

$sudo gedit /home/hduser/.bashrc

 

Then you will add the following configurations at the end of .bachrc file

 

# Set Hadoop-related environment variables

 

export HADOOP_HOME=/home/hduser/hadoop

 

# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)

 

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

 

# Some convenient aliases and functions for running Hadoop-related commands

 

unalias fs &> /dev/null

alias fs=”hadoop fs”

unalias hls &> /dev/null

alias hls=”fs -ls”

 

# If you have LZO compression enabled in your Hadoop cluster and

# compress job outputs with LZOP (not covered in this tutorial):

# Conveniently inspect an LZOP compressed file from the command

# line; run via:

#

# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo

#

# Requires installed ‘lzop’ command.

#

lzohead () {

    hadoop fs -cat $1 | lzop -dc | head -1000 | less

}

 

# Add Hadoop bin/ directory to PATH

export PATH=$PATH:$HADOOP_HOME/bin

 

 

hadoop-env.sh

We need only to update the JAVA_HOME variable in this file. Simply you will open this file using a text editor using the following command:

 

$sudo gedit /home/hduser/hadoop/conf/hadoop-env.sh

 

or

 

nano /home/hduser/hduser/hadoop/conf/hadoop-env.sh

 

Then you will need to change the following line

 

# export JAVA_HOME=/usr/lib/j2sdk1.5-sun

 

To 

 

export JAVA_HOME=/usr/lib/jvm/java-7-oracle

 

Note: if you faced “Error: JAVA_HOME is not set” Error while starting the services, then you seems that you forgot toe uncomment the previous line (just remove #).

 

core-site.xml

First, we need to create a temp directory for Hadoop framework. If you need this environment for testing or a quick prototype (e.g. develop simple hadoop programs for your personal test …), I suggest to create this folder under /home/hduser/ directory, otherwise, you should create this folder in a shared place under shared folder (like /usr/local …) but you may face some security issues. But to overcome the exceptions that may caused by security (like java.io.IOException), I have created the tmp folder under hduser space.

 

To create this folder, type the following command:

 

$ sudo mkdir  /home/hduser/tmp

 

Please note that if you want to make another admin user (e.g. hduser2 in hadoop group), you should grant him a read and write permission on this folder using the following commands:

 

 

$ sudo chown hduser:hadoop /home/hduser/tmp

 

$ sudo chmod 755 /home/hduser/tmp

Now, we can open hadoop/conf/core-site.xml to edit the hadoop.tmp.dir entry.

We can open the core-site.xml using text editor:

 

$sudo gedit /home/hduser/hadoop/conf/core-site.xml

 

or

 

nano /home/hduser/hduser/hadoop/conf/core-site.xml

 

Then add the following configurations between .. xml elements:

 

  hadoop.tmp.dir

  /home/hduser/tmp

  A base for other temporary directories.

 

  fs.default.name

  hdfs://localhost:54310

  The name of the default file system.  A URI whose

  scheme and authority determine the FileSystem implementation.  The

  uri’s scheme determines the config property (fs.SCHEME.impl) naming

  the FileSystem implementation class.  The uri’s authority is used to

  determine the host, port, etc. for a filesystem.

mapred-site.xml

We will open the hadoop/conf/mapred-site.xml using a text editor and add the following configuration values (like core-site.xml)

nano /home/hduser/hduser/hadoop/conf/mapred-site.xml

 

  mapred.job.tracker

  localhost:54311

  The host and port that the MapReduce job tracker runs

  at.  If “local”, then jobs are run in-process as a single map

  and reduce task.

 

hdfs-site.xml

Open hadoop/conf/hdfs-site.xml using a text editor and add the following configurations:

 

nano /home/hduser/hduser/hadoop/conf/hdfs-site.xml

 

  dfs.replication

  1

  Default block replication.

  The actual number of replications can be specified when the file is created.

  The default is used if replication is not specified in create time.

 

Formatting NameNode

~/hduser/hadoop/bin/hadoop namenode -format

 

You should format the NameNode in your HDFS. You should not do this step when the system is running. It is usually done once at first time of your installation.

Run the following command

 

$/home/hduser/hadoop/bin/hadoop namenode -format

 


NameNode Formatting

Starting Hadoop Cluster

You will need to navigate to hadoop/bin directory and run ./start-all.sh script.

cd ~/hduser/hadoop/bin/

./start-all.sh

 


Starting Hadoop Services using ./start-all.sh

 

There is a nice tool called jps. You can use it to ensure that all the services are up.

SHIVA TRILOGY PART 1

IMMORTALS OF MELUHA

Image

Om Namah Shivayi.

The universe bows to Lord Shiva. I bow to Lord Shiva.

Amish, an IIM graduate created a gripping mythological plot by breaking the rooster coop of the traditional writing style. He combined the essence of mythology, philosophy and adventure in Shiva Trilogy. The author contemplates the readers to think about the possibility of Lord Shiva in blood and flesh, who was raised to be a god because of his karma. He urges us to think out of the realm of human imagination and learn the lessons of life with the humble Shiva. Thus the nomadic journey begins.

‘The Immortals of Meluha’ is the primary key that connects us to the world of Meluha.Amish introduces Shiva as a village head who was forced to take a shelter in the metro politician city of Meluha far away from his Himalayan roots. When the author describes Meluha, we may find him narrating the old tales of our land Bharath.The very old rich traditions, customs and the decline of the country and the rise after hitting the rock bottom will show the connections between meluha and Bharath.

The social norms, the culture, the sophisticated and modernized civilization of meluhians show one side of the coin. The well built laws and social systems of meluha are quiet interesting and terrific.Those laws can be found as the exact replica of which existed in our olden systems.Maika system of meluha can be found as the fairest system that validates a person based on his talents and not on his ancestry.This can be an antidote to the old system of cast discrimination or to the existing reservation system.

You will come across the familiar characters like Nandi, ayurvathi, Brahaspathi and parvatehswar who will unleash the story which their land beholds. The arrival of the Neelakand, the title which Shiva couldn’t accept just because his throat turned to be blue will lead him to the prophecy, untouched stories and the ugly face of Meluha.In the beginning we can find a Shiva who is an absolute slacker but as the story progress we can find him as a listener, a student, a rebel and a policy maker.

The most interesting part is, which is rich in philosophy and full of banters of the vasudev pandits.The way they make Shiva understand the real meaning of the title which he is bestowed and the way they generate a carving in Shiva and of course the readers for the lessons, make us think, are leaves really green in colour?

The precision of passing knowledge as if measured, only to quench the thirst and not more will leave us in search of more and more.

The innocent and the impatient Shiva who tries to confirm if he is the real neelkand, and the answers of the vasudaves create a mysterious ambience to the whole story.
The simplicity of arguments is the strength of this story. All passages are eye opener and will make the reader ponder in wild thoughts.

His meeting with Lady Sati was not written in an extraordinary manner where the male shows the dominance but with the natural human emotions. There is surely a spark and it is kept in limit. There is a specific unwritten line which both keeps up.

“Sati had a calm confidence which had not covered the ugly distance towards arrogance” this was Shiva’s first impression about Sati. Sati is one such formidable character in this story who will make us awe in respect. Shiva’s urge to impress his lady love and the mistakes he makes is an eye opener to the cruel Vikarma system.

Vikarma system is another representation of the untouchable culture prevailed in our society. Shiva took a stern stand against the law by taking the blessings of the impure .The teasing of pundits about his concern for all the vikarma or just the one, brings a small smile on the readers…
Sati’s challenge of agnipareekaha to save Shiva’s dignity was a thrilling experience enriched with Shiva’s planning and sati’s braveness in execution.When sati takes the Agni baan to protect her love, Shiva was tangled at the “sun and earth” poem.
 

Lord Daksha playing the cupid between Shiva and Sati and his efforts to withdraw the rules of vikarma all shows his immense love for his daughter.Daksha is shown as a character who spend his whole life trying to impress others. You can find him as a weak character who is a little selfish and a great manipulator.

It’s with great respect that one can view the character of Parvateshwar.His loyalty to Lord Rama and his nation are worth admirable. His reluctance to accept neelkand and the violations of Rama’s laws proves his conventional mind set and tough exterior.
 
Never think that it’s just a Shiva tale, we can see lord Rama, the maryada purushotam who established order out of the chaos. We can also find the suryavanshis and chandravansis who are the arch nemesis. And their differences in approach reflect in their administration and socio-political life. If one is the sun other is the moon, without both existences are impossible. They are the two faces of the same coin.

Shiva’s journey from a nomad to a mahadev gives an insight into the role of a mahadev.A man become mahadev only when he fights for good. He is the one who absorbs the negatives. He is forged into the heat of battle when he wages a war to destroy the evil.
 Har har mahadev
This slogan resounds in our ears telling that each of us is Mahadev.Strive to be a Mahadev and you will be one. Its ones karma that determines mahadev   no gender, no cast, no creed can block you from unleashing yourself.
 
 

The strength of this book is the wonderful meaning that each passage holds which make us to map them to the current socio-political ambience. A satire is created which criticizes the society and its deeds in a curt manner.

Another important characteristic of this book is its fast pace, it will attack you with the twists and turns that you will be forced to go along with the flow. The constant attacks on meluha by hooded figures will put the readers in a maze.as the story progress the mystery will entwine and will take the story to new levels.


The splendid adventure that Shiva provides and the innocence with which he tries to analyze the situation and read through the unread , seeking the help of Nandi to understand the unknown shows the true Mahadev.The pandemonium of Shiva while he understands the basis of his journey  will keep the readers at tangling points. His search to find what is evil will make the readers to interrogate the story to solve the puzzle. It’s truly a magical voyage.
 

This book is one such journey where we can expect anything. Expect the unexpected that’s what the story holds with it. Serendipity causes major twists and turns in the story. It will bombard with you in such a way that it will make you think what will happen next. Refrain yourself and enjoy the journey of the Shiva from a nomad to a Neelakand.If you read the first book of this series then it’s sure that you won’t be able to hold yourself back from grabbing other two books. The sly voices of vasudev pandits will enchant our mind even after the story ends.

Revealing the original Shiva will surely engross the readers and make them forget that they have reached the end leaving a frown, why did it stop here??