Monday, 4 April 2016

HADOOP 2  INSTALLATION  STEPS
  1. Deploy HA Cluster
HDFS High Availability (HA) feature and how to configure and manage an HA HDFS cluster, using the Quorum Journal Manager (QJM) feature to prevent hadoop master namenode from SPOF(Single point of failure).

Create Dedicated User
Firstly create a dedicated user for Hadoop.We can create a Hadoop user executing following commnd.

# addgroup hadoop
#adduser   --ingroup hadoop hadoop

Then,open /etc/sudoers file.This grant all permission to hadoop user like root.
#vi /etc/sudoers & add following line,
 hadoop ALL=(ALL) ALL

Then switch user hadoop by typing
#su – hadoop
  1. Decide where you want to keep your haddop installation.
  2. I have user created by name hadoop. For testing purpose, I am keeping all the hadoop components in /usr/local/. I also have to setup ssh. Connect to user hadoop and run the following commands.
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
  1. Update /etc/hosts file
Sometimes hadoop does not work because if some unexpected entry in /etc/hosts file. Check if you have the following entry in /etc/hosts file
IP_Addresshostname1
IP_Addresshostname2
IP_Addresshostname3

Install Hadoop
Save all the files to /usr/local/, or your preferred directory.

cd /usr/local
Extract Hadoop common files to the same folder. Run the following command.

sudo tar xzf hadoop-2.2.0.tar.gz
ln –s hadoop-2.2.0 hadoop (make it simple by creating link)
(Change the permission and owner of the directory by running following command)
sudo chown -R hduser:hadoop hadoop
sudo chmod -R 777 hadoop

Set the following environment variable in your startup script. In my case, I have to set it on/home/hadoop/.bashrc
#vi .bashrc
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

Configuration
Edit the configuration files for hadoop. As this is a basic setup we just need to edit the file core-site.xml,hdfs-site.xml,yarn-site.xml and slaves.found in
$HADOOP_HOME/etc/hadoop. Add the following content between theconfigration tags.

cd/usr/local/hadoop/etc/hadoop/conf/

Now you modifyhadoop-env.sh
#vi hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-i386/
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
Now you modify core-site.xml
#vicore-site.xml
<configuration>
<property>
   <name>hadoop.tmp.dir</name>
   <value>/home/hadoop/double/tmp/hadoop-${user.name}</value>
</property>
<description>A base for other temporary directories</description>
        <property>
   <name>fs.defaulFS</name>
   <value>hdfs://mycluster</value>
</property>
<description>the default path prefix used by the Hadoop FS client when none is given</description>
     <property>
 <name>dfs.journalnode.edits.dir</name>
 <value>/home/hadoop/mydata/hdfs/journalnode/mycluster</value>
</property>
<description>the path where the JournalNode daemon will store its local state</description>
<property>
<name>ha.zookeeper.quorum</name>
<value>zk1.example.com:2181,zk1.example.com:2181,zk1.example.com:2181</value>
</property>
<description>This lists the host-port pairs running the ZooKeeper service(we are using 2222 port in place of default port)</description>
</configuration>
Then,modify hdfs-site.xml
#vi hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.permissions.superusergroup</name>
<value>hadoop</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/mydata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/mydata/hdfs/datanode</value>
     </property>
<property>
 <name>dfs.nameservices</name>
 <value>mycluster</value>
</property>
<description>the logical name for this new nameservice
</description>
<property>
 <name>dfs.ha.namenodes.mycluster</name>
 <value>nn1,nn2</value>
</property>
<description>unique identifiers for each NameNode in the nameservice,"nn1" and "nn2" as the individual IDs of the NameNodes</description>
<property>
 <name>dfs.namenode.rpc-address.mycluster.nn1</name>
 <value>master_node1:8020</value>
</property>
<property>
 <name>dfs.namenode.rpc-address.mycluster.nn2</name>
 <value>master_node2:8020</value>
</property>
<description>the fully-qualified RPC address for each NameNode to listen on</description>
<property>
 <name>dfs.namenode.http-address.mycluster.nn1</name>
 <value>master_node1:50070</value>
</property>
<property>
 <name>dfs.namenode.http-address.mycluster.nn2</name>
 <value>master_node2:50070</value>
</property>
<description>the fully-qualified HHTP address for each NameNode to listen on</description>
<property>
    <name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/mycluster</value>
</property>
<description>the URI which identifies the group of JNs where the NameNodes will write/read edits</description>
<property>
    <name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<description>the Java class that HDFS clients use to contact the Active NameNode</description>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<name>dfs.namenode.avoid.read.stale.datanode</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.stale.datanode.interval</name>
<value>20000</value>
</property>
<property>
<name>dfs.namenode.avoid.write.stale.datanode</name>
<value>true</value>
</property>
Now you modify yarn-site.xml
vi yarn-site.xml
<configuration>
    <property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
   </configuration>

Then do the change in slaves
(Here define the list of slave nodes )

#vi slaves
node1.example.com
node2.example.com
node3.example.com

copy the same configurations to all slave node and another master node which act as a standby namenode in cluster.In slave node exclude following property in hdfs-site.xml
<property>
 <name>dfs.ha.fencing.methods</name>
 <value>sshfence</value>
</property>
<property>
 <name>dfs.ha.fencing.ssh.private-key-files</name>
 <value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
  <name>dfs.ha.automatic-failover.enabled</name>
  <value>true</value>
</property>

Note:-Before formate the namenode,start the Journalnode on cluster using following command,
#cd /usr/local/hadoop/etc/hadoop
#sbin/hadoop-daemons.sh start journalnode

Formatting NameNode On Master with NameserviceId NN1
# cd /usr/local/hadoop
# bin/hadoop namenode -format
#sbin/hadoop-daemon.sh start namenode

Formatting NameNode On Master with NameserviceId NN2
#bin/hadoop namenode -bootstrapStandby
#sbin/hadoop-daemon.sh start namenode
Starting Hadoop Cluster
#sbin/hadoop-daemon.sh start namenode
#sbin/hadoop-daemons.sh start datanode
#sbin/yarn-daemon.sh start resourcemanager
#sbin/yarn-daemons.sh start nodemanager
#sbin/mr-jobhistory-daemon.sh start historyserver
Whenever an HA NameNode starts, it is initially in the Standby state.To make active namenode start the zookeeper.
6) Install zookeeper and do the following steps for installing,
$ chown -R hadoop:hadoop zookeeper
$ chmod -R 755  zookeeper

7) now  do the following change,
$cp /usr/local/zookeeper/zoo_sample.cfg/usr/local/zookeeper/zoo.cfg
$vi /usr/local/zookeeper/zoo.cfg
clientPort=2181
dataDir=/home/hadoop/zookeeper
server.1=zk1.example.com:2888:3888
server.2=zk2.example.com:2888:3888
server.3=zk3.example.com:2888:3888
Set the following environment variable in your startup script.
  1.  #vi .bashrc
  2. export ZK_HOME=/usr/local/zookeeper
  3. start zookeeper by using following command,
#cd /usr/local/zookeeper
#bin/zkserver.sh start
After that execute the following command,
On Master with NameserviceId NN1
#bin/hdfs  zkfc -formatZK
#sbin/hadoop-daemon.sh start zkfc
(now namenode with service id NN1 active.)

On Master with NameserviceId NN2
  1. #sbin/hadoop-daemon.sh start zkfc
  2. Once the command is run succesfully, let us verify if everything is working fine.
  3. You should run the following command see the result.
On Namenode output shown like this.
8365 DFSZKFailoverController
6300 JobHistoryServer
31287 Jps
8231 QuorumPeerMain
5861 NodeManager
8628 NameNode
5990 ResourceManager

On Namenode output shown like this.
120748 JournalNode
38539 Jps
127059 HRegionServer
121513 DataNode
123427 QuorumPeerMain
  1. Stoping Hadoop Cluster
#bin/hadoop-daemon.sh stop namenode
#sbin/hadoop-daemons.sh stop datanode
#sbin/yarn-daemon.sh stop resourcemanager
#sbin/yarn-daemons.sh stop nodemanager
#sbin/mr-jobhistory-daemon.sh stop historysever

No comments:

Post a Comment