Hello Friends...
In this blog I am explaining the procedure to install hadoop single node cluster in Linux. I installed hadoop 2.6.0 on Ubuntu 12.04. Hadoop installation needs basic working knowledge of Linux. I hope you have basic working knowledge of linux or have a look on this post for basic understanding of Linux first: Linux administration.
The steps for installing hadoop are as following:
1. First open the terminal by Ctrl+Alt+T.
2. Run the update command: sudo apt-get update
First it will prompt for your password, and then it may take time depending upon your internet
speed and system update status.
3. Then install java in your sytem using $ sudo apt-get install openjdk-6-jdk
Note: I used java version 6, you can opt for higher version 7 or 8.
To change java version in your system you can run the command:
$ update-alternatives --config java
4. check java version by using: $ java -version
5. Add a new group named hadoop: $ sudo addgroup hadoop
6. Then make a new user hduser in that group: $ sudo adduser –ingroup hadoop hduser
It may ask for some details like name, address, etc. Fill these details although you may skip some of these.
7. Now for communication install the ssh: $ sudo apt-get install ssh
8. Generate the RSA public private key pair using SSH and move this to the authorized_keys as shown in following steps:
9. Add localhost as secure channel using ssh: $ ssh localhost
10. Now install the freely available hadoop version from any site ( I downloaded 2.6.0)
11. Untar the downloaded package using the command: $ tar xvzf hadoop-2.6.0.tar.gz
Now make the hadoop directory inside /usr/local by the command: mkdir -p /usr/local/hadoop
12. Now change directory to this folder using: $ cd hadoop-2.6.0
13. Now move all content of this directory to the /usr/local/hadoop
14. This may throw an error like:
hduser is not in the sudoers file. This incident will be reported......
15. To deal with this error add hduser to the sudoers file as shown
16. Now again move the folder as tried previously and change its ownership to the hduser as shown:
17. Now we are almost done, and just need to change the configuration file. The following files
needs to be changed
1. ~/.bashrc
2. hadoop-env.sh
3. core-site.xml
4. mapred-site.xml
5. hdfs-site.xml
18. Open bashrc by the comand ( vim ~/.bashrc) and add the hadoop path to the directory as shown: [If vim is not already installed on your system, install it by following command sudo apt-get install vim (after it again try to open the .bashrc file as shown:) vim ~/.bashrc
19. Now open ( $ vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh) and update hadoop-env.sh as shown
20. Now First make a tmp directory as mentioned in the given step:
Now open and update core-site.xml as shown:
Now first copy content of mapred-site.xml.template to mapred-site.xml by the command shown in the image below:
21. Now open and update mapred-site.xml as shown (opening command in the image above and opened file and necessary changes in the image underneath)
22. Now make two directories for namenode and datanode and then make corresponding updates in hdfs-site.xml
Updates in hdfs-site.xml
23. Now we are done... !!
24. Lets start the hadoop now,
25. first format the namenode
26. Then start the hadoop:
27.Change the directory where start-all.sh file resides:
28. Now start hadoop : $ start-all.sh and check the status of the node using the command $ jps
29. Lets see the web interface of Namenode and Secondary namenodes:
Namenode at port 50070 of localhost:
We are done.... All components are working fine.
30. Last one.... Dont forget to leave hadoop cluster without stoping the services by the following commands:
If you wish to make a multinode hadoop cluster. Please refer the instructions given at following post hadoop multinode installation
*****************************************************************************
Now to run the first program on your hadoop cluster Please follow this blog: Running first program in hadoop
For configuring hbase in your hadoop cluster visit this post
For configuration of pig in your hadoop cluster go to this pig-installation-page
For more frequent updates about Big data Analytics using hadoop please visit and like: DataioticsHub
Thanks and Regards
In this blog I am explaining the procedure to install hadoop single node cluster in Linux. I installed hadoop 2.6.0 on Ubuntu 12.04. Hadoop installation needs basic working knowledge of Linux. I hope you have basic working knowledge of linux or have a look on this post for basic understanding of Linux first: Linux administration.
The steps for installing hadoop are as following:
1. First open the terminal by Ctrl+Alt+T.
2. Run the update command: sudo apt-get update
First it will prompt for your password, and then it may take time depending upon your internet
speed and system update status.
3. Then install java in your sytem using $ sudo apt-get install openjdk-6-jdk
Note: I used java version 6, you can opt for higher version 7 or 8.
To change java version in your system you can run the command:
$ update-alternatives --config java
4. check java version by using: $ java -version
5. Add a new group named hadoop: $ sudo addgroup hadoop
6. Then make a new user hduser in that group: $ sudo adduser –ingroup hadoop hduser
It may ask for some details like name, address, etc. Fill these details although you may skip some of these.
7. Now for communication install the ssh: $ sudo apt-get install ssh
8. Generate the RSA public private key pair using SSH and move this to the authorized_keys as shown in following steps:
9. Add localhost as secure channel using ssh: $ ssh localhost
10. Now install the freely available hadoop version from any site ( I downloaded 2.6.0)
11. Untar the downloaded package using the command: $ tar xvzf hadoop-2.6.0.tar.gz
Now make the hadoop directory inside /usr/local by the command: mkdir -p /usr/local/hadoop
12. Now change directory to this folder using: $ cd hadoop-2.6.0
13. Now move all content of this directory to the /usr/local/hadoop
14. This may throw an error like:
hduser is not in the sudoers file. This incident will be reported......
15. To deal with this error add hduser to the sudoers file as shown
16. Now again move the folder as tried previously and change its ownership to the hduser as shown:
17. Now we are almost done, and just need to change the configuration file. The following files
needs to be changed
1. ~/.bashrc
2. hadoop-env.sh
3. core-site.xml
4. mapred-site.xml
5. hdfs-site.xml
18. Open bashrc by the comand ( vim ~/.bashrc) and add the hadoop path to the directory as shown: [If vim is not already installed on your system, install it by following command sudo apt-get install vim (after it again try to open the .bashrc file as shown:) vim ~/.bashrc
19. Now open ( $ vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh) and update hadoop-env.sh as shown
20. Now First make a tmp directory as mentioned in the given step:
Now open and update core-site.xml as shown:
Now first copy content of mapred-site.xml.template to mapred-site.xml by the command shown in the image below:
21. Now open and update mapred-site.xml as shown (opening command in the image above and opened file and necessary changes in the image underneath)
22. Now make two directories for namenode and datanode and then make corresponding updates in hdfs-site.xml
Updates in hdfs-site.xml
23. Now we are done... !!
24. Lets start the hadoop now,
25. first format the namenode
26. Then start the hadoop:
27.Change the directory where start-all.sh file resides:
28. Now start hadoop : $ start-all.sh and check the status of the node using the command $ jps
Errorfree start of the hadoop environment will show Namenode, SecondaryNameNode,
NodeManager, DataNode, ResourceManager and jps itself as running processes. So we are done.29. Lets see the web interface of Namenode and Secondary namenodes:
Namenode at port 50070 of localhost:
We are done.... All components are working fine.
30. Last one.... Dont forget to leave hadoop cluster without stoping the services by the following commands:
If you wish to make a multinode hadoop cluster. Please refer the instructions given at following post hadoop multinode installation
*****************************************************************************
Now to run the first program on your hadoop cluster Please follow this blog: Running first program in hadoop
For configuring hbase in your hadoop cluster visit this post
For configuration of pig in your hadoop cluster go to this pig-installation-page
For more frequent updates about Big data Analytics using hadoop please visit and like: DataioticsHub
Thanks and Regards
