Solving 0.0.0.0 permission denied (publickey) error message on Hadoop name node

For a Hadoop cluster to work properly, all the hadoop components like NameNode, SecondaryNameNode, JobTracker and ProcessManager must be started. If you happen to create a hadoop cluster by yourself like I mentioned in my previous blog, when you start the name node by using 'start-dfs.sh', you might get the below error

'0.0.0.0 permission denied (publickey)'

This is due to the fact that, by default, the hadoop core process will try to start secondary namenode on the default ip address 0.0.0.0 which by default should be pointing to localhost. In order to do that it tries to establish an SSH connection to the ip address 0.0.0.0 and as the error message states, it fails to make the connection because the passwordless SSH is not setup correctly for the host 0.0.0.0. Below steps will help you fix this

1) If you are using a config file for passwordless ssh (mostly in case of AWS servers)

In order to setup passwordless ssh, you might have created a config file in .ssh folder of your home directory with the entries for hostname, userid and the keyfile to be used while doing ssh, as shown below

Read more »

Read More

Solving jps: command not found error message

jps is a command line utility that comes with jdk which allows you to view all the Java processes running on a host. It is extremely useful when your environment has multiple background java processes running, like it happens in a Hadoop cluster. If you happen to create a hadoop cluster by yourself as mentioned in my previous blogs, 'jps' command is very useful to diagnostic utility to make sure the NameNodes, DataNodes, ProcessManager and ResourceManager processes are running.

Sometimes if you try to use 'jps' and get the error message "jps: command not found error message" here is how you solve it

Read more »

Read More

Setting up a Hadoop Cluster in RHEL 6 - Preparing the servers

In this two part series, we will discuss how to setup a Hadoop cluster on RHEL 6. I was inspired by my friend who did the Big Data Specialization course of University of California San Diego through Coursera and I thought it will be a good learning experience for me to setup a Hadoop cluster myself as well. Even though Cloudera's open source paltform is the most common distribution of Hadoop, I really wanted to understand some low-level details of Hadoop components and how they interact with each other. There fore, I thought it will be useful to install and configure the basic Apache Hadoop distribution from scratch.

In this part, we will discuss the setup of  Linux environment and how to install the Hadoop distribution. In Part 2 we will discuss how to configure the core components of the Hadoop ecosystem and start the server. Okay, lets get started

Read more »

Read More