Hadoop Framework
Knowledge shared is knowledge gained J
Wish you all
a happy day folks. Let us look at the Hadoop architecture in today’s topic.
Hadoop is a
framework developed by Apache in order to handle large volume of data (i.e.)
big data.
Have a look
at the below picture closely.
· Hadoop framework is designed based on
Master/Slave architecture
Let me
explain in it laymen terms about the architecture then let us move on to the
real terminologies used in the Hadoop
Simple
explanation
Just imagine
Master Rob has three slaves named A, B, C.
· Rob will note down all A, B, C
information in his employee register – Where A,B,C lives ? How much potential
each one possess etc
· A,B,C knows who the master after
interacting with Master
· Rob will periodically (1 hour) check
if A/B/C is performing their duty. If anyone is not replying or missing means
there is something wrong with them
· Rob is not sure when A, B or C will
leave him. Hence whatever A knows he orders to share with B and C. Similarly B à share with A and C and C à share with A and B
· A/B/C after performing their duty
will update their current amount of work, pending work etc.
· Rob as a single person cannot monitor
all employees. He appoints Tom as resource manager in order to monitor A/B/C
· Rob maintains who log where who did
what
· If Rob loses his register and log
then it will be very difficult to track A, B, C wages, capability, past
achievements etc. Hence Bob gets this information once in a while
Let
us map the above scenario in Hadoop terminology
1. Rob à Master Node or Name Node
2. A/B/C à Data Node or Slave Node
3. Rob maintains A/B/C info à Metadata (FS Image). This has all the information
about the Cluster
4. Rob’s Log à Edit logs which has transactional information
5. Rob Checks periodically à Heart beat every 30 seconds to make sure the slave is
not down
6. Tom à Resource manager process or daemon to manage Slaves A/B/C
7. Sharing info à Replication factor. This is to make sure the data is
retrieved even if one machine goes down
8. Bob à Secondary name node. This is housekeeping node periodically
gets the metadata information from Name Node
9. A/B/C performing work à Read/Write operation
Process
Explanation
HDFS
- Hadoop File System
We know file systems like UNIX file
system, NTFS, FAT32. Similarly Hadoop file system is a type of file system used
in Big data world. This file system determines
· How the data is stored
· What are the directory structures?
All the Big
data tools developed has to imply with Hadoop file system
YARN
Processing
Yarn – Yet another resource negotiator
YARN was
introduced from Hadoop 2.0. This is a processing unit for Hadoop framework.
Some of the operations are
· How to track the job
· Scheduling the job
