How to find the version of cloudera Hadoop

Read the following file

# cat /usr/lib/hadoop/cloudera/cdh_version.properties

# Autogenerated build properties
version=2.0.0-cdh4.1.2
git.hash=f0b53c81cbf56f5955e403b49fcd27afd5f082de
cloudera.hash=f0b53c81cbf56f5955e403b49fcd27afd5f082de
cloudera.base-branch=cdh4-base-2.0.0
cloudera.build-branch=cdh4-2.0.0_4.1.2
cloudera.pkg.version=2.0.0+552
cloudera.pkg.release=1.cdh4.1.2.p0.27
cloudera.cdh.release=cdh4.1.2
cloudera.build.time=2012.11.02-00:01:31GMT

cloudera.pkg.name=hadoop

The Hadoop version installed is 2.0.0-cdh4.1.2, which means that Cloudera version installed is based on Apache Hadoop 2.0.0 release.

Hadoop Namenode - Image, Checkpoint(fsimage), Journal(edits)

Image

HDFS Namenode keeps the entire namespace in RAM. The inode data and list of blocks belonging to each file comprise the metadata of the name system called the image.

Checkpoint (fsimage)

The persistent record of the image stored in the Namenode's native fielsystem is called a checkpoint.The locations of block replicas may change over time and are not part of the persistent checkpoint. checkpoint also called fsimage(filesystem image).

The fsimage file contains a serialized form of all the directory and file inodes in the filesystem. Each inode is an internal representation of a file or directory’s metadata and contains such information as the file’s

replication level, modification and access times, access permissions, block size, and the blocks a file is made up of. For directories, the modification time, permissions, and quota metadata is stored.

Note: The fsimage file does not record the datanodes on which the blocks are stored. Instead the namenode keeps this mapping in memory, which it constructs by asking the datanodes for their block lists when they join the cluster and periodically afterward to ensure the namenode’s block mapping is up-to-date.

Journal(edit log)

The NameNode also stores the modification log of the image called the journal in the local host’s native file system. journal also called edits. When a filesystem client performs a write operation (such as creating or moving a file), it is first recorded in the edit log. The namenode also has an in-memory representation of the filesystem metadata, which it updates after the edit log has been modified. The in-memory metadata is used to serve read requests.

Upon namenode startup, the fsimage file is loaded into RAM and any changes in the edits file are replayed, bringing the in-memory view of the filesystem up to date

How HDFS is different from traditional filesystems like ext3?

1. Traditional filesystems like ext3 are implemented as kernel modules. HDFS is a userspace filesystem - filesystem code runs outside kernel as OS process and is not registered or exposed via the Linux VFS layer.
2. Traditional filesystems need to mounted. HDFS filesystems need not be mounted, as it just runs as a OS process.
3. HDFS is ditributed filesystem - distributed across many machines. So size of a HDFS file is not limited by the machine capacity. In traditional filesystems, file size cannot exceed the disk space capacity of the machine.
4. Traditional filesystems use block size of 4KB or 8KB. HDFS uses larger block size of 64MB by default.
6. Unlike conventional file systems, HDFS provides an API that exposes the locations of a file blocks. This allows applications like the MapReduce framework to schedule a task to where the data are located, thus improving the read performance.

How to extract .xz file in Linux?

To extract .xz files use the option

tar xJf  <file.xz>

Tools for benchmarking Hard Disk performance(File system Read Write Performance)

dd

dd command can be used to read and write from any block device in Linux.

Using dd command we can perform simple sequential read and sequential write tests

Using dd command to check sequential write speed by writing a 1GB file to disk

dd if=/dev/zero of=/tmp/outputfile bs=1M count=1000

But the above test has a drawback.Any data written to disk is first cached in memory and then written to the disk. So in the above test, it gives the speed at which data was cached into RAM and not the speed at which data was written to the disk. So the above command is tweaked as follows

dd if=/dev/zero of=/tmp/outputfile bs=1M count=1000 conv=fdatasync

So now the dd command will report the write speed only after the data is synced to the disk

Alternative way,

  sync;time bash -c "(dd if=/dev/zero of=/tmp/outputfile bs=1M count=1000; sync)"

Using dd command to check sequential read speed

   dd if=/tmp/outputfile of=/dev/null bs=1M count=1000

hdparm 

It is a performance and benchmarking tool for SATA/IDE drives.

To measure how many MB/s, the hard drive can read

     hdparm -t --direct /dev/sda1

     (or) hdparm -tT /dev/sda1

  -t  : read from cache buffer

  -T : speed of reading without precached buffer

iozone

iozone is an open source file system benchmarking utility.

The default command line option is -a, which stands for full automatic mode. It tests with block sizes ranging from 4k to 16M and file sizes ranging from 64k to 512M.

     iozone -a /dev/sda1

iozone performs 13 types of tests. Some important tests are

Write : Indicates the speed of writing a new file to the filesystem. Creating a new file is always slower than rewriting an existing file owing to the metadata overhead involved while creating a new file (like creating a new inode entry).

ReWrite : Indicates the speed of writing to an existing file

Read : Indicates the speed of reading an existing file

ReRead : Indicates the speed of rereading a file that is already read.

RandomRead : Indicates the speed at which random areas of a single file are read.

RandomWrite : Indicates the speed at which we can write into random areas of a file

Tools for Benchmarking Network Performance

• iperf : To measure bandwidth - Throughput between two nodes
• sockperf ping-pong test : To measure network latency on TCP - https://code.google.com/p/sockperf/
• PING : To measure network latency using ICMP

Difference between Incident and Problem

This piece of information I took from http://www.devopsmastery.com/devops/2014/2/10/whats-the-difference-between-an-incident-and-a-problem

An incident as defined by ITIL is as follows: "An unplanned interruption to an IT Service or a reduction in the Quality of an IT Service. Failure of a Configuration Item that has not yet impacted Service is also an Incident. For example Failure of one disk from a mirror set." A configuration item is just about anything in ITIL Terms.  Everything from a NIC or Hard Drive to a Web Service can be declared a Configuration Item. It's just something to tie incidents and problems to.

A problem as defined by ITIL is as follows: "A cause of one or more Incidents. The cause is not usually known at the time a Problem Record is created, and the Problem Management Process is responsible for further investigation."

Here are some other quick hit differences:
Problems should be traced down to a root-cause level where as incidents should be resolved as quickly as possible to restore the service to operation.

Incidents like a known issue with a Java application memory leak need become a Problem when they repeatedly occur.  

Incidents can generally be resolved by the person doing the work.

Problems must be closed by a manager assigned in the problem management process.