Allen's Blog

In order to execute the next-year plan, I search the research topics and technologies in Hadoop YARN and HDFS, then make a note as follows:

Since Hadoop YARN was proposed, the new generation technology are continusly discussed. For knowing the work of YARN, please refer to the post [1].
The capacity scheduler of YARN[2][3] provides a default capacity scheduler, org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler [4], to let hadoop eco-system manipulates its’ resources. It also provides resource calculator, org.apache.hadoop.yarn.util.resource.DefaultResourseCalculator [5], to calculate the memory usage, the disk usage and the cpu usage of each compute node.
For those system administrators and developers who are curious about the operations of resource allocation and the scheduler, please see the reports [6], [7] and [8].
In scientific area, Project HaSTE [9] proposed a new Hadoop YARN scheduling algorithm, which aims at efficiently utilizing the resources for scheduling map/reduce tasks in Hadoop YARN and improving the makespan of MapReduce jobs.

On the other hand, HDFS [10] is a usually used file system in Hadoop. However, it needs TCP socket connection to read/write data. Due to this reason, the IO performance will be lower than directly reading from local disk without network connection. Therefore, HDFS provides a function called HDFS Short-Circuit Local Reads [11] and also provides a native libaray to directly access the HDFS file system. According to the report [12], the I/O performance of using Short-Circuit is better than TCP.
ps. The other tricky technology to improve the I/O performance of HDFS is to use CombineFileInputFormat [13][14]. But I don’t think this method is better than using Short-Circuit.

Reference

1. Karthik Kambatla, Wing Yew Poon, and Vikram Srivastava, “How Apache Hadoop YARN HA Works,” Cludera. Available: [Online] http://blog.cloudera.com/blog/2014/05/how-apache-hadoop-yarn-ha-works/
2. Hadoop, “Haddop MapReduce Next Generation – Capacity Scheduler”, Apache Hadoop. Available: [Online] http://hadoop.apache.org/docs/r2.5.2/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Configuration
3. skyWalker_ONLY, “Hadoop-2.4.1学习之容量调度器”. Available: [Online] http://blog.csdn.net/skywalker_only/article/details/41351147
4. GrepCode, “CapacityScheduler”, GrepCode.com. Available: [Online] http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-yarn-server-resourcemanager/2.6.0/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java/
5. GrepCode, “ResourceCalculator”, GrepCode.com. Available: [Online] http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-yarn-common/2.6.0/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java/
6. Vinod Kumar Vavilapalli, “Resource Location in YARN: Deep Dive,” Hortonworks. Available: [Online] http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
7. SEQUENCEIQ, “YARN Schedulers demystified – Part 1: Capacity.” Available: [Online] http://blog.sequenceiq.com/blog/2014/07/22/schedulers-part-1/
8. SEQUENCEIQ, “YARN Schedulers demystified – Part 2: Fair.” Available: [Online] http://blog.sequenceiq.com/blog/2014/09/09/yarn-schedulers-demystified-part-2-fair/
9. Bo Sheng, “Project HaSTE: Hadoop YARN Scheduling Based on Task-Dependency and Resource-Demand,” The 7th IEEE International Conference on Cloud Computing, Anchorage, AK, June 2014.  Available: [Online] http://www.cs.umb.edu/~shengbo/research/haste.html
10. Hadoop, “HDFS User Guide,” Apache Hadoop. Available: [Online] http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Related_Documentation
11. Hadoop, “HDFS Short-Circuit Local Reads,” Apache Hadoop. Available: [Online] https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html
12. Colin McCabe, “How Improved Short-Circuit Local Reads Bring Better Performance and Security to Hadoop,” Cloudera. Available: [Online] http://blog.cloudera.com/blog/2013/08/how-improved-short-circuit-local-reads-bring-better-performance-and-security-to-hadoop/
13. Dhruba Borthakur, “HDFS block replica placement in your hands now!” Available: [Online] http://hadoopblog.blogspot.de/2009/09/hdfs-block-replica-placement-in-your.html
14. Hadoop, “Class CombineFileInputFormat<K,V>”. Available: [Online] http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html

Recently I got a requirement for reading Hadoop’s SequenceFile without enabling Hadoop Platform. However, most examples introduce the read/write SequenceFile with Hadoop Platform. How do I read such files without hadoop?
There’s a tricky solution in this case.
1. Download Hadoop binary file from hadoop site. For Linux/Unix please directly download it; for Windows, there’s pre-built archive file – hadoop-common-2.2.0-bin (source code is here) , created by Abhijit Ghosh.
2. Set environment variable HADOOP_HOME by the directory path (suppose the directory is /usr/local/hadoop in Unix ; or C:/hadoop-common-2.2.0-bin in Windows)
3. Append $HADOOP_HOME/bin to the end of environment variable PATH. ( i.e. /usr/local/hadoop/bin in Unix ; or C:/hadoop-common-2.2.0-bin/bin in Windows
4. Write your program like this (Notice that you have to download hadoop-common-2.2+):

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URLEncoder;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.util.ReflectionUtils;

public class ProcessSequenceFile {
	
	public static void readSequenceFile(String sequenceFileName) throws IOException, URISyntaxException {
		Configuration conf = new Configuration();
		String directoryPath = "file:///";
		directoryPath = URLEncoder.encode(directoryPath, "UTF-8");
		FileSystem fs = FileSystem.get(new URI(directoryPath),conf);
		Path file = new Path(fs.getUri().toString() +  sequenceFileName);
		@SuppressWarnings("deprecation")
		SequenceFile.Reader reader = new SequenceFile.Reader(fs, file, conf);
		Text key = (Text) ReflectionUtils.newInstance(reader.getKeyClass(), conf);
		Text value = (Text) ReflectionUtils.newInstance(reader.getValueClass(), conf);
		while(reader.next(key,value)) {
			System.out.println("Key:" + key);
			System.out.println("=================");			
			System.out.println(value);
			
		}
		
	}
	
	public static void main(String[] args) {
		try {
			// If args[0] is the SequenceFile we need to read.
			readSequenceFile(args[0]);
		} catch (IOException | URISyntaxException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
}

Enjoy it!

Reference:
1. Gnosis Runmination, “WIN7下运行hadoop程序报：Failed to locate the winutils binary in the hadoop binary path.” Available: [Online] http://www.cnblogs.com/zq-inlook/p/4386216.html
2. StackOverFlow, “Running Apache Hadoop 2.1.0 on Windows”. Available: [Online] http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows
3.  Abhijit Ghosh, “ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path,” SrcCodes.com. Available: [Online] http://www.srccodes.com/p/article/39/error-util-shell-failed-locate-winutils-binary-hadoop-binary-path
4. Hadoop, “Native Libraries Guide.” Available: [Online] https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html#Native_Hadoop_Library