09-云计算_mapreducev2

• Since 2012, the term "Hadoop" often refers not to just the base
Hadoop package but rather to the Hadoop Ecosystem, which includes all of the additional software packages that can be installed on top of or alongside Hadoop, such as Apache Hive, Apache Pig and Apache Spark.
HDFS API
FileSystem fileSystem = FileSystem.get(conf);Path path = new Path(“/path/to/file.ext”); if (!fileSystem.exists(path)) { System.out.println(“File does not exists”); return; } FSDataInputStream in = fileSystem.open(path); int numBytes = 0; while ((numBytes = in.read(b)) > 0) { System.out.prinln((char)numBytes)); // code to manipulate the data which is read } in.close(); out.close(); fileSystem.close();
•HDFS 2.0
–Federation: Isolation & extensibility via multiple NameNodes –Checkpoint support –Performance improvement
Hadoop Common HDFS Hadoop YARN Hadoop MapReduce
HDFS Read a File
HDFS Write a File
HDFS Commands
• Not a general purpose file system
• doesn’t implement posix • can’t just mount it and view files
• Access via “hdfs dfs” commands
HDFS Application
• PB data storage & management • Applications : write-once-read-many • Applications : insensitive to latency, sensitive to throughput • No suitable for millions of little files. • No suitable for random read
第七章 大数据计算模型及架构
•
主要内容
一．大数据计算模型及架构
1. 2. 3. 4.
Hadoop MapReduce Storm Spark
Motivation
• Large Scale Data Processing • Want to process lots of data ( > 1 TB) • Want to parallelize across hundreds/thousands of CPUs • … Want to make this easy
• Contains Libraries and other modules • Hadoop Distributed File System • Yet Another Resource Negotiator • A programming model for large scale data processing
•YARN: Next Generation Execution
–Improves MapReduce performance –Supports new frameworks beyond MapReduce! –Low latency, Streaming, Services –Do more with a single Hadoop cluster
• Updates are stored to persistent journal • Lots of files bad
cks for client or namenode
File Storage
A given file is broken down into blocks (default=64MB), then blocks are replicated across cluster (default=3).
Optimized for: • Throughput • Put/Get/Delete • Appends Block Replication for: • Durability • Availability • Throughput
Block Replicas are distributed across servers and racks.
HDFS Architecture
Namenode & Datanodes
• Two different types of nodes in the
file system. • Namenode - stores all metadata and block locations in memory.
"Google Earth uses 70.5 TB: 70 TB for the raw imagery and 500 GB for the index data."
Google Solutions
• The Google File System • Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung • Appeared in:19th ACM Symposium on Operating Systems Principles, Lake George, NY, October, 2003. • /archive/gfs-sosp2003.pdf • MapReduce: Simplified Data Processing on Large Clusters • Jeffrey Dean and Sanjay Ghemawat • OSDI'04: Sixth Symposium on Operating System Design and Implementation, • San Francisco, CA, December, 2004. • /archive/mapreduce-osdi04.pdf
Key Characteristics •Scalable –Efficiently store and process petabytes of data –Linear scale driven by additional processing and storage •Reliable –Redundant storage –Failover across nodes and racks •Flexible –Store all types of data in any format –Apply schema on analysis and sharing of the data •Economical –Use commodity hardware –Open source software guards against vendor lock-in
关 雪 峰
测绘遥感国家重点实验室
Data-intensive computing
• Data-intensive computing is a class of parallel computing
applications which use a data parallel approach to processing large volumes of data typically terabytes or petabytes in size and typically referred to as big data. • Computing applications which devote most of their execution time to computational requirements are deemed compute-intensive, whereas computing applications which require large volumes of data and devote most of their processing time to I/O and manipulation of data are deemed data-intensive.
Fault Tolerance in HDFS layer
• Hardware failure is the norm rather than the exception • Detection of faults and quick, automatic recovery from them
is a core architectural goal of HDFS. • Master Slave Architecture with NameNode (master) and DataNode (slave) • Common types of failures
system used by Hadoop applications. HDFS creates multiple