hadoop倒排索引实验报告

大数据技术概论实验报告作业三姓名：郭利强专业：工程管理专业学号： 2015E8009064028目录1.实验要求 (3)2.环境说明 (4)2.1系统硬件 (4)2.2系统软件 (4)2.3集群配置 (4)3.实验设计 (4)3.1第一部分设计 (4)3.2第二部分设计 (6)4.程序代码 (11)4.1第一部分代码 (11)4.2第二部分代码 (17)5.实验输入和结果 (21)实验输入输出结果见压缩包中对应目录 (21)1.实验要求第一部分：采用辅助排序的设计方法，对于输入的N个IP网络流量文件，计算得到文件中的各个源IP地址连接的不同目的IP地址个数，即对各个源IP地址连接的目的IP地址去重并计数举例如下：第二部分：输入N个文件，生成带详细信息的倒排索引举例如下，有4个输入文件：– d1.txt: cat dog cat fox– d2.txt: cat bear cat cat fox– d3.txt: fox wolf dog– d4.txt: wolf hen rabbit cat sheep要求建立如下格式的倒排索引：– cat —>3: 4: {(d1.txt,2,4),(d2.txt,3,5),(d4.txt,1,5)}–单词—>出现该单词的文件个数:总文件个数： {(出现该单词的文件名，单词在该文件中的出现次数，该文件的总单词数),……}2.环境说明2.1系统硬件处理器：Intel Core i3-2350M CPU@2.3GHz×4 内存：2GB磁盘：60GB2.2系统软件操作系统：Ubuntu 14.04 LTS操作系统类型：32位Java版本：1.7.0_85Eclipse版本：3.8Hadoop插件：hadoop-eclipse-plugin-2.6.0.jar Hadoop：2.6.12.3集群配置集群配置为伪分布模式，节点数量一个3.实验设计3.1第一部分设计利用两个Map/Reduce过程，在第一个MR中，读取记录并去除重复记录，第二个MR按照辅助排序设计方法，根据源地址进行分组，统计目的地址数量。

第一个MR设计：自定义StringPair{源地址，目的地址}类型，实现WritableComparable，在map过程读取文件，输出<StringPair,NullWritable>,reduce过程去除重复记录输出<StringPair.toString, NullWritable >。

在第二个MR设计：1.在Map过程读取第一个MR的输出，对value值进行拆分，并以拆分得到的源地址和目的地址初始化StringPair对象作为输出键，输出值为1。

public void map(Object key, Text value, Context context)throws IOException, InterruptedException {String[] records = value.toString().split("\t");String sourceip = records[0];String desip=records[1];context.write(new StringPair(sourceip,desip),one);}2.定义GroupComparator类，继承WritableComparator类，并重载compare方法，对Map过程输出按照StringPair.first排序，完成按照源地址分组。

public static class GroupComparator extends WritableComparator {protected GroupComparator() {super(StringPair.class, true);}@Overridepublic int compare(WritableComparable w1,WritableComparable w2) {StringPair ip1=(StringPair)w1;StringPair ip2=(StringPair)w2;return ip1.getFirst().compareTo(ip2.getFirst());}}3.在Reduce过程统计分组中的所有值，得到源地址连接不同目的地址数量。

public void reduce( StringPair key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {int sum = 0;for (IntWritable val:values) {sum += val.get();}statistic.set(sum);context.write(key.getFirst(), statistic);}}3.2第二部分设计利用两个Map/Reduce过程，第一个MR统计各个文件中的所有单词的出现次数，以及各个文件单词总数，第二个MR根据统计结果处理加工得到单词倒排索引。

第一个MR设计：1.在Map过程中，重写map类，利用StringTokenizer类，将map 方法中的value值中存储的文本，拆分成一个个的单词，并获取文件名，以两种格式进行输出< filename+word,1>或者<filename,1>。

public void map(Object key, Text value, Context context)throws IOException, InterruptedException {//获取文件名FileSplit fileSplit= (FileSplit)context.getInputSplit();String fileName = fileSplit.getPath().getName();//获取单词在单个文件中出现次数，及文件单词总数StringTokenizer itr= new StringTokenizer(value.toString());for(; itr.hasMoreTokens(); ) {String word =removeNonLetters( itr.nextToken().toLowerCase());String fileWord = fileName+"\001"+word;if(!word.equals("")){context.write(new Text(fileWord), new IntWritable(1));context.write(new Text(fileName), new IntWritable(1));}}}2.在Reduce过程中，统计得到每个文件中每个单词的出现次数，以及每个文件的单词总数，输出<key,count>。

public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}context.write(key,new IntWritable(sum));}}第二个MR设计：1.Map过程读取第一个MR的输出，对value值进行拆分，重新组合后输出键为固定Text类型值index，值为filename+word+count或者filename+count。

public void map(Object key, Text value, Context context)throws IOException, InterruptedException {String valStr = value.toString();String[] records = valStr.split("\t");context.write(new Text("index"),new Text(records[0]+"\001"+records[1]));}}2.Reduce过程中定义四个HashMap，Map<String,Integer> wordinfilescount，key为单词+文件名，value为单词在该文件中出现的次数；Map<String,Integer> filescount ，key为文件名，value为文件的单词总数；Map<String,Integer> wordinfiles，key为单词，value为单词在多少个文件中出现；Map<String,String> indexes，key为单词，value为倒排索引。

读取values值，根据设定分隔符拆分，判断拆分后长度如果为2，则该值为文件名+文件单词总数，将拆分后的文件名及文件单词总数，组成键值对放入Map<String,Integer> filescount；拆分后长度如果为3，则该值为文件名+单词+单词在该文件中出现次数，将拆分后的文件名+单词及单词在该文件中出现次数组成键值对放入Map<String,Integer> wordinfilescount，同时统计单词在多少个文件中出现，并组成键值对放入Map<String,Integer> wordinfiles。

遍历Map<String,Integer> wordinfilescount，将单词作为键，“单词->出现该单词的文件个数:总文件个数：{(出现该单词的文件名，单词在该文件中的出现次数，该文件的总单词数)”作为值，放入Map<String,String> indexes中。

遍历Map<String,String> indexes获取倒排索引并输出全部索引。

public void reduce( Text key, Iterable<Text> values, Context context)throws IOException, InterruptedException {//拆分输入，获取单词出现在几个文件中以及在该文件中出现次数，各个文件的单词总数，总文件数for (Text val : values) {String valStr = val.toString();String[] records = valStr.split("\001");switch(records.length){case 2:filescount.put(records[0], Integer.parseInt(records[1]));break;case 3:{wordinfilescount.put(valStr, Integer.parseInt(records[2]));if(!wordinfiles.containsKey(records[1])){wordinfiles.put(records[1], 1);}else{wordinfiles.put(records[1], wordinfiles.get(records[1])+1);}};break;}}//处理获取倒排索引for (Entry<String, Integer> entry : wordinfilescount.entrySet()) {String valStr = entry.getKey();String[] records = valStr.split("\001");String word = records[1];if(!indexes.containsKey(word)){StringBuilder sb = new StringBuilder();sb.append(word).append("->").append(wordinfiles.get(word)).append(":").append(filescount.size()).append(":").append("{(").append( records[0]).append(",").append(entry.getValue()).append(",").append(filescount.get( records[0])).append(")");indexes.put(word,sb.toString() );}else{StringBuilder sb = new StringBuilder();sb.append(",(").append( records[0]).append(",").append(entry.getValue()).append(",").append(filescount.get( records[0])).append(")");indexes.put(word,indexes.get(word)+sb.toString() ); }}for (Entry<String, String> entry : indexes.entrySet()) {context.write(new Text(entry.getValue()+"}"), NullWritable.get());}}}4.程序代码4.1第一部分代码1.IpStatistics.java/*** Licensed to the Apache Software Foundation (ASF) under one* or more contributor license agreements. See the NOTICE file* distributed with this work for additional information* regarding copyright ownership. The ASF licenses this file* to you under the Apache License, Version 2.0 (the* "License"); you may not use this file except in compliance* with the License. You may obtain a copy of the License at** /licenses/LICENSE-2.0** Unless required by applicable law or agreed to in writing, software* distributed under the License is distributed on an "AS IS" BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.* See the License for the specific language governing permissions and* limitations under the License.*/import java.io.IOException;import java.util.ArrayList;import java.util.Collections;import parator;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Map.Entry;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.io.WritableComparable;import org.apache.hadoop.io.WritableComparator;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;import org.apache.hadoop.fs.Path;public class IpStatistics {//第一个Map/Reduce的map类，用于去重public static class RemoveMapper extendsMapper<Object, Text, StringPair, NullWritable> {public void map(Object key, Text value, Context context)throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {String nextToken =itr.nextToken();String[] records = nextToken.split(",");String sourceip = records[0].replace("<", "");String destinationip = records[1].replace(">", "");context.write(newStringPair(sourceip,destinationip),NullWritable.get() );}}}//第二个Map/Reduce过程map类，用于统计public static class StatisticsMapper extendsMapper<Object, Text, StringPair, IntWritable> {IntWritable one=new IntWritable(1);public void map(Object key, Text value, Context context)throws IOException, InterruptedException {String[] records = value.toString().split("\t");String sourceip = records[0];String desip=records[1];context.write(new StringPair(sourceip,desip),one);}}//按照源地址分组public static class GroupComparator extends WritableComparator {protected GroupComparator() {super(StringPair.class, true);}@Overridepublic int compare(WritableComparable w1,WritableComparable w2) {StringPair ip1=(StringPair)w1;StringPair ip2=(StringPair)w2;return ip1.getFirst().compareTo(ip2.getFirst());}}//第一个Map/Reduce过程reduce过程，去重public static class RemoveReducer extendsReducer<StringPair, IntWritable, Text, NullWritable> {public void reduce( StringPair key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {context.write(new Text(key.toString()), NullWritable.get());}}//第二个Map/Reduce过程reduce过程，统计public static class StatisticsReducer extendsReducer<StringPair, IntWritable, Text, IntWritable> {private IntWritable statistic = new IntWritable();public void reduce( StringPair key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException {int sum = 0;for (IntWritable val:values) {sum += val.get();}statistic.set(sum);context.write(key.getFirst(), statistic);}}//去重任务public static void RemoveTask(String[] args) throws Exception{Configuration conf = new Configuration();String master="127.0.0.1";conf.set("fs.defaultFS", "hdfs://127.0.0.1:9000");conf.set("er", "hadoop");conf.set("","yarn");conf.set("yarn.resourcemanager.address", master+":8032");conf.set("yarn.resourcemanager.scheduler.address", master+":8030");conf.set("mapred.jar","ipstatistics.jar");String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length < 3) {System.err.println("Usage: ipstatistics <in> [<in>...] <out>");System.exit(2);}Job job = new Job(conf, "ipstatisticsRemoving");job.setMapperClass(RemoveMapper.class);job.setReducerClass(RemoveReducer.class);job.setOutputKeyClass(StringPair.class);job.setOutputValueClass(NullWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));job.waitForCompletion(true) ;}//统计任务public static void StatisticsTask(String[] args) throws Exception{ Configuration conf = new Configuration();String master="127.0.0.1";conf.set("fs.defaultFS", "hdfs://127.0.0.1:9000");conf.set("er", "hadoop");conf.set("","yarn");conf.set("yarn.resourcemanager.address", master+":8032");conf.set("yarn.resourcemanager.scheduler.address", master+":8030");conf.set("mapred.jar","ipstatistics.jar");String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length < 3) {System.err.println("Usage: ipstatistics <in> [<in>...] <out>");System.exit(2);}Job job = new Job(conf, "ipstatisticsStatistics");job.setMapperClass(StatisticsMapper.class);job.setGroupingComparatorClass(GroupComparator.class);job.setReducerClass(StatisticsReducer.class);job.setOutputKeyClass(StringPair.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[1]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[2]));System.exit(job.waitForCompletion(true) ? 0 : 1);}public static void main(String[] args) throws Exception {RemoveTask(args);StatisticsTask(args);}}2.StringPair.javaimport org.apache.hadoop.io.Text;import org.apache.hadoop.io.WritableComparable;import java.io.DataInput;import java.io.DataOutput;import java.io.IOException;public class StringPair implements WritableComparable<StringPair> {private Text first;private Text second;public StringPair() {this.first = new Text();this.second = new Text();}public StringPair(String first, String second) {set(new Text(first), new Text(second));}public StringPair(Text first, Text second) {set(first, second);}public void set(Text first, Text second) {this.first = first;this.second = second;}public Text getFirst() {return first;}public Text getSecond() {return second;}public void write(DataOutput out) throws IOException {first.write(out);second.write(out);}public void readFields(DataInput in) throws IOException {first.readFields(in);second.readFields(in);}@Overridepublic int hashCode() {return first.hashCode() * 163 + second.hashCode();}@Overridepublic boolean equals(Object obj) {if (obj instanceof StringPair) {StringPair ip = (StringPair) obj;return first.toString().equals(ip.first.toString())&& second.toString().equals(ip.second.toString());}4.2第二部分代码import java.io.IOException;import java.util.HashSet;import java.util.Map;import java.util.Set;import java.util.StringTokenizer;import java.util.HashMap;import java.util.Map.Entry;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.NullWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.Mapper.Context;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.input.FileSplit;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class InverseV3 {/** 第一个mr的map类，获取每个单词在单个文件中出现次数，输入为每个文件行偏移量，输出为<word+filename,1>* 或<filename,1>*/public static class statisticsMap extendsMapper<Object, Text, Text, IntWritable> {private Text mapKey = new Text("key");@Overridepublic void map(Object key, Text value, Context context)throws IOException, InterruptedException {//获取文件名FileSplit fileSplit= (FileSplit)context.getInputSplit();String fileName = fileSplit.getPath().getName();//获取单词在单个文件中出现次数，及文件单词总数StringTokenizer itr= new StringTokenizer(value.toString());for(; itr.hasMoreTokens(); ) {String word =removeNonLetters( itr.nextToken().toLowerCase());String fileWord = fileName+"\001"+word;if(!word.equals("")){context.write(new Text(fileWord), new IntWritable(1));context.write(new Text(fileName), new IntWritable(1));}}}//去掉字符串非字母字符public static String removeNonLetters(String original){StringBuffer aBuffer=new StringBuffer(original.length());char aCharacter;for(int i=0;i<original.length();i++){aCharacter=original.charAt(i);if(Character.isLetter(aCharacter)){aBuffer.append(aCharacter);}}return new String(aBuffer);}}//第一个mr的reduce类，统计汇总出现单词的文件个数，及每个文件中单词出现个数及每个文件单词个数,public static class statisticsReduce extendsReducer<Text, IntWritable, Text, IntWritable> {@Overridepublic void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}context.write(key,new IntWritable(sum));}}public static class InverseMapper extendsMapper<Object, Text, Text, Text> {public void map(Object key, Text value, Context context)throws IOException, InterruptedException {String valStr = value.toString();String[] records = valStr.split("\t");context.write(new Text("index"),new Text(records[0]+"\001"+records[1]));}}public static class InverseReducer extendsReducer<Text, Text, Text, NullWritable> {private Map<String,Integer> wordinfilescount = newHashMap<String,Integer>();//key为单词+文件名，value为单词在该文件中出现的次数private Map<String,Integer> filescount = new HashMap<String,Integer>();//key为文件名，value为文件的单词总数private Map<String,Integer> wordinfiles = new HashMap<String,Integer>();//key为单词，value为单词在多少的文件中出现private Map<String,String> indexes = new HashMap<String,String>();//key为单词，value为倒排索引public void reduce( Text key, Iterable<Text> values, Context context)throws IOException, InterruptedException {//拆分输入，获取单词出现在几个文件中以及在该文件中出现次数，各个文件的单词总数，总文件数for (Text val : values) {String valStr = val.toString();String[] records = valStr.split("\001");switch(records.length){case 2:filescount.put(records[0], Integer.parseInt(records[1]));break;case 3:{wordinfilescount.put(valStr, Integer.parseInt(records[2]));if(!wordinfiles.containsKey(records[1])){wordinfiles.put(records[1], 1);}else{wordinfiles.put(records[1], wordinfiles.get(records[1])+1);}};break;}}//处理获取倒排索引for (Entry<String, Integer> entry : wordinfilescount.entrySet()) {String valStr = entry.getKey();String[] records = valStr.split("\001");String word = records[1];if(!indexes.containsKey(word)){StringBuilder sb = new StringBuilder();sb.append(word).append("->").append(wordinfiles.get(word)).append(":").append(filescount.size()).append(":").append("{(").append( records[0]).append(",").append(entry.getValue()).append(",").append(filescount.get( records[0])).append(")");indexes.put(word,sb.toString() );}else{StringBuilder sb = new StringBuilder();sb.append(",(").append( records[0]).append(",").append(entry.getValue()).append(",").append(filescount.get( records[0])).append(")");indexes.put(word,indexes.get(word)+sb.toString() );}}for (Entry<String, String> entry : indexes.entrySet()) {context.write(new Text(entry.getValue()+"}"), NullWritable.get());}}}//统计单词在文件中出现次数及单个文件单词总数public static void StatisticsTask(String[] args) throws Exception{ Configuration conf = new Configuration();String master="127.0.0.1";conf.set("fs.defaultFS", "hdfs://127.0.0.1:9000");conf.set("er", "hadoop");conf.set("","yarn");conf.set("yarn.resourcemanager.address", master+":8032");conf.set("yarn.resourcemanager.scheduler.address", master+":8030");conf.set("mapred.jar","invertdindex.jar");String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length < 3) {System.err.println("Usage: ipstatistics <in> [<in>...] <out>");System.exit(2);}Job job = new Job(conf, "invertd index1");job.setMapperClass(statisticsMap.class);job.setReducerClass(statisticsReduce.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs[0]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));job.waitForCompletion(true) ;}//根据统计结果输出倒排索引public static void InverseTask(String[] args) throws Exception{ Configuration conf = new Configuration();String master="127.0.0.1";conf.set("fs.defaultFS", "hdfs://127.0.0.1:9000");conf.set("er", "hadoop");conf.set("","yarn");conf.set("yarn.resourcemanager.address", master+":8032");conf.set("yarn.resourcemanager.scheduler.address", master+":8030");conf.set("mapred.jar","invertdindex.jar");String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();if (otherArgs.length < 3) {System.err.println("Usage: ipstatistics <in> [<in>...] <out>");System.exit(2);}Job job = new Job(conf, "invertd index2");job.setMapperClass( InverseMapper.class);job.setReducerClass( InverseReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(Text.class);FileInputFormat.addInputPath(job, new Path(otherArgs[1]));FileOutputFormat.setOutputPath(job, new Path(otherArgs[2]));System.exit(job.waitForCompletion(true) ? 0 : 1);}public static void main(String[] args) throws Exception {StatisticsTask(args);InverseTask(args);}}5.实验输入和结果实验输入输出结果见压缩包中对应目录。