Data Mining
大数据外文翻译参考文献综述
大数据外文翻译参考文献综述(文档含中英文对照即英文原文和中文翻译)原文：Data Mining and Data PublishingData mining is the extraction of vast interesting patterns or knowledge from huge amount of data. The initial idea of privacy-preserving data mining PPDM was to extend traditional data mining techniques to work with the data modified to mask sensitive information. The key issues were how to modify the data and how to recover the data mining result from the modified data. Privacy-preserving data mining considers the problem of running data mining algorithms on confidential data that is not supposed to be revealed even to the partyrunning the algorithm. In contrast, privacy-preserving data publishing (PPDP) may not necessarily be tied to a specific data mining task, and the data mining task may be unknown at the time of data publishing. PPDP studies how to transform raw data into a version that is immunized against privacy attacks but that still supports effective data mining tasks. Privacy-preserving for both data mining (PPDM) and data publishing (PPDP) has become increasingly popular because it allows sharing of privacy sensitive data for analysis purposes. One well studied approach is the k-anonymity model [1] which in turn led to other models such as confidence bounding, l-diversity, t-closeness, (α,k)-anonymity, etc. In particular, all known mechanisms try to minimize information loss and such an attempt provides a loophole for attacks. The aim of this paper is to present a survey for most of the common attacks techniques for anonymization-based PPDM & PPDP and explain their effects on Data Privacy.Although data mining is potentially useful, many data holders are reluctant to provide their data for data mining for the fear of violating individual privacy. In recent years, study has been made to ensure that the sensitive information of individuals cannot be identified easily.Anonymity Models, k-anonymization techniques have been the focus of intense research in the last few years. In order to ensure anonymization of data while at the same time minimizing the informationloss resulting from data modifications, everal extending models are proposed, which are discussed as follows.1.k-Anonymityk-anonymity is one of the most classic models, which technique that prevents joining attacks by generalizing and/or suppressing portions of the released microdata so that no individual can be uniquely distinguished from a group of size k. In the k-anonymous tables, a data set is k-anonymous (k ≥ 1) if each record in the data set is in- distinguishable from at least (k . 1) other records within the same data set. The larger the value of k, the better the privacy is protected. k-anonymity can ensure that individuals cannot be uniquely identified by linking attacks.2. Extending ModelsSince k-anonymity does not provide sufficient protection against attribute disclosure. The notion of l-diversity attempts to solve this problem by requiring that each equivalence class has at least l well-represented value for each sensitive attribute. The technology of l-diversity has some advantages than k-anonymity. Because k-anonymity dataset permits strong attacks due to lack of diversity in the sensitive attributes. In this model, an equivalence class is said to have l-diversity if there are at least l well-represented value for the sensitive attribute. Because there are semantic relationships among the attribute values, and different values have very different levels of sensitivity. Afteranonymization, in any equivalence class, the frequency (in fraction) of a sensitive value is no more than α.3. Related Research AreasSeveral polls show that the public has an in- creased sense of privacy loss. Since data mining is often a key component of information systems, homeland security systems, and monitoring and surveillance systems, it gives a wrong impression that data mining is a technique for privacy intrusion. This lack of trust has become an obstacle to the benefit of the technology. For example, the potentially beneficial data mining re- search project, Terrorism Information Awareness (TIA), was terminated by the US Congress due to its controversial procedures of collecting, sharing, and analyzing the trails left by individuals. Motivated by the privacy concerns on data mining tools, a research area called privacy-reserving data mining (PPDM) emerged in 2000. The initial idea of PPDM was to extend traditional data mining techniques to work with the data modified to mask sensitive information. The key issues were how to modify the data and how to recover the data mining result from the modified data. The solutions were often tightly coupled with the data mining algorithms under consideration. In contrast, privacy-preserving data publishing (PPDP) may not necessarily tie to a specific data mining task, and the data mining task is sometimes unknown at the time of data publishing. Furthermore, some PPDP solutions emphasize preserving the datatruthfulness at the record level, but PPDM solutions often do not preserve such property. PPDP Differs from PPDM in Several Major Ways as Follows ：1) PPDP focuses on techniques for publishing data, not techniques for data mining. In fact, it is expected that standard data mining techniques are applied on the published data. In contrast, the data holder in PPDM needs to randomize the data in such a way that data mining results can be recovered from the randomized data. To do so, the data holder must understand the data mining tasks and algorithms involved. This level of involvement is not expected of the data holder in PPDP who usually is not an expert in data mining.2) Both randomization and encryption do not preserve the truthfulness of values at the record level; therefore, the released data are basically meaningless to the recipients. In such a case, the data holder in PPDM may consider releasing the data mining results rather than the scrambled data.3) PPDP primarily “anonymizes” the data by hiding the identity of record owners, whereas PPDM seeks to directly hide the sensitive data. Excellent surveys and books in randomization and cryptographic techniques for PPDM can be found in the existing literature. A family of research work called privacy-preserving distributed data mining (PPDDM) aims at performing some data mining task on a set of private databasesowned by different parties. It follows the principle of Secure Multiparty Computation (SMC), and prohibits any data sharing other than the final data mining result. Clifton et al. present a suite of SMC operations, like secure sum, secure set union, secure size of set intersection, and scalar product, that are useful for many data mining tasks. In contrast, PPDP does not perform the actual data mining task, but concerns with how to publish the data so that the anonymous data are useful for data mining. We can say that PPDP protects privacy at the data level while PPDDM protects privacy at the process level. They address different privacy models and data mining scenarios. In the field of statistical disclosure control (SDC), the research works focus on privacy-preserving publishing methods for statistical tables. SDC focuses on three types of disclosures, namely identity disclosure, attribute disclosure, and inferential disclosure. Identity disclosure occurs if an adversary can identify a respondent from the published data. Revealing that an individual is a respondent of a data collection may or may not violate confidentiality requirements. Attribute disclosure occurs when confidential information about a respondent is revealed and can be attributed to the respondent. Attribute disclosure is the primary concern of most statistical agencies in deciding whether to publish tabular data. Inferential disclosure occurs when individual information can be inferred with high confidence from statistical information of the published data.Some other works of SDC focus on the study of the non-interactive query model, in which the data recipients can submit one query to the system. This type of non-interactive query model may not fully address the information needs of data recipients because, in some cases, it is very difficult for a data recipient to accurately construct a query for a data mining task in one shot. Consequently, there are a series of studies on the interactive query model, in which the data recipients, including adversaries, can submit a sequence of queries based on previously received query results. The database server is responsible to keep track of all queries of each user and determine whether or not the currently received query has violated the privacy requirement with respect to all previous queries. One limitation of any interactive privacy-preserving query system is that it can only answer a sublinear number of queries in total; otherwise, an adversary (or a group of corrupted data recipients) will be able to reconstruct all but 1 . o(1) fraction of the original data, which is a very strong violation of privacy. When the maximum number of queries is reached, the query service must be closed to avoid privacy leak. In the case of the non-interactive query model, the adversary can issue only one query and, therefore, the non-interactive query model cannot achieve the same degree of privacy defined by Introduction the interactive model. One may consider that privacy-reserving data publishing is a special case of the non-interactivequery model.This paper presents a survey for most of the common attacks techniques for anonymization-based PPDM & PPDP and explains their effects on Data Privacy. k-anonymity is used for security of respondents identity and decreases linking attack in the case of homogeneity attack a simple k-anonymity model fails and we need a concept which prevent from this attack solution is l-diversity. All tuples are arranged in well represented form and adversary will divert to l places or on l sensitive attributes. l-diversity limits in case of background knowledge attack because no one predicts knowledge level of an adversary. It is observe that using generalization and suppression we also apply these techniques on those attributes which doesn’t need th is extent of privacy and this leads to reduce the precision of publishing table. e-NSTAM (extended Sensitive Tuples Anonymity Method) is applied on sensitive tuples only and reduces information loss, this method also fails in the case of multiple sensitive tuples.Generalization with suppression is also the causes of data lose because suppression emphasize on not releasing values which are not suited for k factor. Future works in this front can include defining a new privacy measure along with l-diversity for multiple sensitive attribute and we will focus to generalize attributes without suppression using other techniques which are used to achieve k-anonymity because suppression leads to reduce the precision ofpublishing table.译文：数据挖掘和数据发布数据挖掘中提取出大量有趣的模式从大量的数据或知识。
data mining 4
TaskTask-relevant data
this is the database portion to be investigated database or data warehouse name database tables or data warehouse cube conditions for data selection relevant attributes or dimensions data grouping criteria
2010-5-20 13
Subjective interestingness measures (I)
Actionability
a pattern is interesting if the user can act on it to his advantage actionability is an elusive concept that is very difficult to capture formally because
accuracy for classification rules
2010-5-20 12
Objective interestingness measures (II)
Utility
the potential usefulness of the pattern e.g support for association rules
2010-5-20 6
Background knowledge (II)
operation-derived hierarchy
a whole or a portion of hierarchy derived from operations specified by users, experts, or the data mining system
data mining 练习题
1.Below is a table representing eight transactions and five items: Beer, Coke, Pepsi,Milk, and Juice. The items are represented by their first letters; e.g., "M" = milk.which of the pairs are frequent itemsets?2.Here is a table with seven transactions and six items, A through F. An "x"indicates that the item is in the transaction.A B C D E Fx x x x xx x x x xx x x xx x xx x x x xx x xx x xAssume that the support threshold is 2. Find all the closed frequent itemsets. Answer: There are many ways to find the frequent itemsets, but the amount of data is small, so we'll just list the results.Among the pairs, all but AF are frequent. The counts are:AC, CE: 5AE, CD: 4AD, BC, BD, BE, BF, DE: 3AB, CF, DF, EF: 2Here are the counts of the frequent triples:ACE: 4ACD, BCE, CDE: 3ABC, ABE, ADE, BCD, BDE, BDF, BCF, BEF, CEF: 2There are four quadruples that are frequent, all with counts of 2: BCEF, BCDE, ACDE, and ABCE. There are no frequent sets of five items.To be closed, the itemset must have a larger count than all of its immediate supersets. Thus, all four of the listed quadruples are closed. A triple with a count of 2 cannot be closed unless it is contained in none of the four frequent quadruples. Among these, only BDF qualifies as closed. However, each of the triples with a count of 3 or 4 is closed, since there are no quadruples with counts this high.Among the pairs, only AC, CD, BD, and BF are closed. Among the singletons, only A and F are not closed. A, which appears 5 times, is contained in AC, which also occurs 5 times, and F, which occurs 3 times, is not closed because BF also appears 3 times.3. Find the set of 2-shingles for the "document":ABRACADABRAand also for the "document":BRICABRACAnswer the following questions:1.How many 2-shingles does ABRACADABRA have?2.How many 2-shingles does BRICABRAC have?3.How many 2-shingles do they have in common?4.What is the Jaccard similarity between the two documents"?Answer: The 2-shingles for ABRACADABRA: AB, BR, RA, AC, CA, AD, DA. The 2-shingles for BRICABRAC: BR, RI, IC, CA, AB, RA, AC.There are 5 shingles in common:AB, BR, RA, AC, CA.As there are 9 different shingles in all, the Jaccard similarity is 5/9.State the correct minhash value of each column.Answer: Look at the rows in the stated order R4, R6, R1, R3, R5, R2, and for each row, make that row be the minhash value of a column if the column has not yet beenassigned a minhash value. We sart with R4, which only has 1 in column C3, so the minhash value for C3 is R4.Next, we consider R6, which has 1 in C2 only. Since C2 does not yet have a minhash value, R6 becomes its value.Next is R1, with 1's in C2 and C3. However, both these columns already have minhash values, so we do nothing.Next, consider R3. It has 1's in C2 and C4. C2 already has a minhash value, but C4 does not. Thus, the minhash value of C4 is R3.When we consider R5 next, we see it has 1's in C1 and C3. The latter already has a minhash value, but R5 becomes the minhash value for C1. Since all columns now have minhash values, we are done.5. Perform a hierarchical clustering of the following six points:using the centroid proximity measure (distance between two clusters is the distance between their centroids). If you do this task correctly, you will find that there is a stage at which there is a tie for which pair of clusters is closest. Follow both choices. You will find that some sets of points are clusters in both cases, some sets are clusters in only one, and some are not clusters regardless of which choice you make. Answer: First, A and B, being the closest pair of points gets merged. The centroid for this pair is at (5,5). The next closest pair of centroids is C and F, so these are merged and their centroid is at (24.5, 13.5). At this time, there is a tie for closest centroids. AB and CF have centroids at distance sqrt(452.5), and so do D and CF. Thus, there are two possible third merges:1.Merge AB and CF, giving three clusters ABCF, D, and E. The centroid of ABCF is(14.75, 9.25). In this case, the next merge is ABCD with E.2.Merge CF with D, giving three clusters CDF, AB, and E. The centroid of CDF is at(27.33, 20). In this case, the next merge is E with AB.As a result, the two sequences of clusters created are:1.AB, CF, ABCF, ABCEF, ABCDEF.2.AB, CF, CDF, ABE, ABCDEF.6.Consider three Web pages with the following links:Suppose we compute PageRank with a β of 0.7, and we introduce the additional constraint that the sum of the PageRanks of the three pages must be 3, to handle the problem that otherwise any multiple of a solution will also be a solution. Compute the PageRanks a, b, and c of the three pages A, B, and C, respectively.Answer: The rules for computing the next value of a, b, or c as we iterate are:a <- .3b <- .7(a/2) + .3c <- .7(a/2+b+c) + .3The reason is that a splits its PageRank between b and c, while b gives all of its to c, and c keeps all its own. However, all PageRank is multiplied by .7 before distribution (the "tax"), and .3 is then added to each new PageRank.In the limit, the assignments become equalities. That immediately tells us a = .3. We can then use the second equation to discover b = .7*.3/2 + .3 = .405. Finally, the third equation simplifies to c = .7(.555 + c) + .3, or .3c = .6885. From this equation we get c = 2.295. It is now a simple matter to compute the subs of each two of the variables: a+b = .705, a+c = 2.595, and b+c = 2.7.7. 分析我们提到的各种算法的优缺点。
data mining 1
Northeastern University
2009-12-3 1
About Book
原版
Morgan Kaufmann Publishers, Inc.
影印版
高等教育出版社
中译版
机械工业出版社
2009-12-3
2
About Author
Professor (2001-) Department of Computer Science Univ. of Illinois at Urbana-Champaign Professor (1987-2001) Intelligent Database Systems Research Lab. School of Computing Science Simon Fraser University, Canada Ph.D. (1985) Department of Computing Science University Of Wisconsin-Madison Current Research Data mining, data warehousing, and knowledge discovery in databases Spatial and multi-media data mining WWW Technology: Weblog mining and Web Mining DNA data mining and bio-informatics Deductive and object_oriented databases
1991-1994 Workshops on Knowledge Discovery in Databases
Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1996)
DataMining
TECS 2007, Data Mining
R. Ramakrishnan, Yahoo! Research Bee-Chung Chen, Raghu Ramakrishnan, Jude Shavlik, Pradeep Tamma
12
What is a Data Mining Model?
A data mining model is a description of a certain aspect of a dataset. It produces output values for an assigned set of inputs.
Bellwether Analysis
Data Mining
(with many slides due to Gehrke, Garofalakis, Rastogi)
Raghu Ramakrishnan Yahoo! Research University of Wisconsin–Madison (on leave)
6ห้องสมุดไป่ตู้
Case Study: Bank (Contd.)
3. Group customers into clusters and investigate clusters
Group 2 Group 3 Group 1
Group 4
TECS 2007, Data Mining R. Ramakrishnan, Yahoo! Research Bee-Chung Chen, Raghu Ramakrishnan, Jude Shavlik, Pradeep Tamma
TECS 2007
R. Ramakrishnan, Yahoo! Research
Data Mining：Concepts and Techniques
Types of Outliers (I)
Three kinds: global, contextual and collective outliers Global Outlier Global outlier (or point anomaly) Object is Og if it significantly deviates from the rest of the data set Ex. Intrusion detection in computer networks Issue: Find an appropriate measurement of deviation Contextual outlier (or conditional outlier) Object is Oc if it deviates significantly based on a selected context o Ex. 80 F in Urbana: outlier? (depending on summer or winter?) Attributes of data objects should be divided into two groups Contextual attributes: defines the context, e.g., time & location Behavioral attributes: characteristics of the object, used in outlier evaluation, e.g., temperature Can be viewed as a generalization of local outliers—whose density significantly deviates from its local area Issue: How to define or formulate meaningful context?
DataMiningofVeryLargeData
• Run a-priori or one of its improvements (for sets of all sizes, not just pairs) in main memory, so you don’t pay for disk I/O each time you increase the size of itemsets.
• We may proceed beyond frequent pairs to find frequent triples, quadruples, . . .
– Key a-priori idea: a set of items S can only be frequent if S - {a } is frequent for all a in S .
– Items that appear together too often could represent plagiarism.
5Leabharlann Applications 3
• “Baskets” = Web pages; “items” = linked pages.
– Pairs of pages with many common references may be about the same topic.
• Data is stored in a file, basket-by-basket.
– As we read the file one basket at a time, we can generate all the sets of items in that basket.
Lecture数据仓库与OLAP技术概述
Data Mining: Concepts and Techniques
8
OLTP
OLAP
用户 功能
员工, IT专业人员 每天的日常操作
知识工作者 决策支持
DB设计 数据
使用
面向应用+ER 当前的，详细的数据
重复的
面向主题+Star
历史的, 汇总的, 多维 的集成的, 整理过的
特定的
访问 工作单元
数据挖掘: 概念与技术
— 第三、四章 —
王家兵 博士 华南理工大学计算机科学与工程学院
E-mail: jbwang@
2020年5月12日星期二
Data Mining: Concepts and Techniques
1
Lecture 3: 数据仓库、 OLAP及数据立方 体计算
读/写、索引
多次扫描
短的, 简单的事务处理 复杂查询
记录数/查询 几十
百万
用户数
上千
百
DB规模
100MB-GB
100GB-TB
metric
transaction throughput
query throughput, response
2020年5月12日星期二
Data Mining: Concepts and Techniques
2020年5月12日星期二
Data Mining: Concepts and Techniques
13
立方体: 方体格
all
time
item
location supplier
0-D(apex) cuboid 1-D cuboids
time,item
搜索引擎的检索方法与技巧
搜索引擎的检索方法与技巧
引擎技巧是可以帮助我们更有效率地信息的方法，可以从简单的关键词、更复杂的组合查询到更复杂的检索方法，有效地定位到结果，以节省
时间，提高检索效率。
下面介绍几种引擎检索方法与技巧。
1、完整词组：使用完整词组来定位相关结果，可以有效避免结果中
返回的非想要的相关内容，多数引擎都支持“单引号”包围定位完整词组，比如“data mining”（data mining）就是data mining这个词组内容，
它会在结果中列出包含这两个词的内容，而不是data和mining两个单词
的内容。
2、相关词：相关词技巧可以有效检索出相关联的结果，比如如果你
要“水平对比”，你可以使用相关词技巧“水平”、“比较”、“对照”
等等词组，这样可以更好地找到想要的结果。
3、通配符：通配符技巧可以使用特殊符号*，?来代替一段字符来相
关的结果，比如使用“data*mining”来
datamining,dataengineering,dataanalysis等词组，使用
“data?mining”来datamining,dataamining,databmining等词组。
4、精确：精确技巧可以使用多个词紧凑组合完成精确，比如使用“data+mining”来data mining这个精确词组。
data mining习题
附件三：课程部分习题附件三：课程部分习题1. The following table consists of training data from an employee database. The data have been generalized. For a given row entry, count represents the number of data tuples having the values for department, status, age, and salary given in that row.Let status be the class label attributes.(a) How would you modify the ID3 algorithm to take into consideration the count of each generalized data tuple (i.e., of each row entry)?(b) Use your modified version of ID3 to construct a decision tree from the given data.2. A database has 4 transactions. Let min-support =60%, min_confidence=80%. Find the longest frequent itemset(s). List all association rules that satisfy the above requirement, with supports and confidence.TID Data Items bought-----------------------------------------------------------T100 10/15/99 K, A, D, BT200 10/15/99 D, A, C, E, BT300 10/19/99 C, A, B, ET400 10/22/99 B, A, D3. Perform the third iteration of the k-means algorithm for the example given in the section “An Example Using K-Means”. What are the new cluster centers?4. Suppose that the data mining task is to cluster the following 8 points (with (x,y) representing location) into 3 clusters.A1(2,10), A2(2,5), A3(8,4), B1(5,8), B2(7,5), B3(6,4), C1(1,2), C2(4,9)The distance function is Manhattan distance. Suppose initially we assign A1, B1, and C1 as the center of each cluster, respectively. Use the k-means algorithm to show only:(a) the three cluster centers after the first round execution;(b) the final three clusters.5. Consider the feed-forward network in Figure 8.1 with the associated connection weights shown in Table 8.1. Apply the input instance [0.5, 0.2, 1.0] to the feed-forward neural network. r=0.5, Tk = 0.65. Specifically,(a) Compute the input to node i and j;(b) Use the sigmoid function to compute the initial output of nodes i and j;(c) Use the ouput values computed in part b to determine the input and putputvalues for node k;(d) Adjust all weights for one epochOther exercises see text book.。
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
六、数据挖掘的过程
1. 确定业务对象 2. 数据准备 1) 数据的选择 2) 数据的预处理 3) 数据的转换 3. 数据挖掘 4. 结果分析 5. 知识的同化 将分析所得到的知识集成到业务信息系统的组 织结构中去。
An OLAM Architecture
Mining query
User GUI API
Filtering
Layer1 Databases
Data Data integration Warehouse
Data Repository
七、数据挖掘的研究热点
• 应用的探索（由于通用数据挖掘系统在处理特定
应用问题时有其局限性以及开发难度，目前的一种趋
势是开发针对特定应用的数据挖掘系统）； • • 可伸缩的数据挖掘方法； 数据挖掘与数据库系统、数据仓库系统和Web数
ID3算法举例
3、预测型知识（Prediction） 它根据时间序列型数据，由历史的和当前的数据 去推测未来的数据，也可以认为是以时间为关键属性 的关联知识。 目前，时间序列预测方法有经典的统计方法、神 经网络和机器学习等。1968年Box和Jenkins提出了一 套比较完善的时间序列建模理论和分析方法，这些经 典的数学方法通过建立随机模型，如自回归模型、自 回归滑动平均模型、求和自回归滑动平均模型和季节 调整模型等，进行时间序列的预测。由于大量的时间 序列是非平稳的，其特征参数和数据分布随着时间的 推移而发生变化。
关联知识挖掘举例
2、分类知识(Classification＆Clustering) 它反映同类事物共同性质的特征型知识和不同事 物之间的差异型特征知识。最为典型的分类方法是基 于决策树的分类方法。它是从实例集中构造决策树， 是一种有指导的学习方法。该方法先根据训练子集 （又称为窗口）形成决策树。如果该树不能对所有对 象给出正确的分类，那么选择一些例外加入到窗口中， 重复该过程一直到形成正确的决策集。最终结果是一 棵树，其叶结点是类名，中间结点是带有分枝的属性， 该分枝对应该属性的某一可能值。最为典型的决策树 学习系统是ID3，它采用自顶向下不回溯策略，能保 证找到一个简单的树。
因此，仅仅通过对某段历史数据的训练，建立单 一的神经网络预测模型，还无法完成准确的预测任务。
为此，人们提出了基于统计学和基于精确性的再训练
方法，当发现现存预测模型不再适用于当前数据时， 对模型重新训练，获得新的权重参数，建立新的模型。 也有许多系统借助并行算法的计算优势进行时间序列 预测。
4、偏差型知识(Deviation)
三、数据挖掘技术的发展历史 从数据库中发现知识（KDD）一词首次出现在 1989年举行的第11届国际联合人工智能学术会议上。 1995年在加拿大召开了第一届知识发现（KDD） 和数据挖掘（DM）国际学术会议以后，“数据挖掘” 开始流行 。 KDD过程定义为：在数据中鉴别出有用模式的 非平凡过程，该模式是新的，可能是有用的和最终可 理解的。KDD过程可用下图表示。而数据挖掘被认 为是KDD过程中的一个特定步骤，它用专门算法从 数据中抽取模式。
商业数据库现在正在以一个空前的速度增长，并 且数据仓库正在广泛地应用于各种行业。另外数据挖
掘算法经过了这10多年的发展也已经成为一种成熟，
稳定，且易于理解和操作的技术。
数据挖掘技术的演变过程 在数据处理的初期，人们就试图通过某些方法来 实现自动决策支持。当时机器学习成为人们关心的焦 点。机器学习的过程就是将一些已知的并已被成功解 决的问题作为范例输入计算机，机器通过学习这些范 例总结并生成相应的规则，这些规则具有通用性，使 用它们可以解决某一类的问题。随后，随着神经网络 技术的形成和发展，人们的注意力转向知识工程。知 识工程不同于机器学习那样给计算机输入范例，让它 生成出规则，而是直接给计算机输入已被代码化的规 则，而计算机是通过使用这些规则来解决某些问题。 专家系统就是这种方法所得到的成果。
的；可以是演绎的，也可以是归纳的。
发现的知识可以被用于信息管理，查询优化，决
策支持和过程控制等，还可以用于数据自身的维护。
因此，数据挖掘是一门交叉学科，它把人们对数
据的应用从低层次的简单查询，提升到从数据中挖掘
知识，提供决策支持。在这种需求牵引下，汇ቤተ መጻሕፍቲ ባይዱ了不
同领域的研究者，尤其是数据库技术、人工智能技术、 数理统计、可视化技术、并行计算等方面的学者和工
• 加强对各种非结构化数据的开采（Data Mining for
据库系统的集成；
•
发现语言的形式化描述，即研究专门用于知识发
现的数据挖掘语言，也许会像SQL语言一样走向形式 化和标准化；
• 寻求数据挖掘过程中的可视化方法，使知识发现的 过程能够被用户理解，也便于在知识发现的过程中进 行人机交互； • 研究在网络环境下的数据挖掘技术（Web Mining）， 特别是在因特网上建立DMKD服务器，并且与数据库 服务器配合，实现Web Mining；
数据挖掘的商业角度定义
数据挖掘是一种新的商业信息处理技术，其主要
特点是对商业数据库中的大量业务数据进行抽取、转
换、分析和其他模型化处理，从中提取辅助商业决策 的关键性数据。 数据挖掘其实是一类深层次的数据分析方法。在 过去，数据收集和分析的目的是用于科学研究，另外
由于当时计算能力的限制，对大数据量进行分析的复
量的数据中经过深层分析，获得有利于商业运作、提
高竞争力的信息，就像从矿石中淘金一样。数据挖掘 也因此而得名。
数据挖掘可以描述为：按企业既定业务目标，对 大量的企业数据进行探索和分析，揭示隐藏的、未知
的或验证已知的规律性，并进一步将其模型化的先进
有效的方法。
数据挖掘与传统分析方法的区别 数据挖掘与传统的数据分析（如查询、报表、联 机应用分析）的本质区别是数据挖掘是在没有明确假 设的前提下去挖掘信息、发现知识。数据挖掘所得到 的信息应具有先前未知、有效和可实用三个特征。 先前未知的信息是指该信息是预先未曾预料到的, 既数据挖掘是要发现那些不能靠直觉发现的信息或知 识，甚至是违背直觉的信息或知识，挖掘出的信息越 是出乎意料，就可能越有价值。在商业应用中最典型 的例子就是一家连锁店通过数据挖掘发现了小孩尿布 和啤酒之间有着惊人的联系。
Mining result
Layer4 User Interface Layer3 OLAP/OLAM
OLAM Engine
Data Cube API
OLAP Engine
Layer2
MDDB
Meta Data
Filtering&Integration
MDDB
Database API
Data cleaning
杂数据分析方法受到很大限制。
现在，由于各行业业务自动化的实现，商业领域 产生了大量的业务数据，这些数据不是为了分析的目 的而收集的，而是由于纯机会的（Opportunistic）商
业运作而产生。分析这些数据也不是单纯为了研究的
需要，更主要是为商业决策提供真正有价值的信息， 进而获得利润。 但所有企业面临的一个共同问题是：企业数据量 非常大，而其中真正有价值的信息却很少，因此从大
何谓知识？从广义上理解，数据、信息也是知识
的表现形式，但是人们更把概念、规则、模式、规律 和约束等看作知识。人们把数据看作是形成知识的源
泉，好像从矿石中采矿或淘金一样。
原始数据可以是结构化的，如关系数据库中的数
据；也可以是半结构化的，如文本、图形和图像数据；
甚至是分布在网络上的异构型数据。 发现知识的方法可以是数学的，也可以是非数学
发展趋势。缺乏挖掘数据背后隐藏的知识的手段，导
致了“数据爆炸但知识贫乏”的现象。
支持数据挖掘的技术基础 数据挖掘技术是人们长期对数据库技术进行研究 和开发的结果。数据挖掘使数据库技术进入了一个更 高级的阶段，它不仅能对过去的数据进行查询，并且
能够找出过去数据之间的潜在联系，从而促进信息的
传递。现在数据挖掘技术在商业应用中已经可以马上 投入使用，因为对这种技术进行支持的三种基础技术
Data Mining: A KDD Process
Pattern Evaluation
– Data mining: the core of
knowledge discovery process.
Data Warehouse Data Cleaning Data Integration Databases
Data Mining
Task-relevant Data Selection
四、数据挖掘的内容
1、关联知识（Association） 它反映一个事件和其他事件之间依赖或关联的知 识。如果两项或多项属性之间存在关联，那么其中一 项的属性值就可以依据其他属性值进行预测。最为著 名的关联规则发现方法是R.Agrawal提出的Apriori算 法。关联规则的发现可分为两步。第一步是迭代识别 所有的频繁项目集，要求频繁项目集的支持率不低于 用户设定的最低值；第二步是从频繁项目集中构造可 信度不低于用户设定的最低值的规则。识别或发现所 有频繁项目集是关联规则发现算法的核心，也是计算 量最大的部分。
数据仓库技术的发展与数据挖掘有着密切的关系。 数据仓库的发展是促进数据挖掘越来越热的原因之一。
但是，数据仓库并不是数据挖掘的先决条件，因为有
很多数据挖掘可直接从操作数据源中挖掘信息
二、数据挖掘的定义 数据挖掘（Data Mining）就是从大量的、不完 全的、有噪声的、模糊的、随机的实际应用数据中， 提取隐含在其中的、人们事先不知道的、但又是潜在 有用的信息和知识的过程。 与数据挖掘相近的同义词有数据融合、数据分析 和决策支持等。这个定义包括好几层含义：数据源必 须是真实的、大量的、含噪声的；发现的是用户感兴 趣的知识；发现的知识要可接受、可理解、可运用； 并不要求发现放之四海皆准的知识，仅支持特定的发 现问题。