网格数据管理技术

and coding system
• Multimedia data in any standard or user
defined binary format
网格计算－Grid Computing
数据网格需求 –3
• 支持自动的认证
❖Multiple Authentication Realms – single sign-on ❖Uniform user name space
• Distributed Cluster Storage
• 网络
❖Characteristics ❖NWS ❖Guaranteed Service
网格计算－Grid Computing
数据网格管理的体系结构
• Heterogeneity Transparency • Location Transparency • Name Transparency • Distribution Transparency • Replication Transparency • Ownership & Costing Transparency
结果:全球知识社区的出现
• 围绕共同的目标组织的一支队伍
❖社区: “Virtual organizations”
• 各种成员和能力
❖异构是一种力量而不是弱点
• 地理上和行政管理区域的分布
❖没有一个点和机构具有所有的能力和资源
• 适应如此环境是系统的功能
❖调整成员、分配职责和资源
网格计算－Grid Computing
• 数据服务
❖Web- accessibility (HTTP GET, WSDL, SOAP) ❖数据存取和API
• 定位数据物理位置
❖根据参数构造搜索规则和匹配标准 ❖构造查询、分布式查询或者异构联邦查询 ❖从多个不同的数据源抽取数据汇集成一个逻辑单一
的数据集 ❖代理操作 (security/access considerations) ❖Bulk Operations - batch ❖交互式操作和异步操作
网格计算－Grid Computing
数据网格需求 –7
• 数据管理操作
❖管理横跨多域、异构的环境中的资源，保证 24x7x52 小时的高可用能力.
❖DGMS must ensure that data resource/data resource content catalogues/registries are always available and the definitions in them are current, accurate, and consistent
数据网格需求 –4
• 虚拟数据组织
❖Data Location Independence，Uniform data name space,
❖persistent identifiers，Collections Hierarchy
• 与元数据的紧密集成
❖数据发布
• 动态地注册和注销数据资源Register and
• 处理异构和多样性
❖Platforms & systems, vendors, types of storage, types of services, types of processes & users
• 可控制的数据移动
❖Demand-driven Data placement ❖Caching, archiving, version and locks ❖Third-party data movement ❖Parallel data transfer
网格计算－Grid Computing
从 1993 到2004
• 巨大的数据量: Petabytes
❖For an increasing number of communities, gating step is not collection but analysis
• 无处不在的因特网:100+ million hosts
Discovery
Storage System Description
Storage Resources
网格计算－Grid Computing
• 授权 – 访问控制
❖Seamless One-stop authorization ❖Roles & Tickets – inheritance & longevity ❖Possible: data owner can grant 、revoke access、
permission ，and delegate authority ❖Flexible ：combinations of restrictions and the level
对Petabytes（Terabytes）的科学数据进行复杂、 计算密集型的分析、处理
❖多个数据提供者 ❖跨网络的最优数据移动 ❖无缝的安全的数据访问 ❖良好的访问控制机制和复杂的使用方式 ❖数据访问的保证 ❖类似电力网格
• Multiple power generators • Complex transmission networks with switching • Simple Usage Interface – plug and play • Guaranteed Supply - Meeting of demands • Complex cost function
网格计算－Grid Computing
数据网格需求 –2
• 数据结构和表示
❖DGMS必须支持访问所有在数据结构和表示 中定义的数据类型
• Numeric data at highest of precision • Text data in any format, structure, language,
❖must maintains referential integrity of DGMS data resources
❖确保Computing
数据网格需求 –8
• 虚拟管理
❖Single-point administration ❖Autonomous local control ❖Multiple-levels of administrations –
• Total about 200 TB data • Largest catalogs near 1B objects
Data and images courtesy Alex Szalay, John Hopkins
网格计算－Grid Computing
数据的集成是一个基本的挑战
Many sources
of data, services, computation
Discovery
RM
R
R
Security & policy must underlie access & management decisions
Registries organize services of interest to a community
RM Access
RM RM
SSeeccuurirtiyty sseervrvicicee
Data integration activities may require access to, & exploration of, data at many locations
RM Resource management is needed to ensure progress & arbitrate competing demands
网格计算－Grid Computing
Open Grid Architecture
Application
Data Model Management
Information Discovery
Data Handling Systems
Remote Procedure Execution
Dynamic Info
deregister data resources dynamically
• Register and deregister can be propagate to
site holding replicates
• 元数据定义、发布和说明过程的自动化
网格计算－Grid Computing
数据网格需求 –5
网格计算－Grid Computing
数据网格需求 –9
• 存储
❖Hierarchical Storage Systems, Tapes, Disks, SAN, NAS, NFS, Databases, FTP servers, HTTP servers, WSDL services, …
❖Integration on Device Characteristics ❖Storage Bricks
❖Collaboration & resource sharing the norm
• 超高速的网络: 10+ Gb/s
❖Global optical networks
• 巨大的计算能力: 100+ Top/s
❖Moore’s law gives us all supercomputers
网格计算－Grid Computing
全球知识社区的出现
7
网格计算－Grid Computing
全球知识社区的出现常常是由数据驱 动: 例如 Astronomy
No. & sizes of data sets as of mid-2002, grouped by wavelength
• 12 waveband coverage of large areas of the sky