基于云的高性能分布式内存数据库的开发技术

基于云的高性能分布式 内存数据库开发技术Copyright © Versant Corp. All rights reserved.0001By Tiger Lau，CTO of Versant China数据库发展简史模型层次化，结构化 网络化 关系型 对象型优点性能 性能 灵活性, 支持查询 性能,灵活性缺点灵活性, 对查询的支持 灵活性, 对查询的支持 性能 随机性较强, 支持既成的查询数据简单 简单 简单 复杂这两点是现有很多系统的核心问题所在Versant数据库系统架构Versant Versant C C Interface Interface Versant Versant C++ C++ Interface Interface Versant Versant Java Java Interfaces Interfaces Other Other Interfaces, Interfaces, Tools, Tools, etc. etc.Versant Object ManagerVersant VersantNetwork NetworkLayer Layer Versant VersantNetwork NetworkLayer LayerVersant ServerVirtual VirtualSystem SystemLayer LayerRaw Raw Devices, Devices, File File Systems, Systems, RAID, SAN, NAS RAID, SAN, NASVersant数据库的双缓存架构User Interface User User Interface Interface Application Application Application Logic Logic Versant Versant Versant Object Object Manager ObjectManager Manager FE FE Profile Profile Object CacheVersant ClientPage Cache Versant Versant Storage Storage Manager Manager RollRollforward forward log log Logical Logical Log Log File File Physical Physical Log Log File File BE BE Profile Profile Database Database Volume(s) Volume(s)Versant Server架构对比：RDBMS vs VERSANTACID Transactions Indexing SMP Support Event Notification Fault Tolerant Server Replication Scale to TerabytesClient Cache Transactional Control QueryQuerySQL ODBC JDBCSQL C Relational Server Query Engine TablesC++ Java .NETSQL ODBC JDBC Query Engine Server cacheLogicalServer cacheObjectsPhysicalLogicalPhysicalDual LoggingDual LoggingVersant数据库可以实现灵活的多线程架构Client Client Thread Thread Client Client Thread Thread Client Client Thread Thread Client Client Thread Thread Session Object / Object Cache Client Process Server Server Thread Thread Session Object / Object Cache Server Process Page CacheLock TableServer Server Thread ThreadServer Server Thread Threadasync I/O of non-commit buffer writesClient Process Client Client Thread Thread Client Client Thread Thread Session Object / Object CacheLog Log Buffer Buffer Thread ThreadBackground Background Page Page Flusher Flusherwrites modified pages to disk传统关系型数据库在OR-Mapping领域面临困难In-MemoryRDBMSSELECT FROM table1 ...; SELECT FROM table1 ...; SELECT FROM table2 ...; SELECT FROM table2 ...; convert_tables_to_memory(); convert_tables_to_memory(); Order order == new Order() Order order new Order() for(i=0; i<100; i++) {{ for(i=0; i<100; i++) OrderLine pp == new OrderLine(); OrderLine new OrderLine(); … … order.addLine(p) order.addLine(p) }} convert_memory_to_tables(); convert_memory_to_tables(); INSERT INTO table1 ...; INSERT INTO table1 ...; UPDATE table2 ...; UPDATE table2 ...;Tables内存模型与存储模型不一致 从存储模型映射成内存模型时有巨大的性能消耗 更多的转换代码 编程语言和数据操作语言不一致，存在“失配”问题Versant数据库能够更好支持复杂数据VERSANTIn-MemoryOrder order == new Order() Order order new Order() for(i=0; i<100; i++) { for(i=0; i<100; i++) { OrderLine p = new OrderLine(); OrderLine p = new OrderLine(); … … order.addLine(p) order.addLine(p) }}内存模型与存储模型相一致 高性能的大数据量对象操作（排序等） 没有额外的转换代码 强化的程序生产力 程序开发语言即DML与DDL例：关系数据库在解决继承对象存储时的困境soc_no emp_name birth_date dept_name soc_no hour_rate over_rate max_over soc_no month_rate bonus_pctcreate table Employee_Table (emp_id number primary key soc_no number uniqe emp_name varchar Birth_date date dept_id number foreign key) create table Hourly_Emp_Table (emp_id number primary key foreign key hour_rate decimal over_rate decimal max_over decimal) create table Salaried_Emp_Table (emp_id number primary key foreign key month_rate deciaml bonus_pct decimal) create table Department_Table (......) create sequence ......Employee soc_no emp_name birth_date dept_nameCompany comp_name employees departmentsHourlyEmp hour_rate over_rate max_overSalariedEmp month_rate bonus_pctEmployee emp = new HourlyEmployee(“Meier”); mit().由于继承对象需要拆分之后才能存储到关系数据库中， 需要额外的编码，维护难度加大，性能大幅降低$$例：Versant数据库解决继承对象存储轻而易举soc_no emp_name birth_date dept_name soc_no hour_rate over_rate max_over soc_no month_rate bonus_pct Employee soc_no emp_name birth_date dept_name Company comp_name employees departmentsEXEC SQL INSERT INTO Employee_Table (soc_no, emp_name, birth_date, dept_name) VALUES (:emp->soc_no, :emp->name, :emp->birth_date, :emp->dept_name); if (emp->type == 1) EXEC SQL INSERT INTO Hourly_emp_Table (soc_no, hour_rate, over_rate, max_over) VALUE (:emp->soc_no, :emp->hour_rate, :emp->over_rate, :emp->max_over); else if (emp->type == 2) EXEC SQL INSERT INTO Salaried_emp_Table (soc_no, month_rate, bonus_pct) VALUES (:emp->soc_no, :emp->month_rate, :emp->bonus_pct) EXEC SQL COMMIT WORK RELEASE;HourlyEmp hour_rate over_rate max_overSalariedEmp month_rate bonus_pctEmployee emp = new HourlyEmployee(“Meier”); mit().由于继承对象需要拆分之后才能存储到关系数据库中， 需要额外的编码，维护难度加大，性能大幅降低$$JAVA程序—主程序import com.versant.trans.*;public class Sample { static public main ( String args ) { TransSession session = new TransSession(args[0]); Department d1 = new Department(“技术部”); Department d2 = new Department(“客服部”); Employee Employee p1 = new HourlyEmployee (“001”,“Joe”, new Date(1980,1,1),d1,...); p2 = new SalariedEmployee(“101”,”man”,new Date(1960,9,1),d1,...);p1.setDepartment(d2); session.makePersistent(p1); mit(); } }例： Versant数据库解决继承对象存储轻而易举config.jvi配置文件c c c c a n Employee HourlyEmployee SalariedEmployee Department Sample **Employee Employee .java .javajavac javacEmployee Employee .class .classEclipse插件Employee Employee .class .class Enhancer EnhancerConfig Config File Filejava java Employee Employee mydb mydbVersant数据库能帮助开发者降低开发难度降低开发成本 – 代码更少 – 构想更多 – 运行更快Java .NET C++ classes Database DesignAnalyse DesignDatabase mapping layerApplication testDatabase testVersant数据库能通过减少源代码降低开发工作量2000Create 0 37Store 32 137List 15 54Query 28 59Versant Java JDBCVersant数据库能提高复杂应用的性能，降低硬件投入Ericsson Network Management Worldcom Fraud Management50xBritish Airways Yield Managementex l p m Co 复 杂ING Barings Risk Management30xPerformance Improvement 性能提高比例10xe pl m Si 单 简um i ed 等 M 中CIBC Global Markets Front Office Trading5xIBM Benchmark3xSimple Queries Complex transactions 简单数据库查询请求 复杂数据库事务 Complexity level of Data Model for transactions数据库事务中使用的复杂数据模型与RDBMS + OR-MAPPING的性能比较70,000PolePosition提供的性 能比较 相对关系型数据库比有 10 – 50x 性能提高 相对JPA 和Hibernate 比有10 – 50x性能提高60,00050,00040,00030,00020,00010,0000 Melbourne Sepang Bahrain Imola Barcelona Montreal NurburgringJDO/Versant/VOD-8.0JDO/DataNucleus/MySQL-5.1Versant数据库能更好利用系统资源，降低硬件要求Versant数据库测试样例CPU占用率图测试内容 数据库中没有工单 （时间单位：毫秒） Versant 单条提交，向工单表中插入10万条工单 每2000条工单提交一次，向工单表中插 入10万条工单 25152 6248 40322 9079 Oracle 69749 12999 107939 31772同比关系数据库测试样例CPU占用率图数据库中有1000万条工单 （时间单位：毫秒） Versant 26112 6226 46367 10467 Oracle 89383 18573 117654 32198每次select 1000条，单条修改状 态，先改为1，然后改为3 每次select 1000条，每次提交 1000条，先改为1，然后改为 3Versant数据库能更好适应新的开发方法和开发手段快速创建新业务模型——最适合开发人员使用的数据库敏捷开发等快速开发方法正在日渐成为主流。