the ATLAS DAQEF Prototype-1 project.

Atlas DAQTest Report of the Configuration Databases for the Atlas DAQ Prototype-1Authors:Igor SolovievKeywords:Test, Unit Test, OKS, Configuration Databases, Benchmark, OO7AbstractThis document presents the results of the unit tests of the configuration databases component of the ATLAS DAQ/EF Prototype -1 project.NoteNumber:114Version:1.0Date:12 February 1999Reference:http://atddoc.cern.ch/Atlas/Notes/114/Note114-1.html1 IntroductionThis document presents the results of the unit tests outlined in the test plan [1] for the configu-ration databases components of the back-end subsystem within the ATLAS DAQ/Event Filter prototype -1 project.2 The Configuration DatabasesA DAQ system needs a large number of parameters to describe its system architecture, hard-ware and software components, running modes and the system running status. One of the major design issues of Atlas DAQ is to be as flexible as possible, externally parenthesized by configuration databases.The configuration databases are implemented on top of OKS [2] in-memory persistent object manager (which is used during run of DAQ system) and Objectivity/DB [3] OODBMS (is planned to be used for offline configuration management). The configuration databases have data access libraries to hide details of underlying persistent object managers and details of database schema when needed. The schema of configuration databases is designed and gener-ated using StP/OMT [4] CASE tool. For more information about the configuration database library see the high level design [5] and data access library implementation [6] documents.3 Test ObjectivesThe tests were separated by the following groups:•tests of the underlying persistence object managers (i.e. OKS [and partly Objectivity/DB]) and DAQ configuration databases implemented on top of them including functionality, reli-ability, performance and interfaces (GUI, API)•tests of the part of configuration databases generated by StP/OMT CASE tool (schemes, data access libraries and data movers)•tests of the remote configuration databases accessCurrently, the configuration databases do not have a dedicated GUI - just standard data editor of underlying persistent object manager.It was not intended to test the facilities provided by persistent object managers that are not used by configuration databases. The tests are not intended to prove completeness of persist-ence object managers API and GUI.The details of the objectivities of the tests are described in test plan document [1].4 Testware and ApproachIn order to perform the tests several approaches were used:•standard persistent object manager OO7 benchmark;•configuration database generator (simulates expected DAQ configurations);•configuration database benchmark (manipulates DAQ configurations and performs bench-marks);•code coverage CASE tools available at CERN (Insure++[7]);•memory testing CASE tools available at CERN (Purify[8] and Insure++).4.1 The OO7 BenchmarkThe OO7 Benchmark represents a comprehensive test of object oriented database management system (OODBMS) performance.For the OKS object manager (which is not a commercial product) the OO7 benchmark has additional meaning: successful passing of this benchmark proves that object model supported by OKS is comparable with object model supported by the commercial object databases and that OKS is reliable and robust to perform quite complicated benchmark tests.4.1.1 OO7 Benchmark DescriptionRecently, a number of OODBMS systems have become publicly available. However, perhaps since the technology is so new, it is not yet clear precisely how these systems differ in their performance characteristics. In fact, it is not even clear what performance metrics should be used to give a useful profile of an OODBMS’s performance. The OO7 Benchmark [9] has been designed as a first step toward providing such a comprehensive OODBMS performance profile. Among the performance characteristics tested by OO7 are:•the speed of many different kinds of pointer traversals,•the efficiency of many different kinds of updates, including updates to indexed and unin-dexed object fields, repeated updates, sparse updates, and the creation and deletion of objects,•the performance of the query processor on several different types of queries.The OO7 Benchmark produces a set of numbers rather than a single number. A single number benchmark has the advantage that it is very catchy and easy to use for system comparisons. However, a benchmark that returns a set of numbers gives a great deal more information about a system than does one that returns a single number. A single number benchmark is only useful if the benchmark itself precisely mirrors the application for which the system will be used.4.1.2 OO7 Database parametersThere are three standard sizes of the OO7 Benchmark database:small,medium, and large. To test OKS specific properties new tiny configuration1 has been created. The parameters of the OO7 benchmark databases are summarized by Table 1:Table 1: OO7 benchmark database parameters1. For DAQ configuration databases ‘tiny configuration’ corresponds expected DAQ P-1 configuration4.1.3 OO7 Database SchemaSince the OO7 Benchmark is designed to test many different aspects of system performance, its database structure and operations are nontrivial. The OO7 Benchmark is intended to be sug-gestive of many different CAD/CAM/CASE applications, although in its details it does not model any specific application. Recall that the goal of the benchmark is to test many aspects of system performance, rather than to model a specific application. The description used OO7 database schema is summarized by Figure 1:A key component of the OO7 benchmark database is a set of composite parts. The set of all composite parts forms “design library” within the OO7 database. The number of composite parts in the design library, which is controlled by the parameter NumCompPerModule, is 500. Each composite part has a number of attributes, including the integer attributes and buildDate, and a small character array type. Associated with each composite part is a doc-ument object, which models a small amount of documentation associated with the composite part. Each document has an integer attribute id, a small character attribute title, and a char-acter string attribute text. The length of the string attribute is controlled by the parameter NumCompPerModule. A composite part object and its document object are connected by a bi-directional association.In addition to its scalar attributes and its association with a document object, each composite part has an associated graph of atomic parts. Intuitively, the atomic parts within a composite part are the units out of which the composite part is constructed. In the small benchmark, each composite part’s graph contains 20 atomic parts, while in the medium and large benchmarks, each composite part’s graph contains 200 atomic parts (this number is controlled by the param-eter NumCompPerModule). One atomic part in each composite part’s graph is designated as the “root part”.Each atomic part has the integer attributes id,buildDate,x,y, and docId, and the small character array type. In addition to these attributes, each atomic part is connected via a bi-directional association to several (3, 6, or 9) other atomic parts, as controlled by the parameter NumCompPerModule. The initial idea was to connect the atomic parts within each composite part in a random fashion. However, random connections do not ensure complete connectivity. To ensure complete connectivity, one connection is initially added to each atomic part to con-nect the parts in a ring; more connections are then added at random. In addition, initial plans did not specify a 3/6/9 interconnection variation. This variation was included to ensure that OO7 provides satisfactory coverage of the OODBMS performance space, as OO7 preliminary tests indicated that some systems can be very sensitive to the value of this particular bench-mark parameter.The connections between atomic parts are implemented by interposing a connection object between each pair of connected atomic parts. Here the intuition is that the connections them-selves contain data; the connection object is the repository for that data. A connection object contains the integer field length and the short character array type.The design library, which contains the composite parts and their associated atomic parts (including the connection objects) and documents, accounts for the bulk of the OO7 database. However, a set of composite parts by itself is not sufficiently structured to support all of the wished operations. Accordingly, an “assembly hierarchy” was added to the database. Intui-tively, the assembly objects correspond to higher-level design constructs in the application being modelled in the database. Each assembly is either made up of composite parts (in which case it is a base assembly) or it is made up of other assembly objects (in which case it is a com-plex assembly).The first level of the assembly hierarchy consists of base assembly objects. Base assembly objects have the integer attributes id and buildDate, and the short character array type. Each base assembly has a bidirectional association with three “shared” composite parts and three “unshared” composite parts. (The number of both shared and unshared composite parts per base assembly is controlled by the parameter NumCompPerModule.) The OO7 benchmarkdatabase is designed to support multiuser workloads as well as single user tests; the distinction between the “shared” and “unshared” composite parts was added to provide control over shar-ing/conflict patterns in the multiuser workload. This benchmark only deals with the single user tests; only the “unshared” composite part associations are used in the single user benchmark. The “unshared” composite parts for each base assembly are chosen at random from the set of all composite parts.Higher levels in the assembly hierarchy are made up of complex assemblies. Each complex assembly has the usual integer attributes,id and buildDate, and the short character array type; additionally, it has a bidirectional association with three subassemblies (controlled by the parameter NumCompPerModule), which can either be base assemblies (if the complex assembly is at level two in the assembly hierarchy) or other complex assemblies (if the com-plex assembly is higher in the hierarchy). There are seven levels in the assembly hierarchy (controlled by the parameter NumCompPerModule).Each assembly hierarchy is called a module. Modules are intended to model the largest subu-nits of the database application. They are not used explicitly in the small and medium data-bases, each of which consists of just one single module. Modules have several scalar attributes: the integers id and buildDate, and the short character array type. Each module also has an associated manual object, which is a larger version of a document. Manuals are included for use in testing the handling of very large (but simple) objects.4.1.4 Programs4.1.4.1 OriginThe original implementation was written by Mike Carey, David DeWitt and Jeff Naughton (see ftp:///oo7/implementations/ojby.OO7.tar) for Objectivity/DB version 2.1.0 and was modified by the author of this paper for version 4.0.2 to satisfy changes of Objectiv-ity/DB C++ API.The implementation of OKS OO7 benchmark was written in a similar manner to the Objectiv-ity/DB implementation.4.1.4.2 SpecificOKS is in-memory object manager, hence:•it does not use any type of cache and hot/cold tests can be emulated by purging the file sys-tem cache only,•it requires quite a long time to load the database in-memory so it loads database once and then performs all the tests (Objectivity benchmark application is called once per each test)For Objectivity/DB tests benchmark application has been running in standalone mode without lock server.4.1.4.3 InstallationThe OO7 benchmark parameter files are installed under SRT on confdb/data/oo7-conf directory and used for both OKS and Objectivity/DB.The OO7 benchmark programs and scripts for Objectivity/DB are not installed under SRT and located in private directory (see/home-users/isolov/OO7 on sunatdaq01.cern.ch for details):•application gendb is used to generate required database configuration,•application bench is used to perform OO7 benchmark,•shell script do-objy-tests.init is used to create federated database, define benchmark data-base schema and build benchmark applications,•shell script do-objy-tests is used to perform benchmark.The OO7 benchmark library, programs and scripts for OKS are installed under SRT:•oo7-oks library is generated from StP CASE tool and used by benchmark applications,•application oo7GenerateDB-oks is used to generate required database configuration,•application oo7BenchDB-oks is used to perform OO7 benchmark,•shell script oo7-tests/do-oks-tests is used to perform benchmark.4.2 Performance Benchmark of DAQ Configuration Databases using OKSIt was decided that ATLAS DAQ configuration databases will use a two-tier architecture:• a light-weight in-memory persistent object manager to support the real-time requirements during system run,• a full ODBMS as a back-up and for long-term data management.The OKS has been selected as the run-time persistent object manager and hence it’s perform-ance defines the performance of DAQ Configuration Databases during the DAQ system run.This benchmark will be used to receive performance results for a single Read-Out Crate, expected ATLAS DAQ Prototype -1 project and final ATLAS DAQ configurations for each available OS/Compiler on different computers (powerful, average, slow).The goal of this benchmark is to determine what are requirements from OKS to hardware resources to achieve acceptable response time for the configuration databases and what we can achieve now.4.2.1 Benchmark Database SchemaThe schema of DAQ Configuration databases is defined in StP and automatically generated from it. The details of this schema are described in DAQ Configuration Databases Design doc-ument.This benchmark uses only part of classes from this schema. The relations between these classes are shown below on Figure 2:Figure 2: The class schema of benchmark database4.2.2 Database DescriptionThe benchmark generator program tries to model relationships between instances of classes shown above to describe real configuration. For each instance of SW_Object class it gener-ates 5 instances of Program class, 2 instances of SW_Resource class, 8 instances of Process class1, 4 instances of Computer class, 8 instances of Application class and 12 instances of Environment class. After this process has been completed, the generator sets random attribute values of these instances and links them together.The total number of generated objects in an input parameter of generator program.It is difficult to say how many objects will be needed to describe the final ATLAS DAQ con-figuration or even Prototype -1 ATLAS DAQ configuration. We can expect that to describe one Read-Out Crate (ROC) we need about 200-400 objects in configuration database that are:• 1 ROC gives 1 object,• 1 Partition gives 1 object + several environment variables,•ROC contains 20 modules and each of them gives:• 1 module object + one 1 CPU board + 1 i/o device,• 1 application object + 1 sw_object + 1 program + few environment variable + few param-eters•General objects like connection table description, Run Control tree structure, etc.1. Of course, instances of Process class are not persistent in the true configuration databaseWe can expect that P-1 Configuration will contain 10 ROC, Event Builder, several worksta-tions and data storage system. Probably it will be described by 3,000-6,000 objects in configu-ration database.Finally, complete ATLAS DAQ configuration will have about 200 crates, Event Builder,Event Filter, tens of workstations and data storage systems. It will include description of detec-tor hierarchy tree and some detector parameters. It is expected that about 50,000-100,000objects in configuration database will be enough to describe the configuration.figuration. They may be DAQ supervisor, Resource Manager, Run Bookkeeper and GUI Configuration database browser/editor and hence should be started on the most powerful workstations. All the other DAQ application running on ordinary computers will require loading of partial configuration only.4.2.3 Configuration Database ParametersThe configuration databases performance benchmark is parametrized by single value which is number of objects in databases. This number N is:4.2.4 InstallationThe DAQ performance benchmark programs and scripts for OKS are installed under SRT:•OKS schema file DAQ-Configuration.schema is generated from StP CASE tool,•application oks/tests/oks_generate_data is used to generate DAQ configuration database,•application oks/tests/oks_time_tests is used to perform tests,•shell script oks/tests/do-tests is used to perform tests of all DAQ configurations.5 Test Conditions and Results 5.1 Test Environment 5.1.1 Systems EnvironmentThe OO7 benchmark tests have been performed on sunatdaq01.All the DAQ configuration databases performance benchmarks have been done on several available workstations. The idea was to test computers with different performance, number of processors and size of physical memory. All available operating systems had been tested as well.The code coverage and memory analysis tests have been done on sunatdaq01.N 1002n⋅=n 110{,}=The parameters of used workstations are summarized by Table 2:b. WNT kernel + desktop applications + file system caches takes about 40 MB that resultsbenchmark application swapping after 20-25 MBDuring the tests all data files were created on local file systems of tested workstation (/tmp on UNIX or D:/TEMP on WNT). The schema file was located on AFS.All the workstations were not isolated from network and were shared by other users.5.1.2 Time Measurement ApproachesThe OO7 benchmark uses getrusage() system call which retrieves information about the resources utilized by the programs. It’s resolution is dependent on the clock tick interval. Some of the tests have duration about tens of seconds and their response time has been calculated using a single call. Otherwise the tests were performed many times inside a loop and the result-ing time was divided by the number of loop iterations.For DAQ configuration databases benchmark the RWTimer class from Rogue Wave Tools.h++ library was used to implement time measurement. It uses clock() system call to calculate amount of CPU time used by the process. The resolution of this call differs for tested operating systems and can be from a few microseconds to a few milliseconds. When it was possible, the tested function was called many times inside a loop to be sure that the total time is at least ten seconds. Then measured time was divided to the number of loop iterations.5.2 Test Results5.2.1 OO7 Benchmark ResultsThis section presents results of OO7 benchmark.5.2.1.1 Database SizesIn the following, the size of the databases in each of the systems is useful for interpreting the results. The following sizes were measured:1,474,560 bytes length.5.2.1.2 Initialization and Shutdown TimesThe OKS object manager reads the whole contents of the database from disk to memory and binds loaded data. The Objectivity/DB keeps all the data on disk and has relatively small cache only. To generate oo7 configuration, save it and load back the following times were measured:The same results are presented below in graphical form:Figure 3: Times to generate (left picture) and save (right picture) databaseFigure 4: Times to load (left picture) and close (right picture) OO7 benchmark databaseb. time to commit generated database and close federated databasec. there is no single point to measure this timeoks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567890.11101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567891101001000Connections/AtomicPart R e s p o n s e T i m e (s e c o n d s )oks-small oks-mediumoks-tiny 234567890.11101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium5.2.1.3 Traversals5.2.1.3.1 Traversal T1Raw Traversal Speed traverses the assembly hierarchy. As each base assembly is visited, vis-its each of its referenced unshared composite parts. As each composite part is visited, per-forms a depth first search (DFS) on its graph of atomic parts. Returns a count of the number of atomic parts visited when done.The following table compares the performance of the systems on the hot traversal with the cold and hot traversals run as a single transaction (“one”) and as multiple transactions (“many”).These traversals were run over the small database with fanout 3.Figure 5: Results of Traversal T1.5.2.1.3.2 Traversal T6Sparse Traversal Speed traverses the assembly hierarchy. As each base assembly is visited,visits each of its referenced unshared composite parts. As each composite part is visited, visits the root atomic part. Returns a count of the number of atomic parts visited when done .Figure 6: Results of Traversal T6.oks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567890.00.20.40.60.81.01.2Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumposite part, T6 just visits one (the root part). This test coupled with T1 provides inter-esting insight into the costs and benefits of the full swizzling approach to providing persistent virtual memory.5.2.1.3.3 Traversal T2Traversal With Updates repeats T1, but updates objects during the traversal. There are three types of update patterns in this traversal. In each, a single update to an atomic part consists of swapping its (x,y) attributes. The three types of updates are:•Update one atomic part per composite part,•Update every atomic part as it is encountered,•Update each atomic part in a composite part four times.When done, returns the number of update operations that were actually performed.The results of these traversals on the tested database configurations are presented below:Figure 7: Results of Traversal T2A.Figure 8: Results of Traversals T2B (left) and T2C (right).oks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-medium5.2.1.3.4 Traversal T3Traversal With Indexed Field Updates repeats T2, except that now the update is on the date field, which is indexed. The specific update is to increment the date if it is odd, and decrement the date if it is even .The goal here is to check the overhead of updating an indexed field. This should be done using the same three variants used in T2, and again the number of updates should be returned.Figure 9: Results of Traversal T3A.Figure 10: Results of Traversal T3B and T3C.5.2.1.3.5 Traversals T8 and T9Traversal T8scans the manual object counting the number of occurrences of the character “I”.Traversal T9checks to see if the first and last character in the manual object are the same .The results of traversals should be independent from the atomic part fanout.oks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567891101001000Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumFigure 11: Results of Traversals T8 and T9.5.2.1.3.6 Traversals OmittedThere is a lot of other experiments with traversals that changed the size of document objects,traversals that scanned documents instead of traversing atomic part subgraphs, and “reverse-traversals” that go from an atomic part to the root of the assembly hierarchy. These tests were deleted from the final benchmark because, after experimentation, it was discovered that they did not provide additional insight beyond that provided by the other traversals.5.2.1.4 QueriesThe queries are operations that ideally would be expressed as queries in a declarative query language. Not all of the OO7 queries could be expressed entirely declaratively in all of the sys-tems; whenever a query could not be expressed declaratively in a system it was implemented as a “free ” procedure that essentially represents a hand coded version of what the query execu-tion engine would do in order to evaluate the query.Like the traversals, the queries should be run both cold and hot. In each case, OO7 also tests the coupling of the query facility with the application programming language by requiring a “do nothing ” function to be called with field values from each qualifying object. All of the queries are read-only. Again, gaps in the numbering correspond to queries that were tested but eliminated from this paper due to space constraints or a lack of useful additional insights.5.2.1.4.1 Query Q1Exact Match Lookup Query generates 10 random atomic part id’s; for each part id gener-ated, lookups the atomic part with that id. Returns the number of atomic parts processed when done .In each case, the systems used the index to avoid scaling the response time with the database;the relative ordering of the systems’ performance is consistent with that of the small results.oks-tiny 234567890.00010.0010.010.11Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567890.00010.0010.010.11Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-medium5.2.1.4.2 Queries Q2, Q3, and Q7.These queries are most interesting when considered together:•Query Q2:Choose a range for dates that will contain the last 1% of the dates found in the database’s atomic parts. Retrieve the atomic parts that satisfy this range predicate.•Query Q3:Choose a range for dates that will contain the last 10% of the dates found in the database’s atomic parts. Retrieve the atomic parts that satisfy this range predicate.•Query Q7:Scan all atomic parts.Note that queries Q2 and Q3 are candidates for Objectivity/DB B+ tree lookup.Figure 13: Results of Queries Q2 and Q3.oks-tiny 234567890.0010.010.11Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567890.0010.010.11Connections/AtomicPart R e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-mediumoks-tiny 234567890.0010.010.1110Connections/AtomicPartR e s p o n s e T i m e (s e c o n d s )oks-small oks-medium objy-tiny objy-small objy-medium。