A Quantitative Method for Quality Evaluation of Web Sites and Applications

A Quantitative Method for Quality Evaluation of Web Sites andApplicationsLuis Olsina1, Gustavo Rossi2 31 GIDIS, Department of Informatics, Faculty of Engineering, at UNLPam,Calle 9 y 110, (6360) General Pico, La Pampa, ArgentinaE-mail olsinal@.ar2 LIFIA at UNLP, Calle 50 y 115, (1900) La Plata, Buenos Aires, Argentina3 CONICET - ArgentinaE-mail grossi@.arAbstract.In this paper, a quantitative evaluation strategy to assess the quality of Web sites and applications (WebApps) is discussed. We give an overview of the WebQEM (Web Quality Evaluation Method) and its supporting tool by presenting an E-commerce case study. The methodology is useful to systematically assess characteristics, subcharacteristics and attributes that influence product quality. We show that the implementation of the evaluation yields global, partial and elementary quality indicators that can help different stakeholders in understanding and improving the assessed product. Concluding remarks and in-progress research are finally presented.Keywords:Web Engineering, Quantitative Evaluation, WebQEM, Quality Characteristics and Attributes. 1. IntroductionThe Web is playing a central role in diverse application domains such as business, education, industry, and entertainment. As a consequence, there are increasing concerns about the ways in which WebApps are developed and the degree of quality delivered. Thus, there are compelling reasons for a systematic and disciplined use of engineering methods and tools for developing and evaluating Web sites and applications [9]. We need sound evaluation methods for obtaining reliable information about the product’s quality. These methods should identify which attributes and characteristics should be used to obtain meaningful indicators for assuring specific evaluation goals given a user viewpoint.It is widely known that the quality of software products can be described in terms of quality characteristics as defined in the ISO/IEC 9126-1 standard [5]. “However, the state of the art in software measurement is such that, in general, the direct measurement of these characteristics is not practical. What is possible is to assess these characteristics based on the measurement of lower abstraction attributes of the product” ([6], p.3). We consider an attribute as a direct or indirect measurable (tangible or abstract) property of an entity (a WebApps in our case). In addition, we can use a quality model (in the form of a quality requirement tree) in order to specify such characteristics, subcharacteristics and attributes. These quality, cost, or productivity requirements are often quoted as non-functional requirements in the literature.In this context, stakeholders should consider which are the characteristics and attributes that influence the product quality and quality in use (though ensuring product quality is not often sufficient to guarantee quality in use –however, this discussion is beyond the scope of this article). Specifically, there are some characteristics that influence product quality as those prescribed in the ISO 9126-1 standard, i.e., usability, functionality, reliability, efficiency, portability, and maintainability. In order to define and specify the quality requirement tree for a given assessment goal and user viewpoint, we should consider diverse attributes –e.g., Broken Links, Orphan Pages, Quick Access Pages, Table of Contents, Site Map, Links Colour Style Uniformity, Permanence of Main Controls, just to quote a few of them. It might also be admitted, however, that designing a rigorous non-functional requirement model that gives us a strong correlation between attributes and characteristics is a hard endeavour.In this paper, we present the Web Quality Evaluation Method (QEM) [14, 15] and some aspects of its supporting tool, WebQEM_Tool [16]. We show that, by using the methodology for assessment purposes, we can give recommendations both by controlling quality requirements in new Web development projects and by evaluating requirements in operational phases. We show that we can discover either absent features, or requirements poorly implemented, i.e., design and implementation drawbacks related to the interface, navigation, accessibility, search mechanisms, content, reliability and performance, among others.Though our method can be applied for assessing all aspects of Web sites and application, we focus on those that are perceived by the user (navigation, interface, reliability, etc.) instead of other product attributes (such as quality of code, design, etc). In this sense we emphasize the Web site characteristics and attributes from a general visitor viewpoint.The rest of this paper proceeds as follows. In Section 2, we describe the evaluation process to which WebQEM adheres. In Section 3, we discuss a comprehensible example in the field of E-commerce in order to illustrate both the methodology and the first version of the supporting tool. Finally, some concluding remarks and future work are drawn.2. Overview of the Evaluation Process in the WebQEM MethodologyThe WebQEM process steps are grouped in the following four major technical phases:1.Quality Requirements Definition and Specification;2.Elementary Evaluation (both Design and Implementation stages);3.Global Evaluation (both Design and Implementation stages);4.Conclusion of the Evaluation(regarding Recommendations).Figure 1, shows the evaluation process underlying the methodology including the phases, stages, main steps, inputs and outputs. This model is inspired in the ISO’s process model for evaluators [6]. We next give an overview of the major technical phases (the evaluation process has also planning and scheduling steps).2.1 Quality Requirements Definition and Specification.In this phase, evaluators must clarify the evaluation goals and the intended user viewpoint. They should select a quality model, for instance, the ISO-prescribed characteristics in addition to attributes customized to the Web domain. The relative importance of these components should be identified considering the WebApps audience and the extent of the coverage required.Regarding the user profile, at least three abstract evaluation views of quality may be defined, i.e., visitors, developers and managers views. For example, the visitor category can be decomposed in general and expert visitor subcategories.Thus, taking into accounts the domain and product descriptions, the agreed goals, and the selected user view (i.e., the explicit and implicit user needs), characteristics, subcharacteristics and attributes should be specified in a quality requirement tree. In the end of this phase, a quality requirement specification document is produced.2.2 The Elementary Evaluation.In this phase, two major stages are defined as depicted in Fig. 1: The design and the implementation of the elementary evaluation.For each measurable attribute Ai from the requirement tree, we can associate a variable Xi, which will take a numerical value from a direct or indirect metric. However, the value of this metric will not represent the level of satisfaction of this elementary requirement at all. For that reason, it is necessary to define an elementary criterion function that will result afterwards in an elementary indicator or preference value.For instance, let us consider the Broken Links attribute, which measure (count) links that lead to missing destination pages. A possible indirect metric is: X = #Broken_Links / #Total_Links_of_Site. Now, how do we interpret the measured value?; what are the best, worst and intermediate preferred values? The next formula represents a possible criterion function to determine the elementary quality preference EP:EP= 1 (or 100%) if X = 0; EP = 0 (or 0%) if X >= X max ;otherwise EP= (X max – X) / X max if 0 < X < X maxwhere X max is some agreed upper threshold such as 0.06So the elementary quality preference EP is frequently interpreted as the percentage of satisfied requirement for a given attribute, and it is defined in the range between 0, and 100% (so the scale type and the unit of metrics become normalized [20]). Furthermore, to ease the interpretation of preferences, we primarily group them in three acceptability levels, namely: unsatisfactory (from 0 to 40%), marginal (from 40 to 60%), and satisfactory (from 60 to 100%) –this is exemplified in Section 3.4.In the implementation stage, the selected metrics are applied to the Web application as shown in Fig. 1. Some values can be measured observationally, while others can be obtained automatically by using computerized tools.Figure 1. The evaluation processes underlying in the WebQEM methodology. The phases, stages, main processes, inputs and outputs are shown.2.3 The Global Evaluation Phase.Again, two major stages are defined: The design and the implementation of the partial/global quality evaluation. In the design stage, aggregation criteria and a scoring model should be selected. The goal of quantitative aggregation and scoring models is to make the evaluation process well structured, accurate, and comprehensible by evaluators. There are at least two type of models: for example those based on linear additive scoring models [2], and those based on nonlinear multi-criteria scoring models [1] where different attributes and characteristics relationships can be designed. In both cases, the relative importance of indicators is considered by means of weights. For example, if our procedure is based on a linear additive scoring model the aggregation and computing of partial/global indicators or preferences (P/GP), considering relatives weights (W) is based on the following formula:P/GP= (W1 EP1 + W2 EP2+ ... + W m EP m); (1)such that if the elementary preference (EP) is in the unitary interval range the following is held:0 <= EP i <= 1 ; or given a percentage scale, 0 <= EP i <= 100 ;and the sum of weights must fulfill that(W1 + W2 + ... + W m ) = 1; if W i > 0 ; to i = 1 ... m;The basic arithmetic aggregation operator for inputs is the plus (+, or A) connector. The above (1) expression cannot be used to model simultaneity or replaceability of inputs, among other limitations, as we discuss later.Therefore, once a scoring model has been selected, the aggregation process follows the hierarchical structure as defined in the non-functional requirement tree (as the one shown in table 2), from bottom to top. Applying a stepwise aggregation mechanism, a global schema can be obtained in the end. This model allows us to compute partial and global indicators in the implementation stage. The global quality preference represents ultimately the global degree of satisfaction in meeting the stated requirements.2.4 The Conclusion of the Evaluation.In this phase, the documentation of Web product components, the specification of quality requirements, metrics, criteria, elementary and final results are recorded. In the end, the strengths and weaknesses of the assessed product with regard to established goals and user viewpoint can be analyzed and understood by requesters and evaluators. Recommendations can be suggested and justified.2.5 Automating the Process using WebQEM_Tool.The evaluation and comparison processes require both methodological and technological support. We have developed a Web-based tool [16] that supports the administration of evaluation projects. It allows editing and relating non-functional requirements. For instance, in our E-commerce case study, more than ninety attributes have intervened [15]. Then, by means of the automatic or manual edition of elementary indicators, WebQEM_Tool allows to aggregate the elements in order to yield a schema and calculate a global quality indicator for each site. This enables evaluators to assess and compare the quality of Web products. The WebQEM_Tool relies on a Web-based hyperdocument model that supports traceability of evaluation aspects as we will see in Section 3.2. The results of an evaluation are shown through linked pages with textual, tabular and graphical information, which are dynamically generated from tables stored in the data layer.3. Putting WebQEM to WorkIn this section we discuss some issues related with an e-bookstore case study [15]. We have also made evaluations in the museum and academic domains; they can be seen in [12, 13].3.1 About WebApps Quality Requirements.There are many potential attributes, both general and domain specific that contribute to the quality of WebApps. Figure 2, shows a screenshot of an e-store home page (.ar) pointing just some few attributes generally available in this kind of sites. Table 2 documents a wider list of tailorable quality requirements considering a general visitor profile.The requirement tree shown in Table 2 is intended to be reusable among domains. For instance, the Usability characteristic is split up in subcharacteristics such as Global Site Understandability, Feedback and Help Features, and Interface and Aesthetic Features. The Functionality characteristic is decomposed in Searching and Retrieving Issues, Navigation and Browsing Issues, and Domain Specific Functionality and Content. This last component of the tree (where Functionality is the supercharacteristic) should be customized among domains, therefore is no totally intended for reuse. Table 1 outlines the schema we used in the e-bookstore study. We have identified five main components for e-stores (see also the categorization in [8]), namely: Product Information (2.3.1 coded), Purchase Features (2.3.2), Customer Features (2.3.3), Store Features (2.3.4), and Promotion Policies (2.3.5).Figure 2. A screenshot of Cúspide’s home page where some attributes are highlighted.Table 1. The Domain Specific Functionality and Content subcharacteristic for E-bookstore sites (the italic style represents direct or indirect measurable attributes).2.3 Domain Specific Functionality and Content (for E-bookstores)2.3.1 Product Information 2.3.1.1 Product Description 2.3.1.1.1 Basic Book Description 2.3.1.1.2 Book Content & Structure 2.3.1.1.2.1 Book’s Table of Contents 2.3.1.1.2.2 Content Description 2.3.1.1.3 Product Image 2.3.1.1.3.1 Image Availability 2.3.1.1.3.2 Zooming 2.3.1.2 Price Evaluation2.3.1.2.1 Price Comparison Availability 2.3.1.3 Product Rating Availability 2.3.1.4 Related Titles /Authors Recommendation2.3.1.5 Catalog Download Facility 2.3.2 Purchase Features 2.3.2.1 Purchase Mode 2.3.2.1.1 On -line2.3.2.1.1.1 Shopping Basket2.3.2.1.1.1.1 Shopping Basket Availability 2.3.2.1.1.1.2 Continue Buying Feedback 2.3.2.1.1.1.3 Edit/Recalculate Feature 2.3.2.1.1.2 Quick Purchase (1-click or similar)2.3.2.1.1.3 Checkout Features 2.3.2.1.1.3.1 Checkout Security 2.3.2.1.1.3.2 Canceling Feedback 2.3.2.1.2 Off-line2.3.2.1.2.1 Printable Checkout Form 2.3.2.1.2.2 Fax/TE/E-mail Purchase 2.3.2.2 Purchase Policies2.3.2.2.1 Purchase Cancellation Policy 2.3.2.2.2 Return Policy Information2.3.2.2.3 Shipping & Handling Information 2.3.2.2.4 Payment Policy Information 2.3.2.2.5 Resent Purchase (Gift service)2.3.3 Customer Features 2.3.3.1 E-subscriptions2.3.3.2 Customized Recommendations 2.3.3.3 Account Facility 2.3.3.3.1 Account Availability 2.3.3.3.2 Account Security 2.3.3.3.3 Account Configuration 2.3.3.3.3.1 Order History/Status 2.3.3.3.3.2 Account Settings 2.3.3.3.3.3 Address Book2.3.3.4 Customer Revision of a Book 2.3.4 Store Features2.3.4.1 Title Availability Rate 2.3.4.2 Store Ranking 2.3.4.2.1 The Top Books2.3.4.2.2 The Best Seller Books 2.3.5 Promotion Policies2.3.5.1 With-sale Promotion Availability 2.3.5.2 Appetizer Promotion Availability (Contests, Miles, etc.)Though the subtree of table 1 has been specified for the e-bookstore field, it can be easily seen that many of its parts are reusable for a more general e-commerce domain. For instance, in the case of the Purchase Features (2.3.2), we can see two main subfactors: Purchase Mode (2.3.2.1), and Purchase Policies (2.3.2.2). Regarding the Purchase Mode subcharacteristic, online and offline modes are feasible, though the former is becoming more popular as long as confidence in security was increasing [3]. For online purchase, the Shopping Basket , Quick Purchase , and Checkout features are modeled.As remarked elsewhere [18], the shopping basket mechanism is generally used to decouple the selection process from the checkout process of products or services. It is interesting to compare many of these criteria with existing navigation and interface patterns. We can easily argue that when we record and reuse design experiencewe can obtain valuable information for specifying quality attributes or subcharacteristics.Table 2.Tailorable quality requirement tree for a general visitor standpoint (the italic style represents direct or indirect measurable attributes)1. Usability .1.1 Global Site Understandability1.1.1 Global Organization Scheme1.1.1.1Table of Contents1.1.1.2 Site Map1.1.1.3 Global Indexes1.1.1.3.1Subject Index1.1.1.3.2 Alphabetical Index1.1.1.3.3 Chronological Index1.1.1.3.4 Geographical Index1.1.1.3.5 Other Indexes (by audience, by format, hybrid -like alphabetical and subject oriented)1.1.2 Quality of Labeling System1.1.3 Audience-oriented Guided Tour 1.1.3.1Conventional Tour1.1.3.2VR Tour1.1.4 Image Map (Metaphorical, Building, Campus, Floor and Room Imagemaps) 1.2 Feedback and Help Features1.2.1 Quality of Help Features1.2.1.1 Global Help (for first-time visitors) 1.2.1.2 Specific Help (for searching, checking out, etc.)1.2.2 Addresses Directory1.2.2.1 E-mail Directory1.2.2.2 Phone-Fax Directory1.2.2.3 Post mail Directory1.2.3 Link-based Feedback1.2.3.1 FAQ Feature1.2.3.2 What’s New Feature1.2.4 Form-based Feedback1.2.4.1 Questionnaire Feature1.2.4.2 Comments/Suggestions1.2.4.3 Subject-Oriented Feedback1.2.4.4 Guest Book1.3 Interface and Aesthetic Features 1.3.1 Cohesiveness by Grouping Main Control Objects1.3.2 Presentation Permanence and Stability of Main Controls 1.3.2.1Direct Controls Permanence (Main,Search, Browse Controls)1.3.2.2Indirect Controls Permanence1.3.2.3Stability1.3.3Style Issues1.3.3.1Links Color Style Uniformity1.3.3.2Global Style Uniformity1.3.4 Aesthetic Preference1.4 Miscellaneous Features1.4.1Foreign Language Support1.4.2 Website Last Update Indicator1.4.2.1 Global1.4.2.2 Scoped (per sub-site or page)1.4.3Screen Resolution Indicator2. Functionality .2.1 Searching and Retrieving Issues2.1.1 Website Search Mechanisms2.1.1.1Global Search2.1.1.2Scoped Search (e.g., MuseumCollections, Books, Academic Personnel)2.1.2 Retrieve Mechanisms2.1.2.1Level of Retrieving Customization2.1.2.2 Level of Retrieving Feedback2.2 Navigation and Browsing Issues2.2.1 Navigability2.2.1.1 Orientation2.2.1.1.1Indicator of Path2.2.1.1.2 Label of Current Position2.2.1.2 Average of Links per Page2.2.2 Navigational Control Objects2.2.2.1 Presentation Permanence andStability of Contextual (sub-site) Controls2.2.2.1.1 Contextual Controls Permanence2.2.2.1.2 Contextual Controls Stability2.2.2.2 Level of Scrolling2.2.2.2.1 Vertical Scrolling2.2.2.2.2 Horizontal Scrolling2.2.3 Navigational Prediction2.2.3.1Link Title (link with explanatory help)2.2.3.2Quality of Link Phrase2.2.4 Browse Mechanisms2.2.4.1Quick Browse Controls2.3 Domain Specific Functionality andContentNote: see, for example, the specification toe-bookstores in Table 1.3. Reliability .3.1 Non-deficiency3.1.1 Link Errors3.1.1.1 Broken Links3.1.1.2 Invalid Links3.1.1.3 Unimplemented Links3.1.2 Spelling Errors3.1.3 Miscellaneous Errors or Drawbacks3.1.3.1 Deficiencies or absent features dueto different browsers3.1.3.2 Deficiencies or unexpected results(e.g., non-trapped search errors, frameproblems, etc.) independent of browsers3.1.3.3Orphan Pages3.1.3.4Destination Nodes (unexpectedly)under Construction4. Efficiency .4.1 Performance behavior4.1.1 Quick Static Pages4.2 Accessibility4.2.1 Information Accessibility4.2.1.1 Support for text-only version4.2.1.2 Readability by deactivating theBrowser Image Feature4.2.1.2.1 Image Title4.2.1.2.2 Global Readability4.2.2 Window Accessibility4.2.2.1 Number of panes regarding frames4.2.2.2 Non-frame Version3.2 Designing and Implementing the Elementary Evaluation.As mentioned in Section 2.2, the evaluators should define, for each quantifiable attribute, the basis for the elementary evaluation criterion, and perform the measurement and preference mapping process.In order to record the information that is needed during the evaluation processes, we have defined a descriptive specification framework as exemplified in tables 3 and 4. Specific information about definition of attributes, subcharacteristics and characteristics as well as metrics, elementary preference criteria, scoring model components and calculations are recorded. (Notice that codes in the templates in tables 3 and 4 correspond to those shown in the requirement tree).Once evaluators have designed and implemented the elementary evaluation process they should be able to model attributes, subcharacteristics, and characteristics relationships. They should consider not only the relative importance of each attribute in the group but also if the attribute (or subcharacteristic) is a mandatory, an alternative or a neutral one. For this purpose, a robust aggregation and scoring model is needed as we discuss in the next subsection.Table 3.Template and example with the characteristic items. WebQEM_Tool uses this information.Title (code):Reliability (3)Type:CharacteristicFactor:QualitySubcharacteristic/s (code/s):Non-deficiency (3.1)Definition / Comments:“The capability of the software product to maintain a specified level ofperformance when used under specified conditions”[5]Model to determine the Global/Partial Computation:Nonlinear multi-criteria scoring model; specifically, the Logic Scoring of Preference model [1]Employed Tool/s:WebQEM_ToolArithmetic / Logic Operator: C - - (Note: The arithmetic or logic operator item for thesubcharacteristic and characteristic aggregation will be explained in thesubsection 3.3)Weight:0.2Calculated Preference Value/s: A set of values for Reliability, as shown in table 5.Table 4.Template and example with the attribute items.Code / Title: 3.1.1.1. Broken LinksType:AttributeHighest level characteristic:Reliability (3)Supercharacteristic/s (code):Link Errors (3.1.1)Definition / Comments It represents found links that lead to missing destination pages bothinternal and external static pages of a site (known also as dangling links).“Users get irritated when they attempt to go somewhere, only to get theirreward snatched away at the last moment by a 404 or otherincomprehensible error message”. /alertbox/980614.htmlTemplate of Metric and Parameters:Note: The metric and parameters item links another template with information of the selected metric criterion, the expected and planned values, measurement dates among other fields. For instance, the metric criterion is:X = #Broken_Links / #Total_Links_of_Site.For each e-store in the field study, we got the respective X valueData Collection Type:Note: The data collection type item records whether the data are gatheredmanually or automatically and the kind of the employed tool (if that isdone automatically, as for the Broken Links attribute).Employed Tool/s:our Website MA tool, among othersElementary Preference Function:EP= 1 (or 100%) if X = 0;EP = 0 (or 0%) if X >= X max ;otherwise EP= (X max – X) / X max if 0 < X < X maxX max = 0.06Weight:0.5Elementary Preference Value/s:Cúspide’s site yielded an elementary preference of 99.83 %, Amazon,98.40 %, Barnes and Noble 97.45 %, Borders, 76.34, and Díaz de Santos60.07 %3.3 Designing and Implementing the Partial/Global Evaluation.In these stages (see Fig. 1), an aggregation and scoring model should be selected and applied. The hierarchically grouped attributes, subcharacteristics, and characteristics will then be related by arithmetic or logic operators accordingly.As commented in section 2.3, we can choose between (1) a linear additive scoring model; and (2) a nonlinearmulti-criteria scoring model.The additive scoring model cannot be used to model simultaneity or replaceability of inputs; they are not useful to express for example simultaneous satisfaction of several requirements as inputs. Additivity assumes that insufficient presence of a specific attribute (input) can always be compensated by sufficient presence of any other attribute. Furthermore, additive models are unable to model mandatory requirements; i.e., the total absence of a necessary attribute or subcharacteristic cannot be well compensated by means of the high presence of others. Instead, if we use a nonlinear multi-criteria scoring model we can deal with simultaneity, neutrality, replaceability, and other input relationships using aggregation operators based on the weighted power means mathematical model [1]. This model, so-called Logic Scoring of Preferences (LSP), is a generalization of the additive-scoring model, and can be expressed as follow:P/GP(r)= (W1 EP r1 + W2 EP r 2+ ... + W m EP r m)1/ r ; (2)where -∞ <= r <= +∞ ; P/GP(- ∞)= min (EP1 , EP2 , ... , EP m) and;P/GP(+∞) = max (EP1 , EP2, ... , EP m);The power r is a parameter (a real number) selected in order to achieve the desired logical relationship and intensity of polarization of the aggregation function. If P/GP(r) is closer to the minimum then such a criterion specifies the requirement for the simultaneity of inputs. Conversely, if it is closer to the maximum then it specifies the requirement for the replaceability of inputs.As the reader may see, the formula (2) is additive when r = 1, which models the neutrality relationship; i.e., the formula remains the same as in the first additive model. In addition, (2) is supra-additive for r > 1 which models the disjunction or replaceability of inputs. And it is sub-additive for r < 1, (with r != 0) which models the conjunction or simultaneity of inputs.In the case study, the use of this last model was selected. A seventeen-level approach of conjunction-disjunction operators was used, as defined by Dujmovic [1]. Each operator in the model corresponds to a particular value of the r parameter. When r = 1 the operator is tagged with A (or the + sign). The C or conjunctive operators range from weak (C-) to strong (C+) quasi-conjunction functions, i.e., from decreasing values of r, starting from r < 1.In general, the conjunctive operators imply that, a low quality of an input preference can never be well compensated by a high quality of some other input to output a high quality preference (in other words, a chain is as strong as its weakest link). Conversely, disjunctive operators (D operators) imply that a low quality of an input preference can always be compensated by a high quality of some other input.Figure 3. Once the weights and operators were agreed, and the schema checked, the WebQEM_Tool could yield the partial and global preferences as shown in the right-side pane.In order to design the LSP aggregation schema, the following key basic questions (which are part of the Global Preference Criteria Definition task, in Fig. 1), should be answered : (1) what is the kind of relationship among this group of related attributes/subcharacteristic, etc.?, is it either a conjunctive, or disjunctive or neutral relationship?; (2) what is the level of intensity of the logic operator from a weak to strong conjunctive/disjunctive polarization?; (3) what is the relative importance or weight of each element into the group?The WebQEM_Tool allows evaluators to select the aggregation and scoring model. When using the additive scoring model, the aggregation operator is A for all the tree composites (subcharacteristics and characteristics). Instead, if evaluators select the LSP model, the operator must be indicated for each subcharacteristic and characteristic. Fig. 3, shows a partial view of the enacted schema for as generated by our tool.3.4 Analyzing and Recommending.Once the final execution of the evaluation was performed and agreed, decision-makers are ready to analyze the results and draw the conclusions.Table 5, shows the final values for Usability, Functionality, Reliability and Efficiency characteristics and the global Quality indicator in our case study in the e-commerce domain. It also shows the Domain Specific Functionality and Content subcharacteristic (2.3 in Table 1) for the Functionality characteristic.The colored quality bars at the right side in Fig. 4, indicate the acceptability levels; evaluators can easily see the quality level each e-bookstore has reached. For instance, a scoring within a gray bar shows the need for improvement actions; meanwhile, an unsatisfactory rating level may indicate that urgent change actions must be performed. A scoring within a green bar can be interpreted as a satisfactory quality for the analyzed feature. This e-bookstore study was deeply discussed in [15].Table 5.Summary of partial and global preferences for each e-bookstore assessed in late 1999. (For legibility reasons, this table is not shown in the tool screenshot as in Fig. 4)Characteristic and Sub-characteristics Amazon B&N Cúspide Díaz Stos Borders1. Usability76.1682.6275.9356.0972.672. Functionality83.1580.1261.6928.6461.452.1 Searching and Retrieving Issues1001009142.6772.062.2 Navigation and Browsing Issues70.7169.8573.2564.1251.952.3 Domain Specific Functionality and Content81.9976.5345.8114.4261.552.3.1 Product Information63.7242.2040.6410.2015.982.3.2 Purchase Features91.7684.8467.7217.1181.922.3.3 Customer Features100852028.08652.3.4 Store Features10096.8071.2033.6093.572.3.5 Promotion Policies601004001003. Reliability99.4499.1190.9778.5191.664. Efficiency96.8874.5490.1786.0190.90 Global Quality Preference86.8182.9575.5250.3774.86Figure 4.The screenshots show the diverse type of information (as textual, tabular and graphical) yielded by the WebQEM_Tool. The right side shows a graph with the final rankings to each evaluated e-bookstore.。