AN EVALUATION AND SELECTION METHODOLOGY FOR DISCRETE-EVENT SIMULATION SOFTWARE
Anintegratedmethodofselectingenvironmentalcovariat
Journal of Integrative Agriculture 2019, 18(2): 301–315RESEARCH ARTICLEAvailable online at ScienceDirectAn integrated method of selecting environmental covariates for predictive soil depth mappingLU Yuan-yuan1, 2, LIU Feng1, ZHAO Yu-guo1, 2, SONG Xiao-dong1, ZHANG Gan-lin1, 21 State Key Laboratory of Soil and Sustainable Agriculture, Institute of Soil Science, Chinese Academy of Sciences, Nanjing 210008, P.R.China2 University of Chinese Academy of Sciences, Beijing 100049, P.R.ChinaAbstractEnvironmental covariates are the basis of predictive soil mapping. Their selection determines the performance of soil mapping to a great extent, especially in cases where the number of soil samples is limited but soil spatial heterogeneity is high. In this study, we proposed an integrated method to select environmental covariates for predictive soil depth mapping. First, candidate variables that may influence the development of soil depth were selected based on pedogenetic knowledge. Second, three conventional methods (Pearson correlation analysis (PsCA), generalized additive models (GAMs), and Random Forest (RF)) were used to generate optimal combinations of environmental covariates. Finally, three optimal combinations were integrated to produce a final combination based on the importance and occurrence frequency of each environmental covariate. We tested this method for soil depth mapping in the upper reaches of the Heihe River Basin in Northwest China. A total of 129 soil sampling sites were collected using a representative sampling strategy, and RF and support vector machine (SVM) models were used to map soil depth. The results showed that compared to the set of environmental covariates selected by the three conventional selection methods, the set of environmental covariates selected by the proposed method achieved higher mapping accuracy. The combination from the proposed method obtained a root mean square error (RMSE) of 11.88 cm, which was 2.25–7.64 cm lower than the other methods, and an R2 value of 0.76, which was 0.08–0.26 higher than the other methods. The results suggest that our method can be used as an alternative to the conventional methods for soil depth mapping and may also be effective for mapping other soil properties. Keywords: environmental covariate selection, integrated method, predictive soil mapping, soil depth1. IntroductionSoil mapping, classification, and pedologic modelinghave been important drivers in the advancement of ourunderstanding of soil from the earliest scientific studies ofsoils (Brevik et al. 2016). Digital (or predictive) soil mappingmethods provide a rapid and inexpensive way to predictsoil types or properties over large areas (Scull et al. 2003;Yang et al. 2015; Minasny and McBratney 2016; Pásztoret al.2016; Zhang et al.2017). Soil-landscape models Received 21 December, 2017 Accepted 27 March, 2018LU Yuan-yuan, E-mail: exlimit00@; CorrespondenceZHANG Gan-lin, Tel: +86-25-86881279, E-mail: glzhang@issas.© 2019 CAAS. Published by Elsevier Ltd. This is an openaccess article under the CC BY-NC-ND license (http:///licenses/by-nc-nd/4.0/)doi: 10.1016/S2095-3119(18)61936-7302LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315that characterize soils as a function of environmental variables, including climate, organisms, topography, parent material, and time, are the theoretical basis for predictive soil mapping. Specific landscape factors determine the formation of specific soil properties. The spatial distribution patterns of soil types or properties result from the interactive action of multiple soil-forming factors (Jenny 1994; Hudson 1992; McBratney et al. 2003). With the rapid development of geographical information techniques and remote sensing products, numerous datasets of environmental variables have been acquired. These variables can characterize soil-forming factors from multiple aspects. However, a problem also arises: how to select an optimal combination of environmental variables to achieve accurate and steady soil mapping performance? This is an important issue in digital soil mapping because not all environmental covariates have equal predictive capability in modeling, and some covariates may cause noise that reduces the predictive capability of the employed models (Bui et al. 2016). Variable selection attempts to identify and remove the variables that penalize the performance of a model because they are useless, noisy, redundant, or correlated by chance (Zou et al. 2010; Liu et al. 2016). Liu et al. (2009, 2012) discussed the issue of environmental covariates in low relief areas. Behrens et al. (2010) observed that the selection of features appears to aid in the interpretation of data about soil formation and that only a small number of features are typically required to achieve good predictions. Variable selection aims to work with fewer covariates and thus simplify the predictive model, allowing for better computing efficiency without decreasing the performance of model (Lacoste et al. 2016). Variable selection can also provide robust models that may be readily transferred, and it allows non-expert users to build reliable models with only limited expert intervention (Zou et al. 2010).Pearson correlation analysis (PsCA) is the most common method for variable selection. As commonly used modeling methods, generalized additive models (GAMs) and Random Forest (RF) algorithms not only can select variables in their own ways but can also be used for digital soil mapping. Rodriguez-Galiano et al. (2014) applied PsCA in combination with multiple linear regression to identify variables that were significantly correlated with soil organic matter. Tesfa et al. (2009), Zhi et al. (2017), and Heung et al. (2014) selected environmental variables that affected soil development based on variable importance determined by RF. Tesfa et al. (2008) and Deng et al. (2015) extracted optimal environmental variables using GAMs. These studies demonstrated that the removal of non-informative variables can produce better prediction and simpler models. However, the three variable selection methods described above quantify the predictive capability of environmental variables based on different mechanisms. Each method has its strengths and weaknesses, and no studies to date have determined which method is the best suited to a particular data, so it is difficult to choose an optimal method of variable selection. In addition, the digital soil mapping performances of the corresponding optimal combinations selected by the different methods are not steady. The performance of a combination is often related to its original selection method. For example, for an optimal combination selected by GAMs, the prediction result of GAMs usually outperforms other methods (such as RF), and vice versa. Therefore, there is still a need to explore a new variable selection method for digital soil mapping studies.We choose soil depth as the target soil property for studying environmental variable selection and digital soil mapping. One reason for investigating soil depth is that it is an important input parameter for hydrological and ecological modeling. It can affect hydrological processes (Liang et al. 1996; Schenk and Jackson 2005; Tesfa et al. 2009), influence soil quality and productivity (Power et al. 1981), determine the soil’s capacity to store and hold moisture (Boer et al. 1996), and control the soil’s surface and subsurface processes (Heimsath et al. 2001). Soil depth is also a necessary parameter when calculating carbon and other elemental stocks. Additionally, it is often difficult to accurately map soil depth due to its high heterogeneity and difficulty of data collection, especially in mountainous areas. Being able to predict the soil depth in the complex landscape areas can lead to a better understanding of soil erosion or water storage (Hudson 1992; Dietrich et al. 1995; Lu et al. 2014; Scarpone et al. 2016).The objective of this study is to propose a new method of selecting environmental covariates, test its effectiveness through the use of the selected covariates for predictive soil depth mapping and compare it with current variable selection methods.2. Materials and methods2.1. Study areaThe study area covered approximately 30000 km2 in the upper reaches of the Heihe River Basin in Northwest China, on the northeast margin of the Tibetan Plateau (latitude 37.71° to 40.03°N, longitude 96.78° to 101.2°E) (Fig. 1). This region is dominated by the Qilian Mountains, which range in elevation from 1684 to 4600 m above sea level with an average of 3535 m. The study area has a typical plateau continental climate, and the mean annual temperature ranges from –12.3 to 6.6°C. The mean annual precipitation ranges from 480 mm in the southeast to 80 mm303LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315in the northwest, with a mean value of 300 mm.As a result of the special hydrothermal conditions and the undulating relief, the region has a notable vertical zonal vegetation pattern (Wang et al. 2016). There are a rich variety of soil types in this study area, mainly including Inceptisols, Gelisols, Entisols, Aridisols, and Histosols, according to Soil Taxonomy (SSS 2010). In addition, the parent materials are complex and diverse, mainly including moraine deposits, slope deposits, residual deposits, alluvial-diluvial deposits, gully deposits, alluvial deposits, etc. (Li X et al. 2017). Due to the complex topographical conditions, various types of landforms are dominated by high and medium-height mountains, hills, plains, platforms, river terraces, and floodplains, which were formed by various processes, such as glaciation, periglaciation, and erosion.2.2. Soil samplingSoil surveys were conducted during July and August of 2012 and 2013, and 129 soil profiles were studied (Fig. 1). Due to the low accessibility of the alpine environment, a purposive sampling strategy (Zhu et al. 2008) was adopted to determine representative sampling sites. During the sampling process, detailed information about the locations of the sampling sites, including the characteristics of the landscape environment, vegetation, and rock distribution, was recorded. In addition, the soil profile at each sampling site was described in several pedogenetic horizons based on main morphology of horizons. Based on the above information, the soil depth at the sampling site was determined. In this study, soil depth refers to the solum thickness, which is the distance from the surface to the upper boundary of the “non-soil mass”, which is either bed rock or material that contains >75% by volume of >2 mm gravel. For more details, readers can refer to Yi et al. (2015).2.3. Environmental variablesIn this study, the following environmental variables were selected to set up and test the selection methods.A digital elevation model (DEM) and its derivatives A freely available DEM was acquired from a Shuttle Radar Topography Mission DEM (SRTM DEM) with a resolution of 90 m. Seven topographic attributes, including elevation (ELE), slope, aspect, plan curvature (Plancur), profile curvature (Procur), topographic wetness index (TWI), and vertical distance to channel network (VDCN), were derived from the DEM using the System for Automated Geoscientific Analyses (SAGA) geographic information system. Three types of transformation were employed for the aspect. The original aspect had a value between 0 and 360°. The first transformation was expressed in absolute values between 0 and 180°, which represented north and south, respectively, and was denoted by deviN. The second transformation, in which a cosine transformation was applied to converting the aspect into a range from 0 to 1, was denoted by cos_Asp. The third was the cosine transformation of deviN, which was denoted by cos_dN.Climate data Mean annual temperature (MAT), mean annual precipitation (MAP), and drought index (DI) during the previous 30 years were derived as 1-km grids from the interpolation of data from 673 meteorological stations in China.Landsat 5 Thematic Mapper (TM) imagery Compared to the other remotely sensed images, the Landsat 5 TM image was chosen to represent the “organism” soil-forming factor in this study by its ability of multichannel observation with a frequent revisiting period, fine high spatial resolutions, and free availability. A 30 m Landsat 5 TM mosaic image was acquired from the Cold and Arid Regions Sciences Data Center in China (DCHP 2011). The visible-red band (B3, 0.63–0.69 µm), near-infrared band 4 (B4, 0.76–0.96 µm) and shortwave infrared band 5 (B5, 1.55–1.75 µm) were retained. To represent the vegetation intensity, we used the normalized difference vegetation index (NDVI), which was calculated as (B4–B3)/(B4+B3).T o take advantage of relatively detailed terrain information, the environmental variables were collected and converted into raster format with a 90-m resolution using ArcGIS 9.3 (ESRI Inc., USA). The spatial distributions of the main environmental variables are presented in Fig. 2.2.4. An integrated method for covariate selectionThe integrated method (IM) consisted of three main steps (Fig. 3). First, we applied pedogenetic knowledge in the area of interest to determine candidate environmental variables for soil depth prediction. Second, three97°E 98°E 99°E 100°E101°E97°E 98°E99°E100°E101°E38°N39°N40°N 38°N39°N40°NFig. 1 Location of study area and soil sampling site.304LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315conventional variable selection methods (PsCA, GAMs and RF) were used to derive three optimal combinations of environmental variables, namely, PsCA_v, GAMs_v, and RF_v, respectively. Third, a comprehensive process was conducted on the three optimal variable combinations to generate a set of comprehensive covariates, which was denoted by IM_v.Selection of candidate covariates based on pedogenetic knowledge The genesis and development of soil depth are the result of the interactive action of topographical, climatic, biological, and other factors. Therefore, selecting the environmental variables affecting soil depth development is the key to digital soil mapping. Pedogenetic knowledge and previous research on the soil depth indicate that topography is an important factor that affects soil formation and development, particularly in alpine areas (Gessler et al.1995; Böhner and Selige 2006; Körner 2007; Buol et al.2011; Sarkar et al.2013). Topography influences soil development by affecting the redistribution of water-temperature conditions and soil matter. In mountainous areas, soil at the tops of slopes generally moves outward and downward due to the effects of flowing water and gravity, and it ultimately deposits in valleys or lower and more gently sloped areas. Hence, many factors, including elevation, slope gradient and orientation, surface roughness, water distribution, and distance from watercourses, may affect the transport distance and velocity as well as the deposition location. Moreover, temperature and precipitation vary with elevation in alpine areas; therefore, the vegetation varies with elevation and climatic factors. The interaction of a multitude of factors comprehensively affects the development of soil depth.This study analyzes the relationship between soil depth and the soil-forming environment. Environmental variables that affected the genesis and development of soil depth were selected based on pedogenetic knowledge and are referred to as candidate variables. This method of selecting candidate variables is called the pedogenetic knowledge (PG) method. The combination of candidate variables is denoted by cand_v. A total of 17 variables, elevation, slope, aspect, cos_Asp, deviN, cos_dN, Plancur, Procur, TWI, VDCN, MAP, MAT, DI, B3, B4, B5, and NDVI, were selected for cand_v.Derivation of multiple optimal covariate combinationsELE (m)46001684Procur3.26–4.02B425510NDVI0.78–0.47MAT (°C)4.8–18.2300km15075NTWI13.42.6VDCN (m)1650Slope (°)69Aspect (°)360Fig. 2 Spatial distributions of environmental variables in this study area. ELE, elevation; Procur, profile curvature; TWI, topographic wetness index; VDCN, vertical distance to channel network; B4, Landsat 5 TM band 4; NDVI, normalized difference vegetation index; MAT, mean annual temperature.305LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315from different selection algorithms PsCA, GAMs and RFs methods were used to select optimal combinations of covariates. The basic principle and selection processes of the three methods are as follows:(1) Pearson correlation analysis (PsCA)The Pearson correlation coefficient can be used to determine the degree of linear correlation between two series and is obtained by dividing the standard deviations of two variables by their covariance. The calculation function is defined as following (Tang 2008):ρX, Y =corr (X, Y)=σX σY cov (X, Y)=σX σY E [(X–μX )(Y–μY )](1)Where, ρX , Y is the Pearson correlation coefficient; X and Y are two variables; cov (X , Y ) is the covariance between X and Y ; σX and σY are the standard deviations of X and Y , respectively; μX and μY are the means of X and Y , respectively; and E is the expectation.The PsCA can be used to calculate the correlation coefficient and P -value between the predictor and response variables or among the predictor variables. First, the variables significantly correlated with soil depth at a statistical level of 0.05 were selected. Subsequently, the multicollinearity among variables was checked, and the variables with the lowest correlation coefficient were removed. Finally, theenvironmental variables were gradually selected to form an optimal combination of environmental covariates (denoted by PsCA_v). The PsCA method was implemented using the “psych” package in the Statistical Software R 3.1.2 (R Development Core Team 2014).(2) Generalized additive models (GAMs)GAMs, which was proposed and developed by Hasitie and Tibshirani (1990), is non-parametric extensions of generalized linear models. The advantages of GAMs are that they are capable of directly dealing with the nonlinear relationships, identifying the data structure and determining the data pattern using non-parametric methods, and “talking” using data instead of models, and they do not require an assumed data distribution. In addition, GAMs are capable of examining the importance of each factor and selecting the optimal model. The expression is as follows:g(μ)=α+f j (x j )mj =1∑(2)Where, g () is expressed as a sum of the link functions of each variable; μ is the expectation of the response variable; α is the intercept or constant term; m is the number of independent variables; and f j is an unspecified smoothing function, which can be obtained by a smoothing method, such as a locally weighted regression or spline function.PsCA method RF methodGAMs method Soil depth mappingElevation PsCA_v GAMs_vRF_vIM_vSlope Curvature … …Precipitation The flowchart of integrated methodPsCA_vGAMs_vRF_vcand_vCalidation and comparison PedogeneticKnowledgecand_vImportance levelPreliminary covariatesOccurrence frequency Fig. 3 Flowchart diagram of this study. PsCA, GAMs, RF represent the covariate selection methods of Pearson correlation analysis, generalized additive models and Random Forest; cand_v, PsCA_v, GAMs_v, RF_v, IM_v represent the combinations of environmental covariates derived from pedogenetic knowledge, PsCA, GAMs, RF and the integrated method.306LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315The spline smooth function is applied in this study.Based on the combinations of cand_v, the single-factor analysis and multi-factor analysis were performed to select the environmental factors that played dominant roles in the model. According to the size and level of the estimated degrees of freedom and concavity, the optimal combination of environmental factors was obtained based on the GAMs method (denoted by GAMs_v). The GAMs method was implemented using the “mgcv” Package in R (Wood 2001).(3) Random Forest (RF)RF is a powerful ensemble-learning algorithm based on the classification and regression tree (Breiman 2001). The principle of RF is to grow several tree models using random samples and random feature selection techniques and then aggregate these tree models into a comprehensive classifier. During the generation process of RF, each tree is grown using approximately two-thirds of the training data, and the other third is used as out-of-bag (OOB) error data for validation. In this study, we adopted the mean decrease in accuracy (MDA) as the variable importance index. For the regression, the MDA was the average increase in the squared OOB residuals when the variable was permuted (Liaw and Wiener 2002).The optimal covariates were selected based on the variable importance calculation by RF. In other words, a model was first established based on the candidate variables, and the variables were then sorted based on their importance. Subsequently, the variable with the lowest importance was eliminated. This process was repeated until an optimal combination of covariates (denoted by RF_v) was selected. During this process, the relative importance of each environmental variable was evaluated based on the average of 100 repetitive calculations. The “randomForest” package in R was used to establish the relationships between soil depth and predictor variables.Generation of final covariate combination through a comprehensive process The three optimal combinations of environmental covariates (i.e., PsCA_v, GAMs_v, and RF_v) were obtained using conventional selection methods (PsCA, GAMs, and RF, respectively). Subsequently, based on the combinations of PsCA_v, GAMs_v, and RF_v, the comprehensive process was created in two stages (Fig. 3). First, based on the relative importance of each covariate in the different combinations, the integrated score was calculated by the sum of each covariate divided by the corresponding frequency. Then, the covariates were ranked under the rule of high scores to low scores, and some of the covariates with lower scores were removed. The combination of preliminary covariates was then constructed. Second, the occurrence frequencies of preliminary covariates were summarized, and the highest-frequency covariates were used in the final combination (denoted by IM_v).2.5. Evaluation of the methodThe integrated method was evaluated by comparing its performance in predictive soil depth mapping with those of the three conventional variable selection methods (PsCA, GAMs, and RF) and the PG. Two types of prediction methods, RF and support vector machine (SVM), were applied to predict soil depth using the different variable selection methods.As a relatively new data mining method, RF is not only capable of selecting optimal variables based on the importance ranking of the variables, but also able to deal with various types of predictor variables and avoid overfitting phenomena. In addition, RF outperforms other methods for predictive soil mapping due to its ability to deal with nonlinear problems and has been extensively used in research on predictive soil mapping (Breiman 2001; Heung et al. 2014; Zeng et al. 2017; Zhi et al. 2017). Therefore, RF was used in this study to perform soil depth mapping based on the combinations obtained from the different variable selection methods. The number of variables used to grow each tree (mtry) and the number of trees in the forest (ntree) are two user-defined parameters that must be optimized because they can influence the predictive performance of the RF model (Rad et al. 2014). To fit an RF model, a default value of mtry (one-third of the total number of predictors) was used. The default value of ntree (500) has been proven that it can not yield stable results (Grimm et al. 2008). Thus, ntree=1000 was applied in the RF model.To further verify the reliability of the integrated method, SVM was introduced to compare the prediction performances of the five different selection methods. SVM is a machine-learning method based on statistical learning theory that transforms the original input space into a higher-dimensional feature space to find an optimal separating hyperplane (Vapnik 1998; Bui et al.2009; Abe 2010). The “e1071” package in R was used to implement SVM. The radial basis function, which is one of the kernel functions, was selected. The model with the highest 10-fold cross-validation overall accuracy was selected as the optimal settings.The performances of the RF and SVM models were evaluated using a 10-fold cross-validation with 100 iterations. Four statistical indices, including the coefficient of determination (R2), Lin’s concordance correlation coefficient (LCCC) (Lin 1989), mean error (ME)and root mean square error (RMSE), were calculated to assess the mapping reliability. These indices were calculated as follows:R2=(Oi–O)2(Pi–O)2ni=1∑ni=1∑(3)307LU Yuan-yuan et al. Journal of Integrative Agriculture 2019, 18(2): 301–315LCCC=σ2O+σ2P +(–)22rσO σP(4)ME=n 1(P i –O i )ni =1∑(5)RMSE=n 1(P i –O i)2ni =1∑ (6)Where, P i and O i are the predicted and observed soil depth for i th observation, respectively; n is the number of samples; and are the means of the predicted and observed values, respectively; σ2p and σ2o are the variances of the predicted and observed values, respectively; and r is the Pearson correlation coefficient between the predicted and observed values.R 2 measures the matching of the linear relationship between the predicted and observed values. The closer the R 2 is to 1, the better the results are. LCCC provides a measure of how the predictions and observations adhere to a 1:1 relationship, where 1 corresponds to a perfect match between the predicted and observed values. ME indicates whether the model as a whole over- or under-predicted values (bias), whereas RMSE provides an indication of the prediction accuracy. Smaller ME and RMSE values indicate better model performances.3. Results3.1. Descriptive statistics of soil depth and environmental covariatesThe basic statistics of soil depth and environmental covariates are summarized in T able 1. Soil depth varied from 0 to 200 cm with an average of 76.75 cm, and showed a moderately high correlation of variation, with a variance coefficient of 63.59%. Based on the skewness coefficient of 0.15, soil depth had a positively-skewed distribution. The complex landscape environment was reflected by the larger amplitude of variations of some environmental covariates and the significant discrepancies between the different covariates.Pairwise coefficients of correlation were computed to determine the significances of correlations between soil depth and quantitative predictors. Soil depth was positively correlated with B4 (r =0.50), NDVI (r =0.31), MAT (r =0.30), and Procur (r =0.25), and negatively correlated with elevation (r =–0.43), VDCN (r =–0.46), and Plancur (r =–0.23) (P <0.01 level). Soil depth was positively correlated with TWI (r =0.19) and B5 (r =0.20), and negatively correlated with slope (r =–0.18) (P <0.05 level). The correlation coefficients between environmental covariates were also computed. ELE and MAT , slope and TWI, and DI and MAP were negatively correlated at the 0.01 level, whereas B3 was positively correlated with NDVI and DI at 0.01 the level.3.2. Different combinations of environmental covariatesFive combinations of environmental variables were derived based on different variable selection methods (Table 2).As shown in Table 2, the number of variables in the combinations ofT a b l e 1 S u m m a r y s t a t i s t i c s o f s o i l s a m p l e s a n d e n v i r o n m e n t a l c o v a r i a t e s 1)P r o p e r t yD e p t h (c m )E L E(m )S l o p e (d e g r e e )A s p e c t (d e g r e e )d e v i N (d e g r e e )c o s _A s p c o s _d NP l a n c u rP r o c u rT W IV D C N (m )M A T (°C )M A P (m m )D IB 3B 4B 5N D V IM i n i m u m 01 827211–1–1–2.15–1.382.580–10.82644193812–0.13M e d i a n 723 3651719366–0.020.15–0.110.123.8529–4.582916.2838831010.37M e a n 76.753 3181618475–0.020.03–0.080.113.957–4.452541043821000.33M a x i m u m 2004 59439357178111.61.326.093054.55403461221361620.69S t a n d a r d d e v i a t i o n 48.496608.76117520.660.70.580.520.79713.91057.712019280.24V a r i a n c e c o e f fi c i e n t (%)63.1819.8954.7563.5969.333 3002 33372547320.2612587.6441.3477.146.5123.172872.73S k e w n e s s 0.15–0.330.35–0.030.37–0.03–0.13–0.12–0.130.71.750.48–0.232.11.10.1–0.24–0.22K u r t o s i s–1.03–0.43–0.65–1.48–1.11–1.38–1.541.04–0.340.062.44–0.51–1.534.740.89–0.170.29–1.371)E L E , e l e v a t i o n ; d e v i N , c o s _A s p a n d c o s _d N , t h e t r a n s f o r m a t i o n s o f a s p e c t ; P l a n c u r , p l a n c u r v a t u r e ; P r o c u r , p r o fi l e c u r v a t u r e ; T W I , t o p o g r a p h i c w e t n e s s i n d e x ; V D C N , v e r t i c a l d i s t a n c et o c h a n n e l n e t w o r k ; M A T , m e a n a n n u a l t e m p e r a t u r e ; M A P , m e a n a n n u a l p r e c i p i t a t i o n ; D I , d r o u g h t i n d e x ; B 3, L a n d s a t 5 T M b a n d 3; B 4, L a n d s a t 5 T M b a n d 4; B 5, L a n d s a t 5 T M b a n d 5; N D V I , n o r m a l i z e d d i f f e r e n c e v e g e t a t i o n i n d e x .。
优劣解距离综合评价法英文
优劣解距离综合评价法英文English:The Comprehensive Evaluation Method of Advantages and Disadvantages Resolution Distance, also known as the CODAS method, is a multi-criteria decision-making technique used to assess alternatives based on their performance across multiple criteria. This method involves several steps, including the determination of criteria, the normalization of criteria weights, the establishment of a decision matrix, the calculation of the performance scores for each alternative, and the final ranking of alternatives. One of the key features of the CODAS method is its ability to handle both qualitative and quantitative criteria, allowing decision-makers to incorporate a diverse range of factors into the evaluation process. Additionally, the CODAS method provides a systematic framework for resolving trade-offs between advantages and disadvantages, enabling decision-makers to make informed decisions that align with their objectives and preferences. By considering the relative importance of criteria and the performance of alternatives across those criteria, the CODAS method facilitates a comprehensive and transparent evaluation process that can support robust decision-making in various contexts.Translated content:综合评价法优劣解距离方法，也称为CODAS方法，是一种多准则决策技术，用于根据不同标准的绩效评估替代方案。
国际学术会议发言稿(完整版)
国际学术会议发言稿国际学术会议发言稿第一篇：国际学术会议发言稿1. prologuethank ou, mr. hairman, for our graious introdution. i am honored to have the hane to address ou on this speial oasion. the topi of m paper is “transation ost and farmers’ hoie of agriultural produts selling”. the outline of m talk as follos. the first part i ant to introdue the bakground ofthis researh. the seond part suggests a simple household hoie model .the third part overs the data used in this researh. and then, e introdue the empirial results. finall, a simple onlusion is given.introdutionell, let’s move on the first part of this topi .the motivation of this ork like this. institutional eonomis posits that agents making deisions on different tpes of transations do so in a ostl a .for example , farmers deiding sell a partiular rop to hom base their deisions not onl on the prie the expet to reeive in eah market hoie but also on additional osts related to transating in these markets.i ant to use a piture to illustrate it. for example, given some market h annels, farmers’ hoies an be regarded as equilibrium beteen the surplus and the additional osts that related to transating .espeiall in developing ountries, high-value rop produers full partiipate in the market and the transation ost has been the hard onstraint to farmers. furthermore, farmers’ market hoies an be taken as a hoie dilemma of transation ost and prodution surplus. onsequentl, the sientifi question of this researh is ho transation ost affets planters’ hoies.3. methodologlet’s move to the theor etial model of our researh. onsider a household model in one rotation. in stage 1 , famer η needs to alloate the input fators .this proess an qbe set into a funtion like this q? ? q, qη means the output farmers deide qto produe .p implies the output prie implies input prie and.z: ? is fixed input. one produe hat and produe ho man are deided, next question to be onsidered is ho muh produts to be transated in market. here e use three ,are being promoted.2、installing the intelligent eletroni station board at the bus stop station for shoing the distribution of transit lines related to the station, the distane of the bus belong to some transit line and their predition arrival time, anhelp passengers kno the loation of eah bus and determinetheir travel arrangements.3、the urban publi transportation information serviesstem is an important part of apts, also reflets the modernization of urban publi transportation; it onstitutes a part of the modern urban publi transportation sstem, ollaborating ith the intelligent transit dispathing sstem,the transit optimization sstem and the transit evaluation sstem.（二）tehnial analsis of bus arrival time1、the aura of the bus arrival predition time is diretl related to the preision of information shoed on theintelligent eletroni station board. therefore, the aura ofthe next bus’s arrival predition time shoed on theintelligent eletroni station board is a ver ke indiator.2、the predition model of bus arrival time displaed onthe intelligent eletroni station board need to be improved urgentl.meanhile, theor researh，sstem design and implementationof the advaned publi transportation sstems,are being promoted.3、the tehnolog ponents of the bus arrival time aredivided into the folloing five parts:???? the average travel time through the setions; queuing dela time as a result of the impat of signal ontrol; the time the bus travel through the intersetion; stop time at the last fe stop stations as a result of the passengers get off and on before the preditionstop station;? lost time of slo don and speed up beause of bus entering and leaving the stop at the lastfe stop stations before the predition stop station.the predition model of bus arrival timea. the travel time on setionstravel time is defined as pure running time on setions, does not ontain short delaed time beause of traffi signal ontrol, the time for passengers getting on and off on eah stop station and the stop time for vehile tehnial problems.b. the dela time on stop stations before the predition stop stationthe dela time at stop station denotes the lost time hen the slo don and speed up beause of bus entering and leaving the stop station, the time for opening onlusion???? the average travel time through the setions; queuing dela time as a result of the impat of signal ontrol; the time the bus travel through the intersetion; stop time at the last fe stop stations as a result of the passengers get off and on before the preditionstop station;? lost time of slo don and speed up beause of bus entering and leaving the stop at the lastfe stop stations before the predition stop station.? finall, i find m explanation about m major is too professional for m lassmates to understand, so ifind it is important to make use of examples if i ant to make m audiene lear.this major's main job is making and optimizing mahines. students of this major ill learn the theor of mahines and build the mahine based on the knoledge. in addition, analzation optimization of the mahine is also required inthis major, hih ill make the mahine orks ell.this major deals ith the hole plan of the projet inluding investment regulation, risk ontrol, qualit and quantit. allof the ork makes the plan goes ell and get rih profit.第五篇：参加国际学术会议总结参加国际学术会议总结高兵201X 3rd international onferene on advaned puter theor and engineering 于201X年8月20日至201X年8月22日在四川成都的四川大学召开，本次会议由四川省计算机学会和iasit （international assoiation of puter siene and information tehnolog）联合发起，由ieee、四川大学，电子科技大学，西南交通大学，西南民族大学提供技术协助。
英文推荐信称呼
英文推荐信称呼英文推荐信称呼英文信一名国际贸易专业学生的英语老师为其写的一封英文信。
表达了这名学生的优秀给自己留下了深刻的印象。
To Whom It Ma Conern：It is m great pleasure to remend Miss Lili Zhang to ou,as she as one of m finest students in our department. Miss Zhang began attending m English lasses in the Department of International Trade Universit in92 and graduated in springof96. Though it has been over eight ears sine I last sa her, the deep impression she made on me has not faded in the least. She is ver intelligent, honest, reative, artiulate, adaptive person. Her high XXdemi ahievement speaks for itself: she onsistentl sored in the top 5% in lass. I am ertain that Miss Zhang ould make great ontributions to our pan, and I strongl remend her for the position. Please do not hesitate toinquire further if I an be of help to ou. Sinerel, Tim u Dr. Tim Wu 是为好朋友写的一篇120字左右的信，信中简单地地介绍了该朋友的一些个人背景，工作经历和语言能力。
分析化学考研面试问题。
药物分析实验典型问题1、鉴别检查在药品质量控制中的意义及一般杂质检查的主要项目是什么? What are thepurposes of drug identification and test? What are the usual items of drug tests?.2、比色比浊操作应遵循的原则是什么? What are the standard operation procedures forthe clarity test?3、试计算葡萄糖重金属检查中标准铅溶液的取用量。
How much of the lead standardsolution should be taken for the limit test for heavy metals in this experiment?4、古蔡氏试砷法中所加各试剂的作用与操作注意点是什么? What precautions shouldbe taken for the limit test for arsenic(Appendix VIII J，method 1)? And what is the function for each of the test solutions added?5、根据样品取用量、杂质限量及标准砷溶液的浓度，计算标准砷溶液的取用量。
Figure outthe amount of the arsenic standard solution that should be taken for the limit test for arsenic(Appendix VIII J，method 1) (0.0001%) in this experiment with the specified quantity of 2.0 g of sample.6、炽灼残渣测定的成败关键是什么？什么是恒重？What is the key step during thedetermination of residue on ignition? What does ‘ignite or dry to constant weight’mean?7、盐酸普鲁卡因的鉴别原理是什么？What are the principles of the identification ofProcaine Hydrochloride.8、盐酸普鲁卡因注射液中为什么要检查对氨基苯甲酸？Why is the limit of4-aminobenzoic acid tested for Procaine Hydrochloride?9、薄层色谱法检查药物中有关物质的方法通常有哪几种类型?本实验属于哪种?与其它方法有何异同点? How many kinds of the limit tests for related compounds are there?What are the differences between them? Which one is used for the limit test of 4-amino-benzoic acid in Procaine Hydrochloride Injection?10、醋酸氢化可的松的鉴别原理是什么？What are the principles of the identification ofhydrocortisone acetate?11、甾体激素中“其它甾体”检查的意义和常用方法是什么？What are the commonly usedmethod for and the significance of the limit test for other steroids for the steroidal drugs?12、哪类甾体激素可与四氮唑蓝产生反应，是结构中的何种基团参与了反应，反应式是什么？What kind of steroidal drugs can react with the alkaline tetrazolium blue TS?What is the chemical reaction equation?13、氯贝丁酯的鉴别原理是什么？What are the principles of the identification ofclofibrate?14、氯贝丁酯中为什么要检查对氯酚？其方法及原理是什么？Why is the limit ofp-Chlorophenol tested for clofibrate? What kind of method is employed for the test and what is the principle?15、气相色谱法检查杂质有哪些方法，试比较各种方法的特点？How many types ofmethods are there for the test of related compounds by the gas chromatography?What are the differences between them?16、抗生素类药物的鉴别和检查有何特点？What are the characteristics for theidentification and tests of antibiotics?17、钠盐的焰色反应应注意什么？What precautions should be taken during the flamereaction of sodium salts?18、本品吸收度检查的意义是什么？What is the purpose of the light absorption tests forbenzylpenicillin sodium?19、药物晶型测定的常用方法有哪些，各有什么特点？What are the commonly usedmethods for the test of polymorphism? And what are the characteristics of each of them?20、吸收系数测定方法与要求？What are the standard operation procedures for theestablishment of specific absorbance?21、写出异烟肼与溴酸钾的滴定反应式和滴定度的计算过程。
美国FDA分析方法验证指引中英文对照
美国FDA分析方法验证指南中英文对照美国FDA分析方法验证指南中英文对照八、、I.INTRODUCTIONThis guida nee provides recomme ndati ons to applica nts on submitt ing an alytical procedures, validati on data, and samples to support the docume ntati on of the identity, strength, quality, purity, and potency of drug substances and drug products.1.绪论本指南旨在为申请者提供建议，以帮助其提交分析方法，方法验证资料和样品用于支持原料药和制剂的认定，剂量，质量，纯度和效力方面的文件。
This guida nce is in ten ded to assist applica nts in assembli ng in formati on, submitt ing samples, and prese nti ng data to support an alytical methodologies. The recomme ndati ons apply to drug substa nces and drug products covered in new drug applicati ons (NDAs), abbreviated new drug applicati ons (ANDAs), biologics license applications (BLAs), product license applications (PLAs), and supplements to these即plicatio ns.本指南旨在帮助申请者收集资料，递交样品并资料以支持分析方法。
METHOD FOR SELECTION AND DISPLAY OF AT LEAST ONE
专利名称：METHOD FOR SELECTION AND DISPLAY OF AT LEAST ONE PIECE OF ADDITIONALINFORMATION发明人：SCHWEIER, René申请号：EP2007000865申请日：20070201公开号：WO07/090560P1公开日：20070816专利内容由知识产权出版社提供摘要：According to the invention, the selection and display of a piece of information in addition to other information which may be transmitted from a server (3) to a client (2) may be improved such that a reduction in the load on the communication network is possible, by means of automatic analysis of a profile assigned to the user (7) of the client (2), wherein at least one piece of partial information is automatically selected from the information depending on the result of the analysis of the profile, one piece of additional information is automatically selected from the available additional information depending on the analysis of at least one property of the partial information, an activation element which may be activated is automatically assigned to the partial information, wherein on an activation of the activation element an action assigned to the additional information is automatically carried out and the activation element is automatically provided with a representation element, by means of which at least one parameter relating to the representation of the activation element on the client (2) may be set.申请人：SCHWEIER, René地址：DE,DE国籍：DE,DE代理机构：WÖRZ, Volker 更多信息请下载全文后查看。
METHOD FOR THE SELECTION OF NUCLEIC ACID LIGANDS
专利名称：METHOD FOR THE SELECTION OF NUCLEIC ACID LIGANDS发明人：EULBERG, Dirk,KLUSSMANN, Sven申请号：EP2003010053申请日：20030910公开号：WO04/024950P1公开日：20040325专利内容由知识产权出版社提供摘要：The invention relates to a method for the selection of one or more nucleic acid ligands from a mixture of candidate - nucleic acids. The nucleic acid ligands can bind to a target molecule. Said inventive method comprises the following steps a) placing the mixture in contact with the immobilised target molecule, b) separating the nucleic acid(s) binding to the target molecule with a higher affinity from the remainder of the candidate mixture, whereby the nucleic acid(s) binding to the target molecule with a higher affinity are provided in a complex with the target molecule, c) eluting the nucleic acid(s) binding to the target molecule with a higher affinity from the target molecule and d) optionally amplifying the binding nucleic acid(s) with a higher affinity, whereby an enriched nucleic acid product is produced. The target molecule is immobilised on the surface and the method also comprises an ultra filtration step.申请人：EULBERG, Dirk,KLUSSMANN, Sven地址：Max-Dohrn-Strasse 8-10 10589 Berlin DE,Thulestrasse 12 13189 BerlinDE,Paulsborner Str. 83A 10709 Berlin DE国籍：DE,DE,DE代理机构：BOHMANN, Armin, K.更多信息请下载全文后查看。
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
Proceedings of the 2002 Winter Simulation ConferenceE. Yücesan, C.-H. Chen, J. L. Snowdon, and J. M. Charnes, eds. ABSTRACTFor large international companies with their own simula-tion team it is often hard to select new discrete event simu-lation software. Often, preferences and application areasbetween countries differ, and simulation software alreadyin use influences the outcome of the selection process.Available selection methods do not suffice in such cases.Therefore, a two-phase evaluation and selection methodol-ogy is proposed. Phase one quickly reduces the long-list toa short-list of packages. Phase two matches the require-ments of the company with the features of the simulationpackage in detail. Different methods are used for a detailedevaluation of each package. Simulation software vendorsparticipate in both phases.The approach was tested for the Accenture world-widesimulation team. After the study, we can conclude that themethodology was effective in terms of quality and efficientin terms of time. It can easily be applied for other large or-ganizations with a team of simulation specialists.1 INTROD U CTIONCurrently the market offers a variety of discrete-eventsimulation software packages. Some are less expensivethan others. Some are generic and can be used in a widevariety of application areas while others are more specific.Some have powerful features for modeling while othersprovide only basic features. Modeling approaches andstrategies are different for different packages. There aremany properties that make each discrete-event simulationpackage different. All discrete-event simulation packageshave their particular strengths and weaknesses. This makesthe selection and purchase of a simulation package diffi-cult. Buying the appropriate discrete-event simulationpackage is important and can save a lot of money.Accenture is the one of the world’s leading manage-ment and technology services organizations. Accenture has a large team of simulation experts, who operate on a world wide basis in many types of projects for a wide range of customers. Currently, the Accenture simulation team uses two different simulation packages (Arena and ProModel), but they want to standardize on one package. The team ap-plies simulation techniques to many different problem do-mains. The main ones are helping clients in defining strategies, designing processes, analyzing performance and extracting customer’s experience. For the team, having one generic discrete-event package instead of several packages increases model reusability, staff interchangeability and reduces model development time, training costs and pur-chasing costs. The outcome of the research is the basis for a decision which discrete-event simulation package Accen-ture will use for the next three years. This paper is structured as follows. First, we describe the background of the project, introducing the state of the field and presenting selection methods available from the scientific community. Second, the methodology that was used here is described. Third, results obtained by following the methodology are covered. Fourth, results are dis-cussed. Finally, conclusion and issues for further research are presented. 2 BACKGRO U ND Evaluation of discrete-event simulation packages is not new. Many researchers have carried out surveys on available packages for different purposes. H owever, there is only a limited number of papers that describe methods to perform an evaluation of discrete-event simulation packages. H lupic (1997) developed a software tool (SimSelect) that selects a simulation software given the required features. Nikoukaran, H lupic, and Paul (1999)AN EVALUATION AND SELECTION METHODOLOGY FORDISCRETE-EVENT SIMULATION SOFTWARETamrat W. TewoldeberhanAlexander VerbraeckEdwin ValentinSystems Engineering DepartmentFaculty of Technology, Policy and ManagementDelft University of TechnologyJaffalaan 5Delft, 2628BX , THE NETHERLANDSGilles Bardonnet Accenture Avenue George V 55 75379 Paris Cedex 08, FRANCEcreated a framework of criteria to be considered when evaluating discrete-event simulation software. Other researchers such as Banks (1991) and Pidd (1992) already showed a similar framework in earlier literature.The need for having an efficient selection method for discrete-event simulation packages is increasing as the simulation application domain broadens (Shannon 1992) and as the number and type of discrete-event simulation packages increases (OR/MS Today 1999). Companies and institutions that use simulation do some research for their own use, and they use different methodologies and ap-proaches. However, projects and published results on how to effectively conduct evaluation and selection process of discrete-event simulation software are limited.On of the most elaborate frameworks is described in Nikoukaran, Hlupic, and Paul (1999). This framework is structured, and pays attention to a rich set of criteria on which simulation packages can be compared. It is, how-ever, difficult to base a decision for a large multinational company on these criteria, as it is only a comparison, with-out weighing and without a method to determine the rela-tive weights, and the weight differences between parts of the simulation team.3 METHODOLOGYTo evaluate simulation packages, and to select the best one for a large company, is a time consuming task unless an efficient methodology is used. Usually, choosing from a list of alternatives requires detailed knowledge of the selec-tion criteria, and on the score of the alternatives on these selection criteria. If there are many alternatives and if the criteria list is long, the evaluation becomes a challenging task. To accomplish this task efficiently, a two-phase evaluation and selection methodology is proposed, which will be explained below. The methodology was designed to be fast and as objective as possible. Of course it should re-spond to the specific needs of the simulation team that wants to acquire a new simulation tool.In the first phase, simulation packages are selected based on the existence of the most important features and criteria. In the second phase, detailed evaluation and analy-sis are done for packages that satisfy the requirements of the first phase.During the two phases, numerous interactions take place between the simulation team, management team, se-lection team and simulation package vendors. The selec-tion team is defined here as the analysts who are responsi-ble for carrying out the research and making the final recommendations for selection. They can be employees of the company or external consultants. The interactions be-tween the actors are indicated in Figure 1. The vendors have only a limited interaction with company management (price) and the simulation team (demos), and most of their interaction with the selection team.Figure 1: Interactions during the Selection Process3.1 Phase One: Feature CheckIn phase one, a list of required features is created, and a wide list of discrete-event simulation software packages are checked for availability of these required features. To accomplish this ‘feature-check’ phase, the following steps are taken: vision and requirement identification, criteria ex-traction, criteria weighing, characteristics of discrete-event simulation software identification, and screening and rank-ing the simulation software. Those packages that satisfy the first phase are transferred to the second phase. The de-tailed methodology of phase one is as follows.3.1.1 Vision and Requirement Identification Identifying the overall vision of the simulation team for the near future is necessary for identifying the functional re-quirement for the simulation packages. The vision covers items like: current and near future application area of dis-crete-event simulation, type of product/service, types of customers, business process or work routine, business ob-jective and similar aspects. A questionnaire is one of the main methods to be used to extract the necessary informa-tion from the team. In other cases where the selection is carried out internally within a company, other methods such as workshops or a brainstorming session may be used (see Figure 2). The exact purpose of the questionnaire is to get the following items: future goal and plan of the simula-tion team regarding discrete-event simulation studies, main fields on simulation in which the team is involved, current use of discrete event simulation package in the team as well as near future use, features the team experts need from a simulation package, constraints the team expert have in the current simulation packages they use, and a criteria list for evaluation of packages with a ranking. In addition, in-terviews and library research can be used. Results that are found to be inconsistent during the analysis are discussed again with the team. The criteria list, then, is categorized using the Nikoukaran, Hlupic, and Paul (1999) framework.Figure 2: Vision and Requirement Identification3.1.2 Criteria ExtractionBased on the vision identified and additional sources, crite-ria are extracted by the selection team, in close co-operation with the simulation team. The additional sources for input are external experts on simulation, the company’s clients, internal project findings and reports, and literature. Criteria are extracted from the vision by asking questions such as: “What are the main features of a simulation pack-age that make it applicable to a wide range of problems?”, “What functionalitie s are ofte n use d during an e ngage-ment?”, “How does the team address a client’s problem?”, “How doe s the te am use the simulation package s?”, and “What are the main obje ctive s of the simulation proje cts carried out?”.For the additional inputs, literature, research projects and clients are consulted. From literature and research pro-jects, the importance of certain criteria for particular appli-cation areas can be extracted and discussed with the simu-lation team and management. The reason behind using this methodology is that different features of simulation pack-ages are required for different application areas. For exam-ple, optimization might be considered very important for supply chain management whereas external connectivity might be considered to be more important for real time simulation.The initial list of criteria is discussed with the simula-tion team for additional feedback and consistency.3.1.3 Criteria WeighingAfter the extraction of the criteria, weights based on the level of importance are given to each criterion. In addition, to increase the efficiency of the selection, hard (criteria that must be satisfied by any means) and soft criteria are identi-fied. To cope with the various locations of the simulation team members – even when the simulation team is based in one location, the team members will be external with cli-ents on various projects – questionnaires are used to weigh the criteria. The result is then analyzed and presented to the team for additional feedback. E-mail is ideal to update the team with the latest progress.For giving weights to the criteria, each member input from the simulation team is considered. Weighing is done by first defining the scale. A five level scale is defined us-ing scales 1 to 5. Five indicates very important and one in-dicates least important. After weight definition, members of the team give their personal weight to each of the crite-ria. For each criterion, the average weight is then calcu-lated from the weights obtained from the members. After-wards, standard deviations and averages are analyzed and compared with the raw data. The results of the analysis and additional input gathered from other sources (e.g., litera-tures, clients) are then presented to the team members. Fi-nally results are discussed to decide if they need to be ad-justed. Modification of the weights is then done based on the feedback of the team.In addition to assigning weights to the criteria, identi-fication of hard and soft criteria is also made. Hard criteria are obtained from the weights. Those criteria that are very important (close to 5) are considered to be hard criteria. In this way, hard and soft criteria are separated. The hard cri-teria obtained are then discussed with the group so as to be sure there is nothing to be added or removed. The team should be well aware that a simulation package that does not satisfy just one of the exit criteria immediately leaves the race.3.1.4 Characteristics Identification of Discrete-Event Simulation SoftwareFor the evaluation processes, discrete-event simulation packages available in the market are collected. The sources of information used are conference proceedings, research papers, vendor websites, input from the simulation team, and simulation practitioners.Since the focus is on discrete-event simulation pack-ages, packages that are used specifically for continuous simulation and Monte Carlo simulations are not consid-ered. H owever, packages that have discrete-event simula-tion capability as well as other capability are considered as possible candidates.After the final list of discrete-event simulation pack-ages is prepared, the characteristics of the packages are identified. Different ways can be used to extract the char-acteristics. One of the ways is consulting the vendors, ex-perimenting with the demo version of the packages, refer-ring to research papers that describe the experiences with simulation packages, and consulting simulation experts (outside and inside the company).In most of the cases, however, vendors can best be consulted using questionnaires. The questionnaire is pre-pared to address general issues and specific questions re-garding the packages. The general questions addresses is-sues such as modeling approach, simulation software class, simulation type, and application area.Specific questions about the features of the packages, supplied by the vendors, are categorized based on a framework. The criteria categories are:• Mode l de ve lopme nt: questions that are related to model development and modeling approach. Thisincludes features like model building tools, reus-ability of libraries, coding aspects, conditional rout-ings, queuing policies, and other related aspects • Input mode s: This includes input modes such as interactive mode, batch mode, from external files(spreadsheets, database, text files, etc.) and ran-dom variate generation.• Testing and efficiency: The questions in this cate-gory include debugging features and error control.• Execution: The questions in this category covers features such as multiple replications, automaticbatch run, warm up period, and reset capability.• Animation: The questions in this category cover animation development features, animation runningfeatures, display features, and icon development.• Output: The questions in this category include features used for displaying outputs either interms of numbers or business graphics. It also in-cludes capability to communicate with externalpackages.• User: The questions in this category include cost, compatibility of packages with different operatingsystems and hardware.The questionnaire should be designed to gather as much data as possible. Since the data obtained from vendors can be incorrect, validation of the data is done by using the other methods listed above i.e., experimenting with the package, reading research papers, consulting simulation experts, and checking vendors website. For the experimen-tation, demo and full versions of the packages are used. Experiments are done by building a small model and trying to address each criterion (and feature) in the criteria list, by checking most of the commands available, reading help files about the features of the package, running and debug-ging small models and demo models, and viewing and ex-perimenting with already existing demo models provided by the vendor or by other users.Another method used for extracting the characteristics of the packages is library research. Different research pa-pers, conference proceedings and vendor websites are used. 3.1.5 Screening and Ranking ofSimulation SoftwareOnce the characteristics of the packages are identified, screening and ranking is performed. Screening is done by using the hard criteria that are obtained in the previous steps. The packages that don’t satisfy one or more of the hard criteria are removed from the potential list. Those that satisfy all the hard criteria are kept for further ranking. Testing is done by checking whether the packages satisfy the requirement set by each criterion.After the packages are filtered (by using the hard crite-ria), further ranking is done on the remaining packages (us-ing soft criteria) to select a maximum of 10 best packages. The number 10 was chosen to restrain the solution domain, yet being representative. For ranking, the following proce-dure is used.• Define score: if a package satisfies a particular criterion, it scores 1 otherwise 0.• Give score to each package for all the criteria us-ing 1 and 0.• By following a simple Multi Criteria Decision Making (MCDM) method called Simple Multi-Attribute Rating Technique (SMART), criteriaweights were multiplied by the scores of eachpackage for all the criteria.• The sum of the product of weight and score for each package is compared.• The first 10 packages that have the highest scores are selected for further investigationTwo points need to be mentioned here. First of all, the ranking is done by only checking whether a package con-tains a particular feature or not. It is not based on the qual-ity of a feature. The quality of each feature is checked in the next phase of the analysis. Second, the aim of the rank-ing is only to filter the best 10 packages in the list. This, however, doesn’t necessarily mean that the one on the top of the list is going to be the best of the overall evaluation process. The best package for the company is known once the quality of each feature is tested for all the remaining packages in the second phase of research.3.2 Phase Two: Quality CheckIn phase two, discrete-event simulation packages are evaluated for their quality. To accomplish this quality-check phase, the following steps are taken: Criteria selec-tion, criteria weighing, designing a case study, conduct ex-periments, gather additional information, ranking of soft-ware, sensitivity analysis. The detailed methodology of phase two is described hereafter.3.2.1 Criteria SelectionTo evaluate the packages quality wise, some criteria have to be selected. The criteria address features that are impor-tant to analyze the quality of the package, considering the use that the simulation team plans to make of the simula-tion package. Features that need to be evaluated are se-lected by brainstorming with team members, selection team judgments and information already available from the first phase of the project.3.2.2 Criteria WeighingThe criteria obtained are weighed by following the proce-dure mentioned in section 3.1.3.3.2.3 Designing Case StudyThe purpose of the case study is to help evaluate the fea-tures of interest by carrying out a small simulation study using each of the packages that remained after the first phase. The case study is prepared in such a way that it ad-dresses most of the criteria in the list of the first step of phase 2. It does not require a specific modeling approach – as we do not want to introduce a bias to a certain modeling approach. The case study should be small but fairly com-plicated, and it should represent a typical problem that is representative for the type of simulation study carried out by the simulation team.The case study can best be constructed from scratch; when taking a previous project as case study, there can eas-ily be a bias towards the currently used simulation package – in positive or negative sense depending on the success of the project. After choosing a typical case, it needs to be ex-tended to address the criteria that need to be evaluated. The case study evolves by using feedback from the team mem-bers and, if necessary, outside experts. As the case study is on paper, it is again easy to involve the simulation team members in the selection process because the case descrip-tion can be distributed using e-mail.3.2.4 Conduct ExperimentExperiment on the packages is done by modeling the case study. The modeling steps are as follows:• Conceptualization: conceptual model design based on the package’s modeling approach. Thiscan be difficult, because the package may ask fora different modeling approach than what theselection team is used to.• Specification: actual model construction. All ex-periments with the different simulation packagesshould use the same data sets and distributionfunctions as input, so the results in the output aretruly comparable. It is important to keep a log-book for immediately writing down the experi-ences with the simulation packages.• Execution: running the model and creating output to analyze further. Run-time and animation speedcan, for instance, be analyzed here if these are partof the quality criteria.• Output: Analyzing the output provided. If the quality criteria include specific aspects of outputtypes or post-processing of the output, it can bedone in this step.• Scenario management: analyze different scenarios if needed.For the experiments, vendors should be involved. Ven-dor participation is important because they can show the most efficient way of modeling the case using their pack-ages. Furthermore, their level of cooperation in the project is one way of investigating the support level of the vendor. The vendors can be given three options for participation: • Vendor’s work on the case study is transparent to other vendors (vice versa) and they know withwhom they are competing.• Vendor’s work is not transparent to other vendors and they only know with whom they are compet-ing.• Vendors only work on the case study and vendors don’t know with whom they are competing.Consensus should be reached among the vendors on the way of working. The experiment is conducted based on their cooperation strategy.3.2.5 Gather Additional InformationIn addition to conducting experiment, additional information has to be collected to enrich the evaluation quality. The addi-tional information collected is mainly considering the limits of the packages, and advantages and disadvantages of the packages. The additional information is collected from arti-cles and research papers, and simulation experts who have used the packages. A very good source of input here can be a user group of the simulation package.3.2.6 Ranking of SoftwareBy using the output of the experiments and the additional information collected from various sources, a score for all the packages can be given. The following method is used: • Define a scale for scoring: in this case, a 4 level scale is defined. Scaling is done using scale 0 to 3.3 indicates “Good”, 2 indicates “Sufficient”, 1 in-dicates “Insufficient”, and 0 indicates “Featuredoesn’t exist”.• For all the packages, assign scores for each crite-rion based on the defined scale. The selectionteam, which has followed all efforts for workingout the case study in detail, does the scoring.• The detailed scores for all the criteria is averaged based on the criteria categories.• Again, by following the Multi Criteria Decision Making (MCDM) method SMART (SimpleMulti-Attribute Rating Technique), criteriaweights are multiplied by the scores of each pack-age for all the criteria.• The sum of the product of weight and score for each package is compared.The result of the ranking is then given to the team members to get feedback. After this ranking, the packages receiving the highest score can be considered better than the ones scoring lower, as the comparison has been based on the (perceived) quality of the simulation package in a representative case study.3.2.7 Sensitivity AnalysisAfter ranking is done and the best package is known, sensi-tivity analysis has to be done in order to assess the robust-ness of the result. A sensitivity analysis is performed be-cause both scores and criteria weights are often subjectively generated in Multi Criteria Decision Making (MCDM). The following approach is used for sensitivity analysis1. Changing weights of criteria: Criteria weights arechanged and how much the decision is sensitive tothe change is observed. Changes made area. Make all criteria weights equalb. Change each criteria weights to minimum oneat a timec. Change each criteria weights to maximumone at a time2. Limiting criteria: The first few important criteriawere considered. The following considerations aremadea. The first important criterionb. The first two important criteria are consideredc. The first three important criteria are consid-ered and so on.3. Changing scores of alternative packages: Makesome reasonable changes to scores of the differentalternatives and observe the impact it would haveon the recommended decision. 4 PRACTICAL APPLICATIONOF THE METHODOLOGYThe methodology has been applied in practice for selectinga new discrete event simulation package for Accenture’sworld-wide simulation team. Following the methodologyexplained above, a detailed vision of the team was ex-tracted in the first phase. The vision includes the team’sstrategy, work environment, type of clients, type of project,and team expertise. The criteria were generated with theweights, and categorized with the Nikoukaran, Hlupic, andPaul (1999) framework criteria framework. Based on theweights, the hard criteria are in boldface in the tables.Table 1: Model Development and Input category CriteriaCriteria Weight Graphical model building 5Merging models 4Conditional routing 4Statistical distribution 5Queuing policies 4Reuse of user defined modules 3Built-in functions 3Link to other languages 3Coding tools and utilities 3Input from text files 5Input from database 4Input from spreadsheets 5Automatic data collection 3Batch input mode 3Interactive input mode 5Random number generators 5Program generator 3The obtained model development criteria in Table 1reflect the vision of the team. Since the team is involved indifferent application areas of simulation, flexibility andeasy of use in modeling are very important. Therefore, thehigh score for “Graphical model building” reflects this vi-sion. In addition, due to the different application areas, theprecision and numbers involved vary from project to pro-ject, making the “random number generator” criterion im-portant. “Input from external files” criteria scores veryhigh reflecting the vision of the team.Table 2: Vendor Category CriteriaCriteria Weight Documentation 4 Maintenance support 5Pedigree 3 Pre-purchase facility 2In Table 2 it can be seen that maintenance support anddocumentation is very important because time is a very im-portant issue. Efficient and fast support is highly required in a work environment where deadlines are numerous. In addi-tion, detailed and good documentation is very necessary.Table 3: Execution Category Criteria Criteria Weight Multiple runs 5 Automatic batch runs 3 Reset capability 4 Start in non-empty state 3 Interaction with user (in running mode) 2 Warm up period 5 Ability to calculate appropriate warm-up pe-riod and replications 3 Speed control 5 Self executable versions 3 The “Multiple runs” criterion in Table 3, which indi-cates repeating simulation runs many times, scores very high compared to others because of variance reduction test performance. For the team the “speed control” criterion is important because the speed of the simulation can vary from as fast as possible for getting numerical results to a slow speed for demonstrating results to clients. The “Warm up period” criterion is also important for Accenture be-cause some of the systems the group deals with are non-terminating systems.Table 4: Animation Category CriteriaCriteria WeightIntegration of animation 3Library of icons 3Screen layout 3Concurrent animation mode 3Animation on/off feature 53D animation 1Animation development feature 3In the animation criteria of Table 4, the “3D anima-tion” criterion is not important for the team because thegroup mainly carries out business process simulation asopposed to manufacturing where it could be more impor-tant. “Animation on/off” is important just because turningthe animation off can increase the simulation speed.Table 5: Testing and Efficiency Category CriteriaCriteria WeightError checker 5Interacting debugger 5Multitasking 2Display features 3Tracing 3 Breakpoints 4 Running backwards 1 Limits 2In Table 5, “Error checker” and “Interactive debug-ger” criteria score high because most of the projects man-aged by the group are complex and large. Without a gooddebugger, it is considered difficult to fine-tune the model.Table 6: Output Category Criteria Criteria Weight Standard report generation 4 Report customization 5 Integration with statistical packages 3 Integration with other simulation packages 3 Feature for exporting data to database 3 Feature for exporting data to spreadsheets 5Feature for exporting data to text files or word processors 5 Optimization 3Output analysis feature 4Business graphics 4For the output criteria in Table 6, “Customization”, “Export to spreadsheets” and “Export to word processors” are considered important because the output features are common tools for displaying results to clients. Table 7:User Category CriteriaCriteria WeightCost 2Connectivity with internet 2Package interoperability 2Package link to different animation packages 2Package has open source code 1Package application area 5Flow oriented modeling approach 4High level architecture 2Capability for continuous simulation 2Simulation strategy 3For the user criteria in Table 7, “Package application area” criterion scored high for the reason that the package has to be generic enough to be applied in different domains. For the phase 1 evaluation, more than 50 packages were considered. A few software vendors were not able to participate in the project. By using the hard criteria from tables 1 to 7, the packages were screened. Those packages that didn’t satisfy all the hard criteria were eliminated. Packages that satisfied all the hard criteria in alphabetical order were Arena 5.0 (Rockwell Software), AutoMod 9.1 (Brooks Automation), Enterprise Dynamics 3.1(EnterpriseDynamics), Extend 5.0 (Imagine That Inc.), ProModel2001 (Promodel Corporation), Quest (DELMIA Corpora-tion), Simul8 6.0 (Simul8 Corporation), and Witness 2000(Lanner Group, Inc.). Furthermore, these packages were ranked based on de-tailed feature they offer, i.e., availability of features. The。