1-The relationship between language testing and SLA revisited

The relationship between language testing and second language acquisition,revisitedElana Shohamy *School of Education,Tel Aviv University,69978Tel Aviv,IsraelReceived 5January 2000;received in revised form 1March 2000;accepted 7March 2000AbstractThe paper examines the relationship between and the relevance of second language acqui-sition (SLA)and language testing (LT).Based on three dimensions of potential contributions of LT to SLA [(1)de®ning the construct of language ability;(2)applying LT ®ndings to test SLA hypotheses;and (3)providing SLA researchers with quality criteria for tests and tasks]and three dimensions of potential contribution of SLA to LT [(1)identifying language com-ponents for elicitation and criteria assessment;(2)proposing tasks for assessing language;and (3)informing language testers about di erences and accommodating these di erences],this paper examines the interfaces of the two ®elds based on articles published in recent issues of the journals ``Language Testing''and ``Studies in Second Language Acquisition''.The rele-vance of LT to SLA is examined based on written interviews with leading scholars in SLA who were asked about the relevance of LT to their work.The results indicate very limited interfaces between the two ®elds as well as limited relevance of LT to SLA.The conclusions and implications discuss to the potential need of LT to broaden its focus and scope by addressing broader views of language learning and language processing such as:viewing language in its complexities and dynamics;involving the learners and test takers;marketing better LT theories to those out of the ®eld;expanding the context beyond psychometrics;expanding the types of instruments used beyond tests;addressing educational issues;and working towards relevance.#2000Elsevier Science Ltd.All rights reserved.Keywords:Second language acquisition;Language testing;Relevance;Interaction;Data collection;Second languageresearchSystem 28(2000)541±/locate/system0346-251X/00/$-see front matter #2000Elsevier Science Ltd.All rights reserved.P I I :S 0346-251X(00)00037-3*Fax:+972-3-6409-477.E-mail address:elana@post.tau.ac.il (E.Shohamy).542 E.Shohamy/System28(2000)541±5531.IntroductionThe disciplines of language testing(LT)and second language acquisition(SLA) belong to the same®eld,that of language learning.They share similar goals of understanding the process of language learning,assessing it and looking for ways to improve it.It is,therefore,expected that the two disciplines would interact,share and contribute to one other.The nature of the relationship between the two®elds was discussed in a book entitled``Interfaces Between Second Language Acquisition and Language Testing Research''(Bachman and Cohen,1998)which is based on papers presented in a symposium of the same title that took place at the1993conference of the American Association of Applied Linguistics in Seattle.The book provides insightful analyses of the relationship between LT and SLA from various per-spectives.Chapelle(1998)discusses the construct de®nition and validity inquiry of SLA research;Tarone(1998)examines research on interlanguage variations and its implications to LT;Cohen(1998)provides insight into the use of strategies and processes of test taking;Brindley(1998)focusses on the relationship between rating scales and SLA;Douglas(1998)addresses the issue of testing method in context-based second language research;Shohamy(1998)uses the case of dis-course analysis to demonstrate how each of the disciplines does not take into account the various developments and research®ndings of the other;and®nally, Bachman(1998)provides a general perspective of the interfaces between the two disciplines.The conclusions drawn from most of the papers are that there is room for more cooperation between the two®elds and that these can lead to bene®ts for both.This paper will revisit the issue of the interfaces and relevance of LT and SLA based on two sources:(1)research articles appearing in recent issues of LT and SLA journals;and(2)interviews with leading scholars in SLA who were asked to present their views regarding the relevance of LT to their work.2.Potential areas of relevanceIn the chapter mentioned above,Shohamy(1998)identi®ed six areas of potential contributions of LT to SLA and SLA to LT.These areas are described below as they provide a framework for examining and analyzing the interfaces between the two ®elds.2.1.Areas where LT can contribute to SLA2.1.1.De®ning the construct of language abilityA major component of the research in LT focuses on various de®nitions and theories of language ability.It is claimed that theories of language ability can be useful for SLA researchers in examining the validity of their®ndings.E.Shohamy/System28(2000)541±5535432.1.2.Applying LT®ndings to test SLA hypothesesLT researchers obtain various®ndings about language behavior from tests.It often includes research that examines how speci®c features of SLA manifest them-selves in testing situations.It is claimed that such®ndings can be useful for SLA researchers in order to generate,test,and con®rm various SLA hypotheses.2.1.3.Providing SLA researchers with quality criteria for tests and tasksOne important contribution of LT to SLA is developing quality criteria for tasks and tests that are being used for collecting SLA nguage testers have expert-ise and experience in development and validation of tests so that reliable and valid SLA data can be obtained.At the same time the generalizability and external validity of language data depend on the reliability and validity of the language data obtained via the di erent SLA tests and tasks.It is,therefore,important that these data will be obtained via tests and tasks which are reliable and valid.It is claimed,therefore,that SLA researchers need to employ the variety of techniques and procedures developed by language testers to ensure that they obtain valid and reliable SLA data.2.2.Areas where SLA can contribute to LT2.2.1.Identifying language components for elicitation and criteria for their assessmentSLA research can identify the necessary constructs needed for language testers in order to develop second language tests.Thus,research into task variability can provide valuable insights into the roles that variables such as interlocutors,topic, social status,and discourse-domain play in pro®ciency tests.Similarly,when research indicates that writing is a process that requires revisions and re-writing, testers can use such information in order to develop writing tests that incorporate the revision process.2.2.2.Proposing tasks for assessing languageSLA researchers have used varied and creative tasks beyond traditional tests for collecting SLA data.One potential contribution that SLA research and experience can make to LT is to broaden the repertoire of measurement instruments that lan-guage testers could use.Tasks such as judgment tests,observation,documentation, narration,elicited imitations,modi®ed interaction,and small group interactions are only a few of the varied and creative elicitation tasks used by SLA researchers that can be applicable for language testers.2.2.rming language testers about di erences and accommodating thesedi erencesSLA researchers can alert language testers to problematic areas in test construc-tion.For example,if SLA research has concluded that the L1makes a di erence in SLA,then language testers should not treat test takers from varied L1backgrounds the same.There may be a need to make accommodation for these di erences by devising di erent versions of tests for test takers depending on their L1.544 E.Shohamy/System28(2000)541±5533.Method3.1.Research questionGiven the potential contributions of one®eld to the other,the research questions posed in this paper refer to two dimensions:the interfaces between SLA and LT and the relevance of LT to SLA as perceived by SLA researchers.3.2.Data collection procedures3.2.1.InterfacesThe extent to which the two®elds contribute to one another is based on a number of published articles in SLA and LT journals.Three issues of``Studies in Second Language Acquisition''(SSLA,Vol.21,issue Nos.1,3,and4)and three issues of ``Language Testing''(Vol.13,issue Nos.1,2,and3)were selected as sources of data for they represent the two®elds,respectively.These articles were analyzed according to the six areas of potential contribution mentioned above.3.2.2.RelevanceThe relevance of LT to SLA was examined based on written interviews with leading researchers who have familiarity and experience with LT.An attempt was made to select researchers from diverse areas in SLA who are knowledgeable about the®eld of LT.The researchers come from areas of applied linguistics,SLA and LT;they are all of high stature in the®eld and have national and/or inter-national reputations.Each of the respondents was asked to express his/her per-ceptions with regards to the relevance of LT to their area of expertise in response to the written electronic mail letter.Ten people were contacted,eight responded. Information regarding their background and areas of research in SLA are included.4.Results4.1.Interfaces4.1.1.Contribution of LT to SLA researchThe analysis of the interfaces is based on10articles appearing in the three issues of SSLA(Vol.21,1999ÐWilliams,1999;Myles et al.,1999;Moyer,1999and Ortega,1999in issue No.1;Pienemann and Hakansson,1999;Izumi et al.,1999and Saito,1999in issue No.3and Rosa and O'Neill,Mackey,and Rott,issue No.4). The following patterns were observed regarding the contribution of LT to SLA.In terms of de®ning constructs of language ability identi®ed in LT,none of the SLA articles focussed on any issue of LT and none of the articles applied®ndings obtained in LT to their research.Yet,there were often references to problems and issues relating to testing,especially with regards to elicitation procedures.ForE.Shohamy/System28(2000)541±553545 example,Rosa and O'Neill(1999)discuss the inappropriateness of tasks such as protocols,cloze tests,grammaticality judgements and rule formation for their type of research and the fact that there was external interference in measuring intake. Similarly,Mackey(1999)discusses the di culty in measuring development and especially in tapping the speci®c structures the researcher is interested in with regards to:(1)di culties associated with devising tests that can directly measure development of the structures that occurred in the interaction;and(2)di culties associated with operationalizing second language development(p.565).Yet,no reference is made to such discussions in the LT literature.In terms of applying LT®ndings to SLA research,none of the articles make any reference to research in LT and,as noted above,even within the context of discus-sions about elicitation procedures.It almost seems like research on elicitation pro-cedures is not viewed as part of the LT discipline.In terms of applying quality criteria(i.e.reliability and validity)developed in LT to the various data collection tools used in the SLA research,there is very little evi-dence of such applications.While all the SLA research articles use a variety of data collection procedures,often very innovative,very rarely are there any uses,infor-mation and applications of the psychometric techniques developed in LT for exam-ining the quality of these SLA tools.Thus,most of the articles do not provide any information with regards to reliability,validity and items of the tests and tasks. For example,in the article by Rosa and O'Neill(1999,p.528),in the section where the assessment tasks and the pre-test and post-test design are described,there is only a short description of the multiple-choice recognition task that is designed to measure the intake of the target structure.They mention that the test included22 sentences,10containing the target subordinate clause structure and10which were distracters of conditional sentences of other types.Yet,there is no information about the reliability or validity of these items or any other psychometric criteria related to the type of distracters.It is interesting to note that in the discussion that the authors provide(p.532)of the limitation of the di erent elicitation tasks,they relate to various criticisms but fail to mention the psychometric properties of various elicitation tasks.Similarly,in the article by Rott(1999)there is a description of two types of tests,a vocabulary checklist,vocabulary acquisition and retention measures.Yet,no infor-mation is supplied with regards to the development of these tools.While the author does mention the scoring procedure and the fact that there was``full agreement'' between the two raters,there is no information as to the procedures or exact®gures that this high agreement refers to(p.596).In the article by Izumi et al.(1999)there is a description of two types of tests,a grammaticality judgement and a picture-cued production test.Yet,while there is detailed description of the number of sentences and what they contain,there is no information as to the reliability of the instruments.The only article that contained such information is the one by Ortega(1999) which includes measures of reliability and explanations as to the speci®c pro-cedures followed in order to determine the psychometric qualities of the instruments (p.124).546 E.Shohamy/System28(2000)541±553The lack of emphasis on psychometric properties of the instruments is problematic given the fact that the results of these tests provide the basis and criteria for deter-mining the results of the studies.4.1.2.Contribution of SLA to LTThis analysis is based on the articles appearing in three issues of``Language Testing''(Vol.16,1999ÐFreedle and Kostin,1999;Laufer and Nation,1999and Papjohn,1999in issue No.1;Belgar and Hunt,1999;Kormos,1999;Schmitt,1999 and Brown,1999in issue No.2;and Marquardt and Gillam,1999;Holm et al., 1999;Windsor et al.,1999;Cardell and Chenery,1999in issue No.3).The following patterns were observed regarding the contribution of SLA to LT.In terms of identifying components from SLA for the elicitation procedures,most articles in LT are based on phenomena identi®ed in SLA research such as vocabu-lary acquisition or oral performance but applied to testing situations.For example, the article by Kormos(1999)focusses on the type of conversation conducted in oral tests.Similarly,Belgar and Hunt(1999)focus on a speci®c test for measuring voca-bulary.Schmitt(1999)examines issues of construct validity of vocabulary.Thus,the type of analysis that is done is indeed based on topics identi®ed in SLA research.In that respect,research in SLA is very relevant to LT.In terms of identifying tasks for assessing language,there is no evidence that SLA has any impact on the methods used for assessment as all LT articles employ tests exclusively and not any other methods of language elicitation.For example,while in the article by Mackey(1999,issue No.4of SSLA,p.568)there is a description of a variety of instruments such as story completion,picture sequencing,picture di er-ences,and picture drawing,there are no LT articles that conduct research on these types of procedures,although it is mentioned that the di erent phases of language development require the use of such a variety of tools.Yet,research in LT,as appearing in the1999volume,does not focus on elicitation instruments needed in SLA research.Most articles just do not focus on such issues.In terms of accommodating di erences found in SLA in designing tests,there is hardly any evidence of this as most studies refer to the learners as homogenous groups.The only article that addresses the issue is the one by Brown(1999)which discusses the relative importance of language background of individual test takers to test performance.4.2.RelevanceThe following views were expressed by researchers in SLA regarding the relevance of LT.Interviewee1is a prominent scholar and researcher in SLA.She has written a well-known and widely used book on SLA and conducted ample research on various SLA topics.With regards to the relevance of LT to SLA,Interviewee1claimed that in theory LT is relevant,but in practice it is not as there are still areas in SLA that could bene®t much more from what language testers have to o er.She mentions the lack of tests which are sensitive to inter-language and to the various developmental stages.E.Shohamy/System28(2000)541±553547She also noted that most language tests still have a very narrow view of grammar and rarely include pragmatics of language structure use.Speci®cally,she claimed that language tests fail to understand the complexity of language and that``F F F there is a need therefore for a language test that would do justice to the dynamics of language and would understand that language is a complex,dynamic,non-linear system''. Interviewee2is also a major leader in research in SLA who has conducted research in various areas of applied linguistics,SLA,language teaching and LT. While her major work is in SLA and teaching she has written a number of well-respected articles in LT.In her response she noted that LT does not provide much insight into the processes of second language learning.She noted that``F F F if we consider one of the main goals of SLA research to provide insight into what it is that students do and do not know,and especially HOW they come to know what they know,then I don't think the testing®eld has much to say''.She is skeptical about the ability of any test to provide information regarding language learning since all tests require that what is tested is pre-determined;it is therefore not based on what learners actually do in a particular learning situation that might provide learning opportunities.She provides examples from her own research to demonstrate that in situations when instructional interventions were set up,focussing on particular aspects of the target language they have learned,that students determined their own goals and set their own agendas with the material while paying no attention to the researchers'goals.Interviewee2is critical about the®eld of LT for not paying enough attention to the learners and for not considering their views. Interviewee3is a scholar from Europe with diverse interests in SLA,who provides an international perspective on the relevance of LT.He claims that LT is based on old views of SLA as it does not relate to practical needs.Regarding relevance,he notes that LT does not relate to language processing and that``LT still seems to hold on to the black box approach pretending that there is no idea of what's going on within that machine.But we do''.The other point Interviewee3mentioned related to the lack of attention of lan-guage testers to the current political needs.``Also the statistics that go with it will have to be reviewed,because more than ever we seem to be going from a norm referenced approach to criterion referenced approach''.Interviewee4is a well-known scholar in foreign language education who has recently been responsible for introducing an innovative curriculum in a major higher education institution.She argues that LT is extraordinarily relevant,especially in the political and curricular context and has a pivotal power position.Yet,she notes that:F F F the profession as a whole,researchers and practitioners in all areas,from program development,supervision,assessment to classroom practice have ten-ded to take whatever assessment or testing practices prevailed as the``natural'' situation,a state of a airs that tended to blind us to this pervasive in¯uence of testing on absolutely everything we do,from research foci,to research design, to the reporting and analyzing of results,to potential implications for curricu-lum development,teaching practice,materials development,and,last but cer-tainly not least,to major educational and social policy decisions.548 E.Shohamy/System28(2000)541±553She also argued that the in¯uence/relevance of assessment is more crucial now than ever,and will likely continue to be so into the future since very little in the®eld can change unless some key assumptions and practices in assessment/testing will change and can become institutionalized.She added that any argument one way or the other about language learning and language teaching needs to be grounded in some form of``evidence'',and that``evidence''will always be based on some form of assessment.But for that purpose there is a need to devise more sophisticated assessment practices.Interviewee5is a prominent researcher in SLA who has written articles and con-ducted research both in SLA and in LT.He claimed that in many cases SLA and LT researchers have been working in parallel paths to solve problems of learning second languages.He noted that it is a pity since SLA researchers are not better informed about work in LT because many breakthroughs in LT theory are centrally relevant to SLA.In too few cases has SLA research been explicitly informed by LT but,in many cases,breakthroughs have occurred®rst in LT and,several years later,work on similar lines has been done by SLA researchers.One example has to do with instances of property theories where LT and SLA research interact.He mentions recent attempts to investigate,®rst,it has been a preoccupation of many SLA researchers to characterize L2knowledge in terms of access to cer-tain universals of Universal Grammar.Researchers'initial attempts to do this by means of asking learners to evaluate the grammaticality of certain test sen-tences were uninformed by language testing and produced,in general,contra-dictory and unstable results.More recent attempts have investigated bilinguals' strategies for judging the grammaticality of certain items and concluded(1)that bilinguals and monolinguals use di erent strategies for assessing the gramma-ticality of sentences and(2)bilinguals use di erent strategies according to the kind of grammatical item with which they are presented.Interviewee6is a distinguished researcher who has published many articles in areas of second language learning,language acquisition,language teaching and LT. He noted that LT is not sensitive to teaching and learning and in many ways not relevant in the educational context.His position,however,is that LT has a key role to play.Perhaps for too long SLA researchers have at times inadvertently violated certain basic principles of sound testing.By the same token,language testers have sometimes been culpable of preparing tests that violate basic SLA principles(such as the concern for contextual variables)that would have made the tests more valid. Another dimension of relevance that he added relates to the status of LT in applied linguistics as judged by whether a given graduate program in English as a second language(ESL),foreign language education,and/or applied linguistics in general,requires one or more testing courses.Accordingly,the bulk of applied lin-guistics graduate programs still do not require LT,thus its low relevance.Thus,LT is relevant but the®eld of applied linguistics has not yet realized it. Interviewee7is a prominent scholar working speci®cally in the area of foreign language learning.She has published a lot in the®eld and is currently a director ofE.Shohamy/System28(2000)541±553549 a language center that includes a large number of languages in a prominent university.She claimed that the LT industry is not sensitive at all to the needs of individual instructors and to the realities of running language programs.The emphasis in LT is too much on statistics and that tends to harm the creativity of tests and teachers'developments.The technical information is revered and con-structed as if it holds more importance than teacher knowledge and the context in which teachers work.``The governence by`good items'with the appropriate test statistics while overlooking issues of learners and learning is a major problem with testing as the theories developed by testing experts are irrelevant to language learn-ing in the classroom''.She concludes by stating that,``If a teacher had to calculate a test statistic every time she wanted to assess a student F F F well,that's just an immoral message as far as I'm concerned and gets us nowhere F F F except for the testing industrial complex F F F it keeps it in business''.Interviewee8is a researcher working on issues of language learning from political and bureaucratic perspectives.She noted that the work in LT is not as relevant as it could be to language education as long as it is labeled language testing,being dis-tinct from the broader term``language assessment''.Speci®cally,she argued that LT had a fairly narrow and carefully de®ned history and set of traditions,rituals,pro-cedures and criteria by which its practitioners judge their workÐits own govern-mentality.The commitment to clarity and rigor in this work has much to o er, especially in high stakes contexts where languages are taught and/or where language pro®ciency/competence is at issue.In addition,she wrote that the hard-headed ``culture''of LT can o er a useful antidote to the sentimental wish harbored by many educators to avoid the realities that attach to the gatekeeping situations of which they are inevitably part.Yet,for this``o ering''to have maximum impact she thinks that those in the®eld of LT need to continue and strengthen their work on the ethical and ideological meanings and e ects of LT.In other words,F F F those involved in language testing can strengthen the relevance of their work for language teaching if they try to look at what they are doing beyond the narrow technicalities of their work.This does not mean that they should aban-don these technicalities and attempt to improve them.Quite the reverse:the mean-ing of these technicalities and their concomitant discipline needs to be understood and furthered,and,thus understood,to be directed to where they are needed.She also stated that the latter point is why she perceived LT to be placed in the broader,looser®eld of language assessment.In educationalÐand hence gate-keepingÐsettings,assessment includes the observations of students'performance on any day in class;pro®ling of learners'development;traditional examinations; portfolios;performance-based assessment/testing;and standardized tests.Not all this assessment entails tests or approaches that draw from the traditional culture and requirements of testing.Without locating and understanding language testing within a broad spectrum of assessment situations,those concerned with the work of language testing550 E.Shohamy/System28(2000)541±553are in danger of,on the one hand,attempting to colonise areas that are inappropriate(by expanding the use of tests and testing procedures to contexts where other forms of assessment may be more useful for teachers,learners, parents and administrators),and,on the other,ignoring their potential con-tribution to other(often highly complex)arenas in which language assessment is taking place(by failing to alert naive practitioners of language assessment to the technical dimensions of what they are doing or propose to do).Table1summarizes the results obtained from scholars regarding the relevance of LT to SLA.Table1Summary of results regarding relevance of language testing(LT)to second language acquisition (SLA)researchInterviewee1:In theory LT is very relevant but not in practice,there is much more that could be done. Not enough assessment of inter-language,pragmatics,and LT holds a narrow view of nguage tests do not address the complexity and the dynamics of language in a non-linear wayInterviewee.2:LT determines in advance what people should know,yet this is not what test takers really nguage testers are not paying enough attention to the learner and his/her attitudes towards testsInterviewee.3:LT is based on old views of SLA and does not relate to practical needs.It does not relate to language processing and holds on to the black-box approach pretending that there is no idea of what's going on within that machine.Also,there is lack of attention to current needs in EuropeInterviewee.4:LT is very relevant,especially in the political and curricular context and has a pivotal power position.Yet,the profession tends to take whatever assessment or testing practices prevail as the `natural'situation,thus testing absolutely everything we do,from research foci,to research design,to the reporting and analyzing of results,to potential implications for curriculum development,teaching practice,materials development,and,last but certainly not least,to major educational and social policy decisions.The in¯uence/relevance of assessment is more crucial now than ever since things do not change unless some key assumptions and practices in assessment/testing will change and can become institutionalized.A need to devise more sophisticated assessment practices and to depart from what has been the customary playground,conceptual apparatus,methodology,and power position of the`psychometric folks'Interviewee.5:SLA researchers and LT researchers work in parallel to solve problems of learning second languages.SLA researchers should be better informed about work in LT as many breakthroughs in language testing theory are centrally relevant to SLAInterviewee.6:LT is not sensitive to teaching and learning and to the educational context.SLA researchers have inadvertently violated certain basic principles of sound testing.By the same token, language testers have sometimes been culpable of preparing tests that violate basic SLA principles(such as the concern for contextual variables)that would have made the tests more validInterviewee.7:LT is not sensitive to the needs of individual instructors and to the realities of running language programs.The emphasis on statistics by testing communities sti¯es creativity in test developmentInterviewee.8:Not relevant as long as it is testing and not assessment.A need to work for relevance。