Language assessment meeting 15

Series editors’ preface to Assessing Grammar

Grammar, the structural glue, the “code” of language, is arguably at the heart of language use, whether this involves speaking, listening, readingor writing. Grammar has also been central to language teaching andassessment historically, from the Middle Ages, when “rhetoric” was a key component of a university education, to the “skills-and-components” models of the 1960s that informed both language pedagogy and languagetesting.

Acknowledgments

In the late 1990s when Lyle Bachman and Charles Alderson invited me to write a book on assessing grammar, the resurging interest in grammar in the field of applied linguistics had already been well underway, and I was delighted. I knew there was no other book on assessing grammatical ability, and I knew this would be a challenge. In the next five years, I worked continuously on this book and am deeply grateful that Lyle, Charles and Mickey Bonin, then of Cambridge University Press, never lost faith that I would finish the manuscript. Having now completed it, I have many people to thank. First, I would like to thank Lyle, Charles and Mickey for their endless patience and steadfast support and enthusiasm for this volume. I would also like to thank them for reading the manu script carefully and for providing constructive and thought-provoking comments. I would especially like to acknowledge Lyle, who selflessly read and reread each chapter, and then provided detailed comments, feedback and guidance. I just don’t know that many people who are willing to engage in a three-year discussion and sometimes impassioned debate on the meaning of ‘meaning’. I am sincerely grateful to Lyle for contributing to that discussion and for his ongoing mentorship – and, of course, his good humo. 

Differing notions of ‘grammar’ for assessment

Introduction

The study of grammar has had a long and important role in the history of second language and foreign language teaching. For centuries, to learn another language, or what I will refer to generically as an L2, meant toknow the grammatical structures of that language and to cite prescriptions for its use. Grammar was used to mean the analysis of a languagesystem, and the study of grammar was not just considered an essential feature of language learning, but was thought to be sufficient for learners to actually acquire another language (Rutherford, 1988). Grammar in and of itself was deemed to be worthy of study – to the extent that in the Middle Ages in Europe, it was thought to be the foundation of all knowledge and the gateway to sacred and secular understanding (Hillocks and Smith, 1991). Thus, the central role of grammar in language teaching remained relatively uncontested until the late twentieth century. Even a few decades ago, it would have been hard to imagine language instruction without immediately thinking of grammar. 

What is meant by ‘grammar’ in theories of language?

Grammar and linguistics

Before attempting to define what it means to ‘know’ grammar or to be able to ‘use’ it to communicate in second or foreign language contexts, we first need to discuss what is commonly meant by ‘grammar’. This is important given the different definitions and conceptualizations of grammar that have been proposed over the years, and the diverse ways in which these notions of grammars have influenced L2 educators. When most language teachers, second language acquisition (SLA) researchers and language testers think of ‘grammar’, they call to mind one of the many paradigms (e.g., ‘traditional grammar’ or ‘universal grammar’) available for the study and analysis of language. Such linguistic grammars are typically derived from data taken from native speakers and minimally constructed to describe well-formed utterances within an individual framework. These grammars strive for internal consistency and are mainly accessible to those who have been trained in that particular paradigma. 

Form-based perspectives of language

Form-based perspectives of language

Several syntactocentric, or form-based, theories of language have provided grammatical insights to L2 teachers. I will describe three: traditional grammar, structural linguistics and transformational-generative grammar.

Form- and use-based perspectives of language

The three theories of linguistic analysis described thus far have provided insights to L2 educators on several grammatical forms. These insights provide information to explain what structures are theoretically possible in a language. Other linguistic theories, however, are better equipped to examine how speakers and writers actually exploit linguistic forms during language use. For example, if we wish to explain how seemingly similar structures like I like to read and I like reading connote different meanings, we might turn to those theories that study grammatical form and use interfaces. This would address questions such as: Why does a language need two or more structures that are similar in meaning? Are similar forms used to convey different specialized meanings? To what degree are similar forms a function of written versus spoken language, or to what degree are these forms characteristic of a particular social group or a specific situation? It is important for us to discuss these questions briefly if we ultimately wish to test grammatical forms along with their meanings and uses in context.

Communication-based perspectives of language

Other theories have provided grammatical insights from a communicationbased perspective. Such a perspective expresses the notion that language involves more than linguistic form. It moves beyond the view of language as patterns of morphosyntax observed within relatively decontextualized sentences or sentences found within natural-occurring corpora. Rather, a communication-based perspective views grammar as a set of linguistic norms, preferences and expectations that an individual invokes to convey a host of pragmatic meanings that are appropriate, acceptable and natural depending on the situation. The assumption here is that linguistic form has no absolute, fixed meaning in language use (as seen in sentences 1.5 and 1.7 above), but is mutable and open to interpretation by those who use it in a given circumstance. Grammar in this context is often co-terminous with language itself, and stands not only for form, but also for meaningfulness and pragmatic appropriacy, acceptability or naturalness – a topic I will return to later since I believe that a blurring of these concepts is misleading and potentially problematic for language educators.

What is pedagogical grammar?

Many language teachers who have taken courses in linguistic analysis and learned to examine language within the frameworks of formal, grammatical theories have often felt that these courses did not adequately meet their immediate needs. This is often because courses in linguistic analysis rarely address classroom concerns such as what grammar to teach, how to teach it and how to test it. Furthermore, it is unlikely that language teachers would attempt to teach phrase-structure rules, parameter-setting conditions or abstract notions of time and space, and certainly, they would never test students on these principles. As a result, many language teachers feel that knowledge of formal grammatical theory has little to offer their practice, and they have some misgivings asto how relevant this is for language assessment. Instead, in my experience, they prefer to draw on an experiential knowledge base derived from a familiarity with language textbooks, from their own hands-on experience of what actually works in classrooms, from a critical reflection oftheir practice and from informal discussions of their practice with cool language. 

Summary

In this chapter, I have attempted to answer the question ‘What do wemean by grammar?’ In this respect, I have differentiated between language and language analysis or linguistics. I have also discussed several schools of linguistics and have shown how each has broadened our understanding of what is meant by ‘grammar’. Finally, I have shown how these different notions of grammar provide complementary information that could be drawn on for purposes of teaching or assessing grammar. In the next chapter I will discuss how second language grammatical knowledge is acquired. In this respect, we will examine how grammatical ability has been conceptualized in L2 grammar teaching and learning, and how L2 grammar teaching and learning are intrinsically linked to assesment. 

The place of vocabulary in language

assessment

Introduction

At glance, it may seem that assessing the vocabulary knowledge of second language learners is both necessary and reasonably straightforward. It is necessary in the sense that words are the basic building blocks of language, the units of meaning from which larger structures such as sentences, paragraphs and whole texts are formed. For native speakers, although the most rapid growth occurs in child- hood, vocabulary knowledge continues to develop naturally in adult life in response to new experiences, inventions, concepts, social trends and opportunities for learning. For learners, on the other hand, acquisition of vocabulary is typically a more conscious and demand- ing process. Even at an advanced level, learners are aware of limita- tions in their knowledge of second language (or L2) words. They experience lexical gaps, that is words they read which they simply do not understand, or concepts that they cannot express as adequately as they could in their ®rst language (or L1). Many learners see second language acquisition as essentially a matter of learning vocabulary, so they devote a great deal of time to memorising lists of L2 words and rely on their bilingual dictionary as a basic communicative resource.

Recent trends in language testing However, scholars in the ®eld of language testing have a rather differ-

 ent perspective on vocabulary-test items of the conventional kind. Such items neatly into what language testers call the discrete- point approach to testing. This involves designing tests to assess whether learners have knowledge of particular structural elements of the language: word meanings, word forms, sentence patterns, sound contrasts and so on. In the last thirty years of the twentieth century, language testers progressively moved away from this approach, to the extent that such tests are now quite out of step with current thinking about how to design language tests, especially for prociency assessment.

A number of criticisms can be made of discrete-point vocabulary tests.

. It is dificult to make any general statement about a learner's voca-

bulary on the basis of scores in such a test. If someone gets 20 items correct out of 30, what does that say about the adequacy of the learner's vocabulary knowledge?

. Being proficient in a second language is not just a matter of knowing a lot of words ± or grammar rules, for that matter ± but being able to exploit that knowledge effectively for various commu- nicative purposes. Learners can build up an impressive knowledge of vocabulary (as re¯ected in high test scores) and yet be incapable of understanding a radio news broadcast or asking for assistance at an enquiry counter.

. Learners need to show that they can use words appropriately in their own speech and writing, rather than just demonstrating that they understand what a word can mean. To put it another way, the standard discrete-point items test receptive but not productive competence.

Three dimensions of vocabulary assessment

Up to this point, I have outlined two contrasting perspectives on the role of vocabulary in language assessment. One point of view is that it is perfectly sensible to write tests that measure whether learners know the meaning and usage of a set of words, taken as independent semantic units. The other view is that vocabulary must always be assessed in the context of a language-use task, where it interacts in a natural way with other components of language knowledge. To some extent, the two views are complementary in that they relate to different purposes of assessment. Conventional vocabulary tests are most likely to be used by classroom teachers for assessing progress in vocabulary learning and diagnosing areas of weakness. Other users of these tests are researchers in second language acquisition with a special interest in how learners develop their knowledge of, and ability to use, target-language words. On the other hand, researchers in language testing and those who undertake large testing projects tend to be more concerned with the design of tests that assess learn- ers' achievement or prociency on a broader scale. For such pur- poses, vocabulary knowledge has a lower prole, except to the extent that it contributes to, or detracts from, the performance of commu- nicative tasks. 

Discrete embedded

The first dimension focuses on the construct which underlies the assessment instrument. In language testing, the term construct refers to the mental attribute or ability that a test is designed to measure. In the case of a traditional vocabulary test, the construct can usually be labelled as `vocabulary knowledge' of some kind. 

Context-independent ± context-dependent

The role of context, which is an old issue in vocabulary testing, is the basis for the third dimension. Traditionally contextualisation has meant that a word is presented to test-takers in a sentence rather than as an isolated element. From a contemporary perspective, it is necessary to broaden the notion of context to include whole texts and, more generally, discourse. In addition, we need to recognise that contextualisation is more than just a matter of the way in which vocabulary is presented. 

An overview of the book

The three dimensions are not intended to form a comprehensive model of vocabulary assessment. Rather, they provide a basis for locating the variety of assessment procedures currently in use within a common framework and, in particular, they offer points of contact between tests which treat words as discrete units and ones that assess vocabulary morint integratively in a task-based testing context. At various points through the book I refer to the dimensions and exem- plify them. Since a large proportion of work on vocabulary assessment to date has involved instruments which are relatively discrete, selec-tive and context independent in nature, this approach may seem to be predominant in several of the following chapters. However, my aim is to present a balanced view of the subject, and I discuss mea- sures that are more embedded, comprehensive and context depen- dent wherever the opportunity arises, and especially in the last two chapters of the book.