Raters’ accent-familiarity levels and their effects on pronunciation scores and intelligibility on high-stakes English tests

Some current high-stakes tests of English have abandoned native-speaker models of pronunciation for scoring purposes, and instead rely largely on raters’ estimations of ‘listener effort’ needed to cope with test-takers’ speech in order to determine pronunciation scores. Recent studies within the field of language testing have revealed significant score variance occurring on such tests due to raters’ differing familiarities with test-takers’ accents. The studies that investigated raters’ accent-familiarity differences as a threat to reliability and validity of scores on highstakes tests have only determined significant score differences can occur, but have offered little more than speculation concerning why accent-familiarity impacts raters’ score decisions. The purpose of this thesis was to investigate not only the veracity of the threat, but also attempt to provide an explanation why raters’ accent-familiarity differences affect scores. A strong rationale exists supporting a hypothesis that exposure to the speech of a particular group of speakers, or accent, positively affects listeners’ speech processing abilities of utterances in that accent by increasing intelligibility. In order to determine the veracity of the hypothesis two studies were conducted: a pilot study examined the pronunciation scores and intelligibility differences between raters with different levels of accent-familiarity with Japanese-English, and a larger study investigated pronunciation score and intelligibility differences with Arabic-English, Spanish-English and Dhivehi-English. Many-Facets Rasch Measurements of the data revealed significant differences in both pronunciation scores and intelligibility occurred between accent-familiarity rater groups with all accents. The findings also showed significant correlations between level of accent-familiarity and score leniency, as well as accent-familiarity level and increased intelligibility, though the measures and effect sizes were not equal with each accent. Raters’ accent-familiarity differences were confirmed as a valid threat to pronunciation scores.