Abstracts
Barras, Malgorzata & Karges, Katharina: Giving test takers a voice: Pre-operational testing of listening (and reading) assessment tasks
To validate the use of assessment tasks, e.g. in listening tests, test developers need to know what is going on in the minds of test takers while they are solving test items. To this end, introspective data collection procedures can be used in addition to quantitative test data when piloting new test instruments. This can provide valuable insights into the cognitive processes and strategies used in task solving.
In our contribution, we will share findings from the pre-operational testing of scenario-based assessment tasks designed to assess receptive language skills in Swiss 9th-graders learning French and English as foreign languages. Our computer-based tasks, assessing listening, reading or a combination of the two, intended to emulate relevant language use within larger scenarios, e.g. preparing a trip to another city. To complete a scenario, the students solved several tasks, such as getting advice on sights to see from peers in an instant messaging chat. These tasks correspond to CEFR levels A2/B1, instructions and questions are presented in the common language of instruction (German).
We investigated the functioning of the tasks in a mixed-methods design. Qualitative data was collected through think-aloud protocols and/or retrospective interviews of more than 80 students, 30 of which participated in 2-hour in-depth interviews. Quantitative data was gathered involving 631 students.
Our presentation emphasizes qualitative insights into students' perceptions of key features of our scenario-based listening tasks, particularly the use of the language of instruction for questions and instructions.
Gießler, Ralf: Decoding challenges in EFL listening
The “massive and time consuming apparatus of training skills and strategies” (Swan & Walter 2017: 235) has become a default approach in teaching EFL listening. Still, the proportion of ninth-grade students in Germany who perform below the minimum standards in listening comprehension in English, ranges between 17 percent (nationwide) and 28.4% (Saxony-Anhalt) (IQB 2015: 11). Qual- itative analyses of state wide mandatory tests reveal that learners rely on contextual plausibility and top down processing strategies rather than on the cues provided by specific lexical items in the text (Qua-Lis 2017: 21). Other research evidence shows that L2 listeners tend to use L1 seg- mentation procedures involuntarily for segmenting L2 speech (Goh & Vandergrift 2021: 159f.)
In this talk, it will be argued that difficulties in listening comprehension are primarily caused by in- accurate and disfluent decoding and parsing. The interplay of lexical segmentation and vocabulary knowledge has been identified as a decisive variable for successful L2 listening comprehension re- cently (Lange & Matthews 2020). The complex process of word segmentation and word activation will be illustrated in some detail with examples from listening comprehension texts. In conclusion, implications for teaching L2 listening will be brought up for discussion.
List of references
Böhme, K.; Schipolowski, S.; Haag, N. (Eds.). (2015). IQB Trends in Student Achievement 2015. The Second National Assessment of Language Proficiency at the End of the Ninth Grade. Summary. Münster: Waxmann. Online available: https://www.iqb.hu-berlin.de/bt/BT2015/Bericht/
Lange, K. & Matthews, J. (2020). Exploring the relationships between L2 vocabulary knowledge, lexical segmentation, and L2 listening comprehension. Studies in Second Language Learning and Teaching, 10 (4), 723-749.
Vandergrift, L., & Goh, C. C. (2012). Teaching and learning second language listening: Metacogni-
tion in action. Routledge.
Swan, M. & Walter, C. (2017). Misunderstanding comprehension. Elt Journal, 71(2), 228-236.
Qua-LisNRW (2017). Fachdidaktische Rückmeldung zu den zentralen Prüfungen am Ende der
Klasse 10 (ZP10) im Fach Englisch. Online available: https://www.schulentwicklung.nrw.de/cms/upload/Faecher_Seiten/eng
Guggenbichler, Elisa: Exploring the effect of self-paced listening on test scores of learners with differing L1 literacy skills
As learners with low-level literacy skills demonstrate decreased L2 listening comprehension (e.g., Kormos et al., 2019) and higher levels of anxiety (e.g., Piechurska-Kuciel, 2008), traditional administrator-controlled listening tests seem to pose particular challenges for learners with reading-related learning difficulties. To accommodate these learners, test providers may allow candidates to self-pace, i.e., stop and rewind, the audio during tests. However, research on the effect of self-pacing as a test accommodation for learners with low-level L1 literacy skills yielded inconclusive results (Eberharter et al., 2023). This study presents preliminary findings on the differential effect of self-paced listening on (a) test scores and (b) anxiety of students with varying L1 literacy skills, L2 vocabulary knowledge, and metacognitive awareness across response formats.
To expand research on self-paced listening, 200 secondary-school learners of English in Austria (grades 11/12) took a subset of items from a standardized B2 listening exam in a counter-balanced design. They completed an anchor task and then took two tasks in four different conditions regarding administration mode (standard vs. self-paced listening) and task format (multiple-choice items vs. open answers). Between the conditions, participants completed a questionnaire on test-taking strategies and listening anxiety. They also took a standardized L1 literacy test, the MALQ (Vandergrift et al., 2006), and an L2 vocabulary test, which served as a proxy for L2 proficiency. Data was analysed using Generalized Linear Mixed-Effects Modelling and multi-facet Rasch analyses. The findings provide insights into the complexity and viability of self-pacing to increase fairness in L2 listening tests for students with low-level literacy skills.
List of references
Eberharter, K., Kormos, J., Guggenbichler, E., Ebner, V. S., Suzuki, S., Moser-Frötscher, D., Konrad, E., & Kremmel, B. (2023). Investigating the impact of self-pacing on the L2 listening performance of young learner candidates with differing L1 literacy skills. Language Testing, 02655322221149642. https://doi.org/10.1177/02655322221149642
Kormos, J., Košak Babuder, M., & Pižorn, K. (2019). The role of low-level first language skills in second language reading, reading-while-listening and listening performance: A study of young dyslexic and non-dyslexic language learners. Applied Linguistics, 40(5). https://doi.org/10.1093/applin/amy028
Piechurska-Kuciel, E. (2008). Input, processing and output anxiety in students with symptoms of developmental dyslexia. In J. Kormos & E. H. Kontra (Eds.), Language learners with special needs: An international perspective (pp. 86–109). Multilingual Matters.
Vandergrift, L., Goh, C. C. M., Mareschal, C. J., & Tafaghodtari, M. H. (2006). The metacognitive awareness listening questionnaire: Development and validation. Language Learning, 56(3), 431–462. https://doi.org/10.1111/j.1467-9922.2006.00373.x
Jones: How do listening input modality and feedback timing affect vowel acquisition and listening comprehension?
L1 users perceive vowels more accurately than L2 users (Flege & McKay, 2004) and novel contrasts may be perceived differently depending upon L1 influence (Iverson & Evans, 2007). Research on second language phonology acquisition (Hardison, 2003; Sueyoshi & Hardison, 2005; Motohashi-Saigo & Hardison, 2009) shows that visual modality can influence consonant acquisition. One reason for this may be due to the same region of the brain being activated by both acoustic speech information and visual speech articulatory gestures (Glanz et al, 2018). However, there is little work on the use of articulatory gestures to facilitate visual salience of vowels and how this affects acquisition (exceptions being Masapollo, Polka & Menard, 2017; and Polka, Ruan & Masapollo, 2019). Video use for second language listening input is widespread, therefore understanding whether modality affects language acquisition and listening comprehension can inform more effective media use in teaching and learning. We investigated the following research question: Does input modality affect perceptual acquisition of vowels? This quasi-experimental study examines the relationships between input modality, vowel acquisition and listening comprehension. 36 L1 Japanese learners in the first year of study at a Japanese university completed in-class online listening tasks for L2 English that differed only in input modality, with half exposed to audiovisual input and the remaining half exposed to audio-only input in order to assess acquisition of the vowels TRAP, STRUT, THOUGHT and NURSE. Gains in post-test and delayed post-test scores were analysed using Bayesian general linear models. No evidence was found for visual salience increasing vowel acquisition between conditions, which may be due to factors such as screen size, attendance to visual images, and time/frequency of exposure to the target vowels in an audiovisual modality.
List of references
Flege, J. E., & MacKay, I. R. A. (2004). Perceiving vowels in a second language. Studies in Second Language Acquisition, 26(01). https://doi.org/10.1017/S0272263104026117
Glanz, O., Derix, J., Kaur, R., Schulze-Bonhage, A., Auer, P., Aertsen, A., & Ball, T. (2018). Real-life speech production and perception have a shared premotor-cortical substrate. Scientific Reports, 8(1). https://doi.org/10.1038/s41598-018-26801-x
Hardison, D. M. (2003). Acquisition of second-language speech: Effects of visual cues, context, and talker variability. Applied Psycholinguistics, 24(4), 495–522. https://doi.org/10.1017/S0142716403000250
Hardison, D. M. (2020). Multimodal input in second-language speech processing. Language Teaching, 54(2),206-220. https://doi.org/10.1017/S0261444820000592
Iverson, P., & Evans, B. G. (2007). Learning English vowels with different first-language vowel systems: Perception of formant targets, formant movement, and duration. The Journal of the Acoustical Society of America, 122(5), 2842-2854. https://doi.org/10.1121/1.2783198
Masapollo, M., Polka, L., & Ménard, L. (2017). A universal bias in adult vowel perception – By ear or by eye. Cognition, 166, 358–370. https://doi.org/10.1016/j.cognition.2017.06.001 Motohashi-Saigo, & Hardison, D. M. (2009). Acquisition of L2 Japanese geminates: Training with waveform displays. Language Learning & Technology, 13(2), 29–47.
Polka, L., Ruan, Y., & Masapollo, M. (2019). Understanding vowel perception biases – it’s time to take a meta-analytic approach. In A. M. Nyvad, M. Hejná, A. Højen, A. B. Jespersen, & M. Hjortshøj (Eds.), A sound approach to language matters: In honor of Ocke-Schwen Bohn (pp. 561–582). Dept. of English, School of Communication & Culture, Aarhus University.
Sueyoshi, A., & Hardison, D. M. (2005). The Role of Gestures and Facial Cues in Second Language Listening Comprehension. Language Learning, 55(4), 661–699. https://doi.org/10.1111/j.0023-8333.2005.00320.x
Krämer, Viktoria, Dillmann, Lisa Marie & Rossa, Henning: The interactive nature of item difficulty in L2 listening assessment: The predictive value of task characteristics
This study investigates the nature of item difficulty in listening comprehension tasks used in the annual national language assessment project VERA-8 English. As VERA-8 assesses language levels ranging from A1 to C1 of the CEFR (Council of Europe, 2020), task and item difficulties are correspondingly varied. Test developers and item writers aim to judge and predict item difficulty as accurately as possible, however a generic effective rating system is not supported by the research literature on task difficulty. Aiming at developing a suitable system for our specific project, we consider how task and item difficulty can be anticipated based on task characteristics and test specifications. We further explore the relevance of the respective characteristics and to what extent these task characteristics, as well as the cognitive processes involved in solving a task, can be used to predict and explain psychometric parameters such as difficulty and discrimination based on pilot testing task performance data.
To this end, a descriptive analysis of trialed listening comprehension tasks will be conducted regarding the representativeness and distribution of the required cognitive processes by coding the cognitive operations elicited by a task or an item (as proposed by the theoretical specifications, such as input decoding, lexical search, parsing, meaning construction and discourse construction etc.) as well as various task characteristics such as length and complexity of stimulus, lexis, information density, ambiguities in the text, number of speakers, speakers’ accents, speech rate, task format, length and complexity of items etc. (see also Nold et al., 2008). Developing a rating system to assess cognitive processes demanded by the individual items will be an integral part of the coding process while also providing a more reliable instrument for prediction of item and task difficulty.
Latimer, Nicola: The Road to Understanding in Lecture Listening
The listening tasks used in most university admissions language tests (e.g. IELTS, TOEFL iBT, PTE Academic, CAE) use audio material as the primary/sole source of information for comprehension. Test-takers are required to read questions and answer options (e.g. in MCQs) but not to read information and integrate it with the audio input to build a coherent understanding of content. Concerns have been raised about the artificial nature of such test methods (Field, 2019, pp.73-86) as this is not what students are required to do once they start their courses.
Most academic lectures combine auditory information with textual information on slides (Hallewell & Crook, 2019; Roberts, 2018). Understanding academic lecture content is likely to involve (near-)synchronous integration of auditory input from the lecturer and textual (and other visual) information on presentation slides. However, there has been a lack of research into the way that students process and integrate these two streams of information.
This presentation reports on a study that used eye-tracking and stimulated recall to investigate how audio and textual information are integrated in lectures. We collected and analysed 5 real-life lectures to identify discourse relations between the lecturer's speech and slide text. We then did eye-tracking and stimulated recall with 4 x B2 and 4 x C1 university students to see how they integrate audio and textual information when watching lecture clips, under two conditions: (1) with headings only, (2) headings + some elaboration.
The findings provided insights into factors affecting ease/difficulty of lecture comprehension, and relevant metacognitive strategies. We then discuss implications for how to make lectures easier for L2 students to follow, and for designing test tasks assessing lecture listening.
Siepmann, Philipp & Walper, Katharina: More than just talking – Considering listening comprehension in oral communication exams
Oral communication exams have been introduced as a mandatory part of summative assessment in foreign language classrooms in many federal states of Germany in the wake of the establishment of national educational standards (KMK 2012). While the official guidelines frame oral examinations as a "speaking exam" (Sprechprüfung, e.g., Niedersächsisches Kultusministerium 2023; see also MSW NRW 2014), it is obvious that successful foreign language communication in real-world contexts not only requires sufficient speaking proficiency, but also listening comprehension competence.
This contribution will report on a design-based research project carried out between 2019 and 2023 in the German federal state of North Rhine-Westphalia. As part of a complex intervention in the practice of oral communication exams, the assessment task was redesigned in a way that it allows for the candidates to demonstrate, and for examiners to evaluate, learners' transactional and interactive listening comprehension competence. Employing conversation analysis, a comparison between discourse structures in the pre-intervention and post-intervention assessment designs will reveal how task structures can afford positive mutual interdependence in communication and hence require active and close listening. Thus, it will be shown that performance in these examinations will depend as much on good listening as on elaborate speaking.
List of references
Ministerium für Schule und Weiterbildung des Landes Nordrhein-Westfalen [MSW NRW] (2014). Mündliche Prüfungen in den modernen Fremdsprachen in der gymnasialen Oberstufe. Handreichung September 2014. https://www.standardsicherung.schulministerium.nrw.de/cms/upload/ angebote/muendliche_kompetenzen/docs/1503_Handreichung_Muendliche_ Pruefungen.pdf [3 September 2023].
Niedersächsisches Kultusministerium (2023). Die Sprechprüfung im Fremdsprachenunterricht. Online. https://bildungsportal-niedersachsen.de/allgemeinbildung/unterrichtsfaecher/die-sprechpruefung-im-fremdsprachenunterricht [3 September 2023].
Steinlen, Anja & Piske, Thorsten: The development of English listening skills of minority and majority language students in a bilingual and a regular elementary school program in Germany
Foreign language (FL) listening skills are among the basic skills acquired during the elementary school years. However, conflicting results have been reported regarding English listening skills of students with a minority language background, for whom the English often constitutes the L3 (e.g. Elsner, 2007; Keßler & Paulick, 2010). In addition, little information is available as to how students’ FL listening skills develop over time in different programs with different FL intensity. In this longitudinal study, we will focus on English listening skills of third and fourth grade students with and without majority language background, who have attended either a regular or an intensive bilingual elementary school program in Germany since grade 1.
We will present data of 100 children (50 students per program, 50% with a minority language background), whose English listening skills have been assessed using the listening subtest of the Primary School Assessment Kit (PSAK-L, Little et al. 2003). These data are supplemented by cognitive variables (e.g. nonverbal intelligence, Raven et al. 2002) and a parents’ questionnaire.
The results indicate that the proficiency level depends on the intensity of the program: The 4th graders in the regular FL program reach level A1 whereas their peers in the bilingual program obtain level A2/B1. In addition, statistical analyses do not indicate any significant differences of language background in either grade and in either program. These findings lends support to the assumption that any type FL programs in elementary schools may be beneficial for minority language children, even those programs with a higher intensity of FL input such as intensive bilingual programs.
List of references:
Elsner, Daniela (2007). Hörverstehen im Englischunterricht der Grundschule. Ein Leistungsvergleich zwischen Kindern mit Deutsch als Muttersprache und Deutsch als Zweitsprache. Frankfurt am Main: Peter Lang.
Keßler, Jörg-Uwe & Paulick, Christian (2010). Mehrsprachigkeit und schulisches Englischlernen bei Lernern mit Migrationshintergrund. In: Ahrenholtz, Bernt (Eds.). Fachunterricht und Deutsch als Zweitsprache. Tübingen: Narr, 257-278.
Little, D.; Simpson, B.; Catibusic, B. (2003). PSAK. Primary School Assessment Kit. Dublin: Integrate Ireland Language and Training.
Raven, John C. (1976b). Standard Progressive Matrices (SPM). Sets A, B, C, D & E. Prepared by JC Raven. San Antonio, TX: Harcourt.
Thaler, Isabelle Sophie: Investigating the pedagogical content knowing (PCKing) of listening comprehension in pre-service teacher education
This single-case study investigated a pre-service foreign language teacher’s pedagogical content knowing (PCKing) of teaching listening from an interactionist perspective. Contrary to the ‘cognitivist’ (Kubanyiova & Feryok, 2015) perspective, considering cognition as fixed and static and neglecting influences of and on context and action, the ‘interactionist’ (Li, 2020) perspective emphasises “situated, dynamic… knowing in action” (Kubanyiova & Feryok, 2015, p. 438).
This idiographic research, set at a Bavarian secondary school, was driven by the following three research questions, which mirror the three phases of teaching (prospective, enactive and retrospective, Shulman, 1986):
Cognition-for-interaction: What does lesson planning reveal about an EFL teacher’s pedagogical content knowledge (PCK) of scaffolding listening competence?
Cognition-in-interaction: How does an EFL teacher’s pedagogical content knowing (PCKg) of scaffolding listening competence unfold during teaching?
Reflection on and for cognition-in-interaction: What does an EFL teacher’s reflection on her cognition-in-interaction reveal about factors influencing this unfolding, and any potential transformation of her PCK?
Interactionist studies of language teacher cognition investigate “what teachers say and do” (Li, 2020, p. 28), which is why oral data in the form of semi-structured interviews and lesson observation was collected. The findings were analysed using Shulman’s (1987) model of pedagogical reasoning. One of the most striking findings was the strong congruence between the teacher’s cognition-for-interaction and her cognition-in-interaction, which is rather unusual for pre-service teachers. Furthermore, it was remarkable how the pre-service teacher helped her class develop metacognitive awareness, what one might almost call pupil-PCK of listening, along the lines of Shulman’s (1986) two categories: understanding difficulties of listening and ways of making listening “comprehensible” (p. 9).
List of references
Borg, S. (2003). Teacher cognition in language teaching: A review of research on what language teachers think, know, believe, and do. Language Teaching, 36(2), 81–109.
Borg, S. (2009). Language teacher cognition. In A. Burns & J. C. Richards (Eds.), Second Language Teacher Education (pp. 163–171). Cambridge University Press.
Borg, S. (2012). Current approaches to language teacher cognition research: A methodological analysis. In R. Barnard & A. Burns (Eds.), Researching language teacher cognition and practice (pp. 11–29). Multilingual Matters.
Kubanyiova, M., & Feryok, A. (2015). Language teacher cognition in applied linguistics research: Revisiting the territory, redrawing the boundaries, reclaiming the relevance. The Modern Language Journal, 99(3), 435–449.
Li, L. (2017). Social interaction and teacher cognition. Edinburgh University Press.
Li, L. (2020). Language teacher cognition: A sociocultural perspective. Palgrave Macmillan.
Shulman, L. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–23.
Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.
Vandergrift, L. (2007). Recent developments in second and foreign language listening comprehension research. Language Teaching, 40(3), 191–210.
Wilden, Eva, Porsch, Raphaela & Guttke, Joel: Analyzing the potential for communicative-cognitive activation in primary EFL listening tasks
One of the three basic dimensions of instructional quality, cognitive activation is assumed to be an influencing factor of students’ learning outcomes (Praetorius et al., 2018). One way of assessing the degree of cognitive activation is through task analysis. As the building blocks of teaching English as a foreign language (EFL), tasks provide information about the potential for cognitive activation in a lesson (Robinson, 2011). Due to inconclusive empirical evidence on the effectiveness of cognitive activation, subject-specific operationalizations of the construct have been proposed in order to consider the unique criteria of quality in teaching in different subjects.
This talk defines communicative-cognitive activation in the context of primary EFL education and presents a coding scheme developed to analyze listening tasks for their potential for communicative-cognitive activation (CCA). Using qualitative content analysis (Mayring, 2022), 711 tasks taken from five primary EFL textbooks used in Germany were analyzed regarding their potential for CCA. Results indicate rather consistent levels of CCA across all textbooks regarding overall textbook and task characteristics. However, textbooks vary considerably regarding their potential for CCA in listening tasks. This hints at the crucial role of teachers in achieving CCA in the primary EFL classroom.
List of references
Mayring, P. (2022). Qualitative content analysis: a stey-by-step guide. SAGE Publications.
Praetorius, A.-K., Klieme, E., Herbert, B. & Pinger, P. (2018). Generic dimensions of teaching quality: the German framework of Three Basic Dimensions. ZDM Mathematics Education 50, 407-426. https://doi.org/10.1007/s11858-018-0918-4
Robinson, P. (2011). Second language task complexity, the Cognition Hypothesis, language learning, and performance. In P. Robinson (Ed.), Second Language Task Complexity (pp. 3-38). John Benjamins.