MULTIMODAL AI IN FOREIGN LANGUAGE TEACHING METHODOLOGY

ABSTRACT

This article is devoted to the application of multimodal artificial intelligence in learning foreign languages and how it helps to improve understanding, increase motivation, and develop all four language skills. The article presents tools for creating educational materials and an example lesson. This demonstrates how multimodal AI makes the language learning process more efficient and interesting.

Keywords: Multimodal AI, foreign language teaching, lesson plan, interactive learning.

In the context of global digitalization and the intensive introduction of artificial intelligence technologies, it is critically important to identify effective methods of teaching foreign languages. Modern pedagogical paradigms focus on the formation of students' communicative competence, which necessitates the use of interactive and personalized educational tools. Within the framework of this concept, multimodal artificial intelligence integrating textual, visual and auditory data is positioned as a promising tool that helps to increase motivation and optimize the learning process of language material.

“Advances in so-called transformer - based deep neural networks led to the creation of a Generative AI systems capable of accepting natural language prompts as input, called chatbots - computer programs that process human conversation, allowing humans to interact with digital devices as if they were communicating with a real person — of which ChatGPT (Chat Generative Pre-trained Transformer) is the most widely-used one (at present).” [1, p 28].

The purpose of this article is to analyze the theoretical foundations of multimodal learning and artificial intelligence, as well as to evaluate the effectiveness of their integration into foreign language teaching methods based on a practical experiment.

This approach is based on R. Mayer's cognitive theory of multimedia learning, which explains why the simultaneous use of different sensory channels, when carefully integrated, makes learning more effective.

“Multimodal learning is grounded in the Cognitive Theory of Multimedia Learning, developed by Richard E. Mayer. According to this theory, learners process information through two primary channels: a verbal channel and a visual channel, each with limited capacity.” [2, p. 31].

When applied to the study of foreign languages, a multimodal approach allows for the creation of a language environment that is as close as possible to natural communication.

The use of images, audio and video materials not only promotes a deep understanding of the meaning of linguistic units, but also comprehensively develops all aspects of speech activity: from listening to writing.

“ChatGPT, an artificial intelligence application, has emerged as a promising educational tool with a wide range of applications, attracting the attention of researchers and educators. This qualitative case study, chosen for its ability to provide an indepth exploration of the nuanced effects of AI on the foreign language learning process within its realworld educational context, aimed to utilize ChatGPT in foreign language education, addressing a gap in existing research by offering insights into the potential, benefits, and drawbacks of this innovative approach. ” [3, p. 9]. As a result, learners can wisely use it and be are able to construct mental connections between meaning, form, and use and learning environments appear that have both adaptive and cognitively rich capabilities.

“Today’s students are growing up in an era increasingly dominated by AI applications. Familiarity with these technologies helps students to use AI effectively, engage with it critically, and understand its applications.” [4 p. 6]. The introduction of multimodal artificial intelligence into language education opens up new horizons and is a significant step forward in teaching methodology.

“The Artificial Intelligence (AI) is being used in every field around the world, including education. For example, in the United States, the companies such as Google and Microsoft are investing heavily in AI education technology, while the government emphasized the importance of integrating AI technology into the context of foreign language learning. Microsoft developed AI-powered education tools, such as Learning Tools for OneNote, which uses to improve reading and writing skills. With the implementation of speech recognition technology, students can practice both speaking and listening while receiving precise feedback.” [5, p.1].

To begin implementing this approach in foreign language teaching, you can apply a step-by-step method, starting with a small group of students or even with one student. This will allow you to carefully monitor the process and promptly make the necessary changes. At the initial stage, it is extremely important to determine the current level of language proficiency through diagnostic testing. The test should cover grammar, vocabulary, listening, speaking, and writing skills. Such an initial analysis will help identify the strengths and weaknesses of the student, which will form the basis for developing an individual learning plan.

A variety of tasks can be used to develop all four basic language skills (reading, writing, listening, speaking). For example, dialogues generated by artificial intelligence, followed by questions, will help improve speech perception skills by ear. Conversational skills can be developed by answering questions from a chatbot or by participating in AI-enabled role-playing games. In written assignments, students can describe images created by artificial intelligence or analyze video materials. To expand the vocabulary, AI can offer illustrations and hints, helping to establish associations between words and images. In addition, artificial intelligence provides instant feedback, which increases students' confidence, promotes self-correction, and makes the learning process more interactive. [Table 1]

Table 1.

Multimodal AI Tools for developing language skills

The table above illustrates how various AI tools can be effectively utilized to enhance all four essential language skills: listening, speaking, writing, and vocabulary. Through the integration of various modalities —audio, text, visual, and interactive elements — students encounter a more immersive and engaging environment for language learning.

“This multimodal approach not only reinforces comprehension but also facilitates active production and self-correction, aligning with cognitive principles of multimedia learning”. [2, p. 55]

In practice, applying these tools involves creating lessons centered on particular themes or subjects. For example, one lesson can involve hearing an AI-generated conversation, answering questions through a chatbot, composing a description of an AI-produced image, and doing vocabulary tasks with visual cues. This integration guarantees that students cultivate various skills at the same time while staying motivated and involved with interactive materials.

Ultimately, consistent evaluation and tracking of student advancement are advised. By integrating pre and post-tests with observational data on motivation and engagement, educators can assess both quantitative and qualitative results of multimodal AI interventions, offering proof of this method's effectiveness in foreign language teaching.

The effective application of multimodal AI in English language classes includes organizing lessons around specific thematic topics, like Travel, Food, or Hobbies. Every lesson aims to combine textual, visual, auditory, and interactive elements to enhance the four language abilities: listening, speaking, writing, and vocabulary.

Activity options for the lesson can consist of:

1. Listening & Understanding: Learners engage with AI-created conversations and respond to comprehension queries, receiving prompt feedback.

2. Conversation Exercises / Role-playing: Engaging with AI chatbots or enacting situations that mimic real-life contexts, like placing an order at a restaurant or requesting directions.

3. Writing Tasks: Detailed descriptions of images generated by AI or concise summaries of brief AI videos, utilizing AI tools for feedback on grammar and vocabulary.

4. Vocabulary Activities: Engaging flashcards and tests featuring visuals and AI-generated clues, strengthening newly learned words through context.

Anticipated Results:

1.  Improve understanding and creation abilities in all four language domains.
2.  Boost motivation and involvement by utilizing interactive tasks with visual support.
3.  Enhance learner independence by offering immediate, tailored feedback.
4.  Enhance vocabulary memory by using multiple modes of association (pictures, sounds, and written words).

You can see an example of a lesson on the table below.

Table 2.

Teacher’s Lesson Plan: 45-Minute English Lesson Using Multimodal AI

Practical application shows that AI tools can improve all four language abilities — listening, speaking, writing, and vocabulary — while offering immediate feedback for self-adjustment. These tools promote learner independence, enhance confidence, and stimulate active involvement, rendering lessons more engaging and effective.

In conclusion, multimodal AI supports rather than substitutes educators, enhancing the educational experience, improving skills acquisition, and equipping learners for a more engaging and tech-forward language learning journey.

References:

1. Miles, M. B., Huberman, A. M., & Saldaña. Qualitative data analysis: A methods sourcebook (4th ed.). SAGE. - 2020.
2. Mayer, R. E. (2020). Multimedia Learning (3rd ed.). Cambridge: Cambridge University Press.
3. Ozek Filiz Gunyel, Fatih Karataş, Faramarz Yaşar Abedi, Derya Karadeniz Yasemin, Kuzgun Received: March 2024.
4. Panteion of Athens Aristotle University of Thessaloniki, Artificial Intelligence in Foreign Language Education/Amalia Maria Fyka/ Greece.2024.
5. ResearchGate/AI tools in foreign language teaching/Marhabo Avazmatova/June 2024.