WHEN AI GETS WRITING WRONG: A REVIEW ARTICLE OF HALLUCINATIONS AND INACCURATE FEEDBACK IN EFL WRITING

ABSTRACT

Generative AI is increasingly used to provide feedback on EFL student writing, promising to ease teacher workload, yet its fluent, confident output can hide significant inaccuracies. This review synthesizes eight studies (2019–2026), drawn from an initial pool of 47 sources across Google Scholar, ResearchGate, and Elibrary, to examine hallucinations and inaccurate feedback generated by tools such as ChatGPT, GPT-4, and PaLM 2. It proposes a taxonomy of errors, including false corrections, missed errors, misleading explanations, redundant feedback, inconsistency, and hallucination, with hallucination rates reaching roughly a quarter of all feedback sentences. AI feedback proves relatively reliable at the sentence level but far less so for coherence, argumentation, and meaning and since inaccurate feedback sounds as confident as accurate feedback, EFL learners may struggle to tell the two apart. The review concludes that AI feedback should serve as a supervised, supplementary tool rather than a replacement for instructor judgment.

Keywords: artificial intelligence; large language models; ChatGPT; hallucination; automated writing evaluation; automated feedback; EFL writing; written corrective feedback; feedback accuracy; AI in language education.

Introduction. The dream of automating the assessment and feedback of student writing is nearly as old as educational technology itself. In 1966, Ellis Page complained about the «exorbitant» burden placed on English teachers who spent numerous hours grading student essays, imagining a future where computers might shoulder this responsibility [1, p. 1]. Even 50 years after Ellis Page's claim, objections over teacher workload persist, as do dreams of offloading that work to automation [1, p. 1]. The challenge is still, throughout the centuries, one of the most highly relevant and discussed. Moreover, the complexity is relatively acute; teachers are typically reluctant to assign more writing practice because doing so requires the time to read and respond that teachers do not usually have. Regular workload at educational institutions limits the time for checking students' writing outcomes as well as controlling the overall writing competence and its progress. Providing students with formative feedback during the writing process is a key instructional practice that helps students improve as writers, yet the considerable time and effort it takes to provide students with feedback, especially multiple students across multiple classes, is discouraging for many educator [2, p. 2]. The large number of students and the specific format of writing regularly limit the ability to provide individual feedback to students, restricting the development of advanced writing skills. Recent progress in generative AI has offered a potential solution to this long-lasting problem. Generative AI systems, like OpenAI's GPT-4, are large transformer-based deep neural network large language models developed with large amounts of text from the Internet  [1, p. 2]. Furthermore, they provide required information with the reliance on algorithms they were coded with. Unlike previous Automated Writing Evaluation (AWE) systems, which were limited to specific features of writing and typically restricted to academic genres, generative AI is not restricted to specific scoring metrics and can instead analyze a text based on whatever parameters are input by the user [1, p. 2]. By now, there is a large quantity of AI, which is mostly available and has free trials, that teachers as well as students can use for specific purposes. ChatGPT, in particular, has gathered great attention, attracting more than one million users within just five days of its launch [4]. Moreover, ChatGPT achieved 900 million active users per week in February 2026, growing twice from the previous year and 40 % of it is used for writing goals, editing, communication, etc. Its capabilities open an innovative direction in language learning [4]. Yet, despite the noticeable strengths of LLMs, they are not specifically designed for AES, nor is it clear how reliable they may be if utilized for such purposes [1, p. 2]. As researchers have noted, ChatGPT's original purpose was to simulate human conversation, but the program's functions go far beyond this; it has the ability to create new text from scratch [4]. This creates a paradox at the heart of AI-assisted writing instruction. AI can provide immediate, sophisticated, and personalized feedback, but the fluency of its answers can have inaccuracies and inappropriate recommendations. While LLM-based feedback systems generate feedback that is «highly fluent, coherent, and human-like,» they are «susceptible to generating spurious text» [3, p. 491]. In other words, while these systems are capable of producing feedback that is often indistinguishable from instructor feedback in terms of fluency and coherence, they may generate prose that is misleading or completely irrelevant to the input student assignments [3, p. 491]. The stakes are substantional. Hallucinated content can «potentially lead students to believe what is false, impacting their understanding of knowledge and even their motivation to learn» [3, p. 491]. Furthermore, concerns about hallucination «impede the deployment of feedback systems in real classes» . Generative AI tools «can produce fabricated information that appears authentic,» with a Stanford HAI study finding that general-purpose AI chatbots hallucinated on 58–82 % of legal research questions. Even specialized legal AI tools built on retrieval-augmented generation hallucinated more than 17 % of the time  [8]. Therefore, this review examines the nature and frequency of inaccurate feedback generated by AI writing tools, the reasons for inaccuracies' appearance and the implications for EFL writing instructiоn. So, it proposes a taxonomy of AI feedback errors and considers whether students can recognize incorrect AI feedback- an educational problem that may be more significant than the hallucinations themselves.

Methodology. To identify relevant sources on the accuracy of AI-generated feedback in EFL writing instruction, this review followed a systematic search process across several academic databases: Google Scholar, ResearchGate and Elibrary, with additional searching of key journals in applied linguistics and computer-assisted language learning. Searching processes were combined in various layouts and included artificial intelligence, large language models, ChatGPT, GPT-4, hallucination, automated writing evaluation, automated feedback, written corrective feedback, EFL writing, ESL writing, error correction, accuracy, reliability, and bias. The search covered publications from 2019 through 2026. The previous sources were chosen to capture early foundational work on AI bias and fairness, while the latest studies included the most recent empirical work on AI generative tools in educational settings. Sources were included if they (a) addressed AI-generated feedback on student writing in EFL, ESL, or broader educational settings; (b) offered empirical findings on the accuracy, reliability, hallucination or overall quality of AI feedback; (c) drew comparisons between AI and human feedback where relevant; (d) were written in English; and (e) were accessible in full text. Sources were excluded if they (a) dealt only with AI-generated content rather than feedback specifically; (b) applied to domains outside writing instruction; (c) lacked peer review or empirical grounding; or (d) predated 2019. This process initially returned 47 potentially relevant sources. After screening titles and abstracts, 23 remained for full-text review. Applying the inclusion and exclusion criteria to these 23 sources narrowed the final focus to eight studies. Together, these eight sources offered a sufficiently varied and substantive base for examining hallucinations and inaccuracies in AI-generated feedback on EFL writing.

Literature review. Before it can be evaluated how accurate AI feedback actually is, it is needed to be clear about what «inaccurate» even means in this context. Analyzing the phenomena across the nine sources reviewed here, several distinct types of error keep showing up and they are not similar . The most straightforward type is the false correction: the AI flags something as wrong when it was basically fine. [4, p 7-8] Moreover, ChatGPT marked the correct sentence as an error, then produced a «correction» that was literally identical to the original. When researchers dismissed and asked it to point out exactly what was wrong, the model retracted and admitted the sentence had been correct all along. The authors describe this pattern as «overflagging» [4] . Similarly a good deal of ChatGPT's feedback was technically accurate but pointless-grammatically fine, yet not doing anything to actually improve the writing [5, p. 9-11]. That kind of feedback functions almost like a false correction in practice since it nudges students to revise language that never needed revising and can leave them second-guessing choices that were already sound. Nevertheless, the opposite problem is when the AI simply fails to catch a real mistake. Here the numbers are hard to ignore: ChatGPT missed 64 % of unclear-message errors, 52 % of faulty connectors, 46 % of non-idiomatic phrasing, and 38 % of structural sentence problems. [4, p. 7-8]. These aren't minor gaps and notably, they group around the higher-order issues that require actually following the writer's meaning rather than scanning for surface slips. However across 24 student paragraphs, human instructors flagged 670 errors in total, while ChatGPT caught only 333-roughly half.[6]. A mechanistic explanation for at least part of this: the BART-based system it studied sometimes «misinterprets spurious correlations in the data,» which can cause it to generate feedback that contradicts what's actually in the student's text rather than responding to it. Sometimes the correction itself is fine, but the reasoning behind it isn't. [3, p. 496]. ChatGPT gave unusually detailed explanations-more detailed, one instructor noted, that the depth didn't guarantee accuracy. The same study documented a tendency toward over-correction, where phrasing or sentence structure got rewritten even though the original was grammatically sound, often with an explanation that didn't quite hold up.[5, с. 9-11]. Similarly, feedback that reads as fluent and confident but is actually not connected to what the student eventually wrote. [3, p. 491]. That combination is arguably more dangerous than an error that's obviously wrong, because trustworthy-sounding feedback is exactly the kind students are most likely to accept without question. Hallucination is a feedback that references something not actually present in the text [3, p. 495]. In this study, it is defined as generated text that's either nonsensical or unfaithful to the source material and finds that hallucinations turned up in roughly a quarter of all feedback sentences: 27.1 % for the data-driven system and 23.5 % for the prompt-driven one. Furthermore there two types of hallucinations. Intrinsic hallucinations directly contradict the input, while telling a student their work lacks a test plan when one is right there. Extrinsic hallucinations are murkier: claims that can't be checked against the source either way, like commenting on screenshot sizing when there's no image in the document at all [3, p. 495]. In addition, PaLM 2 claimed a student had «developed 2–3 paragraphs» of writing that had no visible paragraph. Even more unsettling, the same study found that when students left their writing unfinished cut off mid-sentence, GPT-4 would sometimes simply finish the sentence itself and then score the (partly AI-written) result, in some cases inflating the score as a consequence [1, p. 2]. Moreover, in Mata v. Avianca, a New York attorney used ChatGPT for case research, and the tool fabricated citations and quotes, insisting they were real and locatable in standard legal databases. That's not an isolated incident either; between 2023 and 2025, courts worldwide issued hundreds of rulings addressing AI hallucinations in legal filings, with about 90 % of those (790 out of 863) occurring in 2025 alone [8]..

Discussion. To sum up, these findings point to a reasonably consistent conclusion: generative AI can produce feedback that looks like expert commentary while occasionally carrying very little relationship to what a student actually wrote. This is the paradox flagged in the introduction -fluency and reliability are not the same thing, and the sources reviewed here suggest that current systems have solved the first problem far more convincingly than the second. The pattern that emerges from the taxonomy is not random. AI feedback tends to be reasonably trustworthy at the level of individual sentences – grammar, spelling, word choice and considerably less trustworthy the moment it has to reason about meaning, coherence, or the writer's overall argument [4, p. 7-8; 5, p. 9-11; 6]. This is worth taking seriously pedagogically, because it means AI feedback is probably least reliable exactly where advanced EFL writers need the most support: not in fixing subject-verb agreement, but in shaping how ideas connect and whether an argument actually holds together. A tool that is strong on the easy layer and weak on the hard layer risks reinforcing a narrow, mechanical idea of what «good writing» means. The deeper issue, though, may not be the errors themselves but their presentation. Because hallucinated or redundant feedback is delivered in the same confident, fluent register as accurate feedback [3, p. 491], there is no obvious signal to an EFL learner that something is wrong. A native-level command of English is normally what lets a reader sense when a claim rings false; the students most likely to rely heavily on AI feedback are, almost by definition, the ones least equipped to catch it when that feedback misleads them. This raises a question this review can flag but not resolve: whether EFL students can reliably distinguish accurate AI feedback from confidently wrong AI feedback at all. If they generally cannot, then even a modest hallucination rate of 20–27 % [3, с. 496] is not a minor technical flaw - it is a recurring risk of quiet misinstruction. None of this means AI feedback has no place in EFL writing instruction. Prompt engineering, model choice, and low-temperature settings all measurably reduce error rates k [1; 2; 3], and for surface-level revision AI tools already perform close to a useful standard. But the evidence here argues against unsupervised use, particularly for higher-order writing skills, and points toward a hybrid model in which AI feedback is treated as a first pass to be checked, not a final verdict with the instructor, not the algorithm, remaining responsible for the feedback a student ultimately trusts.

Conclusion. This review examined how and why AI-generated feedback on EFL writing goes wrong. Across the eight sources, a clear taxonomy emerge: false corrections, missed errors, misleading explanations, redundant feedback, inconsistency, and hallucination with one consistent pattern: AI feedback is fairly reliable at the sentence level but far less so when it comes to meaning, coherence, and argument quality. Hallucination rates as high as a quarter of all feedback sentences, along with concrete cases of fabricated content in both educational and legal settings. The core pedagogical concern is that inaccurate feedback is delivered with the same confidence as accurate feedback, leaving EFL learners poorly equipped to tell the difference. Prompt engineering and careful model choice help but don't eliminate this risk. This points toward AI feedback being best used as a supplementary tool under teacher oversight, not a stand-alone replacement for instructor judgment. This review is limited by its relatively small corpus and the fast pace of LLM development, which may date findings tied to specific model versions. Future research should test EFL students' actual ability to detect inaccurate AI feedback and track how reliance on such tools affects writing development over time. Until then, cautious, supervised integration of AI feedback remains the most defensible approach for EFL classrooms.

References:

1. Pack A., Barrett A., Escalante J. Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability // Computers and Education: Artificial Intelligence. 2024. Vol. 6. Art. 100234. DOI: 10.1016/j.caeai.2024.100234.
2. Steiss J., Tate T., Graham S., Cruz J., Hebert M., Wang J., Moon Y., Tseng W., Warschauer M., Olson C.B. Comparing the quality of human and ChatGPT feedback of students’ writing // Learning and Instruction. 2024. Vol. 91. Art. 101894. DOI: 10.1016/j.learninstruc.2024.101894.
3. Jia Q., Cui J., Xi R., Liu C., Rashid P., Li R., Gehringer E. On assessing the faithfulness of LLM-generated feedback on student assignments // Proceedings of the 17th International Conference on Educational Data Mining. Atlanta, GA, 2024. P. 491⁠–⁠499. DOI: 10.5281/zenodo.12729868.
4. Alsaweed W., Aljebreen S. Investigating the accuracy of ChatGPT as a writing error correction tool // International Journal of Computer-Assisted Language Learning and Teaching. 2024. Vol. 14, No. 1. P. 1⁠–⁠18. DOI: 10.4018/IJCALLT.364847.
5. Lin S., Crosthwaite P. The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback // System. 2024. Vol. 127. Art. 103529. DOI: 10.1016/j.system.2024.103529.
6. Alsoweed R.A., AbuSa’aleek A.O. ChatGPT and instructor feedback in EFL paragraph writing: A comparative analysis of error patterns based on Byrd and Benson’s framework // International Journal of Technology in Education. 2026. P. 1105⁠–⁠1123. DOI: 10.46328/ijte.7676.
7. Waked A., Alamr H. Analyzing AI hallucinations to train critical thinking and increase self-efficacy in first-semester EFL university students // Proceedings of the Twelfth International Conference on Higher Education Advances (HEAd’26). 2026. DOI: 10.4995/HEAd26.2026.21805.
8. When AI Gets It Wrong: Addressing AI Hallucinations and Bias // MIT Sloan Teaching & Learning Technologies. 2025. URL: https://mitsloanedtech.mit.edu/ai/basics/addressing-ai-hallucinations-and-bias/ (accessed: 14.09.2026).