AI in Academic Writing: Assessing the Effectiveness, Grading Consistency, and Student Perspectives of ChatGPT and You.com for EFL Students

Abstract

This study evaluates the impact of AI-generated feedback and targeted training sessions on the writing skills of 16 B1-B2 level students majoring in English Language and Literature (ELL) or English Language Teaching (ELT) at Niğde Ömer Halisdemir University, School of Foreign Languages, during the 2023-2024 spring term. Using a mixed-methods approach, the research assessed students' writing across four paragraph types—opinion, cause/effect, reason/result, and advantage/disadvantage—employing a rubric that rates Grammar and Mechanics, Sentence Structure, Cohesion and Coherence, Argumentation/Content Development, Vocabulary Usage, and Task Achievement out of 30. Both ChatGPT and You.com were used to grade the initial and revised drafts of students' paragraphs. Following the first draft feedback, students participated in targeted training sessions aimed at addressing identified weaknesses. Quantitative analysis of draft scores, supported by paired t-tests, independent t-tests, and ANOVA, revealed significant improvements in writing performance from the first to the final drafts, with You.com generally providing higher final draft scores compared to ChatGPT. Correlation analysis affirmed the reliability of both AI models. Qualitative insights, obtained from a focus group interview, highlighted the perceived effectiveness of the AI feedback and training sessions. The study underscores the potential of AI-generated feedback as a valuable educational tool, offering timely, individualized support that enhances writing skills, with supplementary training sessions further boosting the efficacy of AI feedback. This research contributes to understanding the role of AI in enhancing writing instruction and feedback within English as a Foreign Language (EFL) education, aiming to inform pedagogical practices and integrate AI tools effectively in writing classrooms.