Examining Korean EFL College Students’ Experiences and Perceptions of Using ChatGPT as a Writing Revision Tool

1. ChatGPT: Its Affordances for Language Learning and Teaching

Since ChatGPT was first released in November 2022, it has received considerable attention from language educators and researchers. The primary focus of research papers to date has been on the potential of the platform as a language learning and teaching tool. As an example, based on their preliminary analysis, Kohnke et al. (2023) discussed the educational values of the tool. According to them, ChatGPT can effectively facilitate language learning by providing learners with opportunities to engage in authentic interactions through its key features such as word recognition, error correction, and explanation. Furthermore, it is capable of generating texts in a variety of genres and difficulty levels, which can be particularly useful for teachers when preparing teaching materials to align with their students’ varying proficiency levels and needs.

Park (2023) also investigated the positive effects that ChatGPT can have on English education. Specifically, ChatGPT utilizes data-driven algorithms and offers tailored learning experiences by considering individual learners’ interests, proficiency level, and learning goals and providing customized learning materials accordingly. In addition, it can offer instant feedback, assisting in correcting errors in grammar, vocabulary, and structure. Furthermore, the utilization of GPT-4, which has functions for transforming texts into images and sounds, has the potential to enhance students’ engagement in and motivation for language learning.

Shin et al. (2023) focused their analysis on the potential of chatbots as a tool for content-based English learning and teaching. They invited 27 educational specialists to participate in a survey to evaluate three different chatbots: Cleverbot, Kuki, and ChatGPT. They found that ChatGPT had superior language proficiency and provided various information thanks to a DL approach based on massive language models, indicating its potential for use in content-based English learning and teaching. Shin et al. anticipated that the platform can be utilized not only as a self-directed learning tool for language learners but also as a helpful teaching resource for language instructors.

Despite its potential advantages, ChatGPT has limitations. For example, Ahn (2023) examined ChatGPT’s efficacy as a language learning and evaluation tool by analyzing its performance on the English reading sections of the Korean College Scholastic Ability Test. Overall, ChatGPT demonstrated high accuracy rates in answering the questions and abilities to reason and provide explanations for the answers without any specialized input, but it also made some mistakes. Ahn asserted that human interventions and careful interpretations are required for the outputs that ChatGPT generates.

As shown above, the existing body of literature is constrained in that it mostly concentrates on offering recommendations for the use of ChatGPT in language learning and teaching or summarizing the anticipated advantages of ChatGPT based on a feature analysis. More investigation is needed into the experiences and perceptions of English language learners when it is actually implemented in English classrooms. This line of inquiry will contribute to optimizing the use of the tool and thus enhancing the learning experiences of English language learners.

2. Utilization of ChatGPT in the EFL Writing Process

Writing is one of the areas of English language proficiency where ChatGPT can be extremely helpful to learners. This is mostly attributable to its ability to provide prompt, individualized feedback (Hong, 2023) and opportunities for honing writing skills (Kim et al., 2023). Some empirical studies (e.g., Cao & Zhong, 2023; Choe, 2023a, 2023b; Shin, 2023) have examined the use of ChatGPT in the English writing process and assessed its effectiveness from the perspective of the students.

Shin (2023) investigated the possibility of using ChatGPT as a tool to provide guided writing instruction to 21 Korean pre-service English teachers. The main focus of this study was on the perspectives of the students regarding the use of the tool for writing during the planning and revision phases. Most participants found ChatGPT to be very helpful in their writing process; it helped them generate ideas and organize their thoughts, and it provided insightful feedback. Although helpful, some participants advised against relying too much on ChatGPT because they believed it might hinder learning. Shin used these findings to highlight how important it is for teachers to give students thorough instructions on how to use ChatGPT.

Other studies on ChatGPT’s application in English writing have focused on its role in the revision phase. Choe (2023a, 2023b) examined the perceptions of Korean pre-service English teachers regarding the incorporation of ChatGPT into their writing process. Participants were asked to reflect upon how they used ChatGPT to edit their first drafts of the assigned articles. A thematic analysis showed that participants reported greater excitement, more efficient use of time, and increased confidence as affectively positive aspects, and they perceived the feedback on content, organization, and language use as cognitively positive. Negative aspects included worries about possibly hallucinated content and ethical dilemmas regarding plagiarism and over-reliance on the tool. Based on these results, Choe concluded that ChatGPT could be useful in EFL instruction, given that the curriculum includes instruction on its appropriate and ethical use and copyright issues.

Cao and Zhong (2023) investigated ChatGPT’s efficacy in the rewriting process by contrasting it with traditional feedback methods like self- and teacher-feedback. Advanced Chinese ESL/EFL students enrolled in a master’s program in translation and interpretation participated in their study. The participants were required to translate texts from Chinese into English, then edit their translations based on feedback from teachers, ChatGPT, and themselves. ChatGPT performed better than the other two forms of feedback in terms of lexical proficiency and coherence according to an analysis of the translated texts’ lexicon, syntax, and cohesion. However, self- and teacher-feedback were better at assessing syntax, including the appropriate use of passive voice. Cao and Zhong suggested incorporating ChatGPT with conventional feedback methods, highlighting the necessity of a blended feedback strategy.

These studies offer valuable insights into the application of ChatGPT in English writing instruction, outlining both its benefits and drawbacks. Nevertheless, studies on its application in English writing instruction, particularly on the usefulness of ChatGPT-based feedback, are still in their infancy. The goal of this study is to close this research gap and contribute to the body of current knowledge by exploring how EFL students from various academic disciplines use ChatGPT in the context of Korean higher education. It is also important to note that the majority of the studies previously reviewed were conducted with English education and translation majors. Compared to students whose major is English, it is expected that EFL students from a variety of majors enrolled in a general education course will respond differently during the writing revision process. The present study strives to offer insights into the effective utilization of ChatGPT for teaching English writing skills by examining the viewpoints of learners with varying learning experiences and needs.

1. Research Context

This study was conducted at a metropolitan four-year university in South Korea. Students at this university were required to take a mandatory general English course in the spring semester, with a focus on academic writing. One of the authors of this study was an instructor who taught three basic-level academic writing classes. One of the primary activities in these classes was to spend approximately 30 minutes independently writing a paragraph in the classroom after completing each textbook unit. They subsequently submitted their writings to the school Learning Management System (LMS) for evaluation. Oral feedback from the instructor was provided for students online or in person upon request; participation was voluntary.

Starting from the second half of the semester, a new practice was introduced as homework. Students were required to incorporate generative AI Chatbots (e.g., ChatGPT, Bard, or Bing) to proofread and revise their original paragraphs written during class time. Specific prompts were not provided; instead, examples were shared in class, such as “proofread this paragraph” and “revise this paragraph.” Students were also instructed to identify three main difficulties they encountered when writing a paragraph independently, which helped determine whether they could receive assistance from AIs. Five main areas of difficulties were identified from these assignments: grammar, vocabulary/expressions, clarity, coherence, and organization. These areas were addressed in the survey questionnaire to assess the extent to which ChatGPT assisted with writing in these areas. Throughout the second half of the semester, each student submitted three original paragraphs and their revisions to the instructor as homework.

A total of 80 students initially participated in the study, but data from students who used other AI chatbots such as Bard and Bing Chat were filtered out, resulting in a total of 71 student responses being used for this research. Most of the 71 students were in their first year, with the exception of one third-year student. For majors, 28 were computer science majors specializing in computer engineering, software, or information statistics; 26 were law majors, and 17 were art, music, or sports majors. According to the Common European Framework of Reference for Languages (CEFR) scale, the majority of students self-assessed as English beginners, except for one student: 26 students at the A1 level, 35 at the A2 level, nine at the B1 level, and one at the C1 level. In addition, 39 of the students reported receiving oral feedback from the current instructor at least once during the semester, while the remaining 32 did not. All students who reported using ChatGPT used the GPT-3.5 version.

2. Data Collection and Analysis

The primary data for the research were collected from a survey questionnaire designed to investigate students’ experiences and perceptions of using ChatGPT to revise their paragraphs and their opinions on the teacher’s roles in the process. The questionnaire was developed using Google Forms and comprised two sections. The first section gathered information about the students’ background, such as year in school, major, self-assessed English proficiency level, receiving of oral feedback from the instructor, and the type of AI chatbot used for revision. The second section comprised 15 items that investigated students’ perceptions of the use of ChatGPT, including 11 questions answered using a 5-point Likert-type scale, one multiple choice question, and three open-ended questions (see Appendix). The survey link was shared with students both online and offline at the end of the semester and was open from June 12 to 18, 2023. Participation was anonymous and voluntary; no identifying information was requested, and students completed the survey without the instructor’s presence.

Students’ responses to the survey were automatically saved and transferred to an Excel file. Descriptive statistics were used to analyze the Likert-scale and multiple-choice questions, and thematic analysis was used to analyze the open-ended questions. For qualitative analysis, irrelevant information and responses to the open-ended questions, such as periods, commas, and “I don’t know,” were excluded. Subsequently, each researcher in this study individually read and examined the students’ responses, assigning codes to organize information based on its meaning. They then compared their coding results and collaboratively identified patterns in the codes while also noting recurring ideas. In the last phase, they grouped the codes into broader categories or themes and offered their interpretations.

1. Perceived Advantages of Using ChatGPT for Feedback

The first three survey questions1 were developed to understand students’ overall experience with using ChatGPT for paragraph revision. As indicated in Table 1, the responses reflected a generally positive experience with the tool. The first question aimed to assess students’ satisfaction level with the activity of receiving feedback through ChatGPT and revising their writing. Their responses suggest high satisfaction with it (M = 4.13, SD = 0.79). In the subsequent survey questions (2 and 3), students indicated that they found ChatGPT helpful in the process of editing their paragraphs (M = 4.04, SD = 0.92) and expressed an intention to continue using it for paragraph revision in the future (M = 4.21, SD = 0.91). When asked about the quality of its feedback in Questions 4 and 5, students expressed positive perceptions that ChatGPT’s feedback was accurate (M = 4.03, SD = 0.79) and reliable (M = 3.90, SD = 0.86).

TABLE 1

Students’ Overall Evaluation of the Use of ChatGPT During the Revision Process

In order to gain a deeper understanding of their positive experiences and attitudes, an open-ended question (Question 11) was posed about the advantages of using ChatGPT. Table 2 presents the main advantages that students mentioned most: convenience, error correction, availability of various revised versions, and novelty. Many students highlighted the convenience of using ChatGPT as one of its most beneficial aspects. They appreciated the freedom from restrictions in terms of time and place, as well as the ability to receive prompt feedback upon request. Due to these conveniences, students considered the revision process with ChatGPT to be “easy, fast and hassle-free.”

TABLE 2

Advantageous Aspects of Using ChatGPT During the Revision Process

ChatGPT’s next merit was its ability to accurately detect and correct students’ writing errors. Many students viewed error corrections made by ChatGPT as “perfect,” “nothing wrong,” or “model language,” which was also in line with their responses to survey questions 4 and 5 about the quality of its feedback. Regarding the areas of errors where students received help, vocabulary, paragraph flow, grammar, and content enrichment were frequently indicated. In the area of vocabulary, for instance, they felt that ChatGPT helped them learn “new” vocabulary and find “more appropriate words” for their texts. They found ChatGPT helpful in improving the flow of their paragraphs as well. By checking the revised versions of their paragraphs, they were able to see how their ideas could be sequenced more logically and how the inclusion of “linking” or transition words could maintain their writing coherence, ultimately making their texts more readable and improving their overall quality. Grammar was another area where ChatGPT was reported to be helpful due to its ability to easily catch and fix grammatical errors. Lastly, ChatGPT also aided students in finding more supporting details for their paragraphs so that they could back up their main idea within the text.

Likert-scale questions (Questions 6 through 10) were also used to examine students’ perceptions of the quality of ChatGPT’s feedback in five key areas: grammar, clarity, coherence, vocabulary (including expressions), and organization. These areas were selected based on their frequency in students’ reported difficulties during the paragraph writing activity, where they were instructed to identify three areas of difficulty. Table 3 illustrates that students overall agreed that ChatGPT played a critical role in rectifying English errors. The highest agreement was found in the category of English vocabulary and expressions (M = 4.23, SD = 0.87), followed by grammar (M = 4.17, SD = 0.99), organization (M = 4.15, SD = 0.87), clarity (M = 4.10, SD = 0.96), and flow (M = 4.03, SD = 0.94).

TABLE 3

Students’ Perceived Effects of ChatGPT on the Quality of Writing

2. Challenges Faced While Using ChatGPT for Feedback

While students generally had positive experiences with ChatGPT, it was not without its challenges. Students addressed various issues in response to survey question 12, which are summarized in Table 4. One of the main issues was related to how feedback was provided. Firstly, students expressed disappointment that ChatGPT did not automatically offer an explanation about its error corrections unless they explicitly asked for it. As one student commented, “the reason for the correction was unknown,” making it difficult to discern where and why their original sentences or paragraphs were considered incorrect. Additionally, students stated that the vocabulary proposed by ChatGPT was sometimes too advanced or difficult for students to employ. Given that most students considered themselves beginners in English, they frequently noticed a discrepancy between the words suggested by ChatGPT and the ones they had learned or were familiar with in terms of their level of difficulty. Due to its insensitivity to students’ English proficiency, one student mentioned that he had to continually search for other words that he found more familiar and comfortable to use, instead of blindly adopting the vocabulary suggested by ChatGPT. Lastly, students wished that they could have their original text evaluated before ChatGPT created its revision so that they would know how well or poorly their original text was written. Because corrections made by ChatGPT could involve essential changes in areas like grammar, word choice, and style that distinguish one’s writing from another’s, and the modified text often lacked a specific explanation of why the initial text had to be modified, students often found it hard to determine the correctness of their initial attempt. This could result in a missed opportunity for learning English from their own mistakes.

TABLE 4

Issues Students Faced Using ChatGPT During the Revision Process

Another significant challenge that students faced was a communication issue with ChatGPT, which often resulted in misleading feedback. Some students mentioned occasional difficulties communicating with ChatGPT. There were instances where the questions or requests they posed were not accurately understood, leading to answers or responses that did not align with their intended meaning. To minimize miscommunications with ChatGPT, some students chose to use English instead of Korean when making requests, considering ChatGPT’s origin. For instance, one student noted, “It was more accurate to use ChatGPT in its original English form rather than in Korean, so it was a hassle to translate before asking.” This additional step of translation before asking for a more accurate answer appeared to add more stress for this student.

The next issue was students’ doubt about the effectiveness of ChatGPT for their English learning. One instance causing this doubt occurred when students were unsure of their specific areas of weakness and what they needed help with when reviewing the revised text provided by ChatGPT. Lacking the knowledge to critically assess the feedback and therefore feeling compelled to “simply accept” it, they questioned whether ChatGPT’s feedback could be truly effective. Doubts about its effectiveness also arose when students compared it to face-to-face feedback from their instructor. Some students indicated that direct interaction with their instructor led to better comprehension of feedback than when reading it through written text. They also found themselves being less attentive and engaged when asking questions through text chat compared to face-to-face interactions. This raised concerns about whether engagement with ChatGPT could contribute to their actual learning.

The last challenge students encountered was the perception of losing authorship over their writing after undergoing a revision process with ChatGPT. Some students noticed that their work was significantly altered or rewritten, and they expressed mixed feelings about it. While they acknowledged that the modified sentences displayed a more polished and professional quality, they were also disappointed to discover that their initial sentences had not been preserved in any form, whether by retaining more of their original structure, wording, or manner of expression. As one student put it, they sometimes received modified texts by ChatGPT in which their original sentences were no longer recognizable, which led them to feel detached or “foreign” from the revised version.

3. Perceived Teachers’ Role During ChatGPT-Driven Feedback Activity

The remaining part of the survey questionnaire focused on examining students’ thoughts about the instructor’s role in the environment where generative AI chatbots like ChatGPT are used in the process of revising paragraphs. While students had positive experience with ChatGPT and perceived that its feedback was accurate, reliable, and helpful, they responded negatively to survey question 13, which inquired whether they would agree that generative AIs could fully replace face-to-face feedback activities by actual human instructors (M = 2.35, SD = 1.10). In answer to survey question 14, which asked participants to choose between ChatGPT and human instructors for feedback on paragraph writing, a significant portion of students (49.3%) selected a combination of both, 35.2% of the students opted for instructors only, and 15.5% students chose ChatGPT exclusively. These results suggest that students still find value in human feedback, even though they acknowledge the potential usefulness of feedback activities with generative AIs.

When asked in survey question 15 about the roles that teachers can play when AIs like ChatGPT become widely used in revision, students identified three main roles, as depicted in Table 5. The first anticipated role of teachers was to provide feedback on AI-generated responses, clarifying, verifying, and even enhancing their quality. Students expected teachers to address certain aspects of writing that might be challenging for ChatGPT to provide feedback on. For instance, while it may not be necessarily true, some students believed that human teachers are better at explaining subtle differences in the meaning and usage of English words, influenced by cultural factors. They also believed that human English teachers can proofread a text with a deeper understanding of the context in which it is written than ChatGPT can. Moreover, they felt human teachers are better equipped to closely observe an individual student’s unique writing style and needs, and subsequently reflect this understanding when editing the writing. All in all, they suggested that teachers can play an essential role in providing nuanced, culturally sensitive, and individualized feedback, which might be perceived as challenging for ChatGPT.

TABLE 5

Anticipated Teachers’ Roles During ChatGPT-Driven Feedback Activity

Furthermore, students expected teachers to explain AI-generated feedback in an easily accessible, clear, and comprehensible manner. As discussed in the previous section, students frequently struggled to fully understand ChatGPT’s feedback due to its inadequate error descriptions, the use of overly advanced or unfamiliar vocabulary, and responses that did not align with their intentions. These issues raised concerns about its effectiveness in promoting students’ actual learning. Consequently, they anticipated that teachers would provide additional explanations for the feedback generated by AIs.

Besides receiving additional explanations of the AI’s feedback, some students expressed a desire to verify its accuracy and reliability through their teachers. While they acknowledged ChatGPT’s proficiency in many areas, they had concerns about its potential to offer undesired or incorrect feedback. Rather than blindly relying on it, students wanted teachers to actively monitor ChatGPT-generated feedback, identify its errors, and address any questions or concerns that they had.

Secondly, students also highlighted the importance of teachers’ guidance on how to use AIs for text revisions, enabling them to extract specific and desired feedback. They recognized the significance of posing the right questions to elicit desired, high-quality feedback and suggested that teachers provide instruction on formulating appropriate prompts, questions, and requests. Students also pointed out the need for teachers to offer guidance on the responsible and ethical use of AIs to ensure their proper utilization. Given the potential risks of unintentional plagiarism or copyright infringement due to insufficient knowledge or skills in writing and using AIs when incorporating AI feedback into their writing, one student proposed that teachers consistently “guide and supervise” students to prevent any misuse or abuse of AIs.

The last role that students proposed for teachers was to connect with them on an emotional level, offering encouragement and motivation for continued English learning. Students perceived that ChatGPT provided a “perfect” version of their text without detailed explanations for revisions or adequate support for their original writing. This could make them feel anxious about their own English skills and develop a false idea that their current English would not be adequate to show to others. One student pointed out that this could lead to feelings of “frustration” with their own English skills. Therefore, teachers were expected to provide ongoing support for students, being attuned to their emotional needs.