| Abstract: |
Revision provides a window into writers' metacognitive awareness, goal-directed decision making, and linguistic development (Hayes, 2012). Understanding how students revise is therefore critical for both writing research and instruction. However, identifying and evaluating revisions remains challenging: manual annotation is labor-intensive and inconsistent, while existing automated approaches rely largely on surface metrics (e.g., deletion/insertion counts, edit distance) (Tian & Cushing, 2025). These limitations prevent researchers and educators from capturing the semantic and functional nature of revisions, ultimately constraining the usefulness of revision analytics for scalable, instructionally meaningful feedback. Advancing automated methods for identifying and evaluating revisions is essential for generating evidence-based feedback and supporting personalized writing development at scale. This project introduces an automatic workflow that leverages large language models (LLMs) to identify, categorize, and evaluate revisions between students' draft and revised essays (See Appendix A for an overview). The workflow includes four stages: (1) detecting text changes between draft:revision pairs using rule-based approaches, (2) prompting an LLM to explain each change in natural language, (3) using an LLM to categorize revision types (e.g., typographic, grammatical, or content-level), (4) querying an LLM to evaluate each revision as Good, Neutral, or Bad and provide explanation. Two trained human annotators independently code revisions using a custom rubric (Appendix B). The annotations serve as ground truth for evaluating the accuracy and reliability of the automated workflow. Initial analyses indicate that, with carefully designed prompts, state-of-the-art LLMs can reliably identify revision types and evaluate their quality using the textual context. We anticipate that our workflow will achieve performance comparable to human annotation and demonstrate strong potential for integration into writing research and instructional tools. By harnessing LLMs for automated text revision analysis and evaluation, this work advances the development of evidence-based, scalable systems for writing analytics and personalized feedback. The workflow embodies the principles of learning engineering by uniting theory, data, and iterative design to model and support complex learning processes. Anticipated applications include formative feedback systems that highlight meaningful revisions, instructor dashboards that visualize students' writing growth, and adaptive learning environments that guide students toward higher-quality revisions. [This paper was published in: "Proceedings of the Learning Engineering Research Network Convening (LERN 2026)," 2026, pp. 159-162.] |