grading for an essay

• Basics for GSIs
• Advancing Your Skills

Grading Essays

Grade for Learning Objectives Response to Writing Errors Commenting on Student Papers Plagiarism and Grading

Information about grading student writing also appears in the Grading Student Work section of the Teaching Guide. Here are some general guidelines to keep in mind when grading student writing.

Grade for Learning Objectives

Know what the objective of the assignment is and grade according to a standard (a rubric) that assesses precisely that. If the purpose of the assignment is to analyze a process, focus on the analysis in the essay. If the paper is unreadable, however, consult with the professor and other GSIs about how to proceed. It may be wise to have a shared policy about the level of readiness or comprehensibility expected and what is unacceptable.

Response to Writing Errors

The research is clear: do not even attempt to mark every error in students’ papers. There are several reasons for this. Teachers do not agree about what constitutes an error (so there is an unavoidable element of subjectivity); students do not learn when confronted by too many markings; and exhaustive marking takes way too much of the instructor’s time. Resist the urge to edit or proofread your students’ papers for superficial errors. At most, mark errors on one page or errors of only two or three types. One approach to avoid the temptation of marking every error is to read or skim the whole essay quickly once without marking anything on the page – or at least, with very minimal marks. Some instructors find this a useful method in order to get a general sense of the essay’s organization and argument, thus enabling them to better identify the major areas of concern. Your second pass can then focus more in-depth on a few select areas that require improvement.

Commenting on Student Papers

The scholarly literature in this area distinguishes formative from summative comments. Summative comments are the more traditional approach. They render judgment about an essay after it has been completed. They explain the instructor’s judgment of a student’s performance. If the instructor’s comments contain several critical statements, the student often becomes protective of his or her ego by filtering them out; learning from mistakes becomes more difficult. If the assignment is over with, the student may see no reason to revisit it to learn from the comments.

Formative comments, on the other hand, give the student feedback in an ongoing process of learning and skill building. Through formative comments, particularly in the draft stage of a writing assignment, instructors guide students on a strategic selection of the most important aspects of the essay. These include both what to keep because it is (at least relatively) well done and what requires revision. Formative comments let the student know clearly how to revise and why.

For the purposes of this guide, we have distinguished commenting on student writing (which is treated here) from grading student writing (which is treated in the Teaching Guide section on grading ). While it is true that instructors’ comments on student writing should give reasons for the grade assigned to it, we want to emphasize here that the comments on a student’s paper can function as instruction , not simply as justification. Here are ten tips.

• Use your comments on a student’s paper to highlight things the paper accomplishes well and a few major things that would most improve the paper.
• Always observe at least one or two strengths in the student’s paper, even if they seem to you to be low-level accomplishments — but avoid condescension. Writing is a complex activity, and students really do need to know they’re doing something right.
• Don’t make exhaustive comments. They take up too much of your time and leave the student with no sense of priority among them.
• Don’t proofread. If the paper is painfully replete with errors and you want to emphasize writing mechanics, count the first ten errors on the page, draw a line at that point, and ask the student to identify them and to show their corrections to you in office hours. Students do not learn much from instructors’ proofreading marks. Direct students to a writing reference guide such as the Random House Handbook.
• Notice patterns or repeated errors (in content or form). Choose the three or four most disabling ones and direct your comments toward helping the students understand what they need to learn to do differently to correct this kind of error.
• Use marginal notes to locate and comment on specific passages in the paper (for example “Interesting idea — develop it more” or “I lost the thread of the argument in this section” or “Very useful summary here before you transition to the next point”). Use final or end comments to discuss more global issues (e.g., “Work on paragraph structure” or “The argument from analogy is ineffective. A better way to make the point would be…”)
• Use questions to help the student unpack areas  that are unclear or require more explanation and analysis. E.g.: “Can you explain more about what you mean by “x”?”; “What in the text shows this statement?”; “Is “y” consistent with what you’ve argued about “z”?” This approach can help the student recognize your comments less as a form of judgment than a form of dialogue with their work. As well, it can help you avoid “telling” the student how they should revise certain areas that remain undeveloped. Often, students just need a little more encouragement to focus on an area they haven’t considered in-depth or that they might have envisioned clearly in their head but did not translate to the page.
• Maintain a catalogue of positive end comments: “Good beginning for a 1B course.” “Very perceptive reading.” “Good engagement with the material.” “Gets at the most relevant material/issues/passages.” Anything that connects specific aspects of the student’s product with the grading rubric is useful. (For more on grading rubrics , see the Grading section of the Teaching Guide.)
• Diplomatic but firm suggestions for improvement: Here you must be specific and concrete. Global negative statements tend to enter students’ self-image (“I’m a bad writer”). This creates an attitudinal barrier to learning and makes your job harder and less satisfying. Instead, try “The most strategic improvement you could make is…” Again, don’t try to comment on everything. Select only the most essential areas for improvement, and watch the student’s progress on the next draft or paper.
• Typical in-text marks: Provide your students with a legend of your reading marks. Does a straight underline indicate “good stuff”? Does a wavy underline mean something different? Do you use abbreviations in the margins? You can find examples of standard editing marks in many writing guides, such as the Random House Handbook.
• The tone of your comments on student writing is important to students. Avoid sarcasm and jokes — students who take offense are less disposed to learn. Address the student by name before your end-comments, and sign your name after your remarks. Be professional, and bear in mind the sorts of comments that help you with your work.

Plagiarism and Grading

Students can be genuinely uninformed or misinformed about what constitutes plagiarism. In some instances students will knowingly resort to cutting and pasting from unacknowledged sources; a few may even pay for a paper written by someone else; more recently, students may attempt to pass off AI-generated essays as their own work. Your section syllabus should include a clear policy notice about plagiarism and AI so that students cannot miss it, and instructors should work with students to be sure they understand how to incorporate outside sources appropriately.

Plagiarism can be largely prevented by stipulating that larger writing assignments be completed in steps that the students must turn in for instructor review, or that students visit the instructor periodically for a brief but substantive chat about how their projects are developing, or that students turn in their research log and notes at intermediate points in the research process.

All of these strategies also deter students from using AI to substitute for their own critical thinking and writing. In addition, you may want to craft prompts that are specific to the course materials rather than overly-general ones; and you may also require students to provide detailed analysis about specific texts or cases. AI tools like ChatGPT tend to struggle significantly in both of these areas.

For further guidance on preventing academic misconduct, please see Academic Misconduct — Preventing Plagiarism .

You can also find more information and advice about AI technology like ChatGPT at the Berkeley Center for Teaching & Learning.

UC Berkeley has a campus license to use Turnitin to check the originality of students’ papers and to generate feedback to students about their integration of written sources into their papers. The tool is available in bCourses as an add-on to the Grading tool, and in the Assignments tool SpeedGrader. Even with the results of the originality check, instructors are obligated to exercise judgment in determining the degree to which a given use of source material was fair or unfair.

If a GSI does find a very likely instance of plagiarism, the faculty member in charge of the course must be notified and provided with the evidence. The faculty member is responsible for any sanctions against the student. Some faculty members give an automatic failing grade for the assignment or for the course, according to their own course policy. Instances of plagiarism should be reported to the Center for Student Conduct; please see If You Encounter Academic Misconduct .

• PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
• EDIT Edit this Article
• EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
• Browse Articles
• Learn Something New
• Quizzes Hot
• This Or That Game
• Train Your Brain
• Explore More
• Support wikiHow
• About wikiHow
• Log in / Sign up
• Education and Communications
• Teacher Resources

How to Grade a Paper

Last Updated: February 14, 2024 Fact Checked

This article was co-authored by Noah Taxis . Noah Taxis is an English Teacher based in San Francisco, California. He has taught as a credentialed teacher for over four years: first at Mountain View High School as a 9th- and 11th-grade English Teacher, then at UISA (Ukiah Independent Study Academy) as a Middle School Independent Study Teacher. He is now a high school English teacher at St. Ignatius College Preparatory School in San Francisco. He received an MA in Secondary Education and Teaching from Stanford University’s Graduate School of Education. He also received an MA in Comparative and World Literature from the University of Illinois Urbana-Champaign and a BA in International Literary & Visual Studies and English from Tufts University. There are 8 references cited in this article, which can be found at the bottom of the page. This article has been fact-checked, ensuring the accuracy of any cited facts and confirming the authority of its sources. This article has been viewed 85,607 times.

Anyone can mark answers right and wrong, but a great teacher can mark up a paper in such a way as to encourage a student who needs it and let good students know they can do better. As the great poet and teacher Taylor Mali put it: "I can make a C+ feel like a Congressional Medal of Honor and I can make an A- feel like a slap in the face."

Going Through an Essay

• These designations obviously depend upon many things, like the assignment, the grade-level of your students, and their individual concerns. If you're in the middle of a unit on comma usage, it's perfectly fine to call that a "higher" concern. But in general, a basic writing assignment should prioritize the higher concerns listed above.

• Does the student address the prompt and fulfill the assignment effectively?
• Does the student think creatively?
• Does the student clearly state their argument, or thesis?
• Is the thesis developed over the course of the assignment?
• Does the writer provide evidence?
• Does the paper show evidence of organization and revision, or does it seem like a first draft?

• Marking essays in pencil can suggest that the issues are easily fixable, keeping the student looking forward, rather than dwelling on their success or failure. Pencil, blue, or black pen is perfectly appropriate.

• Be as specific as possible when asking questions. "What?" is not a particularly helpful question to scrawl in the margin, compared to "What do you mean by 'some societies'?"

• ¶ = to start a new paragraph
• three underscores under a letter = to lowercase or uppercase the letter
• "sp" = word is spelled incorrectly
• word crossed out with a small "pigtail" above = word needs to be deleted
• Some teachers use the first page as a rule of thumb for marking later concerns. If there are sentence-level issues, mark them on the first page and then stop marking them throughout the essay, especially if the assignment needs more revision.

Writing Effective Comments

• Use marginal comments to point out specific points or areas in the essay the student could improve.
• Use a paragraph note at the end to summarize your comments and direct them toward improvement.
• Comments should not justify a letter grade. Never start a note, "You got a C because...". It's not your job to defend the grade given. Instead, use the comments to look toward revision and the next assignment, rather than staring backward at the successes or failures of the given assignment.

• If you struggle to find anything, you can always praise their topic selection: "This is an important topic! Good choice!"

• When you give your first read-through, Try to determine what these three points might be to make it easier when you're going through the paper and writing comments.

• "In your next assignment, make sure to organize your paragraphs according to the argument you're making" is a better comment than "Your paragraphs are disorganized."

Assigning Letter Grades

• Thesis and argument: _/40
• Organization and paragraphs: _/30
• Introduction and conclusion: _/10
• Grammar, usage, and spelling: _/10
• Sources and Citations: _/10

• A (100-90): Work completes all of the requirements of the assignment in an original and creative manner. Work at this level goes beyond the basic guidelines of the assignment, showing the student took extra initiative in originally and creatively forming content, organization, and style.
• B (89-80): Work completes all of the requirements of the assignment. Work at this level is successful in terms of content, but might need some improvement in organization and style, perhaps requiring a little revision. A B reveals less of the author’s original thought and creativity than A-level work.
• C (79-70): Work completes most of the requirements of the assignment. Though the content, organization, and style are logical and coherent, they may require some revision and may not reflect a high level of originality and creativity on the part of the author.
• D (69-60): Work either does not complete the requirements of the assignment, or meets them quite inadequately. Work at this level requires a good deal of revision, and is largely unsuccessful in content, organization, and style.
• F (Below 60): Work does not complete the requirements of the assignment. In general, students who put forth genuine effort will not receive an F. If you receive an F on any assignment (particularly if you feel you have given adequate effort), you should speak with me personally.

• Some teachers like to hand out papers at the end of the day because they fear discouraging or distracting students during class time. Consider giving the students time to go through the papers in class and be available to talk about their grades afterwards. This will ensure that they read and understand your comments.

Expert Q&A

• Avoid distractions. It can seem like a good idea to grade papers while you watch Jeopardy, but it'll end up taking longer. Set a manageable goal, like grading ten papers tonight, and quit when you've finished and have a drink. Thanks Helpful 0 Not Helpful 0
• Do not keep favorites. Grade everyone equally. Thanks Helpful 0 Not Helpful 0
• Look for more than just grammar. Look for concepts, plots, climax and most importantly...make sure it has a beginning (introduction that catches your attention), a middle (three reasons should have three supportive details to each) and an end(recapture what the paper was about, make a good ending to let the audience remember your story). Thanks Helpful 0 Not Helpful 0

• Always use a rubric to keep yourself safe from grade appeals. You don't want to have to defend subjective grades. Thanks Helpful 12 Not Helpful 4

Things You'll Need

• Something to write with
• A stack of papers
• A stiff beverage

You Might Also Like

Expert Interview

Thanks for reading our article! If you’d like to learn more about grading papers, check out our in-depth interview with Noah Taxis .

• ↑ https://public.wsu.edu/~campbelld/grading.html
• ↑ http://depts.washington.edu/pswrite/grading.html
• ↑ https://phys.org/news/2013-01-red-pen-instructors-negative-response.html
• ↑ https://gsi.berkeley.edu/gsi-guide-contents/student-writing-intro/grading/
• ↑ https://sites.google.com/a/georgetown.edu/prof-william-blattner/resources-for-students/abbreviations-on-returned-papers
• ↑ https://writing-speech.dartmouth.edu/teaching/first-year-writing-pedagogies-methods-design/diagnosing-and-responding-student-writing
• ↑ http://home.snu.edu/~hculbert/criteria.pdf
• ↑ https://teaching.uwo.ca/teaching/assessing/grading-rubrics.html

About This Article

To grade a paper, start by reading it without marking it up to see if it has a clear thesis supported by solid evidence. Then, go back through and write comments, criticism, and questions in the margins. Make sure to give specific feedback, such as “What do you mean by ‘some societies’?” instead of something like “What?” Try to limit yourself to 1 comment per paragraph so you don’t overwhelm the student. You can also write a note at the end, but start with praise before focusing on issues the student should address. For information on how to assign grades to your students’ papers, keep reading! Did this summary help you? Yes No

• Send fan mail to authors

Reader Success Stories

Tracy Nornhold

Jan 17, 2017

Did this article help you?

Apr 1, 2018

Featured Articles

Trending Articles

Watch Articles

• Terms of Use
• Privacy Policy
• Do Not Sell or Share My Info
• Not Selling Info

wikiHow Tech Help Pro:

Develop the tech skills you need for work and life

• University of Pennsylvania
• School of Arts and Sciences
• Penn Calendar

Search form

Evaluation Criteria for Formal Essays

Katherine milligan.

Please note that these four categories are interdependent. For example, if your evidence is weak, this will almost certainly affect the quality of your argument and organization. Likewise, if you have difficulty with syntax, it is to be expected that your transitions will suffer. In revision, therefore, take a holistic approach to improving your essay, rather than focussing exclusively on one aspect.

An excellent paper:

Argument: The paper knows what it wants to say and why it wants to say it. It goes beyond pointing out comparisons to using them to change the reader?s vision. Organization: Every paragraph supports the main argument in a coherent way, and clear transitions point out why each new paragraph follows the previous one. Evidence: Concrete examples from texts support general points about how those texts work. The paper provides the source and significance of each piece of evidence. Mechanics: The paper uses correct spelling and punctuation. In short, it generally exhibits a good command of academic prose.

A mediocre paper:

Argument: The paper replaces an argument with a topic, giving a series of related observations without suggesting a logic for their presentation or a reason for presenting them. Organization: The observations of the paper are listed rather than organized. Often, this is a symptom of a problem in argument, as the framing of the paper has not provided a path for evidence to follow. Evidence: The paper offers very little concrete evidence, instead relying on plot summary or generalities to talk about a text. If concrete evidence is present, its origin or significance is not clear. Mechanics: The paper contains frequent errors in syntax, agreement, pronoun reference, and/or punctuation.

An appallingly bad paper:

Argument: The paper lacks even a consistent topic, providing a series of largely unrelated observations. Organization: The observations are listed rather than organized, and some of them do not appear to belong in the paper at all. Both paper and paragraphs lack coherence. Evidence: The paper offers no concrete evidence from the texts or misuses a little evidence. Mechanics: The paper contains constant and glaring errors in syntax, agreement, reference, spelling, and/or punctuation.

Commenting on and Grading Student Writing

To learn more about how to maximize the feedback you give your students without putting an undue burden on your time, click on items in the list below.

• Focusing your commenting energies

Handling grammar

• Using a grading sheet
• Citation Information

Focus your Commenting Energy

No matter how much you want to improve student writing, remember that students can only take in so much information about a paper at one time. Particularly because writing is such an egocentric activity, writers tend to feel overloaded quickly by excessively detailed feedback about their writing.

Moreover, because most writing can be considered work in progress (because students will continue to think about the content and presentation of their papers even if they don't actively revise), commenting exhaustively on every feature of a draft is counter-productive. Too many comments can make student writers feel as if the teacher is taking control of the paper and cutting off productive avenues for revision.

Focusing your energy when commenting achieves two main goals:

• It leaves students in control of their writing so that they can consider revising--or at least learning from the experience of having written the paper.
• It gives teachers a sense of tackling the most important elements of a paper rather than getting bogged down in detail that might just get ignored by the student.

Typically, we recommend that teachers comment discursively on the one or two most important features of a paper, determined either by your criteria for the assignment or by the seriousness of the effect on a reader of a given paper.

If you assign write-to-learn tasks, you won't want to mark any grammatical flaws because the writing is designed to be impromptu and informal. If you assign more polished pieces, especially those that adhere to disciplinary conventions, then we suggest putting the burden of proofreading squarely where it belongs--on the writer.

You don't need to be an expert in grammar to assign and respond effectively to writing assignments. Click on the list below to read some points to consider as you design your assignments and grading criteria:

Don't Edit Writing to Learn

Editing write-to-learn (WTL) responses is counterproductive. This kind of writing must be informal for students to reap the benefits of thinking through ideas and questioning what they understand and what confuses them. Moreover, most WTL activities are impromptu. By asking students to summarize a key point in the three minutes at the end of class, you get students to focus on ideas. They don't need to edit for spelling and sentence punctuation, and if you mark those errors on their WTL writing, students shift their focus from ideas to form. In other words, marking errors on WTL pieces distracts students from the main goal--learning.

Make Students Responsible for Polishing Their Drafts

Formal drafts do need to be edited, but not necessarily by the teacher. The most efficient way to make sure students edit for as many grammatical and stylistic flaws as they can find is to base a large portion of the grade on how easy the paper is to read. If you get a badly edited piece, you can just hand it back and tell the student you'll grade it when the errors are gone. Or you can take 20-30% off the content grade. Students get the message very quickly and turn in remarkably clean writing.

If a student continues to have problems editing a paper, you can suggest visiting the Writing Center to get some one-on-one help with a writing consultant.

Think of Yourself First as a Reader

Some teachers think that basing 20-30% of the grade on grammatical and stylistic matters is unfair unless they mark all the flaws. We approach this issue from the perspective of readers. If you review a textbook and find editing mistakes, you don't label each one and send the text back to the publisher. No, you just stop reading and don't adopt the textbook. Readers who are not teachers just don't keep reading is a text that is too confusing or if errors are too distracting. Readers who are teachers are perfectly justified in simply noting with an X in the margin where a sentence gets too confusing or where mistaken punctuation leads the reader astray. Students are resourceful (they can get help from an on-campus writing center office or a writing center website) and will figure out the problem once a reader points out where the text stumbles. That's really all it takes.

Use Peer Editing

Perhaps the most helpful tool in getting clean, readable papers from students is the peer editing session. Most students are better editors of someone else's paper than proofreaders of their own, so having students exchange papers and look for flaws helps them find many more glitches than they'll find on their own.

View More about Student Peer Review

Try a Time-Saving Shortcut

If you feel compelled to mark grammatical and stylistic flaws, work out a shorthand for yourself and give students a handout explaining your marks. Most teachers can get by with one symbol for a sentence that gets derailed or confused, another for faulty punctuation of all sorts, and a third for inaccurate words (spelling or meaning). Save your time and energy for commenting on substance rather than form.

Sample Policies on Grading Grammar versus Content

Outdoor Resources 1XX (excerpts)

(Although we don't recommend assigning points for errors (because then you have to mark and count them all), this teacher was clear about expectations.)

Your paper should contain from 1,500 to 2,000 words, or about five to seven pages. The paper must be typewritten, double spaced, and bound. Neatness is essential.

A Check List of Points to Consider:

I. Mechanics

Neatness. Is your report clean, neatly organized, with a look of professional pride about it?

Spelling. Two points will be deducted for each misspelled word.

Grammar and punctuation. Five points will be deducted for each sentence which uses improper grammar or punctuation.

Outline. Did you follow the course outline?

Form. Is your paper in the proper form?

Bibliography. Are the references properly cited?

Binding. Use a cover binding with a secure clasp.

II. Content . . . .

Use a grading sheet

Grading comment sheets or checksheets give teachers and students two advantages over free-form grading:

• Grading sheets of some sort assure that teachers will give students feedback about all the major criteria they set out on the assignment sheet. Even if you decide to use a simple checksheet that ranks students' performance on each criterion on a 1-10 scale, students will be able to see quickly where their strengths and weaknesses are as writers for this assignment.
• Grading sheets, particularly checksheets, typically save teachers time. Even composition teachers don't comment exhaustively about each criterion for each assignment; so, too, disciplinary teachers should be aware that they can comment at some length on just one or two points (typically the major strength and the major weakness) and then rely on the checksheet to fill in for less crucial areas of the paper. If students are concerned about getting more feedback than the checksheet provides, you can encourage them to come to your office hours or send you an e-mail query.

Resource: Sample Grading Sheets

Four sample grading sheets are provided:

• Introductory Composition
• Science Project

Sample Grading Sheet

Composition 1xx Grading Sheet

Grade for essay: ___________

Revision Instructions:

Sample Report Evaluation

Name: _________________

Subject: _________________

__ total points

DETAILED REPORT EVALUATION

Title page:

Table of contents:

Bibliography:

Information page:

Oral presentation:

Sample Evaluation of Written Report

Evaluation of Written Report

Sample Science Project Checksheet

Science Project checksheet

GENERAL 50 POINTS

1. Correct form (15)

Reference list (3)

Citation of sources(2)

Mechanics (order, table of contents, list of tables, list of figures, cover) (5)

2. Composition skills (10)

Spelling (5)

Grammar (5)

3. Log book used to record experimental data, ideas, etc. (10)

4. Abstract (10)

5. Acknowledgments (5)

TOTAL GENERAL: _________

EXHIBIT 50 EXTRA CREDIT POINTS

1. Summarized project well (30)

Problem and hypothesis easy to understand (5)

Experimental method clearly stated (10)

Results summarized in graphs/tables (10)

Conclusion presented (5)

2. Eye appeal (10)

Neat lettering (3)

Pleasing placement of parts (2)

Good use of color (3)

Sturdiness (2)

3. Creativity (10)

TOTAL EXHIBIT POINTS: _______

TOTAL PROJECT: ______

Resource: Sample grading criteria

General Grading Criteria: Composition 1xx

Kate Kiefer, Donna LeCourt, Stephen Reid, & Jean Wyrick. (2018). Commenting on Student Writing. The WAC Clearinghouse. Retrieved from https://wac.colostate.edu/repository/teaching/guides/commenting/. Originally developed for Writing@CSU (https://writing.colostate.edu).

Choose Your Test

Sat / act prep online guides and tips, sat essay rubric: full analysis and writing strategies.

We're about to dive deep into the details of that least beloved* of SAT sections, the SAT essay . Prepare for a discussion of the SAT essay rubric and how the SAT essay is graded based on that. I'll break down what each item on the rubric means and what you need to do to meet those requirements.

On the SAT, the last section you'll encounter is the (optional) essay. You have 50 minutes to read a passage, analyze the author's argument, and write an essay. If you don’t write on the assignment, plagiarize, or don't use your own original work, you'll get a 0 on your essay. Otherwise, your essay scoring is done by two graders - each one grades you on a scale of 1-4 in Reading, Analysis, and Writing, for a total essay score out of 8 in each of those three areas . But how do these graders assign your writing a numerical grade? By using an essay scoring guide, or rubric.

*may not actually be the least belovèd.

Feature image credit: Day 148: the end of time by Bruce Guenter , used under CC BY 2.0 /Cropped from original. 

UPDATE: SAT Essay No Longer Offered

(adsbygoogle = window.adsbygoogle || []).push({});.

In January 2021, the College Board announced that after June 2021, it would no longer offer the Essay portion of the SAT (except at schools who opt in during School Day Testing). It is now no longer possible to take the SAT Essay, unless your school is one of the small number who choose to offer it during SAT School Day Testing.

While most colleges had already made SAT Essay scores optional, this move by the College Board means no colleges now require the SAT Essay. It will also likely lead to additional college application changes such not looking at essay scores at all for the SAT or ACT, as well as potentially requiring additional writing samples for placement.

What does the end of the SAT Essay mean for your college applications? Check out our article on the College Board's SAT Essay decision for everything you need to know.

The Complete SAT Essay Grading Rubric: Item-by-Item Breakdown

Based on the CollegeBoard’s stated Reading, Analysis, and Writing criteria, I've created the below charts (for easier comparison across score points). For the purpose of going deeper into just what the SAT is looking for in your essay, I've then broken down each category further (with examples).

The information in all three charts is taken from the College Board site .

The biggest change to the SAT essay (and the thing that really distinguishes it from the ACT essay) is that you are required to read and analyze a text , then write about your analysis of the author's argument in your essay. Your "Reading" grade on the SAT essay reflects how well you were able to demonstrate your understanding of the text and the author's argument in your essay.

You'll need to show your understanding of the text on two different levels: the surface level of getting your facts right and the deeper level of getting the relationship of the details and the central ideas right.

Surface Level: Factual Accuracy

One of the most important ways you can show you've actually read the passage is making sure you stick to what is said in the text . If you’re writing about things the author didn’t say, or things that contradict other things the author said, your argument will be fundamentally flawed.

For instance, take this quotation from a (made-up) passage about why a hot dog is not a sandwich:

“The fact that you can’t, or wouldn’t, cut a hot dog in half and eat it that way, proves that a hot dog is once and for all NOT a sandwich”

Here's an example of a factually inaccurate paraphrasing of this quotation:

The author builds his argument by discussing how, since hot-dogs are often served cut in half, this makes them different from sandwiches.

The paraphrase contradicts the passage, and so would negatively affect your reading score. Now let's look at an accurate paraphrasing of the quotation:

The author builds his argument by discussing how, since hot-dogs are never served cut in half, they are therefore different from sandwiches.

It's also important to be faithful to the text when you're using direct quotations from the passage. Misquoting or badly paraphrasing the author’s words weakens your essay, because the evidence you’re using to support your points is faulty.

Higher Level: Understanding of Central Ideas

The next step beyond being factually accurate about the passage is showing that you understand the central ideas of the text and how details of the passage relate back to this central idea.

Why does this matter? In order to be able to explain why the author is persuasive, you need to be able to explain the structure of the argument. And you can’t deconstruct the author's argument if you don’t understand the central idea of the passage and how the details relate to it.

Here's an example of a statement about our fictional "hot dogs are sandwiches" passage that shows understanding of the central idea of the passage:

Hodgman’s third primary defense of why hot dogs are not sandwiches is that a hot dog is not a subset of any other type of food. He uses the analogy of asking the question “is cereal milk a broth, sauce, or gravy?” to show that making such a comparison between hot dogs and sandwiches is patently illogical.

The above statement takes one step beyond merely being factually accurate to explain the relation between different parts of the passage (in this case, the relation between the "what is cereal milk?" analogy and the hot dog/sandwich debate).

Of course, if you want to score well in all three essay areas, you’ll need to do more in your essay than merely summarizing the author’s argument. This leads directly into the next grading area of the SAT Essay.

The items covered under this criterion are the most important when it comes to writing a strong essay. You can use well-spelled vocabulary in sentences with varied structure all you want, but if you don't analyze the author's argument, demonstrate critical thinking, and support your position, you will not get a high Analysis score .

Because this category is so important, I've broken it down even further into its two different (but equally important) component parts to make sure everything is as clearly explained as possible.

Part I: Critical Thinking (Logic)

Critical thinking, also known as critical reasoning, also known as logic, is the skill that SAT essay graders are really looking to see displayed in the essay. You need to be able to evaluate and analyze the claim put forward in the prompt. This is where a lot of students may get tripped up, because they think “oh, well, if I can just write a lot, then I’ll do well.” While there is some truth to the assertion that longer essays tend to score higher , if you don’t display critical thinking you won’t be able to get a top score on your essay.

What do I mean by critical thinking? Let's take the previous prompt example:

Write an essay in which you explain how Hodgman builds an argument to persuade his audience that the hot dog cannot, and never should be, considered a sandwich.

An answer to this prompt that does not display critical thinking (and would fall into a 1 or 2 on the rubric) would be something like:

The author argues that hot dogs aren’t sandwiches, which is persuasive to the reader.

While this does evaluate the prompt (by providing a statement that the author's claim "is persuasive to the reader"), there is no corresponding analysis. An answer to this prompt that displays critical thinking (and would net a higher score on the rubric) could be something like this:

The author uses analogies to hammer home his point that hot dogs are not sandwiches. Because the readers will readily believe the first part of the analogy is true, they will be more likely to accept that the second part (that hot dogs aren't sandwiches) is true as well.

See the difference? Critical thinking involves reasoning your way through a situation (analysis) as well as making a judgement (evaluation) . On the SAT essay, however, you can’t just stop at abstract critical reasoning - analysis involves one more crucial step...

Part II: Examples, Reasons, and Other Evidence (Support)

The other piece of the puzzle (apparently this is a tiny puzzle) is making sure you are able to back up your point of view and critical thinking with concrete evidence . The SAT essay rubric says that the best (that is, 4-scoring) essay uses “ relevant, sufficient, and strategically chosen support for claim(s) or point(s) made. ” This means you can’t just stick to abstract reasoning like this:

That explanation is a good starting point, but if you don't back up your point of view with quoted or paraphrased information from the text to support your discussion of the way the author builds his/her argument, you will not be able to get above a 3 on the Analysis portion of the essay (and possibly the Reading portion as well, if you don't show you've read the passage). Let's take a look of an example of how you might support an interpretation of the author's effect on the reader using facts from the passage :

The author’s reference to the Biblical story about King Solomon elevates the debate about hot dogs from a petty squabble between friends to a life-or-death disagreement. The reader cannot help but see the parallels between the two situations and thus find themselves agreeing with the author on this point.

Does the author's reference to King Solomon actually "elevate the debate," causing the reader to agree with the author? From the sentences above, it certainly seems plausible that it might. While your facts do need to be correct,  you get a little more leeway with your interpretations of how the author’s persuasive techniques might affect the audience. As long as you can make a convincing argument for the effect a technique the author uses might have on the reader, you’ll be good.

Say whaaat?! #tbt by tradlands , used under CC BY 2.0 /Cropped and color-adjusted from original.

Did I just blow your mind? Read more about the secrets the SAT doesn’t want you to know in this article . 

Your Writing score on the SAT essay is not just a reflection of your grasp of the conventions of written English (although it is that as well). You'll also need to be focused, organized, and precise.

Because there's a lot of different factors that go into calculating your Writing score, I've divided the discussion of this rubric area into five separate items:

Precise Central Claim

Organization, vocab and word choice, sentence structure, grammar, etc..

One of the most basic rules of the SAT essay is that you need to express a clear opinion on the "assignment" (the prompt) . While in school (and everywhere else in life, pretty much) you’re encouraged to take into account all sides of a topic, it behooves you to NOT do this on the SAT essay. Why? Because you only have 50 minutes to read the passage, analyze the author's argument, and write the essay, there's no way you can discuss every single way in which the author builds his/her argument, every single detail of the passage, or a nuanced argument about what works and what doesn't work.

Instead, I recommend focusing your discussion on a few key ways the author is successful in persuading his/her audience of his/her claim.

Let’s go back to the assignment we've been using as an example throughout this article:

"Write an essay in which you explain how Hodgman builds an argument to persuade his audience that the hot dog cannot, and never should be, considered a sandwich."

Your instinct (trained from many years of schooling) might be to answer:

"There are a variety of ways in which the author builds his argument."

This is a nice, vague statement that leaves you a lot of wiggle room. If you disagree with the author, it's also a way of avoiding having to say that the author is persuasive. Don't fall into this trap! You do not necessarily have to agree with the author's claim in order to analyze how the author persuades his/her readers that the claim is true.

Here's an example of a precise central claim about the example assignment:

The author effectively builds his argument that hot dogs are not sandwiches by using logic, allusions to history and mythology, and factual evidence.

In contrast to the vague claim that "There are a variety of ways in which the author builds his argument," this thesis both specifies what the author's argument is and the ways in which he builds the argument (that you'll be discussing in the essay).

While it's extremely important to make sure your essay has a clear point of view, strong critical reasoning, and support for your position, that's not enough to get you a top score. You need to make sure that your essay  "demonstrates a deliberate and highly effective progression of ideas both within paragraphs and throughout the essay."

What does this mean? Part of the way you can make sure your essay is "well organized" has to do with following standard essay construction points. Don't write your essay in one huge paragraph; instead, include an introduction (with your thesis stating your point of view), body paragraphs (one for each example, usually), and a conclusion. This structure might seem boring, but it really works to keep your essay organized, and the more clearly organized your essay is, the easier it will be for the essay grader to understand your critical reasoning.

The second part of this criteria has to do with keeping your essay focused, making sure it contains "a deliberate and highly effective progression of ideas." You can't just say "well, I have an introduction, body paragraphs, and a conclusion, so I guess my essay is organized" and expect to get a 4/4 on your essay. You need to make sure that each paragraph is also organized . Recall the sample prompt:

“Write an essay in which you explain how Hodgman builds an argument to persuade his audience that the hot dog cannot, and never should be, considered a sandwich.”

And our hypothetical thesis:

Let's say that you're writing the paragraph about the author's use of logic to persuade his reader that hot dogs aren't sandwiches. You should NOT just list ways that the author is logical in support of his claim, then explain why logic in general is an effective persuasive device. While your points might all be valid, your essay would be better served by connecting each instance of logic in the passage with an explanation of how that example of logic persuades the reader to agree with the author.

Above all, it is imperative that you make your thesis (your central claim) clear in the opening paragraph of your essay - this helps the grader keep track of your argument. There's no reason you’d want to make following your reasoning more difficult for the person grading your essay (unless you’re cranky and don’t want to do well on the essay. Listen, I don’t want to tell you how to live your life).

In your essay, you should use a wide array of vocabulary (and use it correctly). An essay that scores a 4 in Writing on the grading rubric “demonstrates a consistent use of precise word choice.”

You’re allowed a few errors, even on a 4-scoring essay, so you can sometimes get away with misusing a word or two. In general, though, it’s best to stick to using words you are certain you not only know the meaning of, but also know how to use. If you’ve been studying up on vocab, make sure you practice using the words you’ve learned in sentences, and have those sentences checked by someone who is good at writing (in English), before you use those words in an SAT essay.

Creating elegant, non-awkward sentences is the thing I struggle most with under time pressure. For instance, here’s my first try at the previous sentence: “Making sure a sentence structure makes sense is the thing that I have the most problems with when I’m writing in a short amount of time” (hahaha NOPE - way too convoluted and wordy, self). As another example, take a look at these two excerpts from the hypothetical essay discussing how the author persuaded his readers that a hot dog is not a sandwich:

Score of 2: "The author makes his point by critiquing the argument against him. The author pointed out the logical fallacy of saying a hot dog was a sandwich because it was meat "sandwiched" between two breads. The author thus persuades the reader his point makes sense to be agreed with and convinces them."

The above sentences lack variety in structure (they all begin with the words "the author"), and the last sentence has serious flaws in its structure (it makes no sense).

Score of 4: "The author's rigorous examination of his opponent's position invites the reader, too, to consider this issue seriously. By laying out his reasoning, step by step, Hodgman makes it easy for the reader to follow along with his train of thought and arrive at the same destination that he has. This destination is Hodgman's claim that a hot dog is not a sandwich."

The above sentences demonstrate variety in sentence structure (they don't all begin with the same word and don't have the same underlying structure) that presumably forward the point of the essay.

In general, if you're doing well in all the other Writing areas, your sentence structures will also naturally vary. If you're really worried that your sentences are not varied enough, however, my advice for working on "demonstrating meaningful variety in sentence structure" (without ending up with terribly worded sentences) is twofold:

• Read over what you’ve written before you hand it in and change any wordings that seem awkward, clunky, or just plain incorrect.
• As you’re doing practice essays, have a friend, family member, or teacher who is good at (English) writing look over your essays and point out any issues that arise. 

This part of the Writing grade is all about the nitty gritty details of writing: grammar, punctuation, and spelling . It's rare that an essay with serious flaws in this area can score a 4/4 in Reading, Analysis, or Writing, because such persistent errors often "interfere with meaning" (that is, persistent errors make it difficult for the grader to understand what you're trying to get across).

On the other hand, if they occur in small quantities, grammar/punctuation/spelling errors are also the things that are most likely to be overlooked. If two essays are otherwise of equal quality, but one writer misspells "definitely" as "definately" and the other writer fails to explain how one of her examples supports her thesis, the first writer will receive a higher essay score. It's only when poor grammar, use of punctuation, and spelling start to make it difficult to understand your essay that the graders start penalizing you.

My advice for working on this rubric area is the same advice as for sentence structure: look over what you’ve written to double check for mistakes, and ask someone who’s good at writing to look over your practice essays and point out your errors. If you're really struggling with spelling, simply typing up your (handwritten) essay into a program like Microsoft Word and running spellcheck can alert you to problems. We've also got a great set of articles up on our blog about SAT Writing questions that may help you better understand any grammatical errors you are making.

How Do I Use The SAT Essay Grading Rubric?

Now that you understand the SAT essay rubric, how can you use it in your SAT prep? There are a couple of different ways.

Use The SAT Essay Rubric To...Shape Your Essays

Since you know what the SAT is looking for in an essay, you can now use that knowledge to guide what you write about in your essays!

A tale from my youth: when I was preparing to take the SAT for the first time, I did not really know what the essay was looking for, and assumed that since I was a good writer, I’d be fine.

Not true! The most important part of the SAT essay is using specific examples from the passage and explaining how they convince the reader of the author's point. By reading this article and realizing there's more to the essay than "being a strong writer," you’re already doing better than high school me.

Change the object in that girl’s left hand from a mirror to a textbook and you have a pretty good sketch of what my junior year of high school looked like.

Use The SAT Essay Rubric To...Grade Your Practice Essays

The SAT can’t exactly give you an answer key to the essay. Even when an example of an essay that scored a particular score is provided, that essay will probably use different examples than you did, make different arguments, maybe even argue different interpretations of the text...making it difficult to compare the two. The SAT essay rubric is the next best thing to an answer key for the essay - use it as a lens through which to view and assess your essay.

Of course, you don’t have the time to become an expert SAT essay grader - that’s not your job. You just have to apply the rubric as best as you can to your essays and work on fixing your weak areas . For the sentence structure, grammar, usage, and mechanics stuff I highly recommend asking a friend, teacher, or family member who is really good at (English) writing to take a look over your practice essays and point out the mistakes.

If you really want custom feedback on your practice essays from experienced essay graders, may I also suggest the PrepScholar test prep platform ? I manage the essay grading and so happen to know quite a bit about the essay part of this platform, which gives you both an essay grade and custom feedback for each essay you complete. Learn more about how it all works here .

What’s Next?

Are you so excited by this article that you want to read even more articles on the SAT essay? Of course you are. Don't worry, I’ve got you covered. Learn how to write an SAT essay step-by-step and read about the 6 types of SAT essay prompts .

Want to go even more in depth with the SAT essay? We have a complete list of past SAT essay prompts as well as tips and strategies for how to get a 12 on the SAT essay .

Still not satisfied? Maybe a five-day free trial of our very own PrepScholar test prep platform (which includes essay practice and feedback) is just what you need.

Trying to figure out whether the old or new SAT essay is better for you? Take a look at our article on the new SAT essay assignment to find out!

Laura graduated magna cum laude from Wellesley College with a BA in Music and Psychology, and earned a Master's degree in Composition from the Longy School of Music of Bard College. She scored 99 percentile scores on the SAT and GRE and loves advising students on how to excel in high school.

Ask a Question Below

Have any questions about this article or other topics? Ask below and we'll reply!

Improve With Our Famous Guides

• For All Students

The 5 Strategies You Must Be Using to Improve 160+ SAT Points

How to Get a Perfect 1600, by a Perfect Scorer

Series: How to Get 800 on Each SAT Section:

Score 800 on SAT Math

Score 800 on SAT Reading

Score 800 on SAT Writing

Series: How to Get to 600 on Each SAT Section:

Score 600 on SAT Math

Score 600 on SAT Reading

Score 600 on SAT Writing

Free Complete Official SAT Practice Tests

What SAT Target Score Should You Be Aiming For?

15 Strategies to Improve Your SAT Essay

The 5 Strategies You Must Be Using to Improve 4+ ACT Points

How to Get a Perfect 36 ACT, by a Perfect Scorer

Series: How to Get 36 on Each ACT Section:

36 on ACT English

36 on ACT Math

36 on ACT Reading

36 on ACT Science

Series: How to Get to 24 on Each ACT Section:

24 on ACT English

24 on ACT Math

24 on ACT Reading

24 on ACT Science

What ACT target score should you be aiming for?

ACT Vocabulary You Must Know

ACT Writing: 15 Tips to Raise Your Essay Score

How to Get Into Harvard and the Ivy League

How to Get a Perfect 4.0 GPA

How to Write an Amazing College Essay

What Exactly Are Colleges Looking For?

Is the ACT easier than the SAT? A Comprehensive Guide

Should you retake your SAT or ACT?

When should you take the SAT or ACT?

Stay Informed

Get the latest articles and test prep tips!

Looking for Graduate School Test Prep?

Check out our top-rated graduate blogs here:

GRE Online Prep Blog

GMAT Online Prep Blog

TOEFL Online Prep Blog

Holly R. "I am absolutely overjoyed and cannot thank you enough for helping me!”

How to Grade Essays Faster | My Top 10 Grading Tips and Tricks

Are you looking for ways to grade essays faster? I get it. Grading essays can be a daunting task for ELA teachers. Following these essay grading tips and tricks can save you time and energy on grading without giving up quality feedback to your students.

Are you Googling “How to Grade Essays Faster” because that never-ending pile of essays is starting to haunt you? (Yup. I’ve been there.) Teachers of all disciplines understand the work-life struggle of the profession. Throw in 60, 80, 100, or more essays, and you’re likely giving up evenings and weekends until that pile is gone.

Truthfully, while there are many aspects of being an ELA teacher I love , grading essays doesn’t quite make the list. However, it’s a necessary aspect of the ELA classroom to hold students accountable and help them improve. But what if I told you there were some tips and tricks you could use to make grading much easier and faster? Because there are. That means saying goodbye to spending your weekends lost in a sea of student essays. It means no more living at school the weeks following students turning in an essay. Instead, prepare to celebrate getting your time (and sanity) back.

Start By Reframing Your Definition of Grading an Essay

Before you can implement my time-saving grading tips and tricks, you need to be willing to shift your mindset regarding grading. Afterall, where does it say we have to give up hours upon hours of our time to get it done? It’s time to start redefining and reframing what it even means to grade an essay.

The key to reframing your definition (and, therefore, expectations) about grading student essays is thinking about helping your students, not correcting them. Of course, there’s nothing wrong with pointing out grammatical and structural errors. However, it’s essential to focus on leaving constructive feedback that can help students improve their craft. Now, how can that be done without spending hours filling the margins with comments?

I’m glad you asked.

Grade Essays Faster with These Tips and Tricks

Since we can’t avoid grading altogether, I hope these tips and tricks can help you grade essays faster and increase student performance. And while I love rubrics, and they can certainly save time grading, they aren’t your only option. So here are eight other tips and tricks to try.

Tip 1: Get Focused.

This has been one of my biggest grading time-savers. And I’m not just talking about limiting your distractions while you grade (more on that in a minute), but I mean narrow your focus on what it is you’re grading. Often, we spend so much time correcting every single grammatical mistake that we miss opportunities to give feedback on the skills we’re currently teaching. Try to focus your feedback on the specific skills your students just learned, like writing a strong thesis, embedding quotations, providing supporting evidence, or transitioning from paragraph to paragraph.

Taking this approach to grading will lead to less overwhelm for both you and your students. In fact, your students will have a clearer understanding of what they need to continue working on. Just be sure to make the specific skill (or skills) that you’re looking for (and grading) clear at the start of the assignment.

Tip 2: Give Student Choice.

Let’s say you’ve been working on a particular skill for a few weeks and have had your students practice using various writing prompts. Instead of feeling forced to provide feedback on every written response, let your students choose their best work for you to grade. I find that this grading technique works best on shorter assignments.

However, that doesn’t mean you can’t apply this to longer essays. If you’ve been working on a certain aspect of essay writing, you can let your students pick the paragraph from their essay they want you to grade. Either way, encourage your students to select the writing they believe best represents their skills and knowledge for the task at hand. Not only will this cut down on your grading time, but it will also encourage a sense of ownership over students’ grades.

Tip 3: Check Mark Revisions.

The checkmark revision approach is a great way to put more ownership and accountability on your students. Instead of grading a student essay by telling them exactly what to fix, turn it into a learning opportunity! As you review the student essay, simply use check marks to note areas that need to be corrected or could be improved. Then, give students time in class to work through their essays, identifying what the check mark indicates and making proper adjustments.

However, make sure your students have a clear list (or rubric) outlining the expectations for the essay. They can use this list to refer to when trying to figure out what revisions they need to make to improve their work. Alternatively, if you’re not ready to jump straight to checkmarks, you can create a comment code that provides a bit more guidance for students without taking up a lot of your time.

Tip 4: Use Conferences.

Have you ever thought about holding student-teacher conferences in lieu of providing written feedback? If not, you totally should! Students are so used to teachers doing the heavy lifting for them. Alternatively, turn the revision process into an active experience for them. Instead of going through the essay on your own, marking errors, and making suggestions, talk it through with each student.

When it comes to student-teacher conferences, make sure to set a reasonable time limit for each conference to ensure you’re not spending days conducting these meetings. Just make sure your time limit is enough to review their written work and provide verbal feedback. I require each student to mark their essay as we review it so they know exactly what to work on. While I’m more than willing to answer questions, I encourage students to make an appointment with me after school if they need extensive help.

Tip 5: Skim and Review

I can’t be the only one who wants to shed a tear of frustration when I watch a student toss a comment-covered essay right into recycling. So, instead of spending hours leaving comments on each and every student’s essay, skim through their rough drafts while noting common errors. That way, instead of waiting until students turn in their final draft to address their mistakes, you can review common errors in class before they submit a final draft.  Trust me. This will make grading those final drafts much easier– especially if you have a clear rubric or grading checklist to follow.

This is a great way to review common grammar mistakes that we don’t always take time to teach at the secondary level. It’s also a great way for you to address aspects of your target skills that students are still struggling with. Lastly, I find this shift in focus from the final product to the revision process helps students better understand (and, perhaps, appreciate) the writing process as more than a grade but a learning experience.

Tip 6: Leave a Comment at the End.

This is a huge time-saver, and it’s pretty simple. Although be warned, it might challenge you to go against all of your grading instincts! We’re so used to marking every single error or making all the suggestions with student essays. But, students are often overwhelmed by the mere look of ink-filled margins. What if, instead, you save your comments for the end and limit yourself to one or two celebrations and one or two areas for improvement? This is a simple yet clear way to provide feedback to your students on a final draft, especially if you’ve already gone through a more in-depth revision process from draft to draft.

Okay fine. If you must, you can fix the grammatical errors using a red pen, but save your energy by avoiding writing the same thing over and over again. If you’ve marked the same error three times, let that be it. If they don’t get it after three examples, they should probably make time to see you after school.

Tip 7: Grade Paragraph-by-Paragraph.

Instead of feeling overwhelmed by grading a tall stack of essays, consider breaking your grading– and writing– process down by paragraph. Assessing a single paragraph is far more time-friendly than an entire essay. So, have your students work on their essay paragraph by paragraph, turning each component in as they are completed. That way, you can provide quick and effective feedback they can apply when revising that paragraph and writing any future paragraphs for the final piece. Take it a step further by breaking it down into specific skills and components of an essay. For example, maybe you grade students’ thesis statements and supporting evidence as two separate steps. Grading each of these components takes far less time and, by the time students put it all together for their final essay, their writing should be much more polished and easier to grade. Plus, since you gave immediate feedback throughout the process, you don’t have to worry about spending hours writing comments throughout their entire paper. Instead, give the students a “final” grade using a simple rubric. And since you gave them opportunities to apply your feedback throughout the writing process, you can even have an “improvement” section of the rubric. This is an easy way to acknowledge student effort and progress with their writing.

Tip 8: Mark-up a Model Paragraph.

Take some of the work off your plate by grading a paragraph and letting the students do the rest. (You read that right.) Here’s how it works: instead of grading an entire paper, rewriting the same comments paragraph after paragraph, just mark up a model paragraph. Alternatively, you can grade the intro and conclusion paragraphs, while marking up one body paragraph as a model for the remaining body paragraphs. Give them a score on a smaller scale, such as 1 to 10, as a phase one grade.

Then, set aside time in class to have your students review your model paragraph and use it to mark up the rest of their paper before fixing their errors. I like giving them time in class to do this so they can ask me any clarifying questions in real-time. Once they turn in their revised essay, you can give them a phase two grade without having to worry about diving too deep into feedback. A comment per paragraph or page would suffice.

More Teacher Tricks to Help You Grade Essays Faster

T ip 9: set realistic goals..

Just like we set our students up for success, set yourself up for success too. If you know you can’t get through a class worth of essays during your prep period, don’t set it as your goal. You’ll only feel overwhelmed, disappointed, and discouraged when you only make it through half of your stack. Instead, only tackle your grading when you have the time to do so, and set realistic goals when you do. Grading more essays than you planned on? You feel on top of the world. Grading fewer? You feel like it’s neverending.

Tip 10: Avoid Distractions.

Instagram? Facebook? I know how easy it is to wander over to your phone and take a scroll break. But, we both know a few minutes can turn into an hour real fast. So, do yourself a favor, and when you know it’s time to grade a stack of essays, free your space of any distractions and set a timer. You’d be surprised by how much you can get done in an hour of uninterrupted essay grading.

The bottom line is that grading is an unavoidable aspect of being an ELA teacher. However, I hope one or more of these ideas can help you grade essays faster. The truth is, with these essay grading tips and tricks, you won’t only grade essays more efficiently, but you’ll provide better feedback for students as well. In fact, the longer we take to grade (or procrastinate grading) those essays, the less effective the feedback is for students, period.

So, here’s to more effective grading– faster!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

The world’s leading AI platform for teachers to grade essays

EssayGrader is an AI powered grading assistant that gives high quality, specific and accurate writing feedback for essays. On average it takes a teacher 10 minutes to grade a single essay, with EssayGrader that time is cut down to 30 seconds That's a 95% reduction in the time it takes to grade an essay, with the same results.

How we've done

Happy users

Essays graded

EssayGrader analyzes essays with the power of AI. Our software is trained on massive amounts of diverse text data, inlcuding books, articles and websites. This gives us the ability to provide accurate and detailed writing feedback to students and save teachers loads of time. We are the perfect AI powered grading assitant.

EssayGrader analyzes essays for grammar, punctuation, spelling, coherence, clarity and writing style errors. We provide detailed reports of the errors found and suggestions on how to fix those errors. Our error reports help speed up grading times by quickly highlighting mistakes made in the essay.

Bulk uploading

Uploading a single essay at a time, then waiting for it to complete is a pain. Bulk uploading allows you to upload an entire class worth of essays at a single time. You can work on other important tasks, come back in a few minutes to see all the essays perfectly graded.

Custom rubrics

We don't assume how you want to grade your essays. Instead, we provide you with the ability to create the same rubrics you already use. Those rubrics are then used to grade essays with the same grading criteria you are already accustomed to.

Sometimes you don't want to read a 5000 word essay and you'd just like a quick summary. Or maybe you're a student that needs to provide a summary of your essay to your teacher. We can help with our summarizer feature. We can provide a concise summary including the most important information and unique phrases.

AI detector

Our AI detector feature allows teachers to identify if an essay was written by AI or if only parts of it were written by AI. AI is becoming very popular and teachers need to be able to detect if essays are being written by students or AI.

Create classes to neatly organize your students essays. This is an essential feature when you have multiple classes and need to be able to track down students essays quickly.

Our mission

At EssayGrader, our mission is crystal clear: we're transforming the grading experience for teachers and students alike. Picture a space where teachers can efficiently and accurately grade essays, lightening their workload, while empowering students to enhance their writing skills. Our software is a dynamic work in progress, a testament to our commitment to constant improvement. We're dedicated to refining and enhancing our platform continually. With each update, we strive to simplify the lives of both educators and learners, making the process of grading and writing essays smoother and more efficient.We recognize the immense challenges teachers face – the heavy burdens, the long hours, and the often underappreciated efforts. EssayGrader is our way of shouldering some of that load. We are here to support you, to make your tasks more manageable, and to give you the tools you need to excel in your teaching journey.

Join the newsletter

Subscribe to get our latest content by email.

The Unwritten Rules of History

• Canadian Historians Online
• Most Popular Posts
• Work With Us
• 10 Tips for Grading Essays Quickly and Efficiently

We’ve all been there. No one likes marking. But as a professor, it’s part of the job description. One of the draft titles of this post was even “How to Grade Essays Without Wanting to Commit Murder.” While there are some great guides on teaching the mechanics of grading available, there isn’t much useful advice on how to make grading easier apart from either having fewer assignments or providing less feedback. In the real world, neither one of these is very useful. But there are strategies that every instructor or professor can follow to make grading essays quicker and more efficient. Here are some of mine.

1) Have Faith in Yourself

One of the biggest problems I’ve faced and continue to face as an instructor is Imposter Syndrome, or the belief that I’ve somehow fooled everyone around me into believing that I am a knowledgeable and competent person. Grading is one area where Imposter Syndrome likes to rear its ugly head. You will have finished reading a paper and then start to doubt that you’ve given it an appropriate grade. Or you worry that your students will get mad at you for giving them a bad grade. Or you’ll worry that this paper will result in a grade dispute, and then real professors will review and judge your work and find you wanting. Resist these thoughts. Remember that you have the expertise and good judgement to evaluate essays. Do not second-guess yourself. Assign a grade, make your comments, and move on. Have faith that you have done your best.

2) Don’t Repeat Yourself

It’s very common in research essays to see that same mistake made more than once. This is particularly the case when it comes to footnotes and bibliographies, which are often filled with tiny mistakes. Don’t spend all your time correcting these mistakes. Fix it once, and explain what you did. If you see it again, circle it and write something like “see previous comment on…” If it’s a systematic problem, I’d then make a note to mention this problem in the comments and say that you’ve only corrected a couple of instances to give them an idea of how to do it properly. This is not high school, and it is not your job to find every single mistake on an essay and correct it. Instead, identify the problem, and give your student an opportunity to apply what they’ve learned. The same goes for grammar and spelling. If it’s a serious issue, I always recommend that students go see the Writing Centre. It’s not your job to teach them how to write (unless it’s a composition class, in which case, good luck!)

3) Create a Comment Bank

You’ll notice that after a while, you will repeat the same sentences over and over again. To save yourself from having to either remember what you said last time or type or to write the same sentence over and over again, create a Word document with your most common comments. This is sometimes referred to as a Comment Bank or a Teaching Toolbox. I will do a whole blog post on this in the near future, but it’s easy to get started. If you save your comments on your computer, read through them and copy and paste the most common into a new Word document. For example, one that I use a lot is “While I can see that you are trying to make an argument here, you spend too much time describing or summarizing your sources rather than analysing them. In general, you should avoid description as much as possible.” The time and frustration you will save is immeasurable

4) Create a Bibliographic Bank

Odds are you will receive several papers on a given topic. Once you’ve been marking for a while, you’ll notice that you keep recommending the same books or articles. Again, to save you from having to remember which sources you want to recommend and/or typing out the full references, create a Word document with a list of topics and some of the most important sources listed for each. This way you only do the research once, rather than a million times. This is also helpful if you want to evaluate whether your students have selected appropriate sources or have missed important ones. Your comps list can be a great starting point.

5) Make a Grading Conversion Chart

In general, most assignments require three different “grades”: a letter grade, a percentage, and a numeric grade (like 7 out of 10). They each have their own purposes, but the odds are you will need to convert between them. Even when working at one institution for many years, it can be hard to do this conversion in your head. Spend several years as a sessional at multiple universities with their own ideas about what each letter grade means, and the problem grows exponentially. My solution is is to use an Excel spreadsheet of grades. This is relative easy to create. Mine look like this:

It’s really easy to do. Each “out of” number has three columns. The first is a numeric grade. The second is that grade converted to a percentage (it’s easier to do with a formula, and then just do “fill down.”) The third column is the corresponding letter grade. You can fill these in manually, or you can use a formula.

Here’s mine, but make sure yours corresponds to your institution’s grading scheme! =IF(K19>=95%,”A+”,IF(K19>90%,”A”,IF(K19>=85%,”A “, IF(K19>=80%,”B+”, IF(K19>=75%,”B”,IF(K19>70%,”B-“, IF(K19>65%,”C+”,IF(K19>60%,”C”,IF(K19>55%,”C”, IF(K19>50%,”P”,IF(K19>0%,”NC”,)))))))))))

6) Mark in Batches

I like to run, and when you’re really tired and facing a long run, thinking of the time remaining in intervals makes it much easier. The same is true for marking. A stack of 100 essays seems insurmountable. So what I do is break that stack down into manageable groups, usually 3 or 5 essays, which is about an hour to an hour and a half of grading, depending on the length of the essay. I sit down, grade those essays, type the comments up, put the grades into my grading sheet, and then take a break of at least 45 minutes. This is part of the SMART goal system (Specific, Measurable, Attainable, Relevant, Time-bound). It really does help make the grading feel achievable while also ensuring that you are giving your mind a break every one in a while. Once you’ve finished your batch, either set them aside in a different location or put a tick or some kind of mark on them so you can easily tell that they are all finished.

7) When in Doubt, Roll Up

Many essays seem to fall in a valley between one grade and the next, like when you’re not sure if it’s a B- or a B. In these cases, I almost always roll up. This was advice that I got when I was a TA, and it stuck with me. Try to give your students the benefit of the doubt. Remember that university is hard. Many students take multiple classes and/or work while in school. If you are dealing with a paper on the borderline between one grade and the next, or your paper is within 1 to 2% of rolling to the next letter grade, then just bump the grade. It’s always better to err on the side of generosity. And giving someone a 69.5% instead of a 70% is just a bit of a dick move.

8) Don’t Waste Your Time

There will be essays that are so bad that they defy all explanation. Either there are no footnotes or bibliography, the essay is 3 pages when it was supposed to be 8, or the student just completely ignored your instructions. In other words, it’s obvious that the student just doesn’t care. Don’t waste your time commenting on these papers. If your student can’t be bothered to read the instructions, then you have no obligation to spend your precious time marking the paper. I usually place a comment to the effect of: “I would strongly recommend that you review the requirements for this assignment, which can be found on the Research Assignment Instructions sheet.” I find that this is firm, but fair. Save your energy for the students who really put effort into their papers, even when they don’t succeed.

9) If You Don’t Have Anything Nice to Say, Say Something Nice Anyways

Students are humans (though it’s easy to forget this sometimes…), and respond best to positive reinforcement. So try to find something good to say about the essay. Some suggestions, courtesy of my good friend Clare include: “Nice margins!” “Excellent choice of font!” On a more serious note, I usually go with something like “This is a great effort!” or “I can see that you are trying here!” I always use the positive-negative-positive sandwich. Put a positive comment, then a negative comment, and then another positive comment. This tends to motivate students to do better rather than just feel defeated. Remember, your job is to encourage students to learn, so make them feel like you are invested in their success.

Expert Tip: One variation on the positive-negative-positive sandwich comes courtesy of my friend Teva Vidal: “The “shit sandwich” is for kids who deserve detailed feedback but who just missed the mark: start off with the main strengths of what they wrote, then lay it on thick with what they screwed up, then end on a positive note in terms of how they can use what they’ve already got going for them to make it better in the future.

10) Try to find some joy in the work

You know how “Time flies when you’re having fun”? Well, this approach can help with marking. Try to have a sense of humour about the whole thing. There will be times when you become angry or frustrated because it seems like students are ignoring your instructions and therefore losing marks unnecessarily. Laughing this off will help. Some professors like to collect so-called “dumb” sentences and post them online. There are a number of ethical problems with that that I will not get into here. But I can and have shared them with my husband when I’m grading in the room with him. We can laugh together and I blow off steam (Saving your marriage through marking! I can see my husband laughing right now). I also like to mark with a bright pink pen, since it’s hard to get mad when you’re writing in pink ink.

—————————————-

So those are my suggestions for making the grading of essays a little more pleasant. I think the most important takeaway is that it’s worth spending the time to create tools. For many years, I would waste time researching lists of sources, writing out the same comments, and using a calculator. But my time, and yours, is precious, so work smart, not hard (this is becoming something of a motto…). Any other tips for grading essays quickly and efficiently? Let me know in the comments below!

So You Want to be a Sessional

essays expert tip grading Sessional work smart not hard

November 18, 2017 at 7:59 am

Many thanks for this! Found it really useful while I’m grading my mid-terms 🙂 The comment about imposter syndrome resonated with me – I’m always second guessing if I should grade higher or lower, or leave it. Most times, I re-read the essay and see that my grading was actually fair first time around.

November 18, 2017 at 5:00 pm

Same here! I still struggle with this, and I’ve been teaching for nearly ten years! Glad I could help!

October 16, 2019 at 3:32 pm

Im a new tertiary level lecturer and I am finding marking the most insightful way to udnerstand how students think. Some of the papers I have marked recently have been indescribable, incomprehnsible and just mere reflections of what I am defining as ‘laziness’. To justify this definition I thought long and hard and finally realised that if it took me truck loads of hours to get it right on essay writing, and to Masters level thats a lot of assignments.  So when I really feel confused I reflect back on my own learning experiences and use that as a secondary standard with the marking rubric the primary standard…I refuse to compromise my standards of learning just to enable a lazy student to maintain theirs.

2 Pingbacks

• To Read or Not to Read? Essay Grading Tips (Graphic) | dino sossi
• Grading Essays Faster and Easier with These 20 Spectacular Tips

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Notify me of follow-up comments by email.

Notify me of new posts by email.

This site uses Akismet to reduce spam. Learn how your comment data is processed .

Contact Me!

unwrittenhistories at gmail dot com

New blog posts every Tuesday and Canadian history roundups every Sunday!

If you would like permission to republish any blog posts from Unwritten Histories, in whole or in part, please contact me at the address above.

Recent Popular Posts

• The Halloween Special - Witchcraft in Canada
• What Should I Call My Professor?
• A Guide to Online Resources for Teaching and Learning Loyalist History
• A Guide to Online Resources for Teaching and Learning about WW1 in Canada
• View unwrittenhistories’s profile on Facebook
• View AndreaEidinger’s profile on Twitter
• View unwrittenhistories’s profile on Instagram
• View andreaeidinger’s profile on LinkedIn

Subscribe to Blog via Email

Email Address

My Twitter Feed

• RSS - Posts

© 2024 Unwritten Histories

Theme by Anders Noren — Up ↑

Center for Teaching

Grading student work.

Print Version

What Purposes Do Grades Serve?

Developing grading criteria, making grading more efficient, providing meaningful feedback to students.

• Maintaining Grading Consistency in Multi-Sectioned Courses

Minimizing Student Complaints about Grading

Barbara Walvoord and Virginia Anderson identify the multiple roles that grades serve:

• as an  evaluation of student work;
• as a  means of communicating to students, parents, graduate schools, professional schools, and future employers about a student’s  performance in college and potential for further success;
• as a  source of motivation to students for continued learning and improvement;
• as a  means of organizing a lesson, a unit, or a semester in that grades mark transitions in a course and bring closure to it.

Additionally, grading provides students with feedback on their own learning , clarifying for them what they understand, what they don’t understand, and where they can improve. Grading also provides feedback to instructors on their students’ learning , information that can inform future teaching decisions.

Why is grading often a challenge? Because grades are used as evaluations of student work, it’s important that grades accurately reflect the quality of student work and that student work is graded fairly. Grading with accuracy and fairness can take a lot of time, which is often in short supply for college instructors. Students who aren’t satisfied with their grades can sometimes protest their grades in ways that cause headaches for instructors. Also, some instructors find that their students’ focus or even their own focus on assigning numbers to student work gets in the way of promoting actual learning.

Given all that grades do and represent, it’s no surprise that they are a source of anxiety for students and that grading is often a stressful process for instructors.

Incorporating the strategies below will not eliminate the stress of grading for instructors, but it will decrease that stress and make the process of grading seem less arbitrary — to instructors and students alike.

Source: Walvoord, B. & V. Anderson (1998).  Effective Grading: A Tool for Learning and Assessment . San Francisco : Jossey-Bass.

• Consider the different kinds of work you’ll ask students to do for your course.  This work might include: quizzes, examinations, lab reports, essays, class participation, and oral presentations.
• For the work that’s most significant to you and/or will carry the most weight, identify what’s most important to you.  Is it clarity? Creativity? Rigor? Thoroughness? Precision? Demonstration of knowledge? Critical inquiry?
• Transform the characteristics you’ve identified into grading criteria for the work most significant to you, distinguishing excellent work (A-level) from very good (B-level), fair to good (C-level), poor (D-level), and unacceptable work.

Developing criteria may seem like a lot of work, but having clear criteria can

• save time in the grading process
• make that process more consistent and fair
• communicate your expectations to students
• help you to decide what and how to teach
• help students understand how their work is graded

Sample criteria are available via the following link.

• Analytic Rubrics from the CFT’s September 2010 Virtual Brownbag
• Create assignments that have clear goals and criteria for assessment.  The better students understand what you’re asking them to do the more likely they’ll do it!
• letter grades with pluses and minuses (for papers, essays, essay exams, etc.)
• 100-point numerical scale (for exams, certain types of projects, etc.)
• check +, check, check- (for quizzes, homework, response papers, quick reports or presentations, etc.)
• pass-fail or credit-no-credit (for preparatory work)
• Limit your comments or notations to those your students can use for further learning or improvement.
• Spend more time on guiding students in the process of doing work than on grading it.
• For each significant assignment, establish a grading schedule and stick to it.

Light Grading – Bear in mind that not every piece of student work may need your full attention. Sometimes it’s sufficient to grade student work on a simplified scale (minus / check / check-plus or even zero points / one point) to motivate them to engage in the work you want them to do. In particular, if you have students do some small assignment before class, you might not need to give them much feedback on that assignment if you’re going to discuss it in class.

Multiple-Choice Questions – These are easy to grade but can be challenging to write. Look for common student misconceptions and misunderstandings you can use to construct answer choices for your multiple-choice questions, perhaps by looking for patterns in student responses to past open-ended questions. And while multiple-choice questions are great for assessing recall of factual information, they can also work well to assess conceptual understanding and applications.

Test Corrections – Giving students points back for test corrections motivates them to learn from their mistakes, which can be critical in a course in which the material on one test is important for understanding material later in the term. Moreover, test corrections can actually save time grading, since grading the test the first time requires less feedback to students and grading the corrections often goes quickly because the student responses are mostly correct.

Spreadsheets – Many instructors use spreadsheets (e.g. Excel) to keep track of student grades. A spreadsheet program can automate most or all of the calculations you might need to perform to compute student grades. A grading spreadsheet can also reveal informative patterns in student grades. To learn a few tips and tricks for using Excel as a gradebook take a look at this sample Excel gradebook .

• Use your comments to teach rather than to justify your grade, focusing on what you’d most like students to address in future work.
• Link your comments and feedback to the goals for an assignment.
• Comment primarily on patterns — representative strengths and weaknesses.
• Avoid over-commenting or “picking apart” students’ work.
• In your final comments, ask questions that will guide further inquiry by students rather than provide answers for them.

Maintaining Grading Consistency in Multi-sectioned Courses (for course heads)

• Communicate your grading policies, standards, and criteria to teaching assistants, graders, and students in your course.
• Discuss your expectations about all facets of grading (criteria, timeliness, consistency, grade disputes, etc) with your teaching assistants and graders.
• Encourage teaching assistants and graders to share grading concerns and questions with you.
• have teaching assistants grade assignments for students not in their section or lab to curb favoritism (N.B. this strategy puts the emphasis on the evaluative, rather than the teaching, function of grading);
• have each section of an exam graded by only one teaching assistant or grader to ensure consistency across the board;
• have teaching assistants and graders grade student work at the same time in the same place so they can compare their grades on certain sections and arrive at consensus.
• Include your grading policies, procedures, and standards in your syllabus.
• Avoid modifying your policies, including those on late work, once you’ve communicated them to students.
• Distribute your grading criteria to students at the beginning of the term and remind them of the relevant criteria when assigning and returning work.
• Keep in-class discussion of grades to a minimum, focusing rather on course learning goals.

For a comprehensive look at grading, see the chapter “Grading Practices” from Barbara Gross Davis’s  Tools for Teaching.

Teaching Guides

• Online Course Development Resources
• Principles & Frameworks
• Pedagogies & Strategies
• Reflecting & Assessing
• Challenges & Opportunities
• Populations & Contexts

Quick Links

• Services for Departments and Schools
• Examples of Online Instructional Modules

Creating and Scoring Essay Tests

FatCamera / Getty Images

• Tips & Strategies
• An Introduction to Teaching
• Policies & Discipline
• Community Involvement
• School Administration
• Technology in the Classroom
• Teaching Adult Learners
• Issues In Education
• Teaching Resources
• Becoming A Teacher
• Assessments & Tests
• Elementary Education
• Secondary Education
• Special Education
• Homeschooling
• M.Ed., Curriculum and Instruction, University of Florida
• B.A., History, University of Florida

Essay tests are useful for teachers when they want students to select, organize, analyze, synthesize, and/or evaluate information. In other words, they rely on the upper levels of Bloom's Taxonomy . There are two types of essay questions: restricted and extended response.

• Restricted Response - These essay questions limit what the student will discuss in the essay based on the wording of the question. For example, "State the main differences between John Adams' and Thomas Jefferson's beliefs about federalism," is a restricted response. What the student is to write about has been expressed to them within the question.
• Extended Response - These allow students to select what they wish to include in order to answer the question. For example, "In Of Mice and Men , was George's killing of Lennie justified? Explain your answer." The student is given the overall topic, but they are free to use their own judgment and integrate outside information to help support their opinion.

Student Skills Required for Essay Tests

Before expecting students to perform well on either type of essay question, we must make sure that they have the required skills to excel. Following are four skills that students should have learned and practiced before taking essay exams:

• The ability to select appropriate material from the information learned in order to best answer the question.
• The ability to organize that material in an effective manner.
• The ability to show how ideas relate and interact in a specific context.
• The ability to write effectively in both sentences and paragraphs.

Constructing an Effective Essay Question

Following are a few tips to help in the construction of effective essay questions:

• Begin with the lesson objectives in mind. Make sure to know what you wish the student to show by answering the essay question.
• Decide if your goal requires a restricted or extended response. In general, if you wish to see if the student can synthesize and organize the information that they learned, then restricted response is the way to go. However, if you wish them to judge or evaluate something using the information taught during class, then you will want to use the extended response.
• If you are including more than one essay, be cognizant of time constraints. You do not want to punish students because they ran out of time on the test.
• Write the question in a novel or interesting manner to help motivate the student.
• State the number of points that the essay is worth. You can also provide them with a time guideline to help them as they work through the exam.
• If your essay item is part of a larger objective test, make sure that it is the last item on the exam.

Scoring the Essay Item

One of the downfalls of essay tests is that they lack in reliability. Even when teachers grade essays with a well-constructed rubric, subjective decisions are made. Therefore, it is important to try and be as reliable as possible when scoring your essay items. Here are a few tips to help improve reliability in grading:

• Determine whether you will use a holistic or analytic scoring system before you write your rubric . With the holistic grading system, you evaluate the answer as a whole, rating papers against each other. With the analytic system, you list specific pieces of information and award points for their inclusion.
• Prepare the essay rubric in advance. Determine what you are looking for and how many points you will be assigning for each aspect of the question.
• Avoid looking at names. Some teachers have students put numbers on their essays to try and help with this.
• Score one item at a time. This helps ensure that you use the same thinking and standards for all students.
• Avoid interruptions when scoring a specific question. Again, consistency will be increased if you grade the same item on all the papers in one sitting.
• If an important decision like an award or scholarship is based on the score for the essay, obtain two or more independent readers.
• Beware of negative influences that can affect essay scoring. These include handwriting and writing style bias, the length of the response, and the inclusion of irrelevant material.
• Review papers that are on the borderline a second time before assigning a final grade.
• Utilizing Extended Response Items to Enhance Student Learning
• Study for an Essay Test
• How to Create a Rubric in 6 Steps
• Top 10 Tips for Passing the AP US History Exam
• UC Personal Statement Prompt #1
• Tips to Create Effective Matching Questions for Assessments
• Self Assessment and Writing a Graduate Admissions Essay
• 10 Common Test Mistakes
• ACT Format: What to Expect on the Exam
• The Computer-Based GED Test
• GMAT Exam Structure, Timing, and Scoring
• Tips to Cut Writing Assignment Grading Time
• 5 Tips for a College Admissions Essay on an Important Issue
• Ideal College Application Essay Length
• SAT Sections, Sample Questions and Strategies
• What You Need to Know About the Executive Assessment

An automated essay scoring systems: a systematic literature review

• Published: 23 September 2021
• Volume 55 , pages 2495–2527, ( 2022 )

Cite this article

• Dadi Ramesh   ORCID: orcid.org/0000-0002-3967-8914 1 , 2 &
• Suresh Kumar Sanampudi 3  

38k Accesses

94 Citations

5 Altmetric

Explore all metrics

Assessment in the Education system plays a significant role in judging student performance. The present evaluation system is through human assessment. As the number of teachers' student ratio is gradually increasing, the manual evaluation process becomes complicated. The drawback of manual evaluation is that it is time-consuming, lacks reliability, and many more. This connection online examination system evolved as an alternative tool for pen and paper-based methods. Present Computer-based evaluation system works only for multiple-choice questions, but there is no proper evaluation system for grading essays and short answers. Many researchers are working on automated essay grading and short answer scoring for the last few decades, but assessing an essay by considering all parameters like the relevance of the content to the prompt, development of ideas, Cohesion, and Coherence is a big challenge till now. Few researchers focused on Content-based evaluation, while many of them addressed style-based assessment. This paper provides a systematic literature review on automated essay scoring systems. We studied the Artificial Intelligence and Machine Learning techniques used to evaluate automatic essay scoring and analyzed the limitations of the current studies and research trends. We observed that the essay evaluation is not done based on the relevance of the content and coherence.

Similar content being viewed by others

Automated Essay Scoring Systems

Automated Essay Scoring System Based on Rubric

Avoid common mistakes on your manuscript.

1 Introduction

Due to COVID 19 outbreak, an online educational system has become inevitable. In the present scenario, almost all the educational institutions ranging from schools to colleges adapt the online education system. The assessment plays a significant role in measuring the learning ability of the student. Most automated evaluation is available for multiple-choice questions, but assessing short and essay answers remain a challenge. The education system is changing its shift to online-mode, like conducting computer-based exams and automatic evaluation. It is a crucial application related to the education domain, which uses natural language processing (NLP) and Machine Learning techniques. The evaluation of essays is impossible with simple programming languages and simple techniques like pattern matching and language processing. Here the problem is for a single question, we will get more responses from students with a different explanation. So, we need to evaluate all the answers concerning the question.

Automated essay scoring (AES) is a computer-based assessment system that automatically scores or grades the student responses by considering appropriate features. The AES research started in 1966 with the Project Essay Grader (PEG) by Ajay et al. ( 1973 ). PEG evaluates the writing characteristics such as grammar, diction, construction, etc., to grade the essay. A modified version of the PEG by Shermis et al. ( 2001 ) was released, which focuses on grammar checking with a correlation between human evaluators and the system. Foltz et al. ( 1999 ) introduced an Intelligent Essay Assessor (IEA) by evaluating content using latent semantic analysis to produce an overall score. Powers et al. ( 2002 ) proposed E-rater and Intellimetric by Rudner et al. ( 2006 ) and Bayesian Essay Test Scoring System (BESTY) by Rudner and Liang ( 2002 ), these systems use natural language processing (NLP) techniques that focus on style and content to obtain the score of an essay. The vast majority of the essay scoring systems in the 1990s followed traditional approaches like pattern matching and a statistical-based approach. Since the last decade, the essay grading systems started using regression-based and natural language processing techniques. AES systems like Dong et al. ( 2017 ) and others developed from 2014 used deep learning techniques, inducing syntactic and semantic features resulting in better results than earlier systems.

Ohio, Utah, and most US states are using AES systems in school education, like Utah compose tool, Ohio standardized test (an updated version of PEG), evaluating millions of student's responses every year. These systems work for both formative, summative assessments and give feedback to students on the essay. Utah provided basic essay evaluation rubrics (six characteristics of essay writing): Development of ideas, organization, style, word choice, sentence fluency, conventions. Educational Testing Service (ETS) has been conducting significant research on AES for more than a decade and designed an algorithm to evaluate essays on different domains and providing an opportunity for test-takers to improve their writing skills. In addition, they are current research content-based evaluation.

The evaluation of essay and short answer scoring should consider the relevance of the content to the prompt, development of ideas, Cohesion, Coherence, and domain knowledge. Proper assessment of the parameters mentioned above defines the accuracy of the evaluation system. But all these parameters cannot play an equal role in essay scoring and short answer scoring. In a short answer evaluation, domain knowledge is required, like the meaning of "cell" in physics and biology is different. And while evaluating essays, the implementation of ideas with respect to prompt is required. The system should also assess the completeness of the responses and provide feedback.

Several studies examined AES systems, from the initial to the latest AES systems. In which the following studies on AES systems are Blood ( 2011 ) provided a literature review from PEG 1984–2010. Which has covered only generalized parts of AES systems like ethical aspects, the performance of the systems. Still, they have not covered the implementation part, and it’s not a comparative study and has not discussed the actual challenges of AES systems.

Burrows et al. ( 2015 ) Reviewed AES systems on six dimensions like dataset, NLP techniques, model building, grading models, evaluation, and effectiveness of the model. They have not covered feature extraction techniques and challenges in features extractions. Covered only Machine Learning models but not in detail. This system not covered the comparative analysis of AES systems like feature extraction, model building, and level of relevance, cohesion, and coherence not covered in this review.

Ke et al. ( 2019 ) provided a state of the art of AES system but covered very few papers and not listed all challenges, and no comparative study of the AES model. On the other hand, Hussein et al. in ( 2019 ) studied two categories of AES systems, four papers from handcrafted features for AES systems, and four papers from the neural networks approach, discussed few challenges, and did not cover feature extraction techniques, the performance of AES models in detail.

Klebanov et al. ( 2020 ). Reviewed 50 years of AES systems, listed and categorized all essential features that need to be extracted from essays. But not provided a comparative analysis of all work and not discussed the challenges.

This paper aims to provide a systematic literature review (SLR) on automated essay grading systems. An SLR is an Evidence-based systematic review to summarize the existing research. It critically evaluates and integrates all relevant studies' findings and addresses the research domain's specific research questions. Our research methodology uses guidelines given by Kitchenham et al. ( 2009 ) for conducting the review process; provide a well-defined approach to identify gaps in current research and to suggest further investigation.

We addressed our research method, research questions, and the selection process in Sect.  2 , and the results of the research questions have discussed in Sect.  3 . And the synthesis of all the research questions addressed in Sect.  4 . Conclusion and possible future work discussed in Sect.  5 .

2 Research method

We framed the research questions with PICOC criteria.

Population (P) Student essays and answers evaluation systems.

Intervention (I) evaluation techniques, data sets, features extraction methods.

Comparison (C) Comparison of various approaches and results.

Outcomes (O) Estimate the accuracy of AES systems,

Context (C) NA.

2.1 Research questions

To collect and provide research evidence from the available studies in the domain of automated essay grading, we framed the following research questions (RQ):

RQ1 what are the datasets available for research on automated essay grading?

The answer to the question can provide a list of the available datasets, their domain, and access to the datasets. It also provides a number of essays and corresponding prompts.

RQ2 what are the features extracted for the assessment of essays?

The answer to the question can provide an insight into various features so far extracted, and the libraries used to extract those features.

RQ3, which are the evaluation metrics available for measuring the accuracy of algorithms?

The answer will provide different evaluation metrics for accurate measurement of each Machine Learning approach and commonly used measurement technique.

RQ4 What are the Machine Learning techniques used for automatic essay grading, and how are they implemented?

It can provide insights into various Machine Learning techniques like regression models, classification models, and neural networks for implementing essay grading systems. The response to the question can give us different assessment approaches for automated essay grading systems.

RQ5 What are the challenges/limitations in the current research?

The answer to the question provides limitations of existing research approaches like cohesion, coherence, completeness, and feedback.

2.2 Search process

We conducted an automated search on well-known computer science repositories like ACL, ACM, IEEE Explore, Springer, and Science Direct for an SLR. We referred to papers published from 2010 to 2020 as much of the work during these years focused on advanced technologies like deep learning and natural language processing for automated essay grading systems. Also, the availability of free data sets like Kaggle (2012), Cambridge Learner Corpus-First Certificate in English exam (CLC-FCE) by Yannakoudakis et al. ( 2011 ) led to research this domain.

Search Strings : We used search strings like “Automated essay grading” OR “Automated essay scoring” OR “short answer scoring systems” OR “essay scoring systems” OR “automatic essay evaluation” and searched on metadata.

2.3 Selection criteria

After collecting all relevant documents from the repositories, we prepared selection criteria for inclusion and exclusion of documents. With the inclusion and exclusion criteria, it becomes more feasible for the research to be accurate and specific.

Inclusion criteria 1 Our approach is to work with datasets comprise of essays written in English. We excluded the essays written in other languages.

Inclusion criteria 2  We included the papers implemented on the AI approach and excluded the traditional methods for the review.

Inclusion criteria 3 The study is on essay scoring systems, so we exclusively included the research carried out on only text data sets rather than other datasets like image or speech.

Exclusion criteria  We removed the papers in the form of review papers, survey papers, and state of the art papers.

2.4 Quality assessment

In addition to the inclusion and exclusion criteria, we assessed each paper by quality assessment questions to ensure the article's quality. We included the documents that have clearly explained the approach they used, the result analysis and validation.

The quality checklist questions are framed based on the guidelines from Kitchenham et al. ( 2009 ). Each quality assessment question was graded as either 1 or 0. The final score of the study range from 0 to 3. A cut off score for excluding a study from the review is 2 points. Since the papers scored 2 or 3 points are included in the final evaluation. We framed the following quality assessment questions for the final study.

Quality Assessment 1: Internal validity.

Quality Assessment 2: External validity.

Quality Assessment 3: Bias.

The two reviewers review each paper to select the final list of documents. We used the Quadratic Weighted Kappa score to measure the final agreement between the two reviewers. The average resulted from the kappa score is 0.6942, a substantial agreement between the reviewers. The result of evolution criteria shown in Table 1 . After Quality Assessment, the final list of papers for review is shown in Table 2 . The complete selection process is shown in Fig. 1 . The total number of selected papers in year wise as shown in Fig. 2 .

Selection process

Year wise publications

3.1 What are the datasets available for research on automated essay grading?

To work with problem statement especially in Machine Learning and deep learning domain, we require considerable amount of data to train the models. To answer this question, we listed all the data sets used for training and testing for automated essay grading systems. The Cambridge Learner Corpus-First Certificate in English exam (CLC-FCE) Yannakoudakis et al. ( 2011 ) developed corpora that contain 1244 essays and ten prompts. This corpus evaluates whether a student can write the relevant English sentences without any grammatical and spelling mistakes. This type of corpus helps to test the models built for GRE and TOFEL type of exams. It gives scores between 1 and 40.

Bailey and Meurers ( 2008 ), Created a dataset (CREE reading comprehension) for language learners and automated short answer scoring systems. The corpus consists of 566 responses from intermediate students. Mohler and Mihalcea ( 2009 ). Created a dataset for the computer science domain consists of 630 responses for data structure assignment questions. The scores are range from 0 to 5 given by two human raters.

Dzikovska et al. ( 2012 ) created a Student Response Analysis (SRA) corpus. It consists of two sub-groups: the BEETLE corpus consists of 56 questions and approximately 3000 responses from students in the electrical and electronics domain. The second one is the SCIENTSBANK(SemEval-2013) (Dzikovska et al. 2013a ; b ) corpus consists of 10,000 responses on 197 prompts on various science domains. The student responses ladled with "correct, partially correct incomplete, Contradictory, Irrelevant, Non-domain."

In the Kaggle (2012) competition, released total 3 types of corpuses on an Automated Student Assessment Prize (ASAP1) (“ https://www.kaggle.com/c/asap-sas/ ” ) essays and short answers. It has nearly 17,450 essays, out of which it provides up to 3000 essays for each prompt. It has eight prompts that test 7th to 10th grade US students. It gives scores between the [0–3] and [0–60] range. The limitations of these corpora are: (1) it has a different score range for other prompts. (2) It uses statistical features such as named entities extraction and lexical features of words to evaluate essays. ASAP +  + is one more dataset from Kaggle. It is with six prompts, and each prompt has more than 1000 responses total of 10,696 from 8th-grade students. Another corpus contains ten prompts from science, English domains and a total of 17,207 responses. Two human graders evaluated all these responses.

Correnti et al. ( 2013 ) created a Response-to-Text Assessment (RTA) dataset used to check student writing skills in all directions like style, mechanism, and organization. 4–8 grade students give the responses to RTA. Basu et al. ( 2013 ) created a power grading dataset with 700 responses for ten different prompts from US immigration exams. It contains all short answers for assessment.

The TOEFL11 corpus Blanchard et al. ( 2013 ) contains 1100 essays evenly distributed over eight prompts. It is used to test the English language skills of a candidate attending the TOFEL exam. It scores the language proficiency of a candidate as low, medium, and high.

International Corpus of Learner English (ICLE) Granger et al. ( 2009 ) built a corpus of 3663 essays covering different dimensions. It has 12 prompts with 1003 essays that test the organizational skill of essay writing, and13 prompts, each with 830 essays that examine the thesis clarity and prompt adherence.

Argument Annotated Essays (AAE) Stab and Gurevych ( 2014 ) developed a corpus that contains 102 essays with 101 prompts taken from the essayforum2 site. It tests the persuasive nature of the student essay. The SCIENTSBANK corpus used by Sakaguchi et al. ( 2015 ) available in git-hub, containing 9804 answers to 197 questions in 15 science domains. Table 3 illustrates all datasets related to AES systems.

3.2 RQ2 what are the features extracted for the assessment of essays?

Features play a major role in the neural network and other supervised Machine Learning approaches. The automatic essay grading systems scores student essays based on different types of features, which play a prominent role in training the models. Based on their syntax and semantics and they are categorized into three groups. 1. statistical-based features Contreras et al. ( 2018 ); Kumar et al. ( 2019 ); Mathias and Bhattacharyya ( 2018a ; b ) 2. Style-based (Syntax) features Cummins et al. ( 2016 ); Darwish and Mohamed ( 2020 ); Ke et al. ( 2019 ). 3. Content-based features Dong et al. ( 2017 ). A good set of features appropriate models evolved better AES systems. The vast majority of the researchers are using regression models if features are statistical-based. For Neural Networks models, researches are using both style-based and content-based features. The following table shows the list of various features used in existing AES Systems. Table 4 represents all set of features used for essay grading.

We studied all the feature extracting NLP libraries as shown in Fig. 3 . that are used in the papers. The NLTK is an NLP tool used to retrieve statistical features like POS, word count, sentence count, etc. With NLTK, we can miss the essay's semantic features. To find semantic features Word2Vec Mikolov et al. ( 2013 ), GloVe Jeffrey Pennington et al. ( 2014 ) is the most used libraries to retrieve the semantic text from the essays. And in some systems, they directly trained the model with word embeddings to find the score. From Fig. 4 as observed that non-content-based feature extraction is higher than content-based.

Usages of tools

Number of papers on content based features

3.3 RQ3 which are the evaluation metrics available for measuring the accuracy of algorithms?

The majority of the AES systems are using three evaluation metrics. They are (1) quadrated weighted kappa (QWK) (2) Mean Absolute Error (MAE) (3) Pearson Correlation Coefficient (PCC) Shehab et al. ( 2016 ). The quadratic weighted kappa will find agreement between human evaluation score and system evaluation score and produces value ranging from 0 to 1. And the Mean Absolute Error is the actual difference between human-rated score to system-generated score. The mean square error (MSE) measures the average squares of the errors, i.e., the average squared difference between the human-rated and the system-generated scores. MSE will always give positive numbers only. Pearson's Correlation Coefficient (PCC) finds the correlation coefficient between two variables. It will provide three values (0, 1, − 1). "0" represents human-rated and system scores that are not related. "1" represents an increase in the two scores. "− 1" illustrates a negative relationship between the two scores.

3.4 RQ4 what are the Machine Learning techniques being used for automatic essay grading, and how are they implemented?

After scrutinizing all documents, we categorize the techniques used in automated essay grading systems into four baskets. 1. Regression techniques. 2. Classification model. 3. Neural networks. 4. Ontology-based approach.

All the existing AES systems developed in the last ten years employ supervised learning techniques. Researchers using supervised methods viewed the AES system as either regression or classification task. The goal of the regression task is to predict the score of an essay. The classification task is to classify the essays belonging to (low, medium, or highly) relevant to the question's topic. Since the last three years, most AES systems developed made use of the concept of the neural network.

3.4.1 Regression based models

Mohler and Mihalcea ( 2009 ). proposed text-to-text semantic similarity to assign a score to the student essays. There are two text similarity measures like Knowledge-based measures, corpus-based measures. There eight knowledge-based tests with all eight models. They found the similarity. The shortest path similarity determines based on the length, which shortest path between two contexts. Leacock & Chodorow find the similarity based on the shortest path's length between two concepts using node-counting. The Lesk similarity finds the overlap between the corresponding definitions, and Wu & Palmer algorithm finds similarities based on the depth of two given concepts in the wordnet taxonomy. Resnik, Lin, Jiang&Conrath, Hirst& St-Onge find the similarity based on different parameters like the concept, probability, normalization factor, lexical chains. In corpus-based likeness, there LSA BNC, LSA Wikipedia, and ESA Wikipedia, latent semantic analysis is trained on Wikipedia and has excellent domain knowledge. Among all similarity scores, correlation scores LSA Wikipedia scoring accuracy is more. But these similarity measure algorithms are not using NLP concepts. These models are before 2010 and basic concept models to continue the research automated essay grading with updated algorithms on neural networks with content-based features.

Adamson et al. ( 2014 ) proposed an automatic essay grading system which is a statistical-based approach in this they retrieved features like POS, Character count, Word count, Sentence count, Miss spelled words, n-gram representation of words to prepare essay vector. They formed a matrix with these all vectors in that they applied LSA to give a score to each essay. It is a statistical approach that doesn’t consider the semantics of the essay. The accuracy they got when compared to the human rater score with the system is 0.532.

Cummins et al. ( 2016 ). Proposed Timed Aggregate Perceptron vector model to give ranking to all the essays, and later they converted the rank algorithm to predict the score of the essay. The model trained with features like Word unigrams, bigrams, POS, Essay length, grammatical relation, Max word length, sentence length. It is multi-task learning, gives ranking to the essays, and predicts the score for the essay. The performance evaluated through QWK is 0.69, a substantial agreement between the human rater and the system.

Sultan et al. ( 2016 ). Proposed a Ridge regression model to find short answer scoring with Question Demoting. Question Demoting is the new concept included in the essay's final assessment to eliminate duplicate words from the essay. The extracted features are Text Similarity, which is the similarity between the student response and reference answer. Question Demoting is the number of repeats in a student response. With inverse document frequency, they assigned term weight. The sentence length Ratio is the number of words in the student response, is another feature. With these features, the Ridge regression model was used, and the accuracy they got 0.887.

Contreras et al. ( 2018 ). Proposed Ontology based on text mining in this model has given a score for essays in phases. In phase-I, they generated ontologies with ontoGen and SVM to find the concept and similarity in the essay. In phase II from ontologies, they retrieved features like essay length, word counts, correctness, vocabulary, and types of word used, domain information. After retrieving statistical data, they used a linear regression model to find the score of the essay. The accuracy score is the average of 0.5.

Darwish and Mohamed ( 2020 ) proposed the fusion of fuzzy Ontology with LSA. They retrieve two types of features, like syntax features and semantic features. In syntax features, they found Lexical Analysis with tokens, and they construct a parse tree. If the parse tree is broken, the essay is inconsistent—a separate grade assigned to the essay concerning syntax features. The semantic features are like similarity analysis, Spatial Data Analysis. Similarity analysis is to find duplicate sentences—Spatial Data Analysis for finding Euclid distance between the center and part. Later they combine syntax features and morphological features score for the final score. The accuracy they achieved with the multiple linear regression model is 0.77, mostly on statistical features.

Süzen Neslihan et al. ( 2020 ) proposed a text mining approach for short answer grading. First, their comparing model answers with student response by calculating the distance between two sentences. By comparing the model answer with student response, they find the essay's completeness and provide feedback. In this approach, model vocabulary plays a vital role in grading, and with this model vocabulary, the grade will be assigned to the student's response and provides feedback. The correlation between the student answer to model answer is 0.81.

3.4.2 Classification based Models

Persing and Ng ( 2013 ) used a support vector machine to score the essay. The features extracted are OS, N-gram, and semantic text to train the model and identified the keywords from the essay to give the final score.

Sakaguchi et al. ( 2015 ) proposed two methods: response-based and reference-based. In response-based scoring, the extracted features are response length, n-gram model, and syntactic elements to train the support vector regression model. In reference-based scoring, features such as sentence similarity using word2vec is used to find the cosine similarity of the sentences that is the final score of the response. First, the scores were discovered individually and later combined two features to find a final score. This system gave a remarkable increase in performance by combining the scores.

Mathias and Bhattacharyya ( 2018a ; b ) Proposed Automated Essay Grading Dataset with Essay Attribute Scores. The first concept features selection depends on the essay type. So the common attributes are Content, Organization, Word Choice, Sentence Fluency, Conventions. In this system, each attribute is scored individually, with the strength of each attribute identified. The model they used is a random forest classifier to assign scores to individual attributes. The accuracy they got with QWK is 0.74 for prompt 1 of the ASAS dataset ( https://www.kaggle.com/c/asap-sas/ ).

Ke et al. ( 2019 ) used a support vector machine to find the response score. In this method, features like Agreeability, Specificity, Clarity, Relevance to prompt, Conciseness, Eloquence, Confidence, Direction of development, Justification of opinion, and Justification of importance. First, the individual parameter score obtained was later combined with all scores to give a final response score. The features are used in the neural network to find whether the sentence is relevant to the topic or not.

Salim et al. ( 2019 ) proposed an XGBoost Machine Learning classifier to assess the essays. The algorithm trained on features like word count, POS, parse tree depth, and coherence in the articles with sentence similarity percentage; cohesion and coherence are considered for training. And they implemented K-fold cross-validation for a result the average accuracy after specific validations is 68.12.

3.4.3 Neural network models

Shehab et al. ( 2016 ) proposed a neural network method that used learning vector quantization to train human scored essays. After training, the network can provide a score to the ungraded essays. First, we should process the essay to remove Spell checking and then perform preprocessing steps like Document Tokenization, stop word removal, Stemming, and submit it to the neural network. Finally, the model will provide feedback on the essay, whether it is relevant to the topic. And the correlation coefficient between human rater and system score is 0.7665.

Kopparapu and De ( 2016 ) proposed the Automatic Ranking of Essays using Structural and Semantic Features. This approach constructed a super essay with all the responses. Next, ranking for a student essay is done based on the super-essay. The structural and semantic features derived helps to obtain the scores. In a paragraph, 15 Structural features like an average number of sentences, the average length of sentences, and the count of words, nouns, verbs, adjectives, etc., are used to obtain a syntactic score. A similarity score is used as semantic features to calculate the overall score.

Dong and Zhang ( 2016 ) proposed a hierarchical CNN model. The model builds two layers with word embedding to represents the words as the first layer. The second layer is a word convolution layer with max-pooling to find word vectors. The next layer is a sentence-level convolution layer with max-pooling to find the sentence's content and synonyms. A fully connected dense layer produces an output score for an essay. The accuracy with the hierarchical CNN model resulted in an average QWK of 0.754.

Taghipour and Ng ( 2016 ) proposed a first neural approach for essay scoring build in which convolution and recurrent neural network concepts help in scoring an essay. The network uses a lookup table with the one-hot representation of the word vector of an essay. The final efficiency of the network model with LSTM resulted in an average QWK of 0.708.

Dong et al. ( 2017 ). Proposed an Attention-based scoring system with CNN + LSTM to score an essay. For CNN, the input parameters were character embedding and word embedding, and it has attention pooling layers and used NLTK to obtain word and character embedding. The output gives a sentence vector, which provides sentence weight. After CNN, it will have an LSTM layer with an attention pooling layer, and this final layer results in the final score of the responses. The average QWK score is 0.764.

Riordan et al. ( 2017 ) proposed a neural network with CNN and LSTM layers. Word embedding, given as input to a neural network. An LSTM network layer will retrieve the window features and delivers them to the aggregation layer. The aggregation layer is a superficial layer that takes a correct window of words and gives successive layers to predict the answer's sore. The accuracy of the neural network resulted in a QWK of 0.90.

Zhao et al. ( 2017 ) proposed a new concept called Memory-Augmented Neural network with four layers, input representation layer, memory addressing layer, memory reading layer, and output layer. An input layer represents all essays in a vector form based on essay length. After converting the word vector, the memory addressing layer takes a sample of the essay and weighs all the terms. The memory reading layer takes the input from memory addressing segment and finds the content to finalize the score. Finally, the output layer will provide the final score of the essay. The accuracy of essay scores is 0.78, which is far better than the LSTM neural network.

Mathias and Bhattacharyya ( 2018a ; b ) proposed deep learning networks using LSTM with the CNN layer and GloVe pre-trained word embeddings. For this, they retrieved features like Sentence count essays, word count per sentence, Number of OOVs in the sentence, Language model score, and the text's perplexity. The network predicted the goodness scores of each essay. The higher the goodness scores, means higher the rank and vice versa.

Nguyen and Dery ( 2016 ). Proposed Neural Networks for Automated Essay Grading. In this method, a single layer bi-directional LSTM accepting word vector as input. Glove vectors used in this method resulted in an accuracy of 90%.

Ruseti et al. ( 2018 ) proposed a recurrent neural network that is capable of memorizing the text and generate a summary of an essay. The Bi-GRU network with the max-pooling layer molded on the word embedding of each document. It will provide scoring to the essay by comparing it with a summary of the essay from another Bi-GRU network. The result obtained an accuracy of 0.55.

Wang et al. ( 2018a ; b ) proposed an automatic scoring system with the bi-LSTM recurrent neural network model and retrieved the features using the word2vec technique. This method generated word embeddings from the essay words using the skip-gram model. And later, word embedding is used to train the neural network to find the final score. The softmax layer in LSTM obtains the importance of each word. This method used a QWK score of 0.83%.

Dasgupta et al. ( 2018 ) proposed a technique for essay scoring with augmenting textual qualitative Features. It extracted three types of linguistic, cognitive, and psychological features associated with a text document. The linguistic features are Part of Speech (POS), Universal Dependency relations, Structural Well-formedness, Lexical Diversity, Sentence Cohesion, Causality, and Informativeness of the text. The psychological features derived from the Linguistic Information and Word Count (LIWC) tool. They implemented a convolution recurrent neural network that takes input as word embedding and sentence vector, retrieved from the GloVe word vector. And the second layer is the Convolution Layer to find local features. The next layer is the recurrent neural network (LSTM) to find corresponding of the text. The accuracy of this method resulted in an average QWK of 0.764.

Liang et al. ( 2018 ) proposed a symmetrical neural network AES model with Bi-LSTM. They are extracting features from sample essays and student essays and preparing an embedding layer as input. The embedding layer output is transfer to the convolution layer from that LSTM will be trained. Hear the LSRM model has self-features extraction layer, which will find the essay's coherence. The average QWK score of SBLSTMA is 0.801.

Liu et al. ( 2019 ) proposed two-stage learning. In the first stage, they are assigning a score based on semantic data from the essay. The second stage scoring is based on some handcrafted features like grammar correction, essay length, number of sentences, etc. The average score of the two stages is 0.709.

Pedro Uria Rodriguez et al. ( 2019 ) proposed a sequence-to-sequence learning model for automatic essay scoring. They used BERT (Bidirectional Encoder Representations from Transformers), which extracts the semantics from a sentence from both directions. And XLnet sequence to sequence learning model to extract features like the next sentence in an essay. With this pre-trained model, they attained coherence from the essay to give the final score. The average QWK score of the model is 75.5.

Xia et al. ( 2019 ) proposed a two-layer Bi-directional LSTM neural network for the scoring of essays. The features extracted with word2vec to train the LSTM and accuracy of the model in an average of QWK is 0.870.

Kumar et al. ( 2019 ) Proposed an AutoSAS for short answer scoring. It used pre-trained Word2Vec and Doc2Vec models trained on Google News corpus and Wikipedia dump, respectively, to retrieve the features. First, they tagged every word POS and they found weighted words from the response. It also found prompt overlap to observe how the answer is relevant to the topic, and they defined lexical overlaps like noun overlap, argument overlap, and content overlap. This method used some statistical features like word frequency, difficulty, diversity, number of unique words in each response, type-token ratio, statistics of the sentence, word length, and logical operator-based features. This method uses a random forest model to train the dataset. The data set has sample responses with their associated score. The model will retrieve the features from both responses like graded and ungraded short answers with questions. The accuracy of AutoSAS with QWK is 0.78. It will work on any topics like Science, Arts, Biology, and English.

Jiaqi Lun et al. ( 2020 ) proposed an automatic short answer scoring with BERT. In this with a reference answer comparing student responses and assigning scores. The data augmentation is done with a neural network and with one correct answer from the dataset classifying reaming responses as correct or incorrect.

Zhu and Sun ( 2020 ) proposed a multimodal Machine Learning approach for automated essay scoring. First, they count the grammar score with the spaCy library and numerical count as the number of words and sentences with the same library. With this input, they trained a single and Bi LSTM neural network for finding the final score. For the LSTM model, they prepared sentence vectors with GloVe and word embedding with NLTK. Bi-LSTM will check each sentence in both directions to find semantic from the essay. The average QWK score with multiple models is 0.70.

3.4.4 Ontology based approach

Mohler et al. ( 2011 ) proposed a graph-based method to find semantic similarity in short answer scoring. For the ranking of answers, they used the support vector regression model. The bag of words is the main feature extracted in the system.

Ramachandran et al. ( 2015 ) also proposed a graph-based approach to find lexical based semantics. Identified phrase patterns and text patterns are the features to train a random forest regression model to score the essays. The accuracy of the model in a QWK is 0.78.

Zupanc et al. ( 2017 ) proposed sentence similarity networks to find the essay's score. Ajetunmobi and Daramola ( 2017 ) recommended an ontology-based information extraction approach and domain-based ontology to find the score.

3.4.5 Speech response scoring

Automatic scoring is in two ways one is text-based scoring, other is speech-based scoring. This paper discussed text-based scoring and its challenges, and now we cover speech scoring and common points between text and speech-based scoring. Evanini and Wang ( 2013 ), Worked on speech scoring of non-native school students, extracted features with speech ratter, and trained a linear regression model, concluding that accuracy varies based on voice pitching. Loukina et al. ( 2015 ) worked on feature selection from speech data and trained SVM. Malinin et al. ( 2016 ) used neural network models to train the data. Loukina et al. ( 2017 ). Proposed speech and text-based automatic scoring. Extracted text-based features, speech-based features and trained a deep neural network for speech-based scoring. They extracted 33 types of features based on acoustic signals. Malinin et al. ( 2017 ). Wu Xixin et al. ( 2020 ) Worked on deep neural networks for spoken language assessment. Incorporated different types of models and tested them. Ramanarayanan et al. ( 2017 ) worked on feature extraction methods and extracted punctuation, fluency, and stress and trained different Machine Learning models for scoring. Knill et al. ( 2018 ). Worked on Automatic speech recognizer and its errors how its impacts the speech assessment.

3.4.5.1 The state of the art

This section provides an overview of the existing AES systems with a comparative study w. r. t models, features applied, datasets, and evaluation metrics used for building the automated essay grading systems. We divided all 62 papers into two sets of the first set of review papers in Table 5 with a comparative study of the AES systems.

3.4.6 Comparison of all approaches

In our study, we divided major AES approaches into three categories. Regression models, classification models, and neural network models. The regression models failed to find cohesion and coherence from the essay because it trained on BoW(Bag of Words) features. In processing data from input to output, the regression models are less complicated than neural networks. There are unable to find many intricate patterns from the essay and unable to find sentence connectivity. If we train the model with BoW features in the neural network approach, the model never considers the essay's coherence and coherence.

First, to train a Machine Learning algorithm with essays, all the essays are converted to vector form. We can form a vector with BoW and Word2vec, TF-IDF. The BoW and Word2vec vector representation of essays represented in Table 6 . The vector representation of BoW with TF-IDF is not incorporating the essays semantic, and it’s just statistical learning from a given vector. Word2vec vector comprises semantic of essay in a unidirectional way.

In BoW, the vector contains the frequency of word occurrences in the essay. The vector represents 1 and more based on the happenings of words in the essay and 0 for not present. So, in BoW, the vector does not maintain the relationship with adjacent words; it’s just for single words. In word2vec, the vector represents the relationship between words with other words and sentences prompt in multiple dimensional ways. But word2vec prepares vectors in a unidirectional way, not in a bidirectional way; word2vec fails to find semantic vectors when a word has two meanings, and the meaning depends on adjacent words. Table 7 represents a comparison of Machine Learning models and features extracting methods.

In AES, cohesion and coherence will check the content of the essay concerning the essay prompt these can be extracted from essay in the vector from. Two more parameters are there to access an essay is completeness and feedback. Completeness will check whether student’s response is sufficient or not though the student wrote correctly. Table 8 represents all four parameters comparison for essay grading. Table 9 illustrates comparison of all approaches based on various features like grammar, spelling, organization of essay, relevance.

3.5 What are the challenges/limitations in the current research?

From our study and results discussed in the previous sections, many researchers worked on automated essay scoring systems with numerous techniques. We have statistical methods, classification methods, and neural network approaches to evaluate the essay automatically. The main goal of the automated essay grading system is to reduce human effort and improve consistency.

The vast majority of essay scoring systems are dealing with the efficiency of the algorithm. But there are many challenges in automated essay grading systems. One should assess the essay by following parameters like the relevance of the content to the prompt, development of ideas, Cohesion, Coherence, and domain knowledge.

No model works on the relevance of content, which means whether student response or explanation is relevant to the given prompt or not if it is relevant to how much it is appropriate, and there is no discussion about the cohesion and coherence of the essays. All researches concentrated on extracting the features using some NLP libraries, trained their models, and testing the results. But there is no explanation in the essay evaluation system about consistency and completeness, But Palma and Atkinson ( 2018 ) explained coherence-based essay evaluation. And Zupanc and Bosnic ( 2014 ) also used the word coherence to evaluate essays. And they found consistency with latent semantic analysis (LSA) for finding coherence from essays, but the dictionary meaning of coherence is "The quality of being logical and consistent."

Another limitation is there is no domain knowledge-based evaluation of essays using Machine Learning models. For example, the meaning of a cell is different from biology to physics. Many Machine Learning models extract features with WordVec and GloVec; these NLP libraries cannot convert the words into vectors when they have two or more meanings.

3.5.1 Other challenges that influence the Automated Essay Scoring Systems.

All these approaches worked to improve the QWK score of their models. But QWK will not assess the model in terms of features extraction and constructed irrelevant answers. The QWK is not evaluating models whether the model is correctly assessing the answer or not. There are many challenges concerning students' responses to the Automatic scoring system. Like in evaluating approach, no model has examined how to evaluate the constructed irrelevant and adversarial answers. Especially the black box type of approaches like deep learning models provides more options to the students to bluff the automated scoring systems.

The Machine Learning models that work on statistical features are very vulnerable. Based on Powers et al. ( 2001 ) and Bejar Isaac et al. ( 2014 ), the E-rater was failed on Constructed Irrelevant Responses Strategy (CIRS). From the study of Bejar et al. ( 2013 ), Higgins and Heilman ( 2014 ), observed that when student response contain irrelevant content or shell language concurring to prompt will influence the final score of essays in an automated scoring system.

In deep learning approaches, most of the models automatically read the essay's features, and some methods work on word-based embedding and other character-based embedding features. From the study of Riordan Brain et al. ( 2019 ), The character-based embedding systems do not prioritize spelling correction. However, it is influencing the final score of the essay. From the study of Horbach and Zesch ( 2019 ), Various factors are influencing AES systems. For example, there are data set size, prompt type, answer length, training set, and human scorers for content-based scoring.

Ding et al. ( 2020 ) reviewed that the automated scoring system is vulnerable when a student response contains more words from prompt, like prompt vocabulary repeated in the response. Parekh et al. ( 2020 ) and Kumar et al. ( 2020 ) tested various neural network models of AES by iteratively adding important words, deleting unimportant words, shuffle the words, and repeating sentences in an essay and found that no change in the final score of essays. These neural network models failed to recognize common sense in adversaries' essays and give more options for the students to bluff the automated systems.

Other than NLP and ML techniques for AES. From Wresch ( 1993 ) to Madnani and Cahill ( 2018 ). discussed the complexity of AES systems, standards need to be followed. Like assessment rubrics to test subject knowledge, irrelevant responses, and ethical aspects of an algorithm like measuring the fairness of student response.

Fairness is an essential factor for automated systems. For example, in AES, fairness can be measure in an agreement between human score to machine score. Besides this, From Loukina et al. ( 2019 ), the fairness standards include overall score accuracy, overall score differences, and condition score differences between human and system scores. In addition, scoring different responses in the prospect of constructive relevant and irrelevant will improve fairness.

Madnani et al. ( 2017a ; b ). Discussed the fairness of AES systems for constructed responses and presented RMS open-source tool for detecting biases in the models. With this, one can change fairness standards according to their analysis of fairness.

From Berzak et al.'s ( 2018 ) approach, behavior factors are a significant challenge in automated scoring systems. That helps to find language proficiency, word characteristics (essential words from the text), predict the critical patterns from the text, find related sentences in an essay, and give a more accurate score.

Rupp ( 2018 ), has discussed the designing, evaluating, and deployment methodologies for AES systems. They provided notable characteristics of AES systems for deployment. They are like model performance, evaluation metrics for a model, threshold values, dynamically updated models, and framework.

First, we should check the model performance on different datasets and parameters for operational deployment. Selecting Evaluation metrics for AES models are like QWK, correlation coefficient, or sometimes both. Kelley and Preacher ( 2012 ) have discussed three categories of threshold values: marginal, borderline, and acceptable. The values can be varied based on data size, model performance, type of model (single scoring, multiple scoring models). Once a model is deployed and evaluates millions of responses every time for optimal responses, we need a dynamically updated model based on prompt and data. Finally, framework designing of AES model, hear a framework contains prompts where test-takers can write the responses. One can design two frameworks: a single scoring model for a single methodology and multiple scoring models for multiple concepts. When we deploy multiple scoring models, each prompt could be trained separately, or we can provide generalized models for all prompts with this accuracy may vary, and it is challenging.

4 Synthesis

Our Systematic literature review on the automated essay grading system first collected 542 papers with selected keywords from various databases. After inclusion and exclusion criteria, we left with 139 articles; on these selected papers, we applied Quality assessment criteria with two reviewers, and finally, we selected 62 writings for final review.

Our observations on automated essay grading systems from 2010 to 2020 are as followed:

The implementation techniques of automated essay grading systems are classified into four buckets; there are 1. regression models 2. Classification models 3. Neural networks 4. Ontology-based methodology, but using neural networks, the researchers are more accurate than other techniques, and all the methods state of the art provided in Table 3 .

The majority of the regression and classification models on essay scoring used statistical features to find the final score. It means the systems or models trained on such parameters as word count, sentence count, etc. though the parameters extracted from the essay, the algorithm are not directly training on essays. The algorithms trained on some numbers obtained from the essay and hear if numbers matched the composition will get a good score; otherwise, the rating is less. In these models, the evaluation process is entirely on numbers, irrespective of the essay. So, there is a lot of chance to miss the coherence, relevance of the essay if we train our algorithm on statistical parameters.

In the neural network approach, the models trained on Bag of Words (BoW) features. The BoW feature is missing the relationship between a word to word and the semantic meaning of the sentence. E.g., Sentence 1: John killed bob. Sentence 2: bob killed John. In these two sentences, the BoW is "John," "killed," "bob."

In the Word2Vec library, if we are prepared a word vector from an essay in a unidirectional way, the vector will have a dependency with other words and finds the semantic relationship with other words. But if a word has two or more meanings like "Bank loan" and "River Bank," hear bank has two implications, and its adjacent words decide the sentence meaning; in this case, Word2Vec is not finding the real meaning of the word from the sentence.

The features extracted from essays in the essay scoring system are classified into 3 type's features like statistical features, style-based features, and content-based features, which are explained in RQ2 and Table 3 . But statistical features, are playing a significant role in some systems and negligible in some systems. In Shehab et al. ( 2016 ); Cummins et al. ( 2016 ). Dong et al. ( 2017 ). Dong and Zhang ( 2016 ). Mathias and Bhattacharyya ( 2018a ; b ) Systems the assessment is entirely on statistical and style-based features they have not retrieved any content-based features. And in other systems that extract content from the essays, the role of statistical features is for only preprocessing essays but not included in the final grading.

In AES systems, coherence is the main feature to be considered while evaluating essays. The actual meaning of coherence is to stick together. That is the logical connection of sentences (local level coherence) and paragraphs (global level coherence) in a story. Without coherence, all sentences in a paragraph are independent and meaningless. In an Essay, coherence is a significant feature that is explaining everything in a flow and its meaning. It is a powerful feature in AES system to find the semantics of essay. With coherence, one can assess whether all sentences are connected in a flow and all paragraphs are related to justify the prompt. Retrieving the coherence level from an essay is a critical task for all researchers in AES systems.

In automatic essay grading systems, the assessment of essays concerning content is critical. That will give the actual score for the student. Most of the researches used statistical features like sentence length, word count, number of sentences, etc. But according to collected results, 32% of the systems used content-based features for the essay scoring. Example papers which are on content-based assessment are Taghipour and Ng ( 2016 ); Persing and Ng ( 2013 ); Wang et al. ( 2018a , 2018b ); Zhao et al. ( 2017 ); Kopparapu and De ( 2016 ), Kumar et al. ( 2019 ); Mathias and Bhattacharyya ( 2018a ; b ); Mohler and Mihalcea ( 2009 ) are used content and statistical-based features. The results are shown in Fig. 3 . And mainly the content-based features extracted with word2vec NLP library, but word2vec is capable of capturing the context of a word in a document, semantic and syntactic similarity, relation with other terms, but word2vec is capable of capturing the context word in a uni-direction either left or right. If a word has multiple meanings, there is a chance of missing the context in the essay. After analyzing all the papers, we found that content-based assessment is a qualitative assessment of essays.

On the other hand, Horbach and Zesch ( 2019 ); Riordan Brain et al. ( 2019 ); Ding et al. ( 2020 ); Kumar et al. ( 2020 ) proved that neural network models are vulnerable when a student response contains constructed irrelevant, adversarial answers. And a student can easily bluff an automated scoring system by submitting different responses like repeating sentences and repeating prompt words in an essay. From Loukina et al. ( 2019 ), and Madnani et al. ( 2017b ). The fairness of an algorithm is an essential factor to be considered in AES systems.

While talking about speech assessment, the data set contains audios of duration up to one minute. Feature extraction techniques are entirely different from text assessment, and accuracy varies based on speaking fluency, pitching, male to female voice and boy to adult voice. But the training algorithms are the same for text and speech assessment.

Once an AES system evaluates essays and short answers accurately in all directions, there is a massive demand for automated systems in the educational and related world. Now AES systems are deployed in GRE, TOEFL exams; other than these, we can deploy AES systems in massive open online courses like Coursera(“ https://coursera.org/learn//machine-learning//exam ”), NPTEL ( https://swayam.gov.in/explorer ), etc. still they are assessing student performance with multiple-choice questions. In another perspective, AES systems can be deployed in information retrieval systems like Quora, stack overflow, etc., to check whether the retrieved response is appropriate to the question or not and can give ranking to the retrieved answers.

5 Conclusion and future work

As per our Systematic literature review, we studied 62 papers. There exist significant challenges for researchers in implementing automated essay grading systems. Several researchers are working rigorously on building a robust AES system despite its difficulty in solving this problem. All evaluating methods are not evaluated based on coherence, relevance, completeness, feedback, and knowledge-based. And 90% of essay grading systems are used Kaggle ASAP (2012) dataset, which has general essays from students and not required any domain knowledge, so there is a need for domain-specific essay datasets to train and test. Feature extraction is with NLTK, WordVec, and GloVec NLP libraries; these libraries have many limitations while converting a sentence into vector form. Apart from feature extraction and training Machine Learning models, no system is accessing the essay's completeness. No system provides feedback to the student response and not retrieving coherence vectors from the essay—another perspective the constructive irrelevant and adversarial student responses still questioning AES systems.

Our proposed research work will go on the content-based assessment of essays with domain knowledge and find a score for the essays with internal and external consistency. And we will create a new dataset concerning one domain. And another area in which we can improve is the feature extraction techniques.

This study includes only four digital databases for study selection may miss some functional studies on the topic. However, we hope that we covered most of the significant studies as we manually collected some papers published in useful journals.

Adamson, A., Lamb, A., & December, R. M. (2014). Automated Essay Grading.

Ajay HB, Tillett PI, Page EB (1973) Analysis of essays by computer (AEC-II) (No. 8-0102). Washington, DC: U.S. Department of Health, Education, and Welfare, Office of Education, National Center for Educational Research and Development

Ajetunmobi SA, Daramola O (2017) Ontology-based information extraction for subject-focussed automatic essay evaluation. In: 2017 International Conference on Computing Networking and Informatics (ICCNI) p 1–6. IEEE

Alva-Manchego F, et al. (2019) EASSE: Easier Automatic Sentence Simplification Evaluation.” ArXiv abs/1908.04567 (2019): n. pag

Bailey S, Meurers D (2008) Diagnosing meaning errors in short answers to reading comprehension questions. In: Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (Columbus), p 107–115

Basu S, Jacobs C, Vanderwende L (2013) Powergrading: a clustering approach to amplify human effort for short answer grading. Trans Assoc Comput Linguist (TACL) 1:391–402

Article   Google Scholar  

Bejar, I. I., Flor, M., Futagi, Y., & Ramineni, C. (2014). On the vulnerability of automated scoring to construct-irrelevant response strategies (CIRS): An illustration. Assessing Writing, 22, 48-59.

Bejar I, et al. (2013) Length of Textual Response as a Construct-Irrelevant Response Strategy: The Case of Shell Language. Research Report. ETS RR-13-07.” ETS Research Report Series (2013): n. pag

Berzak Y, et al. (2018) “Assessing Language Proficiency from Eye Movements in Reading.” ArXiv abs/1804.07329 (2018): n. pag

Blanchard D, Tetreault J, Higgins D, Cahill A, Chodorow M (2013) TOEFL11: A corpus of non-native English. ETS Research Report Series, 2013(2):i–15, 2013

Blood, I. (2011). Automated essay scoring: a literature review. Studies in Applied Linguistics and TESOL, 11(2).

Burrows S, Gurevych I, Stein B (2015) The eras and trends of automatic short answer grading. Int J Artif Intell Educ 25:60–117. https://doi.org/10.1007/s40593-014-0026-8

Cader, A. (2020, July). The Potential for the Use of Deep Neural Networks in e-Learning Student Evaluation with New Data Augmentation Method. In International Conference on Artificial Intelligence in Education (pp. 37–42). Springer, Cham.

Cai C (2019) Automatic essay scoring with recurrent neural network. In: Proceedings of the 3rd International Conference on High Performance Compilation, Computing and Communications (2019): n. pag.

Chen M, Li X (2018) "Relevance-Based Automated Essay Scoring via Hierarchical Recurrent Model. In: 2018 International Conference on Asian Language Processing (IALP), Bandung, Indonesia, 2018, p 378–383, doi: https://doi.org/10.1109/IALP.2018.8629256

Chen Z, Zhou Y (2019) "Research on Automatic Essay Scoring of Composition Based on CNN and OR. In: 2019 2nd International Conference on Artificial Intelligence and Big Data (ICAIBD), Chengdu, China, p 13–18, doi: https://doi.org/10.1109/ICAIBD.2019.8837007

Contreras JO, Hilles SM, Abubakar ZB (2018) Automated essay scoring with ontology based on text mining and NLTK tools. In: 2018 International Conference on Smart Computing and Electronic Enterprise (ICSCEE), 1-6

Correnti R, Matsumura LC, Hamilton L, Wang E (2013) Assessing students’ skills at writing analytically in response to texts. Elem Sch J 114(2):142–177

Cummins, R., Zhang, M., & Briscoe, E. (2016, August). Constrained multi-task learning for automated essay scoring. Association for Computational Linguistics.

Darwish SM, Mohamed SK (2020) Automated essay evaluation based on fusion of fuzzy ontology and latent semantic analysis. In: Hassanien A, Azar A, Gaber T, Bhatnagar RF, Tolba M (eds) The International Conference on Advanced Machine Learning Technologies and Applications

Dasgupta T, Naskar A, Dey L, Saha R (2018) Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring. In: Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications p 93–102

Ding Y, et al. (2020) "Don’t take “nswvtnvakgxpm” for an answer–The surprising vulnerability of automatic content scoring systems to adversarial input." In: Proceedings of the 28th International Conference on Computational Linguistics

Dong F, Zhang Y (2016) Automatic features for essay scoring–an empirical study. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing p 1072–1077

Dong F, Zhang Y, Yang J (2017) Attention-based recurrent convolutional neural network for automatic essay scoring. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) p 153–162

Dzikovska M, Nielsen R, Brew C, Leacock C, Gi ampiccolo D, Bentivogli L, Clark P, Dagan I, Dang HT (2013a) Semeval-2013 task 7: The joint student response analysis and 8th recognizing textual entailment challenge

Dzikovska MO, Nielsen R, Brew C, Leacock C, Giampiccolo D, Bentivogli L, Clark P, Dagan I, Trang Dang H (2013b) SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge. *SEM 2013: The First Joint Conference on Lexical and Computational Semantics

Educational Testing Service (2008) CriterionSM online writing evaluation service. Retrieved from http://www.ets.org/s/criterion/pdf/9286_CriterionBrochure.pdf .

Evanini, K., & Wang, X. (2013, August). Automated speech scoring for non-native middle school students with multiple task types. In INTERSPEECH (pp. 2435–2439).

Foltz PW, Laham D, Landauer TK (1999) The Intelligent Essay Assessor: Applications to Educational Technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1, 2, http://imej.wfu.edu/articles/1999/2/04/ index.asp

Granger, S., Dagneaux, E., Meunier, F., & Paquot, M. (Eds.). (2009). International corpus of learner English. Louvain-la-Neuve: Presses universitaires de Louvain.

Higgins, D., & Heilman, M. (2014). Managing what we can measure: Quantifying the susceptibility of automated scoring systems to gaming behavior. Educational Measurement: Issues and Practice, 33(3), 36–46.

Horbach A, Zesch T (2019) The influence of variance in learner answers on automatic content scoring. Front Educ 4:28. https://doi.org/10.3389/feduc.2019.00028

https://www.coursera.org/learn/machine-learning/exam/7pytE/linear-regression-with-multiple-variables/attempt

Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208.

Ke Z, Ng V (2019) “Automated essay scoring: a survey of the state of the art.” IJCAI

Ke, Z., Inamdar, H., Lin, H., & Ng, V. (2019, July). Give me more feedback II: Annotating thesis strength and related attributes in student essays. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3994-4004).

Kelley K, Preacher KJ (2012) On effect size. Psychol Methods 17(2):137–152

Kitchenham B, Brereton OP, Budgen D, Turner M, Bailey J, Linkman S (2009) Systematic literature reviews in software engineering–a systematic literature review. Inf Softw Technol 51(1):7–15

Klebanov, B. B., & Madnani, N. (2020, July). Automated evaluation of writing–50 years and counting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 7796–7810).

Knill K, Gales M, Kyriakopoulos K, et al. (4 more authors) (2018) Impact of ASR performance on free speaking language assessment. In: Interspeech 2018.02–06 Sep 2018, Hyderabad, India. International Speech Communication Association (ISCA)

Kopparapu SK, De A (2016) Automatic ranking of essays using structural and semantic features. In: 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), p 519–523

Kumar, Y., Aggarwal, S., Mahata, D., Shah, R. R., Kumaraguru, P., & Zimmermann, R. (2019, July). Get it scored using autosas—an automated system for scoring short answers. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 9662–9669).

Kumar Y, et al. (2020) “Calling out bluff: attacking the robustness of automatic scoring systems with simple adversarial testing.” ArXiv abs/2007.06796

Li X, Chen M, Nie J, Liu Z, Feng Z, Cai Y (2018) Coherence-Based Automated Essay Scoring Using Self-attention. In: Sun M, Liu T, Wang X, Liu Z, Liu Y (eds) Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. CCL 2018, NLP-NABD 2018. Lecture Notes in Computer Science, vol 11221. Springer, Cham. https://doi.org/10.1007/978-3-030-01716-3_32

Liang G, On B, Jeong D, Kim H, Choi G (2018) Automated essay scoring: a siamese bidirectional LSTM neural network architecture. Symmetry 10:682

Liua, H., Yeb, Y., & Wu, M. (2018, April). Ensemble Learning on Scoring Student Essay. In 2018 International Conference on Management and Education, Humanities and Social Sciences (MEHSS 2018). Atlantis Press.

Liu J, Xu Y, Zhao L (2019) Automated Essay Scoring based on Two-Stage Learning. ArXiv, abs/1901.07744

Loukina A, et al. (2015) Feature selection for automated speech scoring.” BEA@NAACL-HLT

Loukina A, et al. (2017) “Speech- and Text-driven Features for Automated Scoring of English-Speaking Tasks.” SCNLP@EMNLP 2017

Loukina A, et al. (2019) The many dimensions of algorithmic fairness in educational applications. BEA@ACL

Lun J, Zhu J, Tang Y, Yang M (2020) Multiple data augmentation strategies for improving performance on automatic short answer scoring. In: Proceedings of the AAAI Conference on Artificial Intelligence, 34(09): 13389-13396

Madnani, N., & Cahill, A. (2018, August). Automated scoring: Beyond natural language processing. In Proceedings of the 27th International Conference on Computational Linguistics (pp. 1099–1109).

Madnani N, et al. (2017b) “Building better open-source tools to support fairness in automated scoring.” EthNLP@EACL

Malinin A, et al. (2016) “Off-topic response detection for spontaneous spoken english assessment.” ACL

Malinin A, et al. (2017) “Incorporating uncertainty into deep learning for spoken language assessment.” ACL

Mathias S, Bhattacharyya P (2018a) Thank “Goodness”! A Way to Measure Style in Student Essays. In: Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications p 35–41

Mathias S, Bhattacharyya P (2018b) ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

Mikolov T, et al. (2013) “Efficient Estimation of Word Representations in Vector Space.” ICLR

Mohler M, Mihalcea R (2009) Text-to-text semantic similarity for automatic short answer grading. In: Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009) p 567–575

Mohler M, Bunescu R, Mihalcea R (2011) Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies p 752–762

Muangkammuen P, Fukumoto F (2020) Multi-task Learning for Automated Essay Scoring with Sentiment Analysis. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop p 116–123

Nguyen, H., & Dery, L. (2016). Neural networks for automated essay grading. CS224d Stanford Reports, 1–11.

Palma D, Atkinson J (2018) Coherence-based automatic essay assessment. IEEE Intell Syst 33(5):26–36

Parekh S, et al (2020) My Teacher Thinks the World Is Flat! Interpreting Automatic Essay Scoring Mechanism.” ArXiv abs/2012.13872 (2020): n. pag

Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).

Persing I, Ng V (2013) Modeling thesis clarity in student essays. In:Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) p 260–269

Powers DE, Burstein JC, Chodorow M, Fowles ME, Kukich K (2001) Stumping E-Rater: challenging the validity of automated essay scoring. ETS Res Rep Ser 2001(1):i–44

Google Scholar  

Powers, D. E., Burstein, J. C., Chodorow, M., Fowles, M. E., & Kukich, K. (2002). Stumping e-rater: challenging the validity of automated essay scoring. Computers in Human Behavior, 18(2), 103–134.

Ramachandran L, Cheng J, Foltz P (2015) Identifying patterns for short answer scoring using graph-based lexico-semantic text matching. In: Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications p 97–106

Ramanarayanan V, et al. (2017) “Human and Automated Scoring of Fluency, Pronunciation and Intonation During Human-Machine Spoken Dialog Interactions.” INTERSPEECH

Riordan B, Horbach A, Cahill A, Zesch T, Lee C (2017) Investigating neural architectures for short answer scoring. In: Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications p 159–168

Riordan B, Flor M, Pugh R (2019) "How to account for misspellings: Quantifying the benefit of character representations in neural content scoring models."In: Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

Rodriguez P, Jafari A, Ormerod CM (2019) Language models and Automated Essay Scoring. ArXiv, abs/1909.09482

Rudner, L. M., & Liang, T. (2002). Automated essay scoring using Bayes' theorem. The Journal of Technology, Learning and Assessment, 1(2).

Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetric™ essay scoring system. The Journal of Technology, Learning and Assessment, 4(4).

Rupp A (2018) Designing, evaluating, and deploying automated scoring systems with validity in mind: methodological design decisions. Appl Meas Educ 31:191–214

Ruseti S, Dascalu M, Johnson AM, McNamara DS, Balyan R, McCarthy KS, Trausan-Matu S (2018) Scoring summaries using recurrent neural networks. In: International Conference on Intelligent Tutoring Systems p 191–201. Springer, Cham

Sakaguchi K, Heilman M, Madnani N (2015) Effective feature integration for automated short answer scoring. In: Proceedings of the 2015 conference of the North American Chapter of the association for computational linguistics: Human language technologies p 1049–1054

Salim, Y., Stevanus, V., Barlian, E., Sari, A. C., & Suhartono, D. (2019, December). Automated English Digital Essay Grader Using Machine Learning. In 2019 IEEE International Conference on Engineering, Technology and Education (TALE) (pp. 1–6). IEEE.

Shehab A, Elhoseny M, Hassanien AE (2016) A hybrid scheme for Automated Essay Grading based on LVQ and NLP techniques. In: 12th International Computer Engineering Conference (ICENCO), Cairo, 2016, p 65-70

Shermis MD, Mzumara HR, Olson J, Harrington S (2001) On-line grading of student essays: PEG goes on the World Wide Web. Assess Eval High Educ 26(3):247–259

Stab C, Gurevych I (2014) Identifying argumentative discourse structures in persuasive essays. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) p 46–56

Sultan MA, Salazar C, Sumner T (2016) Fast and easy short answer grading with high accuracy. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies p 1070–1075

Süzen, N., Gorban, A. N., Levesley, J., & Mirkes, E. M. (2020). Automatic short answer grading and feedback using text mining methods. Procedia Computer Science, 169, 726–743.

Taghipour K, Ng HT (2016) A neural approach to automated essay scoring. In: Proceedings of the 2016 conference on empirical methods in natural language processing p 1882–1891

Tashu TM (2020) "Off-Topic Essay Detection Using C-BGRU Siamese. In: 2020 IEEE 14th International Conference on Semantic Computing (ICSC), San Diego, CA, USA, p 221–225, doi: https://doi.org/10.1109/ICSC.2020.00046

Tashu TM, Horváth T (2019) A layered approach to automatic essay evaluation using word-embedding. In: McLaren B, Reilly R, Zvacek S, Uhomoibhi J (eds) Computer Supported Education. CSEDU 2018. Communications in Computer and Information Science, vol 1022. Springer, Cham

Tashu TM, Horváth T (2020) Semantic-Based Feedback Recommendation for Automatic Essay Evaluation. In: Bi Y, Bhatia R, Kapoor S (eds) Intelligent Systems and Applications. IntelliSys 2019. Advances in Intelligent Systems and Computing, vol 1038. Springer, Cham

Uto M, Okano M (2020) Robust Neural Automated Essay Scoring Using Item Response Theory. In: Bittencourt I, Cukurova M, Muldner K, Luckin R, Millán E (eds) Artificial Intelligence in Education. AIED 2020. Lecture Notes in Computer Science, vol 12163. Springer, Cham

Wang Z, Liu J, Dong R (2018a) Intelligent Auto-grading System. In: 2018 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) p 430–435. IEEE.

Wang Y, et al. (2018b) “Automatic Essay Scoring Incorporating Rating Schema via Reinforcement Learning.” EMNLP

Zhu W, Sun Y (2020) Automated essay scoring system using multi-model Machine Learning, david c. wyld et al. (eds): mlnlp, bdiot, itccma, csity, dtmn, aifz, sigpro

Wresch W (1993) The Imminence of Grading Essays by Computer-25 Years Later. Comput Compos 10:45–58

Wu, X., Knill, K., Gales, M., & Malinin, A. (2020). Ensemble approaches for uncertainty in spoken language assessment.

Xia L, Liu J, Zhang Z (2019) Automatic Essay Scoring Model Based on Two-Layer Bi-directional Long-Short Term Memory Network. In: Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence p 133–137

Yannakoudakis H, Briscoe T, Medlock B (2011) A new dataset and method for automatically grading ESOL texts. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies p 180–189

Zhao S, Zhang Y, Xiong X, Botelho A, Heffernan N (2017) A memory-augmented neural model for automated grading. In: Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale p 189–192

Zupanc K, Bosnic Z (2014) Automated essay evaluation augmented with semantic coherence measures. In: 2014 IEEE International Conference on Data Mining p 1133–1138. IEEE.

Zupanc K, Savić M, Bosnić Z, Ivanović M (2017) Evaluating coherence of essays using sentence-similarity networks. In: Proceedings of the 18th International Conference on Computer Systems and Technologies p 65–72

Dzikovska, M. O., Nielsen, R., & Brew, C. (2012, June). Towards effective tutorial feedback for explanation questions: A dataset and baselines. In  Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies  (pp. 200-210).

Kumar, N., & Dey, L. (2013, November). Automatic Quality Assessment of documents with application to essay grading. In 2013 12th Mexican International Conference on Artificial Intelligence (pp. 216–222). IEEE.

Wu, S. H., & Shih, W. F. (2018, July). A short answer grading system in chinese by support vector approach. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications (pp. 125-129).

Agung Putri Ratna, A., Lalita Luhurkinanti, D., Ibrahim I., Husna D., Dewi Purnamasari P. (2018). Automatic Essay Grading System for Japanese Language Examination Using Winnowing Algorithm, 2018 International Seminar on Application for Technology of Information and Communication, 2018, pp. 565–569. https://doi.org/10.1109/ISEMANTIC.2018.8549789 .

Sharma A., & Jayagopi D. B. (2018). Automated Grading of Handwritten Essays 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp 279–284. https://doi.org/10.1109/ICFHR-2018.2018.00056

Download references

Not Applicable.

Author information

Authors and affiliations.

School of Computer Science and Artificial Intelligence, SR University, Warangal, TS, India

Dadi Ramesh

Research Scholar, JNTU, Hyderabad, India

Department of Information Technology, JNTUH College of Engineering, Nachupally, Kondagattu, Jagtial, TS, India

Suresh Kumar Sanampudi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dadi Ramesh .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (XLSX 80 KB)

Rights and permissions.

Reprints and permissions

About this article

Ramesh, D., Sanampudi, S.K. An automated essay scoring systems: a systematic literature review. Artif Intell Rev 55 , 2495–2527 (2022). https://doi.org/10.1007/s10462-021-10068-2

Download citation

Published : 23 September 2021

Issue Date : March 2022

DOI : https://doi.org/10.1007/s10462-021-10068-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Short answer scoring
• Essay grading
• Natural language processing
• Deep learning
• Find a journal
• Publish with us
• Track your research

Home — Essay Samples — Literature — Book Report — Cavalry Crossing A Ford: An Analytical Perspective

Cavalry Crossing a Ford: an Analytical Perspective

• Categories: Book Report Symbolism

About this sample

Words: 685 |

Published: Jun 6, 2024

Words: 685 | Pages: 2 | 4 min read

Table of contents

Imagery and symbolism, structure and form, humanity amidst conflict, conclusion: a moment captured in time.

Cite this Essay

Let us write you an essay from scratch

• 450+ experts on 30 subjects ready to help
• Custom essay delivered in as few as 3 hours

Get high-quality help

Dr. Karlyna PhD

Verified writer

• Expert in: Literature

+ 120 experts online

By clicking “Check Writers’ Offers”, you agree to our terms of service and privacy policy . We’ll occasionally send you promo and account related email

No need to pay just yet!

Related Essays

2 pages / 1072 words

2 pages / 1087 words

4 pages / 1993 words

2.5 pages / 1190 words

Remember! This is just a sample.

You can get your custom paper by one of our expert writers.

121 writers online

Still can’t find what you need?

Browse our vast selection of original essay samples, each expertly formatted and styled

Related Essays on Book Report

Tuesdays with Morrie is book which was composed by one of the subject’s most loved student, Mitch Albom. Mitch is an American writer, columnist, screenwriter, playwright, radio and TV telecaster, and performer. It was committed [...]

Fairy tales have been a part of human culture for centuries, passed down from generation to generation through oral tradition and written literature. These stories often feature magical elements, fantastical creatures, and moral [...]

In the realm of human emotions, love often stands out as an enigmatic and multifaceted phenomenon. The aphorism "love is blind" suggests that love can obscure one's judgment and perception, leading individuals to overlook flaws [...]

In conclusion, "Deaf Like Me" is a testament to the power of communication, understanding, and empathy. Through their personal journey, the Spradleys invite us to reflect on the challenges faced by the Deaf community and [...]

Many modern playwrights seek to connect contemporary issues with ancient themes by updating the stories of mythic stories into a completely modern milieu. With Blood Wedding, Federico Garcia Lorca seeks to explore this idea [...]

Character Development: Explore how the characters in "The Book Thief" are shaped and transformed by the power of words, including Liesel, Hans, and Max. Literary Devices: Analyze the literary devices [...]

Related Topics

By clicking “Send”, you agree to our Terms of service and Privacy statement . We will occasionally send you account related emails.

Where do you want us to send this sample?

By clicking “Continue”, you agree to our terms of service and privacy policy.

Be careful. This essay is not unique

This essay was donated by a student and is likely to have been used and submitted before

Download this Sample

Free samples may contain mistakes and not unique parts

Sorry, we could not paraphrase this essay. Our professional writers can rewrite it and get you a unique paper.

Please check your inbox.

We can write you a custom essay that will follow your exact instructions and meet the deadlines. Let's fix your grades together!

Get Your Personalized Essay in 3 Hours or Less!

We use cookies to personalyze your web-site experience. By continuing we’ll assume you board with our cookie policy .

• Instructions Followed To The Letter
• Deadlines Met At Every Stage
• Unique And Plagiarism Free

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 June 2024

Applying large language models for automated essay scoring for non-native Japanese

• Wenchao Li 1 &
• Haitao Liu 2  

Humanities and Social Sciences Communications volume  11 , Article number:  723 ( 2024 ) Cite this article

129 Accesses

1 Altmetric

Metrics details

• Language and linguistics

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.

Similar content being viewed by others

Accurate structure prediction of biomolecular interactions with AlphaFold 3

Testing theory of mind in large language models and humans

Highly accurate protein structure prediction with AlphaFold

Conventional machine learning technology in aes.

AES has experienced significant growth with the advancement of machine learning technologies in recent decades. In the earlier stages of AES development, conventional machine learning-based approaches were commonly used. These approaches involved the following procedures: a) feeding the machine with a dataset. In this step, a dataset of essays is provided to the machine learning system. The dataset serves as the basis for training the model and establishing patterns and correlations between linguistic features and human ratings. b) the machine learning model is trained using linguistic features that best represent human ratings and can effectively discriminate learners’ writing proficiency. These features include lexical richness (Lu, 2012 ; Kyle and Crossley, 2015 ; Kyle et al. 2021 ), syntactic complexity (Lu, 2010 ; Liu, 2008 ), text cohesion (Crossley and McNamara, 2016 ), and among others. Conventional machine learning approaches in AES require human intervention, such as manual correction and annotation of essays. This human involvement was necessary to create a labeled dataset for training the model. Several AES systems have been developed using conventional machine learning technologies. These include the Intelligent Essay Assessor (Landauer et al. 2003 ), the e-rater engine by Educational Testing Service (Attali and Burstein, 2006 ; Burstein, 2003 ), MyAccess with the InterlliMetric scoring engine by Vantage Learning (Elliot, 2003 ), and the Bayesian Essay Test Scoring system (Rudner and Liang, 2002 ). These systems have played a significant role in automating the essay scoring process and providing quick and consistent feedback to learners. However, as touched upon earlier, conventional machine learning approaches rely on predetermined linguistic features and often require manual intervention, making them less flexible and potentially limiting their generalizability to different contexts.

In the context of the Japanese language, conventional machine learning-incorporated AES tools include Jess (Ishioka and Kameda, 2006 ) and JWriter (Lee and Hasebe, 2017 ). Jess assesses essays by deducting points from the perfect score, utilizing the Mainichi Daily News newspaper as a database. The evaluation criteria employed by Jess encompass various aspects, such as rhetorical elements (e.g., reading comprehension, vocabulary diversity, percentage of complex words, and percentage of passive sentences), organizational structures (e.g., forward and reverse connection structures), and content analysis (e.g., latent semantic indexing). JWriter employs linear regression analysis to assign weights to various measurement indices, such as average sentence length and total number of characters. These weights are then combined to derive the overall score. A pilot study involving the Jess model was conducted on 1320 essays at different proficiency levels, including primary, intermediate, and advanced. However, the results indicated that the Jess model failed to significantly distinguish between these essay levels. Out of the 16 measures used, four measures, namely median sentence length, median clause length, median number of phrases, and maximum number of phrases, did not show statistically significant differences between the levels. Additionally, two measures exhibited between-level differences but lacked linear progression: the number of attributives declined words and the Kanji/kana ratio. On the other hand, the remaining measures, including maximum sentence length, maximum clause length, number of attributive conjugated words, maximum number of consecutive infinitive forms, maximum number of conjunctive-particle clauses, k characteristic value, percentage of big words, and percentage of passive sentences, demonstrated statistically significant between-level differences and displayed linear progression.

Both Jess and JWriter exhibit notable limitations, including the manual selection of feature parameters and weights, which can introduce biases into the scoring process. The reliance on human annotators to label non-native language essays also introduces potential noise and variability in the scoring. Furthermore, an important concern is the possibility of system manipulation and cheating by learners who are aware of the regression equation utilized by the models (Hirao et al. 2020 ). These limitations emphasize the need for further advancements in AES systems to address these challenges.

Deep learning technology in AES

Deep learning has emerged as one of the approaches for improving the accuracy and effectiveness of AES. Deep learning-based AES methods utilize artificial neural networks that mimic the human brain’s functioning through layered algorithms and computational units. Unlike conventional machine learning, deep learning autonomously learns from the environment and past errors without human intervention. This enables deep learning models to establish nonlinear correlations, resulting in higher accuracy. Recent advancements in deep learning have led to the development of transformers, which are particularly effective in learning text representations. Noteworthy examples include bidirectional encoder representations from transformers (BERT) (Devlin et al. 2019 ) and the generative pretrained transformer (GPT) (OpenAI).

BERT is a linguistic representation model that utilizes a transformer architecture and is trained on two tasks: masked linguistic modeling and next-sentence prediction (Hirao et al. 2020 ; Vaswani et al. 2017 ). In the context of AES, BERT follows specific procedures, as illustrated in Fig. 1 : (a) the tokenized prompts and essays are taken as input; (b) special tokens, such as [CLS] and [SEP], are added to mark the beginning and separation of prompts and essays; (c) the transformer encoder processes the prompt and essay sequences, resulting in hidden layer sequences; (d) the hidden layers corresponding to the [CLS] tokens (T[CLS]) represent distributed representations of the prompts and essays; and (e) a multilayer perceptron uses these distributed representations as input to obtain the final score (Hirao et al. 2020 ).

AES system with BERT (Hirao et al. 2020 ).

The training of BERT using a substantial amount of sentence data through the Masked Language Model (MLM) allows it to capture contextual information within the hidden layers. Consequently, BERT is expected to be capable of identifying artificial essays as invalid and assigning them lower scores (Mizumoto and Eguchi, 2023 ). In the context of AES for nonnative Japanese learners, Hirao et al. ( 2020 ) combined the long short-term memory (LSTM) model proposed by Hochreiter and Schmidhuber ( 1997 ) with BERT to develop a tailored automated Essay Scoring System. The findings of their study revealed that the BERT model outperformed both the conventional machine learning approach utilizing character-type features such as “kanji” and “hiragana”, as well as the standalone LSTM model. Takeuchi et al. ( 2021 ) presented an approach to Japanese AES that eliminates the requirement for pre-scored essays by relying solely on reference texts or a model answer for the essay task. They investigated multiple similarity evaluation methods, including frequency of morphemes, idf values calculated on Wikipedia, LSI, LDA, word-embedding vectors, and document vectors produced by BERT. The experimental findings revealed that the method utilizing the frequency of morphemes with idf values exhibited the strongest correlation with human-annotated scores across different essay tasks. The utilization of BERT in AES encounters several limitations. Firstly, essays often exceed the model’s maximum length limit. Second, only score labels are available for training, which restricts access to additional information.

Mizumoto and Eguchi ( 2023 ) were pioneers in employing the GPT model for AES in non-native English writing. Their study focused on evaluating the accuracy and reliability of AES using the GPT-3 text-davinci-003 model, analyzing a dataset of 12,100 essays from the corpus of nonnative written English (TOEFL11). The findings indicated that AES utilizing the GPT-3 model exhibited a certain degree of accuracy and reliability. They suggest that GPT-3-based AES systems hold the potential to provide support for human ratings. However, applying GPT model to AES presents a unique natural language processing (NLP) task that involves considerations such as nonnative language proficiency, the influence of the learner’s first language on the output in the target language, and identifying linguistic features that best indicate writing quality in a specific language. These linguistic features may differ morphologically or syntactically from those present in the learners’ first language, as observed in (1)–(3).

我-送了-他-一本-书

Wǒ-sòngle-tā-yī běn-shū

1 sg .-give. past- him-one .cl- book

“I gave him a book.”

Agglutinative

彼-に-本-を-あげ-まし-た

Kare-ni-hon-o-age-mashi-ta

3 sg .- dat -hon- acc- give.honorification. past

Inflectional

give, give-s, gave, given, giving

Additionally, the morphological agglutination and subject-object-verb (SOV) order in Japanese, along with its idiomatic expressions, pose additional challenges for applying language models in AES tasks (4).

足-が 棒-に なり-ました

Ashi-ga bo-ni nar-mashita

leg- nom stick- dat become- past

“My leg became like a stick (I am extremely tired).”

The example sentence provided demonstrates the morpho-syntactic structure of Japanese and the presence of an idiomatic expression. In this sentence, the verb “なる” (naru), meaning “to become”, appears at the end of the sentence. The verb stem “なり” (nari) is attached with morphemes indicating honorification (“ます” - mashu) and tense (“た” - ta), showcasing agglutination. While the sentence can be literally translated as “my leg became like a stick”, it carries an idiomatic interpretation that implies “I am extremely tired”.

To overcome this issue, CyberAgent Inc. ( 2023 ) has developed the Open-Calm series of language models specifically designed for Japanese. Open-Calm consists of pre-trained models available in various sizes, such as Small, Medium, Large, and 7b. Figure 2 depicts the fundamental structure of the Open-Calm model. A key feature of this architecture is the incorporation of the Lora Adapter and GPT-NeoX frameworks, which can enhance its language processing capabilities.

GPT-NeoX Model Architecture (Okgetheng and Takeuchi 2024 ).

In a recent study conducted by Okgetheng and Takeuchi ( 2024 ), they assessed the efficacy of Open-Calm language models in grading Japanese essays. The research utilized a dataset of approximately 300 essays, which were annotated by native Japanese educators. The findings of the study demonstrate the considerable potential of Open-Calm language models in automated Japanese essay scoring. Specifically, among the Open-Calm family, the Open-Calm Large model (referred to as OCLL) exhibited the highest performance. However, it is important to note that, as of the current date, the Open-Calm Large model does not offer public access to its server. Consequently, users are required to independently deploy and operate the environment for OCLL. In order to utilize OCLL, users must have a PC equipped with an NVIDIA GeForce RTX 3060 (8 or 12 GB VRAM).

In summary, while the potential of LLMs in automated scoring of nonnative Japanese essays has been demonstrated in two studies—BERT-driven AES (Hirao et al. 2020 ) and OCLL-based AES (Okgetheng and Takeuchi, 2024 )—the number of research efforts in this area remains limited.

Another significant challenge in applying LLMs to AES lies in prompt engineering and ensuring its reliability and effectiveness (Brown et al. 2020 ; Rae et al. 2021 ; Zhang et al. 2021 ). Various prompting strategies have been proposed, such as the zero-shot chain of thought (CoT) approach (Kojima et al. 2022 ), which involves manually crafting diverse and effective examples. However, manual efforts can lead to mistakes. To address this, Zhang et al. ( 2021 ) introduced an automatic CoT prompting method called Auto-CoT, which demonstrates matching or superior performance compared to the CoT paradigm. Another prompt framework is trees of thoughts, enabling a model to self-evaluate its progress at intermediate stages of problem-solving through deliberate reasoning (Yao et al. 2023 ).

Beyond linguistic studies, there has been a noticeable increase in the number of foreign workers in Japan and Japanese learners worldwide (Ministry of Health, Labor, and Welfare of Japan, 2022 ; Japan Foundation, 2021 ). However, existing assessment methods, such as the Japanese Language Proficiency Test (JLPT), J-CAT, and TTBJ Footnote 1 , primarily focus on reading, listening, vocabulary, and grammar skills, neglecting the evaluation of writing proficiency. As the number of workers and language learners continues to grow, there is a rising demand for an efficient AES system that can reduce costs and time for raters and be utilized for employment, examinations, and self-study purposes.

This study aims to explore the potential of LLM-based AES by comparing the effectiveness of five models: two LLMs (GPT Footnote 2 and BERT), one Japanese local LLM (OCLL), and two conventional machine learning-based methods (linguistic feature-based scoring tools - Jess and JWriter).

The research questions addressed in this study are as follows:

To what extent do the LLM-driven AES and linguistic feature-based AES, when used as automated tools to support human rating, accurately reflect test takers’ actual performance?

What influence does the prompt have on the accuracy and performance of LLM-based AES methods?

The subsequent sections of the manuscript cover the methodology, including the assessment measures for nonnative Japanese writing proficiency, criteria for prompts, and the dataset. The evaluation section focuses on the analysis of annotations and rating scores generated by LLM-driven and linguistic feature-based AES methods.

Methodology

The dataset utilized in this study was obtained from the International Corpus of Japanese as a Second Language (I-JAS) Footnote 3 . This corpus consisted of 1000 participants who represented 12 different first languages. For the study, the participants were given a story-writing task on a personal computer. They were required to write two stories based on the 4-panel illustrations titled “Picnic” and “The key” (see Appendix A). Background information for the participants was provided by the corpus, including their Japanese language proficiency levels assessed through two online tests: J-CAT and SPOT. These tests evaluated their reading, listening, vocabulary, and grammar abilities. The learners’ proficiency levels were categorized into six levels aligned with the Common European Framework of Reference for Languages (CEFR) and the Reference Framework for Japanese Language Education (RFJLE): A1, A2, B1, B2, C1, and C2. According to Lee et al. ( 2015 ), there is a high level of agreement (r = 0.86) between the J-CAT and SPOT assessments, indicating that the proficiency certifications provided by J-CAT are consistent with those of SPOT. However, it is important to note that the scores of J-CAT and SPOT do not have a one-to-one correspondence. In this study, the J-CAT scores were used as a benchmark to differentiate learners of different proficiency levels. A total of 1400 essays were utilized, representing the beginner (aligned with A1), A2, B1, B2, C1, and C2 levels based on the J-CAT scores. Table 1 provides information about the learners’ proficiency levels and their corresponding J-CAT and SPOT scores.

A dataset comprising a total of 1400 essays from the story writing tasks was collected. Among these, 714 essays were utilized to evaluate the reliability of the LLM-based AES method, while the remaining 686 essays were designated as development data to assess the LLM-based AES’s capability to distinguish participants with varying proficiency levels. The GPT 4 API was used in this study. A detailed explanation of the prompt-assessment criteria is provided in Section Prompt . All essays were sent to the model for measurement and scoring.

Measures of writing proficiency for nonnative Japanese

Japanese exhibits a morphologically agglutinative structure where morphemes are attached to the word stem to convey grammatical functions such as tense, aspect, voice, and honorifics, e.g. (5).

食べ-させ-られ-まし-た-か

tabe-sase-rare-mashi-ta-ka

[eat (stem)-causative-passive voice-honorification-tense. past-question marker]

Japanese employs nine case particles to indicate grammatical functions: the nominative case particle が (ga), the accusative case particle を (o), the genitive case particle の (no), the dative case particle に (ni), the locative/instrumental case particle で (de), the ablative case particle から (kara), the directional case particle へ (e), and the comitative case particle と (to). The agglutinative nature of the language, combined with the case particle system, provides an efficient means of distinguishing between active and passive voice, either through morphemes or case particles, e.g. 食べる taberu “eat concusive . ” (active voice); 食べられる taberareru “eat concusive . ” (passive voice). In the active voice, “パン を 食べる” (pan o taberu) translates to “to eat bread”. On the other hand, in the passive voice, it becomes “パン が 食べられた” (pan ga taberareta), which means “(the) bread was eaten”. Additionally, it is important to note that different conjugations of the same lemma are considered as one type in order to ensure a comprehensive assessment of the language features. For example, e.g., 食べる taberu “eat concusive . ”; 食べている tabeteiru “eat progress .”; 食べた tabeta “eat past . ” as one type.

To incorporate these features, previous research (Suzuki, 1999 ; Watanabe et al. 1988 ; Ishioka, 2001 ; Ishioka and Kameda, 2006 ; Hirao et al. 2020 ) has identified complexity, fluency, and accuracy as crucial factors for evaluating writing quality. These criteria are assessed through various aspects, including lexical richness (lexical density, diversity, and sophistication), syntactic complexity, and cohesion (Kyle et al. 2021 ; Mizumoto and Eguchi, 2023 ; Ure, 1971 ; Halliday, 1985 ; Barkaoui and Hadidi, 2020 ; Zenker and Kyle, 2021 ; Kim et al. 2018 ; Lu, 2017 ; Ortega, 2015 ). Therefore, this study proposes five scoring categories: lexical richness, syntactic complexity, cohesion, content elaboration, and grammatical accuracy. A total of 16 measures were employed to capture these categories. The calculation process and specific details of these measures can be found in Table 2 .

T-unit, first introduced by Hunt ( 1966 ), is a measure used for evaluating speech and composition. It serves as an indicator of syntactic development and represents the shortest units into which a piece of discourse can be divided without leaving any sentence fragments. In the context of Japanese language assessment, Sakoda and Hosoi ( 2020 ) utilized T-unit as the basic unit to assess the accuracy and complexity of Japanese learners’ speaking and storytelling. The calculation of T-units in Japanese follows the following principles:

A single main clause constitutes 1 T-unit, regardless of the presence or absence of dependent clauses, e.g. (6).

ケンとマリはピクニックに行きました (main clause): 1 T-unit.

If a sentence contains a main clause along with subclauses, each subclause is considered part of the same T-unit, e.g. (7).

天気が良かった の で (subclause)、ケンとマリはピクニックに行きました (main clause): 1 T-unit.

In the case of coordinate clauses, where multiple clauses are connected, each coordinated clause is counted separately. Thus, a sentence with coordinate clauses may have 2 T-units or more, e.g. (8).

ケンは地図で場所を探して (coordinate clause)、マリはサンドイッチを作りました (coordinate clause): 2 T-units.

Lexical diversity refers to the range of words used within a text (Engber, 1995 ; Kyle et al. 2021 ) and is considered a useful measure of the breadth of vocabulary in L n production (Jarvis, 2013a , 2013b ).

The type/token ratio (TTR) is widely recognized as a straightforward measure for calculating lexical diversity and has been employed in numerous studies. These studies have demonstrated a strong correlation between TTR and other methods of measuring lexical diversity (e.g., Bentz et al. 2016 ; Čech and Miroslav, 2018 ; Çöltekin and Taraka, 2018 ). TTR is computed by considering both the number of unique words (types) and the total number of words (tokens) in a given text. Given that the length of learners’ writing texts can vary, this study employs the moving average type-token ratio (MATTR) to mitigate the influence of text length. MATTR is calculated using a 50-word moving window. Initially, a TTR is determined for words 1–50 in an essay, followed by words 2–51, 3–52, and so on until the end of the essay is reached (Díez-Ortega and Kyle, 2023 ). The final MATTR scores were obtained by averaging the TTR scores for all 50-word windows. The following formula was employed to derive MATTR:

\({\rm{MATTR}}({\rm{W}})=\frac{{\sum }_{{\rm{i}}=1}^{{\rm{N}}-{\rm{W}}+1}{{\rm{F}}}_{{\rm{i}}}}{{\rm{W}}({\rm{N}}-{\rm{W}}+1)}\)

Here, N refers to the number of tokens in the corpus. W is the randomly selected token size (W < N). \({F}_{i}\) is the number of types in each window. The \({\rm{MATTR}}({\rm{W}})\) is the mean of a series of type-token ratios (TTRs) based on the word form for all windows. It is expected that individuals with higher language proficiency will produce texts with greater lexical diversity, as indicated by higher MATTR scores.

Lexical density was captured by the ratio of the number of lexical words to the total number of words (Lu, 2012 ). Lexical sophistication refers to the utilization of advanced vocabulary, often evaluated through word frequency indices (Crossley et al. 2013 ; Haberman, 2008 ; Kyle and Crossley, 2015 ; Laufer and Nation, 1995 ; Lu, 2012 ; Read, 2000 ). In line of writing, lexical sophistication can be interpreted as vocabulary breadth, which entails the appropriate usage of vocabulary items across various lexicon-grammatical contexts and registers (Garner et al. 2019 ; Kim et al. 2018 ; Kyle et al. 2018 ). In Japanese specifically, words are considered lexically sophisticated if they are not included in the “Japanese Education Vocabulary List Ver 1.0”. Footnote 4 Consequently, lexical sophistication was calculated by determining the number of sophisticated word types relative to the total number of words per essay. Furthermore, it has been suggested that, in Japanese writing, sentences should ideally have a length of no more than 40 to 50 characters, as this promotes readability. Therefore, the median and maximum sentence length can be considered as useful indices for assessment (Ishioka and Kameda, 2006 ).

Syntactic complexity was assessed based on several measures, including the mean length of clauses, verb phrases per T-unit, clauses per T-unit, dependent clauses per T-unit, complex nominals per clause, adverbial clauses per clause, coordinate phrases per clause, and mean dependency distance (MDD). The MDD reflects the distance between the governor and dependent positions in a sentence. A larger dependency distance indicates a higher cognitive load and greater complexity in syntactic processing (Liu, 2008 ; Liu et al. 2017 ). The MDD has been established as an efficient metric for measuring syntactic complexity (Jiang, Quyang, and Liu, 2019 ; Li and Yan, 2021 ). To calculate the MDD, the position numbers of the governor and dependent are subtracted, assuming that words in a sentence are assigned in a linear order, such as W1 … Wi … Wn. In any dependency relationship between words Wa and Wb, Wa is the governor and Wb is the dependent. The MDD of the entire sentence was obtained by taking the absolute value of governor – dependent:

MDD = \(\frac{1}{n}{\sum }_{i=1}^{n}|{\rm{D}}{{\rm{D}}}_{i}|\)

In this formula, \(n\) represents the number of words in the sentence, and \({DD}i\) is the dependency distance of the \({i}^{{th}}\) dependency relationship of a sentence. Building on this, the annotation of sentence ‘Mary-ga-John-ni-keshigomu-o-watashita was [Mary- top -John- dat -eraser- acc -give- past] ’. The sentence’s MDD would be 2. Table 3 provides the CSV file as a prompt for GPT 4.

Cohesion (semantic similarity) and content elaboration aim to capture the ideas presented in test taker’s essays. Cohesion was assessed using three measures: Synonym overlap/paragraph (topic), Synonym overlap/paragraph (keywords), and word2vec cosine similarity. Content elaboration and development were measured as the number of metadiscourse markers (type)/number of words. To capture content closely, this study proposed a novel-distance based representation, by encoding the cosine distance between the essay (by learner) and essay task’s (topic and keyword) i -vectors. The learner’s essay is decoded into a word sequence, and aligned to the essay task’ topic and keyword for log-likelihood measurement. The cosine distance reveals the content elaboration score in the leaners’ essay. The mathematical equation of cosine similarity between target-reference vectors is shown in (11), assuming there are i essays and ( L i , …. L n ) and ( N i , …. N n ) are the vectors representing the learner and task’s topic and keyword respectively. The content elaboration distance between L i and N i was calculated as follows:

\(\cos \left(\theta \right)=\frac{{\rm{L}}\,\cdot\, {\rm{N}}}{\left|{\rm{L}}\right|{\rm{|N|}}}=\frac{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}{N}_{i}}{\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}^{2}}\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{N}_{i}^{2}}}\)

A high similarity value indicates a low difference between the two recognition outcomes, which in turn suggests a high level of proficiency in content elaboration.

To evaluate the effectiveness of the proposed measures in distinguishing different proficiency levels among nonnative Japanese speakers’ writing, we conducted a multi-faceted Rasch measurement analysis (Linacre, 1994 ). This approach applies measurement models to thoroughly analyze various factors that can influence test outcomes, including test takers’ proficiency, item difficulty, and rater severity, among others. The underlying principles and functionality of multi-faceted Rasch measurement are illustrated in (12).

\(\log \left(\frac{{P}_{{nijk}}}{{P}_{{nij}(k-1)}}\right)={B}_{n}-{D}_{i}-{C}_{j}-{F}_{k}\)

(12) defines the logarithmic transformation of the probability ratio ( P nijk /P nij(k-1) )) as a function of multiple parameters. Here, n represents the test taker, i denotes a writing proficiency measure, j corresponds to the human rater, and k represents the proficiency score. The parameter B n signifies the proficiency level of test taker n (where n ranges from 1 to N). D j represents the difficulty parameter of test item i (where i ranges from 1 to L), while C j represents the severity of rater j (where j ranges from 1 to J). Additionally, F k represents the step difficulty for a test taker to move from score ‘k-1’ to k . P nijk refers to the probability of rater j assigning score k to test taker n for test item i . P nij(k-1) represents the likelihood of test taker n being assigned score ‘k-1’ by rater j for test item i . Each facet within the test is treated as an independent parameter and estimated within the same reference framework. To evaluate the consistency of scores obtained through both human and computer analysis, we utilized the Infit mean-square statistic. This statistic is a chi-square measure divided by the degrees of freedom and is weighted with information. It demonstrates higher sensitivity to unexpected patterns in responses to items near a person’s proficiency level (Linacre, 2002 ). Fit statistics are assessed based on predefined thresholds for acceptable fit. For the Infit MNSQ, which has a mean of 1.00, different thresholds have been suggested. Some propose stricter thresholds ranging from 0.7 to 1.3 (Bond et al. 2021 ), while others suggest more lenient thresholds ranging from 0.5 to 1.5 (Eckes, 2009 ). In this study, we adopted the criterion of 0.70–1.30 for the Infit MNSQ.

Moving forward, we can now proceed to assess the effectiveness of the 16 proposed measures based on five criteria for accurately distinguishing various levels of writing proficiency among non-native Japanese speakers. To conduct this evaluation, we utilized the development dataset from the I-JAS corpus, as described in Section Dataset . Table 4 provides a measurement report that presents the performance details of the 14 metrics under consideration. The measure separation was found to be 4.02, indicating a clear differentiation among the measures. The reliability index for the measure separation was 0.891, suggesting consistency in the measurement. Similarly, the person separation reliability index was 0.802, indicating the accuracy of the assessment in distinguishing between individuals. All 16 measures demonstrated Infit mean squares within a reasonable range, ranging from 0.76 to 1.28. The Synonym overlap/paragraph (topic) measure exhibited a relatively high outfit mean square of 1.46, although the Infit mean square falls within an acceptable range. The standard error for the measures ranged from 0.13 to 0.28, indicating the precision of the estimates.

Table 5 further illustrated the weights assigned to different linguistic measures for score prediction, with higher weights indicating stronger correlations between those measures and higher scores. Specifically, the following measures exhibited higher weights compared to others: moving average type token ratio per essay has a weight of 0.0391. Mean dependency distance had a weight of 0.0388. Mean length of clause, calculated by dividing the number of words by the number of clauses, had a weight of 0.0374. Complex nominals per T-unit, calculated by dividing the number of complex nominals by the number of T-units, had a weight of 0.0379. Coordinate phrases rate, calculated by dividing the number of coordinate phrases by the number of clauses, had a weight of 0.0325. Grammatical error rate, representing the number of errors per essay, had a weight of 0.0322.

Criteria (output indicator)

The criteria used to evaluate the writing ability in this study were based on CEFR, which follows a six-point scale ranging from A1 to C2. To assess the quality of Japanese writing, the scoring criteria from Table 6 were utilized. These criteria were derived from the IELTS writing standards and served as assessment guidelines and prompts for the written output.

A prompt is a question or detailed instruction that is provided to the model to obtain a proper response. After several pilot experiments, we decided to provide the measures (Section Measures of writing proficiency for nonnative Japanese ) as the input prompt and use the criteria (Section Criteria (output indicator) ) as the output indicator. Regarding the prompt language, considering that the LLM was tasked with rating Japanese essays, would prompt in Japanese works better Footnote 5 ? We conducted experiments comparing the performance of GPT-4 using both English and Japanese prompts. Additionally, we utilized the Japanese local model OCLL with Japanese prompts. Multiple trials were conducted using the same sample. Regardless of the prompt language used, we consistently obtained the same grading results with GPT-4, which assigned a grade of B1 to the writing sample. This suggested that GPT-4 is reliable and capable of producing consistent ratings regardless of the prompt language. On the other hand, when we used Japanese prompts with the Japanese local model “OCLL”, we encountered inconsistent grading results. Out of 10 attempts with OCLL, only 6 yielded consistent grading results (B1), while the remaining 4 showed different outcomes, including A1 and B2 grades. These findings indicated that the language of the prompt was not the determining factor for reliable AES. Instead, the size of the training data and the model parameters played crucial roles in achieving consistent and reliable AES results for the language model.

The following is the utilized prompt, which details all measures and requires the LLM to score the essays using holistic and trait scores.

Please evaluate Japanese essays written by Japanese learners and assign a score to each essay on a six-point scale, ranging from A1, A2, B1, B2, C1 to C2. Additionally, please provide trait scores and display the calculation process for each trait score. The scoring should be based on the following criteria:

Moving average type-token ratio.

Number of lexical words (token) divided by the total number of words per essay.

Number of sophisticated word types divided by the total number of words per essay.

Mean length of clause.

Verb phrases per T-unit.

Clauses per T-unit.

Dependent clauses per T-unit.

Complex nominals per clause.

Adverbial clauses per clause.

Coordinate phrases per clause.

Mean dependency distance.

Synonym overlap paragraph (topic and keywords).

Word2vec cosine similarity.

Connectives per essay.

Conjunctions per essay.

Number of metadiscourse markers (types) divided by the total number of words.

Number of errors per essay.

Japanese essay text

出かける前に二人が地図を見ている間に、サンドイッチを入れたバスケットに犬が入ってしまいました。それに気づかずに二人は楽しそうに出かけて行きました。やがて突然犬がバスケットから飛び出し、二人は驚きました。バスケット の 中を見ると、食べ物はすべて犬に食べられていて、二人は困ってしまいました。(ID_JJJ01_SW1)

The score of the example above was B1. Figure 3 provides an example of holistic and trait scores provided by GPT-4 (with a prompt indicating all measures) via Bing Footnote 6 .

Example of GPT-4 AES and feedback (with a prompt indicating all measures).

Statistical analysis

The aim of this study is to investigate the potential use of LLM for nonnative Japanese AES. It seeks to compare the scoring outcomes obtained from feature-based AES tools, which rely on conventional machine learning technology (i.e. Jess, JWriter), with those generated by AI-driven AES tools utilizing deep learning technology (BERT, GPT, OCLL). To assess the reliability of a computer-assisted annotation tool, the study initially established human-human agreement as the benchmark measure. Subsequently, the performance of the LLM-based method was evaluated by comparing it to human-human agreement.

To assess annotation agreement, the study employed standard measures such as precision, recall, and F-score (Brants 2000 ; Lu 2010 ), along with the quadratically weighted kappa (QWK) to evaluate the consistency and agreement in the annotation process. Assume A and B represent human annotators. When comparing the annotations of the two annotators, the following results are obtained. The evaluation of precision, recall, and F-score metrics was illustrated in equations (13) to (15).

\({\rm{Recall}}(A,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,A}\)

\({\rm{Precision}}(A,\,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,B}\)

The F-score is the harmonic mean of recall and precision:

\({\rm{F}}-{\rm{score}}=\frac{2* ({\rm{Precision}}* {\rm{Recall}})}{{\rm{Precision}}+{\rm{Recall}}}\)

The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero.

In accordance with Taghipour and Ng ( 2016 ), the calculation of QWK involves two steps:

Step 1: Construct a weight matrix W as follows:

\({W}_{{ij}}=\frac{{(i-j)}^{2}}{{(N-1)}^{2}}\)

i represents the annotation made by the tool, while j represents the annotation made by a human rater. N denotes the total number of possible annotations. Matrix O is subsequently computed, where O_( i, j ) represents the count of data annotated by the tool ( i ) and the human annotator ( j ). On the other hand, E refers to the expected count matrix, which undergoes normalization to ensure that the sum of elements in E matches the sum of elements in O.

Step 2: With matrices O and E, the QWK is obtained as follows:

K = 1- \(\frac{\sum i,j{W}_{i,j}\,{O}_{i,j}}{\sum i,j{W}_{i,j}\,{E}_{i,j}}\)

The value of the quadratic weighted kappa increases as the level of agreement improves. Further, to assess the accuracy of LLM scoring, the proportional reductive mean square error (PRMSE) was employed. The PRMSE approach takes into account the variability observed in human ratings to estimate the rater error, which is then subtracted from the variance of the human labels. This calculation provides an overall measure of agreement between the automated scores and true scores (Haberman et al. 2015 ; Loukina et al. 2020 ; Taghipour and Ng, 2016 ). The computation of PRMSE involves the following steps:

Step 1: Calculate the mean squared errors (MSEs) for the scoring outcomes of the computer-assisted tool (MSE tool) and the human scoring outcomes (MSE human).

Step 2: Determine the PRMSE by comparing the MSE of the computer-assisted tool (MSE tool) with the MSE from human raters (MSE human), using the following formula:

\({\rm{PRMSE}}=1-\frac{({\rm{MSE}}\,{\rm{tool}})\,}{({\rm{MSE}}\,{\rm{human}})\,}=1-\,\frac{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-{\hat{{\rm{y}}}}_{{\rm{i}}})}^{2}}{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-\hat{{\rm{y}}})}^{2}}\)

In the numerator, ŷi represents the scoring outcome predicted by a specific LLM-driven AES system for a given sample. The term y i − ŷ i represents the difference between this predicted outcome and the mean value of all LLM-driven AES systems’ scoring outcomes. It quantifies the deviation of the specific LLM-driven AES system’s prediction from the average prediction of all LLM-driven AES systems. In the denominator, y i − ŷ represents the difference between the scoring outcome provided by a specific human rater for a given sample and the mean value of all human raters’ scoring outcomes. It measures the discrepancy between the specific human rater’s score and the average score given by all human raters. The PRMSE is then calculated by subtracting the ratio of the MSE tool to the MSE human from 1. PRMSE falls within the range of 0 to 1, with larger values indicating reduced errors in LLM’s scoring compared to those of human raters. In other words, a higher PRMSE implies that LLM’s scoring demonstrates greater accuracy in predicting the true scores (Loukina et al. 2020 ). The interpretation of kappa values, ranging from 0 to 1, is based on the work of Landis and Koch ( 1977 ). Specifically, the following categories are assigned to different ranges of kappa values: −1 indicates complete inconsistency, 0 indicates random agreement, 0.0 ~ 0.20 indicates extremely low level of agreement (slight), 0.21 ~ 0.40 indicates moderate level of agreement (fair), 0.41 ~ 0.60 indicates medium level of agreement (moderate), 0.61 ~ 0.80 indicates high level of agreement (substantial), 0.81 ~ 1 indicates almost perfect level of agreement. All statistical analyses were executed using Python script.

Results and discussion

Annotation reliability of the llm.

This section focuses on assessing the reliability of the LLM’s annotation and scoring capabilities. To evaluate the reliability, several tests were conducted simultaneously, aiming to achieve the following objectives:

Assess the LLM’s ability to differentiate between test takers with varying levels of oral proficiency.

Determine the level of agreement between the annotations and scoring performed by the LLM and those done by human raters.

The evaluation of the results encompassed several metrics, including: precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement.

Inter-annotator agreement (human–human annotator agreement)

We started with an agreement test of the two human annotators. Two trained annotators were recruited to determine the writing task data measures. A total of 714 scripts, as the test data, was utilized. Each analysis lasted 300–360 min. Inter-annotator agreement was evaluated using the standard measures of precision, recall, and F-score and QWK. Table 7 presents the inter-annotator agreement for the various indicators. As shown, the inter-annotator agreement was fairly high, with F-scores ranging from 1.0 for sentence and word number to 0.666 for grammatical errors.

The findings from the QWK analysis provided further confirmation of the inter-annotator agreement. The QWK values covered a range from 0.950 ( p  = 0.000) for sentence and word number to 0.695 for synonym overlap number (keyword) and grammatical errors ( p  = 0.001).

Agreement of annotation outcomes between human and LLM

To evaluate the consistency between human annotators and LLM annotators (BERT, GPT, OCLL) across the indices, the same test was conducted. The results of the inter-annotator agreement (F-score) between LLM and human annotation are provided in Appendix B-D. The F-scores ranged from 0.706 for Grammatical error # for OCLL-human to a perfect 1.000 for GPT-human, for sentences, clauses, T-units, and words. These findings were further supported by the QWK analysis, which showed agreement levels ranging from 0.807 ( p  = 0.001) for metadiscourse markers for OCLL-human to 0.962 for words ( p  = 0.000) for GPT-human. The findings demonstrated that the LLM annotation achieved a significant level of accuracy in identifying measurement units and counts.

Reliability of LLM-driven AES’s scoring and discriminating proficiency levels

This section examines the reliability of the LLM-driven AES scoring through a comparison of the scoring outcomes produced by human raters and the LLM ( Reliability of LLM-driven AES scoring ). It also assesses the effectiveness of the LLM-based AES system in differentiating participants with varying proficiency levels ( Reliability of LLM-driven AES discriminating proficiency levels ).

Reliability of LLM-driven AES scoring

Table 8 summarizes the QWK coefficient analysis between the scores computed by the human raters and the GPT-4 for the individual essays from I-JAS Footnote 7 . As shown, the QWK of all measures ranged from k  = 0.819 for lexical density (number of lexical words (tokens)/number of words per essay) to k  = 0.644 for word2vec cosine similarity. Table 9 further presents the Pearson correlations between the 16 writing proficiency measures scored by human raters and GPT 4 for the individual essays. The correlations ranged from 0.672 for syntactic complexity to 0.734 for grammatical accuracy. The correlations between the writing proficiency scores assigned by human raters and the BERT-based AES system were found to range from 0.661 for syntactic complexity to 0.713 for grammatical accuracy. The correlations between the writing proficiency scores given by human raters and the OCLL-based AES system ranged from 0.654 for cohesion to 0.721 for grammatical accuracy. These findings indicated an alignment between the assessments made by human raters and both the BERT-based and OCLL-based AES systems in terms of various aspects of writing proficiency.

Reliability of LLM-driven AES discriminating proficiency levels

After validating the reliability of the LLM’s annotation and scoring, the subsequent objective was to evaluate its ability to distinguish between various proficiency levels. For this analysis, a dataset of 686 individual essays was utilized. Table 10 presents a sample of the results, summarizing the means, standard deviations, and the outcomes of the one-way ANOVAs based on the measures assessed by the GPT-4 model. A post hoc multiple comparison test, specifically the Bonferroni test, was conducted to identify any potential differences between pairs of levels.

As the results reveal, seven measures presented linear upward or downward progress across the three proficiency levels. These were marked in bold in Table 10 and comprise one measure of lexical richness, i.e. MATTR (lexical diversity); four measures of syntactic complexity, i.e. MDD (mean dependency distance), MLC (mean length of clause), CNT (complex nominals per T-unit), CPC (coordinate phrases rate); one cohesion measure, i.e. word2vec cosine similarity and GER (grammatical error rate). Regarding the ability of the sixteen measures to distinguish adjacent proficiency levels, the Bonferroni tests indicated that statistically significant differences exist between the primary level and the intermediate level for MLC and GER. One measure of lexical richness, namely LD, along with three measures of syntactic complexity (VPT, CT, DCT, ACC), two measures of cohesion (SOPT, SOPK), and one measure of content elaboration (IMM), exhibited statistically significant differences between proficiency levels. However, these differences did not demonstrate a linear progression between adjacent proficiency levels. No significant difference was observed in lexical sophistication between proficiency levels.

To summarize, our study aimed to evaluate the reliability and differentiation capabilities of the LLM-driven AES method. For the first objective, we assessed the LLM’s ability to differentiate between test takers with varying levels of oral proficiency using precision, recall, F-Score, and quadratically-weighted kappa. Regarding the second objective, we compared the scoring outcomes generated by human raters and the LLM to determine the level of agreement. We employed quadratically-weighted kappa and Pearson correlations to compare the 16 writing proficiency measures for the individual essays. The results confirmed the feasibility of using the LLM for annotation and scoring in AES for nonnative Japanese. As a result, Research Question 1 has been addressed.

Comparison of BERT-, GPT-, OCLL-based AES, and linguistic-feature-based computation methods

This section aims to compare the effectiveness of five AES methods for nonnative Japanese writing, i.e. LLM-driven approaches utilizing BERT, GPT, and OCLL, linguistic feature-based approaches using Jess and JWriter. The comparison was conducted by comparing the ratings obtained from each approach with human ratings. All ratings were derived from the dataset introduced in Dataset . To facilitate the comparison, the agreement between the automated methods and human ratings was assessed using QWK and PRMSE. The performance of each approach was summarized in Table 11 .

The QWK coefficient values indicate that LLMs (GPT, BERT, OCLL) and human rating outcomes demonstrated higher agreement compared to feature-based AES methods (Jess and JWriter) in assessing writing proficiency criteria, including lexical richness, syntactic complexity, content, and grammatical accuracy. Among the LLMs, the GPT-4 driven AES and human rating outcomes showed the highest agreement in all criteria, except for syntactic complexity. The PRMSE values suggest that the GPT-based method outperformed linguistic feature-based methods and other LLM-based approaches. Moreover, an interesting finding emerged during the study: the agreement coefficient between GPT-4 and human scoring was even higher than the agreement between different human raters themselves. This discovery highlights the advantage of GPT-based AES over human rating. Ratings involve a series of processes, including reading the learners’ writing, evaluating the content and language, and assigning scores. Within this chain of processes, various biases can be introduced, stemming from factors such as rater biases, test design, and rating scales. These biases can impact the consistency and objectivity of human ratings. GPT-based AES may benefit from its ability to apply consistent and objective evaluation criteria. By prompting the GPT model with detailed writing scoring rubrics and linguistic features, potential biases in human ratings can be mitigated. The model follows a predefined set of guidelines and does not possess the same subjective biases that human raters may exhibit. This standardization in the evaluation process contributes to the higher agreement observed between GPT-4 and human scoring. Section Prompt strategy of the study delves further into the role of prompts in the application of LLMs to AES. It explores how the choice and implementation of prompts can impact the performance and reliability of LLM-based AES methods. Furthermore, it is important to acknowledge the strengths of the local model, i.e. the Japanese local model OCLL, which excels in processing certain idiomatic expressions. Nevertheless, our analysis indicated that GPT-4 surpasses local models in AES. This superior performance can be attributed to the larger parameter size of GPT-4, estimated to be between 500 billion and 1 trillion, which exceeds the sizes of both BERT and the local model OCLL.

Prompt strategy

In the context of prompt strategy, Mizumoto and Eguchi ( 2023 ) conducted a study where they applied the GPT-3 model to automatically score English essays in the TOEFL test. They found that the accuracy of the GPT model alone was moderate to fair. However, when they incorporated linguistic measures such as cohesion, syntactic complexity, and lexical features alongside the GPT model, the accuracy significantly improved. This highlights the importance of prompt engineering and providing the model with specific instructions to enhance its performance. In this study, a similar approach was taken to optimize the performance of LLMs. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. Model 1 was used as the baseline, representing GPT-4 without any additional prompting. Model 2, on the other hand, involved GPT-4 prompted with 16 measures that included scoring criteria, efficient linguistic features for writing assessment, and detailed measurement units and calculation formulas. The remaining models (Models 3 to 18) utilized GPT-4 prompted with individual measures. The performance of these 18 different models was assessed using the output indicators described in Section Criteria (output indicator) . By comparing the performances of these models, the study aimed to understand the impact of prompt engineering on the accuracy and effectiveness of GPT-4 in AES tasks.

Based on the PRMSE scores presented in Fig. 4 , it was observed that Model 1, representing GPT-4 without any additional prompting, achieved a fair level of performance. However, Model 2, which utilized GPT-4 prompted with all measures, outperformed all other models in terms of PRMSE score, achieving a score of 0.681. These results indicate that the inclusion of specific measures and prompts significantly enhanced the performance of GPT-4 in AES. Among the measures, syntactic complexity was found to play a particularly significant role in improving the accuracy of GPT-4 in assessing writing quality. Following that, lexical diversity emerged as another important factor contributing to the model’s effectiveness. The study suggests that a well-prompted GPT-4 can serve as a valuable tool to support human assessors in evaluating writing quality. By utilizing GPT-4 as an automated scoring tool, the evaluation biases associated with human raters can be minimized. This has the potential to empower teachers by allowing them to focus on designing writing tasks and guiding writing strategies, while leveraging the capabilities of GPT-4 for efficient and reliable scoring.

PRMSE scores of the 18 AES models.

This study aimed to investigate two main research questions: the feasibility of utilizing LLMs for AES and the impact of prompt engineering on the application of LLMs in AES.

To address the first objective, the study compared the effectiveness of five different models: GPT, BERT, the Japanese local LLM (OCLL), and two conventional machine learning-based AES tools (Jess and JWriter). The PRMSE values indicated that the GPT-4-based method outperformed other LLMs (BERT, OCLL) and linguistic feature-based computational methods (Jess and JWriter) across various writing proficiency criteria. Furthermore, the agreement coefficient between GPT-4 and human scoring surpassed the agreement among human raters themselves, highlighting the potential of using the GPT-4 tool to enhance AES by reducing biases and subjectivity, saving time, labor, and cost, and providing valuable feedback for self-study. Regarding the second goal, the role of prompt design was investigated by comparing 18 models, including a baseline model, a model prompted with all measures, and 16 models prompted with one measure at a time. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. The PRMSE scores of the models showed that GPT-4 prompted with all measures achieved the best performance, surpassing the baseline and other models.

In conclusion, this study has demonstrated the potential of LLMs in supporting human rating in assessments. By incorporating automation, we can save time and resources while reducing biases and subjectivity inherent in human rating processes. Automated language assessments offer the advantage of accessibility, providing equal opportunities and economic feasibility for individuals who lack access to traditional assessment centers or necessary resources. LLM-based language assessments provide valuable feedback and support to learners, aiding in the enhancement of their language proficiency and the achievement of their goals. This personalized feedback can cater to individual learner needs, facilitating a more tailored and effective language-learning experience.

There are three important areas that merit further exploration. First, prompt engineering requires attention to ensure optimal performance of LLM-based AES across different language types. This study revealed that GPT-4, when prompted with all measures, outperformed models prompted with fewer measures. Therefore, investigating and refining prompt strategies can enhance the effectiveness of LLMs in automated language assessments. Second, it is crucial to explore the application of LLMs in second-language assessment and learning for oral proficiency, as well as their potential in under-resourced languages. Recent advancements in self-supervised machine learning techniques have significantly improved automatic speech recognition (ASR) systems, opening up new possibilities for creating reliable ASR systems, particularly for under-resourced languages with limited data. However, challenges persist in the field of ASR. First, ASR assumes correct word pronunciation for automatic pronunciation evaluation, which proves challenging for learners in the early stages of language acquisition due to diverse accents influenced by their native languages. Accurately segmenting short words becomes problematic in such cases. Second, developing precise audio-text transcriptions for languages with non-native accented speech poses a formidable task. Last, assessing oral proficiency levels involves capturing various linguistic features, including fluency, pronunciation, accuracy, and complexity, which are not easily captured by current NLP technology.

Data availability

The dataset utilized was obtained from the International Corpus of Japanese as a Second Language (I-JAS). The data URLs: [ https://www2.ninjal.ac.jp/jll/lsaj/ihome2.html ].

J-CAT and TTBJ are two computerized adaptive tests used to assess Japanese language proficiency.

SPOT is a specific component of the TTBJ test.

J-CAT: https://www.j-cat2.org/html/ja/pages/interpret.html

SPOT: https://ttbj.cegloc.tsukuba.ac.jp/p1.html#SPOT .

The study utilized a prompt-based GPT-4 model, developed by OpenAI, which has an impressive architecture with 1.8 trillion parameters across 120 layers. GPT-4 was trained on a vast dataset of 13 trillion tokens, using two stages: initial training on internet text datasets to predict the next token, and subsequent fine-tuning through reinforcement learning from human feedback.

https://www2.ninjal.ac.jp/jll/lsaj/ihome2-en.html .

http://jhlee.sakura.ne.jp/JEV/ by Japanese Learning Dictionary Support Group 2015.

We express our sincere gratitude to the reviewer for bringing this matter to our attention.

On February 7, 2023, Microsoft began rolling out a major overhaul to Bing that included a new chatbot feature based on OpenAI’s GPT-4 (Bing.com).

Appendix E-F present the analysis results of the QWK coefficient between the scores computed by the human raters and the BERT, OCLL models.

Attali Y, Burstein J (2006) Automated essay scoring with e-rater® V.2. J. Technol., Learn. Assess., 4

Barkaoui K, Hadidi A (2020) Assessing Change in English Second Language Writing Performance (1st ed.). Routledge, New York. https://doi.org/10.4324/9781003092346

Bentz C, Tatyana R, Koplenig A, Tanja S (2016) A comparison between morphological complexity. measures: Typological data vs. language corpora. In Proceedings of the workshop on computational linguistics for linguistic complexity (CL4LC), 142–153. Osaka, Japan: The COLING 2016 Organizing Committee

Bond TG, Yan Z, Heene M (2021) Applying the Rasch model: Fundamental measurement in the human sciences (4th ed). Routledge

Brants T (2000) Inter-annotator agreement for a German newspaper corpus. Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece, 31 May-2 June, European Language Resources Association

Brown TB, Mann B, Ryder N, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems, Online, 6–12 December, Curran Associates, Inc., Red Hook, NY

Burstein J (2003) The E-rater scoring engine: Automated essay scoring with natural language processing. In Shermis MD and Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Čech R, Miroslav K (2018) Morphological richness of text. In Masako F, Václav C (ed) Taming the corpus: From inflection and lexis to interpretation, 63–77. Cham, Switzerland: Springer Nature

Çöltekin Ç, Taraka, R (2018) Exploiting Universal Dependencies treebanks for measuring morphosyntactic complexity. In Aleksandrs B, Christian B (ed), Proceedings of first workshop on measuring language complexity, 1–7. Torun, Poland

Crossley SA, Cobb T, McNamara DS (2013) Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System 41:965–981. https://doi.org/10.1016/j.system.2013.08.002

Article   Google Scholar  

Crossley SA, McNamara DS (2016) Say more and be more coherent: How text elaboration and cohesion can increase writing quality. J. Writ. Res. 7:351–370

CyberAgent Inc (2023) Open-Calm series of Japanese language models. Retrieved from: https://www.cyberagent.co.jp/news/detail/id=28817

Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, 2–7 June, pp. 4171–4186. Association for Computational Linguistics

Diez-Ortega M, Kyle K (2023) Measuring the development of lexical richness of L2 Spanish: a longitudinal learner corpus study. Studies in Second Language Acquisition 1-31

Eckes T (2009) On common ground? How raters perceive scoring criteria in oral proficiency testing. In Brown A, Hill K (ed) Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 43–73). Peter Lang Publishing

Elliot S (2003) IntelliMetric: from here to validity. In: Shermis MD, Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Google Scholar  

Engber CA (1995) The relationship of lexical proficiency to the quality of ESL compositions. J. Second Lang. Writ. 4:139–155

Garner J, Crossley SA, Kyle K (2019) N-gram measures and L2 writing proficiency. System 80:176–187. https://doi.org/10.1016/j.system.2018.12.001

Haberman SJ (2008) When can subscores have value? J. Educat. Behav. Stat., 33:204–229

Haberman SJ, Yao L, Sinharay S (2015) Prediction of true test scores from observed item scores and ancillary data. Brit. J. Math. Stat. Psychol. 68:363–385

Halliday MAK (1985) Spoken and Written Language. Deakin University Press, Melbourne, Australia

Hirao R, Arai M, Shimanaka H et al. (2020) Automated essay scoring system for nonnative Japanese learners. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 1250–1257. European Language Resources Association

Hunt KW (1966) Recent Measures in Syntactic Development. Elementary English, 43(7), 732–739. http://www.jstor.org/stable/41386067

Ishioka T (2001) About e-rater, a computer-based automatic scoring system for essays [Konpyūta ni yoru essei no jidō saiten shisutemu e − rater ni tsuite]. University Entrance Examination. Forum [Daigaku nyūshi fōramu] 24:71–76

Hochreiter S, Schmidhuber J (1997) Long short- term memory. Neural Comput. 9(8):1735–1780

Article   CAS   PubMed   Google Scholar  

Ishioka T, Kameda M (2006) Automated Japanese essay scoring system based on articles written by experts. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 17–18 July 2006, pp. 233-240. Association for Computational Linguistics, USA

Japan Foundation (2021) Retrieved from: https://www.jpf.gp.jp/j/project/japanese/survey/result/dl/survey2021/all.pdf

Jarvis S (2013a) Defining and measuring lexical diversity. In Jarvis S, Daller M (ed) Vocabulary knowledge: Human ratings and automated measures (Vol. 47, pp. 13–44). John Benjamins. https://doi.org/10.1075/sibil.47.03ch1

Jarvis S (2013b) Capturing the diversity in lexical diversity. Lang. Learn. 63:87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x

Jiang J, Quyang J, Liu H (2019) Interlanguage: A perspective of quantitative linguistic typology. Lang. Sci. 74:85–97

Kim M, Crossley SA, Kyle K (2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. Mod. Lang. J. 102(1):120–141. https://doi.org/10.1111/modl.12447

Kojima T, Gu S, Reid M et al. (2022) Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, New Orleans, LA, 29 November-1 December, Curran Associates, Inc., Red Hook, NY

Kyle K, Crossley SA (2015) Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Q 49:757–786

Kyle K, Crossley SA, Berger CM (2018) The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behav. Res. Methods 50:1030–1046. https://doi.org/10.3758/s13428-017-0924-4

Article   PubMed   Google Scholar  

Kyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity using direct judgements. Lang. Assess. Q. 18:154–170. https://doi.org/10.1080/15434303.2020.1844205

Landauer TK, Laham D, Foltz PW (2003) Automated essay scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis MD, Burstein JC (ed), Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 159–174

Laufer B, Nation P (1995) Vocabulary size and use: Lexical richness in L2 written production. Appl. Linguist. 16:307–322. https://doi.org/10.1093/applin/16.3.307

Lee J, Hasebe Y (2017) jWriter Learner Text Evaluator, URL: https://jreadability.net/jwriter/

Lee J, Kobayashi N, Sakai T, Sakota K (2015) A Comparison of SPOT and J-CAT Based on Test Analysis [Tesuto bunseki ni motozuku ‘SPOT’ to ‘J-CAT’ no hikaku]. Research on the Acquisition of Second Language Japanese [Dainigengo to shite no nihongo no shūtoku kenkyū] (18) 53–69

Li W, Yan J (2021) Probability distribution of dependency distance based on a Treebank of. Japanese EFL Learners’ Interlanguage. J. Quant. Linguist. 28(2):172–186. https://doi.org/10.1080/09296174.2020.1754611

Article   MathSciNet   Google Scholar  

Linacre JM (2002) Optimizing rating scale category effectiveness. J. Appl. Meas. 3(1):85–106

PubMed   Google Scholar  

Linacre JM (1994) Constructing measurement with a Many-Facet Rasch Model. In Wilson M (ed) Objective measurement: Theory into practice, Volume 2 (pp. 129–144). Norwood, NJ: Ablex

Liu H (2008) Dependency distance as a metric of language comprehension difficulty. J. Cognitive Sci. 9:159–191

Liu H, Xu C, Liang J (2017) Dependency distance: A new perspective on syntactic patterns in natural languages. Phys. Life Rev. 21. https://doi.org/10.1016/j.plrev.2017.03.002

Loukina A, Madnani N, Cahill A, et al. (2020) Using PRMSE to evaluate automated scoring systems in the presence of label noise. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA → Online, 10 July, pp. 18–29. Association for Computational Linguistics

Lu X (2010) Automatic analysis of syntactic complexity in second language writing. Int. J. Corpus Linguist. 15:474–496

Lu X (2012) The relationship of lexical richness to the quality of ESL learners’ oral narratives. Mod. Lang. J. 96:190–208

Lu X (2017) Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Lang. Test. 34:493–511

Lu X, Hu R (2022) Sense-aware lexical sophistication indices and their relationship to second language writing quality. Behav. Res. Method. 54:1444–1460. https://doi.org/10.3758/s13428-021-01675-6

Ministry of Health, Labor, and Welfare of Japan (2022) Retrieved from: https://www.mhlw.go.jp/stf/newpage_30367.html

Mizumoto A, Eguchi M (2023) Exploring the potential of using an AI language model for automated essay scoring. Res. Methods Appl. Linguist. 3:100050

Okgetheng B, Takeuchi K (2024) Estimating Japanese Essay Grading Scores with Large Language Models. Proceedings of 30th Annual Conference of the Language Processing Society in Japan, March 2024

Ortega L (2015) Second language learning explained? SLA across 10 contemporary theories. In VanPatten B, Williams J (ed) Theories in Second Language Acquisition: An Introduction

Rae JW, Borgeaud S, Cai T, et al. (2021) Scaling Language Models: Methods, Analysis & Insights from Training Gopher. ArXiv, abs/2112.11446

Read J (2000) Assessing vocabulary. Cambridge University Press. https://doi.org/10.1017/CBO9780511732942

Rudner LM, Liang T (2002) Automated Essay Scoring Using Bayes’ Theorem. J. Technol., Learning and Assessment, 1 (2)

Sakoda K, Hosoi Y (2020) Accuracy and complexity of Japanese Language usage by SLA learners in different learning environments based on the analysis of I-JAS, a learners’ corpus of Japanese as L2. Math. Linguist. 32(7):403–418. https://doi.org/10.24701/mathling.32.7_403

Suzuki N (1999) Summary of survey results regarding comprehensive essay questions. Final report of “Joint Research on Comprehensive Examinations for the Aim of Evaluating Applicability to Each Specialized Field of Universities” for 1996-2000 [shōronbun sōgō mondai ni kansuru chōsa kekka no gaiyō. Heisei 8 - Heisei 12-nendo daigaku no kaku senmon bun’ya e no tekisei no hyōka o mokuteki to suru sōgō shiken no arikata ni kansuru kyōdō kenkyū’ saishū hōkoku-sho]. University Entrance Examination Section Center Research and Development Department [Daigaku nyūshi sentā kenkyū kaihatsubu], 21–32

Taghipour K, Ng HT (2016) A neural approach to automated essay scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 1–5 November, pp. 1882–1891. Association for Computational Linguistics

Takeuchi K, Ohno M, Motojin K, Taguchi M, Inada Y, Iizuka M, Abo T, Ueda H (2021) Development of essay scoring methods based on reference texts with construction of research-available Japanese essay data. In IPSJ J 62(9):1586–1604

Ure J (1971) Lexical density: A computational technique and some findings. In Coultard M (ed) Talking about Text. English Language Research, University of Birmingham, Birmingham, England

Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Long Beach, CA, 4–7 December, pp. 5998–6008, Curran Associates, Inc., Red Hook, NY

Watanabe H, Taira Y, Inoue Y (1988) Analysis of essay evaluation data [Shōronbun hyōka dēta no kaiseki]. Bulletin of the Faculty of Education, University of Tokyo [Tōkyōdaigaku kyōiku gakubu kiyō], Vol. 28, 143–164

Yao S, Yu D, Zhao J, et al. (2023) Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

Zenker F, Kyle K (2021) Investigating minimum text lengths for lexical diversity indices. Assess. Writ. 47:100505. https://doi.org/10.1016/j.asw.2020.100505

Zhang Y, Warstadt A, Li X, et al. (2021) When do you need billions of words of pretraining data? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online, pp. 1112-1125. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.90

Download references

This research was funded by National Foundation of Social Sciences (22BYY186) to Wenchao Li.

Author information

Authors and affiliations.

Department of Japanese Studies, Zhejiang University, Hangzhou, China

Department of Linguistics and Applied Linguistics, Zhejiang University, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

Wenchao Li is in charge of conceptualization, validation, formal analysis, investigation, data curation, visualization and writing the draft. Haitao Liu is in charge of supervision.

Corresponding author

Correspondence to Wenchao Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental material file #1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, W., Liu, H. Applying large language models for automated essay scoring for non-native Japanese. Humanit Soc Sci Commun 11 , 723 (2024). https://doi.org/10.1057/s41599-024-03209-9

Download citation

Received : 02 February 2024

Accepted : 16 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1057/s41599-024-03209-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

BusyTeacher.org

HOWTO: 3 Easy Steps to Grading Student Essays

In a world where number two pencils and bubbles on an answer sheet often determine a student’s grade, what criteria does the writing teacher use to evaluate the work of his or her students? After all, with essay writing you cannot simply mark some answers correct and others incorrect and figure out a percentage. The good news is that grading an essay can be just as easy and straightforward as grading multiple-choice tests with the use of a rubric!

What is a rubric?

• A rubric is a chart used in grading essays, special projects and other more items which can be more subjective. It lists each of the grading criteria separately and defines the different performance levels within those criteria. Standardized tests like the SAT’s use rubrics to score writing samples, and designing one for your own use is easy if you take it step by step. Keep in mind that when you are using a rubric to grade essays, you can design one rubric for use throughout the semester or modify your rubric as the expectations you have for your students increase.

How to Grade Student Essays

What should I include?

When students write essays, ESL teachers generally look for some common elements . The essay should have good grammar and show the right level of vocabulary . It should be organized, and the content should be appropriate and effective. Teachers also look at the overall effectiveness of the piece. When evaluating specific writing samples, you may also want to include other criteria for the essay based on material you have covered in class. You may choose to grade on the type of essay they have written and whether your students have followed the specific direction you gave. You may want to evaluate their use of information and whether they correctly presented the content material you taught. When you write your own rubric, you can evaluate anything you think is important when it comes to your students’ writing abilities. For our example, we will use grammar, organization and overall effect to create a rubric .

What is an A?

Using the criteria we selected ( grammar , organization and overall effect ) we will write a rubric to evaluate students’ essays. The most straightforward evaluation uses a four-point scale for each of the criteria. Taking the criteria one at a time, articulate what your expectations are for an A paper , a B paper and so on. Taking grammar as an example, an A paper would be free of most grammatical errors appropriate for the student’s language learning level. A B paper would have some mistakes but use generally good grammar. A C paper would show frequent grammatical errors. A D paper would show that the student did not have the grammatical knowledge appropriate for his language learning level. Taking these definitions, we now put them into the rubric.

The next step is to take each of the other criteria and define success for each of those, assigning a value to A, B, C and D papers. Those definitions then go into the rubric in the appropriate locations to complete the chart.

Each of the criteria will score points for the essay. The descriptions in the first column are each worth 4 points, the second column 3 points, the third 2 points and the fourth 1 point.

What is the grading process?

Now that your criteria are defined, grading the essay is easy. When grading a student essay with a rubric, it is best to read through the essay once before evaluating for grades . Then reading through the piece a second time, determine where on the scale the writing sample falls for each of the criteria. If the student shows excellent grammar, good organization and a good overall effect, he would score a total of ten points. Divide that by the total criteria, three in this case, and he finishes with a 3.33. which on a four-point scale is a B+. If you use five criteria to evaluate your essays, divide the total points scored by five to determine the student’s grade.

Once you have written your grading rubric, you may decide to share your criteria with your students.

If you do, they will know exactly what your expectations are and what they need to accomplish to get the grade they desire. You may even choose to make a copy of the rubric for each paper and circle where the student lands for each criterion. That way, each person knows where he needs to focus his attention to improve his grade. The clearer your expectations are and the more feedback you give your students, the more successful your students will be. If you use a rubric in your essay grading, you can communicate those standards as well as make your grading more objective with more practical suggestions for your students. In addition, once you write your rubric you can use it for all future evaluations.

Like it? Tell your friends:

How to design a rubric that teachers can use and students can understand.

How to Evaluate Speaking

FAQ for Writing Teachers

Subscribe to the PwC Newsletter

Join the community, edit social preview.

Add a new code entry for this paper

Remove a code repository from this paper, mark the official implementation from paper authors, add a new evaluation result row.

• AUTOMATED ESSAY SCORING

Remove a task

Add a method, remove a method, edit datasets, automatic essay multi-dimensional scoring with fine-tuning and multiple regression.

3 Jun 2024  ·  Kun Sun , Rong Wang · Edit social preview

Automated essay scoring (AES) involves predicting a score that reflects the writing quality of an essay. Most existing AES systems produce only a single overall score. However, users and L2 learners expect scores across different dimensions (e.g., vocabulary, grammar, coherence) for English essays in real-world applications. To address this need, we have developed two models that automatically score English essays across multiple dimensions by employing fine-tuning and other strategies on two large datasets. The results demonstrate that our systems achieve impressive performance in evaluation using three criteria: precision, F1 score, and Quadratic Weighted Kappa. Furthermore, our system outperforms existing methods in overall scoring.

Code Edit Add Remove Mark official

Tasks edit add remove, datasets edit, results from the paper edit, methods edit add remove.

Each evening in her tent, researcher Jane Goodall would write up data from her field notebooks, recounting the chimpanzee behavior she observed that day. Immerse yourself in a replica of Jane’s research camp at “Becoming Jane: The Evolution of Dr. Jane Goodall,” an exhibition organized by National Geographic and the Jane Goodall Institute. The exhibition is open at the Natural History Museum of Utah in Salt Lake City, UT from December 7, 2023 through May 27, 2024. Photo by Hugo Van Lawick, Jane Goodall Institute

Murray 7th Grade Student Wins Opportunity to Meet Dr. Jane Goodall

Lily Peterson was selected as the winner of the “Inspired by Jane” Essay Contest  

SALT LAKE CITY, April 1, 2023 – On March 30, Lily Peterson, a Mountain Heights Academy 7th grader from Murray, Utah, was ushered backstage to meet her inspiration, the world-renowned Dr. Jane Goodall, DBE founder of the Jane Goodall Institute and UN Messenger of Peace.  In the meeting, Peterson presented an issue she hopes to address in Utah, and Peterson then wished Goodall a happy 90 th birthday on April 3. This rare opportunity came about after the Utah student won an essay contest hosted by the Natural History Museum of Utah and its partners, the Jane Goodall Institute and my529 Utah’s Educational Savings Plan , 

As the winner of NHMU’s "Inspired by Jane" Essay Contest, Peterson’s essay, which highlights challenges faced by wild mustangs in the Mountain West, rose to the top of more than 280 submissions from 6th, 7th, and 8th graders in Utah who were asked to respond to the writing prompt: “Knowing all that Dr. Jane Goodall as accomplished in her life so far. tell us what positive impact you hope to make in the world by your 90th birthday.” 

“The Natural History Museum of Utah congratulates Lily Peterson for her outstanding accomplishment and wishes her continued success in her academic and professional endeavors,” said Dr. Jason Cryan, The Sarah B. George Executive Director of NHMU. “Lily's passion and determination serve as an inspiration to her peers and the whole community, reminding us all of the power of youth in making a difference.” 

Peterson’s winning essay showcases her passion for veterinary science and her commitment to advocating for the well-being of mustangs in and around Utah. Her dedication is evident in her completion of an online class in equine welfare and management with UC Davis, a remarkable achievement for a student of her age. Peterson’s achievement in winning the "Inspired by Jane" Essay Contest is a testament to her exceptional talent in writing and critical thinking.  

“Being given the opportunity to meet Dr. Jane Goodall has been an incredible experience,” said Peterson. “I admire her because of all she has done as a scientist, conservationist, and activist. As an animal lover I am thankful that she has proven that animals have feelings and emotions too. I loved being able to talk with her one on one about her own childhood experience with horses. It was so special to meet someone who worked so hard to make their own big childhood dreams come true.  It is a moment I will never forget, and for which I am very thankful.” 

For her first-place prize, Lily traveled to Seattle, WA, on an all-expenses-paid trip to attend Goodall’s public lecture on March 30 and meet her after the event. In addition to this once-in-a-lifetime experience, Lily will also receive a $1,000 college savings certificate provided by my529, empowering her to continue her educational journey. my529 Utah’s Educational Savings Plan is designed to assist families, friends, and individuals in investing for a beneficiary’s future higher education. 

Goodall’s legacy in the fields of science and conservation are celebrated in Becoming Jane: The Evolution of Dr. Jane Goodall , a special exhibition open at the Natural History Museum of Utah until May 27, 2024. As featured in the exhibition, Goodall is now as famous for her work inspiring hope and action among the youth of the world, as she is for her groundbreaking research of wild chimpanzees.  

As the world wishes Goodall a 90 th birthday on April 3, NHMU is happy to celebrate her ongoing inspiration on the youth of the world, like Peterson and her own vision for change. 

Becoming Jane is free with the cost of admission to the Museum and always free to members. For tickets and more information about Becoming Jane , please visit: nhmu.utah.edu/jane.      

About the Natural History Museum of Utah    

The Natural History Museum of Utah is one of the leading scientific research and cultural institutions in the country. Established in 1963, the museum’s 10 permanent exhibitions are anchored by its state-of-the-art collections and research facilities containing almost 2 million objects. These collections are used in studies on geological, biological, and cultural diversity, and the history of living systems and human cultures within the Utah region. The museum hosts approximately 300,000 general visitors a year and provides one of the most spectacular private event settings in the Salt Lake City area. NHMU also broadens the reach of its mission through a variety of science-based outreach programs to communities and schools throughout Utah, reaching every school district in the state every other year. 

Download Media

Lily Peterson and Dr. Jane Goodall Eliza Petersen

Lily Peterson's Essay Lily Peterson

Press contacts and links

Press Contact

Margaret Chamberlain

Public Relations

Press Links

• Learn more about the Jane Goodall Institute
• Learn more about my529 Utah's Educational Savings Plan
• Read the winning essay

Related Press Releases

Anthropology

NHMU to Open “Becoming Jane” an Immersive Multimedia Exhibition on the Legacy of Dr. Jane Goodall 

Becoming Jane: The Evolution of Jane Goodall is open at NHMU from December 7, 2023 - May 27, 2024 celebrating Dr. Goodall's legacy of science and conservation.