Below, you’ll find answers to frequently asked questions about AI Scoring.
General Questions
What is AI Scoring?
AI Scoring is an Aware Premium feature that supplements a teacher’s test-scoring practices by using a Large Language Model (LLM) to score students’ constructed response answers against a pre-set rubric.
How can I obtain the AI Scoring feature?
With an Aware Premium license, district administrators with the Create Local Assessment Tests right can select Enable AI Scoring under Assessment Settings > AI Analytics. From there, they can limit the feature to specific test types by selecting Configure Test Types under Assessment Settings, choosing a test type, and then checking the box for Enable AI Scoring for this Test.
Teachers can then make AI Scoring available at the test level by accessing the Online tab under Author and toggling on Enable AI Scoring for Constructed Response Questions. If they want to make the feature available to certain questions, they can access the desired constructed response item, either by creating or editing it, then toggle on Enable AI Scoring.
What rubric does AI Scoring use to grade student responses?
AI Scoring supports Grades 3 through English II for STAAR Informational Text rubrics only and is limited to scoring against the Organization of Ideas (0-3 points) trait and Conventions (0-2 points) trait for a total of 5 possible points.
What is a trait?
A trait is a sub-rubric of the total rubric that has its own score range for a specific aspect of a student’s response. Rubrics can have many traits that add up to a total score. For example, the STAAR Extended Constructed Response Rubrics have a total score range of 0-5. They also consist of two traits. The first trait, Development and Organization of Ideas, is worth 0-3 points. The second trait, Conventions, is worth 0-2 points.
What if I don’t want my district to use AI Scoring?
AI Scoring is optional and is made unavailable by default. To enable or disable AI Scoring, refer to Using AI Scoring as an Administrator.
Usage
What kinds of student responses can I score with AI?
Teachers can use AI Scoring to score constructed response answers linked to a resource with a reading passage. Without a relevant reading passage, AI Scores are liable to be inaccurate and illogical, so be sure to review students’ answers and scores often. As a result, text entry responses cannot be scored with AI.
Does AI Scoring work with multipart items?
If the multipart item contains constructed response items linked to a reading passage, the AI Scoring feature will score students’ responses. It will not score any multiple choice, multiple selection, text entry, or other technology-enhanced items.
Can I change AI Scores I don’t agree with?
Yes, teachers can override any AI-generated Score. AI Scores are highlighted in a different color than manually entered scores. If you need to recover the AI-generated Score, select Revert on the scorecard.
What is my district’s AI Scoring usage limit?
A district’s usage limit can vary depending on the district’s size and needs. On average, we assign a usage limit of 25,000 per paid Aware license, which you can locate on the AI Analytics dashboard.
When does the usage limit reset?
The usage limit resets annually on July 1.
How do I request a higher usage limit?
Reach out to sales@eduphoria.net for inquiries about usage limits.
How does rescoring affect AI-scored answers?
Rescoring captures any saved AI Scores and won’t request them to be scored again. There are some limited exceptions to this, including changing the rubric if the wrong rubric was chosen.
What is the full range of analytics captured on the AI Scoring dashboard?
AI Scoring analysis is limited to the current school year and resets on July 1 each year.
Does rescoring an assessment use up any of my district’s AI Scoring allotment?
No. Rescoring an assessment from the Author screen recalls the saved score that has been submitted, similar to the behavior for non-AI-scored constructed response questions.
Security
How does AI Scoring protect students’ Personally Identifiable Information (PII) and other sensitive data?
Eduphoria does not send information to LLMs that could be used to identify a student, such as student names, student IDs, teacher names, subject or class names, district or school numbers, or other identifiable information.
What information does Aware’s AI Scoring feature share with the LLM?
For AI Scoring, only the student’s answer to a constructed response question is sent. Should the student provide sensitive information within their response, Eduphoria has partnered with entities that do not save or share data, and we employ security for information in transit. If you become aware that a student is sharing sensitive data, be sure to follow your district's procedures.
Troubleshooting
Why can I not enable AI Scoring for my district?
First, ensure your district has an Aware Premium license. Then, ensure you are assigned district-level access permissions and the Create Local Assessment Tests right.
The Score All AI CRs button in Enter Answers is grayed out. What do I do?
There are a few reasons the AI Scoring button might be disabled:
A score was just recently sent. It can take several minutes to process scores. If you’ve recently submitted a score request, come back later to see if the button is available again. If it’s been more than an hour, there may be an issue that you can report to Eduphoria Support.
A test author might not have fully enabled AI Scoring at the question level. Check with the test author to ensure that they have turned on all AI Scoring test and question settings and selected their rubrics.
There are no more AI Scoring requests available for the district. Each district is allocated a limited number of scores each school year. Verify with district administrators that there are scores available. Additional AI Scores are available for purchase.
A scoring job might have failed. When you send an AI Score request, but the job fails, the ability to hit the score button again is blocked for 15 minutes to avoid overloading the servers with score requests. Failure can occur for some of the following reasons:
The scoring service is down.
The AI model is unavailable.
There is a client-side internet outage or connection issue when the user sends the score request.
Support
Where can a district administrator learn more about enacting AI Scoring?
Visit our help article, Using AI Scoring as an Administrator, to learn how to manage access permissions for AI Scoring and view usage information for your district.
Where can a teacher learn more about how to grade student responses with AI Scoring?
Visit our help article, Using AI Scoring as a Teacher, to learn how to turn on the setting for constructed response questions and use AI Scoring to grade student responses on the Enter Answers screen.
Where can I go to get an Error message resolved?
First, determine whether the issue is a recoverable or unrecoverable error. You can resolve a recoverable error by rescoring or by overriding with a manual score. Unrecoverable errors indicate that a detected issue, such as cheating or discriminatory language, requires a teacher’s review of the student response and a manual score.
If you are experiencing a different issue or if the ones above persist, contact Support for further assistance.