Software Interruptions: 2023-11-13 and 2023-11-14

Issue: Performance Level code base changes resulted in a compounded performance impact when using LockDown Browser for Aware Online Testing. 

Event window: 2023-11-13; ~8:45am CST through ~10:00am CST

Event window: 2023-11-14; ~8:45am CST through ~10:00am CST

Full Resolution: 2023-11-15; 1:15pm CST

Status: Resolved

Summary of recent events:

Over the weekend of Nov. 11-12, our teams deployed structural code and database changes related to student assessments, scores, and performance levels. These improvements are a part of supporting more robust assessment analysis tools for Aware users coming in 2024.

While rigorous testing was conducted within areas across the application suite where this data is accessed, two inefficient queries increased database resource utilization and slowed load times. The increased demand on the database servers accelerated rapidly, creating a “traffic jam” for other features in Aware attempting to access online testing and other assessment-related data.

It took longer than expected to locate and diagnose the queries that were causing this issue as compared to those queries simply affected by the issue. While one dedicated team was investigating the root cause issue, the Infrastructure team allocated additional database servers to handle the subsequent increased load.

As of Nov 15th around 1:15pm CST, a fix was deployed for two poor-performing queries within the Online Testing system that disproportionately impacted some LockDown Browser users.

Immediate next steps for Eduphoria:

• Ongoing efforts to improve database query efficiencies with Aware Quick Views.
• Increased server capacity will remain in effect.
• Eduphoria will continuously assess practices that prevent similar types of incidents from occurring in the future.