Data Science capstone project batch #36

We’re happy to celebrate the achievements of our latest graduates from the Full-Time DS Batch #36, who have just wrapped up their Data Science journey with a remarkable, real-world project. 

This round of final presentation showcased how data science and AI can drive tangible impact across industries, from transforming business development workflows to reinventing the way of the new market discovery. 

Take a look at how our graduates are using data science to generate insights, push boundaries, and create real-world impact.

 

Beyond Aggregate Metrics: Reliability of LLMs as Judge

Project by: Shwetha Mangalore, Juan Sebastian Enciso, Thara Shanmugam
 

As artificial intelligence becomes increasingly integrated into education, an important question is whether large language models (LLMs) can assess student work as reliably as human experts.

A capstone project conducted in collaboration with Constructor Tech investigated this question by examining the reliability of LLMs as automated graders. The project focused on identifying the conditions under which an LLM can match human expert judgment and the situations in which human review remains necessary. The motivation lies in the scale and cost of educational assessment: grading student work manually requires trained reviewers and can become particularly time-consuming when large numbers of responses are involved.

The analysis focused on open-ended answers from biology, chemistry, and physics laboratory experiments. The dataset contained approximately 1,090 answer pairs from 20 experiments, with responses written in Spanish, English, and Turkish. Subject-matter expert (SME) grades served as the reference standard against which three LLMs were evaluated: GPT-4o-mini, Gemini 3 Flash, and Claude Sonnet 4.

The initial comparison showed clear differences between the models. Gemini achieved the lowest mean absolute error (MAE), at 5.85 points, compared with 8.1 for Claude and 13.8 for GPT-4o-mini. All three models completed virtually the entire dataset, suggesting that completion rate alone is not sufficient to assess grading reliability.

A more detailed analysis revealed another important finding: grading bias was not consistent across question types. The same model could over-score one type of question and under-score another. This led to an investigation of per-bucket bias correction, in which scores were adjusted according to the observed direction and magnitude of the model's error. However, the analysis also highlighted an important limitation: corrections calculated from one dataset must be validated on held-out data to determine whether they generalize beyond the original sample.

Agreement with human experts provided further evidence of the differences between models. Gemini achieved a weighted Cohen's kappa of 0.753, with 88.3% of its grades falling within 10 points of the SME score. GPT-4o-mini and Claude Sonnet 4 achieved lower agreement scores of 0.568 and 0.548 respectively.

The analysis ultimately points toward a human-in-the-loop approach rather than complete automation. Observation and classification questions showed potential for automated grading, while measurement and calculation questions were better suited to human review. Hypothesis and complex explanation questions continued to require human judgment because they involve scientific reasoning and may allow multiple valid responses.
 

The findings therefore suggest that the most effective role for LLMs in educational assessment may not be to replace expert graders, but to support them. Reliable, straightforward responses can potentially be automated, while more complex or uncertain cases can be escalated to human reviewers. Such an approach could reduce the burden of large-scale assessment while retaining expert judgment where it remains most important.