dataqbs

NEBench R1: a Reasoning Benchmark for Evaluating Nuclear Domain Competence in Large Language Models

· Source: arXiv cs.AI

NEBench R1 is a new test suite created to evaluate both the accuracy and the reasoning quality of large language models (LLMs) in the nuclear domain. The benchmark contains 7,546 multiple‑choice questions drawn from U.S. Nuclear Regulatory Commission reactor operator licensing exams, covering 18 facilities and categorized according to the NRC’s knowledge and skill catalogs. Each item includes the official explanation, which serves as a reference for assessing model reasoning. Six LLMs were evaluated, with accuracy scores ranging from 34 % to 73.9 %, distributed across three performance tiers. An LLM‑based judge applied a multidimensional rubric to correct responses to gauge the robustness of the reasoning. The results show that while the models grasp the problems well, they exhibit shortcomings in factual precision, procedural application, and the handling of distractors, indicating that correct answers are not always supported by sound technical justification. Additionally, longer reasoning traces were found to correlate with lower accuracy, suggesting that the length of the reasoning may reflect question difficulty rather than greater problem‑solving ability. This benchmark provides a tool for diagnosing LLM competence and reliability in critical nuclear engineering tasks, essential information for their safe deployment in regulated settings.

Read the original article on arXiv cs.AI

This summary is an informational synthesis produced by dataqbs.com. All rights to the original content belong to its author and the cited media outlet. We act solely as curators of technology news and claim no authorship.

Read this in Español · Deutsch · Français · Italiano