HPML Course, Columbia · 2026
Evaluating AssetOpsBench for LLM Industrial Agents
A team evaluation of IBM's benchmark for industrial operations agents, focused on whether LLMs use the right evidence and reasoning process for fault diagnosis.
What I built
Our team collected agent trajectories, audited time-series foundation model usage, and evaluated responses with an LLM judge. We translated the findings into an IEEE-format paper for Columbia's High Performance Machine Learning course.
Findings
- Ran 152 industrial operation scenarios.
- Identified a 35.5% TSFM misuse rate.
- Compared trajectory quality, redundancy, and diagnosis reliability.
Open-source contribution
Contributed to the IBM AssetOpsBench open-source project, extending the benchmark evaluation work for industrial LLM agents. View IBM AssetOpsBench on GitHub ↗