Benchmarking an LLM-Powered Chatbot: Beyond Standard Metrics
In today's world of language models, many conventional benchmarks rely on standardized, single-output tasks. While useful for gauging raw language capabilities, they miss the nuanced challenges faced by specialised agents—like our root-cause-analysis chatbot, ell3. We therefore designed a composite benchmark that evaluates the entire system, not just the underlying LLM.
Rethinking Standard LLM Benchmarks
Benchmarks such as MMLU, HumanEval and HellaSwag are great for measuring general language proficiency or coding skills. But an RCA agent needs more: multi-turn reasoning, context retention, and integration with data files. A one-size-fits-all test simply isn't enough.
Unique Requirements of a Specialised AI Agent
- Maintaining context over extended dialogues that involve multiple micro-agents.
- Co-ordinating sub-components such as code-execution and tool invocation.
- Evaluating the impact of role-specific system prompts on overall performance.
- Working with task-specific data files that change from query to query.
Evaluation Criterion
Our composite benchmark combines quantitative, qualitative and operational metrics. A summary is shown below.
| Metric Category | Key Metric | Description | Evaluation Method |
|---|---|---|---|
| Quantitative | Response Accuracy | How often the bot correctly addresses sub-queries related to RCA tasks | Scoring against ground truth |
| Quantitative | Latency | Time taken for each response | Log analysis + count of back-and-forth between micro-agents |
| Qualitative | Context Retention | Ability to recall and maintain context across turns | Human evaluation |
| Qualitative | Dialogue Coherence | Logical flow and narrative consistency | Human evaluation |
| Operational | Robustness & Stability | Consistency under sustained multi-turn dialogue | Stress testing & monitoring |
Early Results
We kept the ell3 system constant and swapped only the LLM backend. Prompts were tuned for gpt-4o-mini; other models were run with the same prompts to gauge out-of-the-box performance.
| LLMs | Response Accuracy | Latency | Context Retention | Dialogue Coherence |
|---|---|---|---|---|
| gpt-4o-mini | 85.96% | Excellent | Excellent | Excellent |
| o4-mini | 82.46% | Excellent | Excellent | Excellent |
| Claude-Sonnet | – | – | – | – |
| Deepseek-r1 | – | Poor (Local host) | Poor | Good |
| Llama-3.2 | 0% | Poor (Local host) | Poor | Poor |
Conclusion: Towards Adaptive, Robust AI Systems
Benchmarking an integrated agent like ell3 means looking well beyond raw LLM scores. By combining automated scoring, human evaluation and operational stress-testing, we gain a holistic view of real-world performance—and a dependable feedback loop for continuous improvement.