Hard to evaluate and test LLM features

But we also have automated evals for AI systems, it's a well developed technique at this point, and it would bootstrap that process by providing some evidence. I think AI/ML systems are unique compared to other software in this regard, because the "how it works" is not inherent in the software itself, and not working once doesn't mean that it doesn't work at all, it could have been luck.

danpalmer · Oct 12, 2026 · comment in “Talorys – A self-hosted personal AI agent on Cloudflare's free tier”

> So by "multiple incidents" you mean a single minor incident involving a third party eval partner. I feel like you struggle to read.

Topfi · Sep 4, 2026 · comment in “Discovery of a new OpenAI agent message board”

It would show the org is paying attention, taking steps to balance model evals between interactive and API use. Also, some empathy for customers that wasted time trying to make opus 5 work for them.

bredren · Sep 1, 2026 · comment in “Claude Fable 5.1 and Claude Mythos 5.1”

Another problem seems to combine results from 2016 and 2019. Ignoring the question of credit, it makes it look like OpenAI doesn't actually understand the eval well in the first place, and undercuts the notion that the results represent great leaps forward in mathematical reasoning.

ctoa · Aug 6, 2026 · comment in “OpenAI's latest math breakthroughs commit research misconduct, experts say”

So they shipped the feature and tested it successfully locally with Claude Code on Fable 5. Then a few days later when preparing their new MCP public release, they ran a suite of evals on Armature and realized that small models could hallucinate audience_ids which would lead their MCP to send the campaign to ALL their contacts by default (which could obviously lead to disasters in prod).

screm · Aug 3, 2026 · story in “Show HN: Product analytics (and evals) for agent sessions on your MCP”

You've got a bunch of hackers in the comments going "But how can I run my own evals?" I think the answer for them is: this isn't for you. Don't want to derail what you're trying to do with Echo in case I'm wide of the mark, but yeah even in that case, if you hadn't considered that use case for it, I reckon there will, probably inside six months, be a substantial market for non-technical users who are sick of seeing …

alex-moon · Jul 24, 2026 · comment in “Show HN: Echo – Fable-level results at 1/3 the cost using open-weight models”

There’s a good eval floating around somewhere and tl;dr they’re awesome but the benchmarks are cooked, you’re better off with Qwen 8B Q4 than 27B 1b or ternary. Thanks for being skeptical, I maintain a llama.cpp-based client and it’s frustrating how high expectations are for local AI bc the median effort level means people mostly assemble their expectations and understanding via marketing soundbites

refulgentis · Jul 16, 2026 · comment in “Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU”

EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool:

efficax · Jul 10, 2026 · comment in “Show HN: Getting GLM 5.2 running on my slow computer”

Likert scale just doesn't work in LLM evals.

siquick · Jul 7, 2026 · comment in “Pruning RAG context down to what the answer actually needs”

It's hard to test this, and we can see why that is with an AI - it may claim to be conscious, or even claim not to be, but how can we trust either answer? Philosophers aren't even sure to trust another human who claims to be conscious, and we struggle constantly to determine at what point creatures the animal kingdom are conscious or not.

Festro · Jun 30, 2026 · comment in “Words Are a Byproduct of Consciousness. For LLMs, It's Backwards”

I'm the creator of hty, as part of my work for building custom eval suites at LatentEvals.com I found that AI agents struggled with interactive CLI / TUIs and would resort to setting flags that may be outdated or not exist at all.

anonfunction · May 20, 2026 · comment in “Show HN: Hty – Puppeteer for the terminal let's AI agents drive interactive CLIs”

Most annoying part is that it makes hard to test the model with your own inputs.

Karliss · May 18, 2026 · comment in “GenCAD”

Task Inference, Task Evaluation, Task Adherence, tool calling, all do very well on my evals. I did however for the first time in a while end my Claude Max subscription because, after their post-mortem [0] I for the first time saw true, reproducible, incredibly frustrating regressions in model output when using Claude Code.

Topfi · May 3, 2026 · comment in “Show HN: State of the Art of Coding Models, According to Hacker News Commenters”

So does it mean the paper only found a bug in some eval code on github that no one actually uses for anything? That doesn't seem to support their claim that AI benchmarks are broken, it only supports the claim that "unused code is often buggy".

_cs2017_ · Apr 11, 2026 · comment in “How We Broke Top AI Agent Benchmarks: And What Comes Next”

I didn't need anything fancy, just 'did the agent answer the question', 'did the agent cover the 5 things it needed to' - that type of thing. I found myself blocked by 'Gemini in Google Sheets', it was too slow and cumbersome, and it didn't handle eval changes well - particularly when trying to associate evals with ground truth.

raviisoccupied · Mar 29, 2026 · story in “Show HN: Beval – Simple evaluations for your AI product”

- 12 built-in eval rules across 4 categories: completeness (output length, coverage), relevance (keyword overlap, hallucination markers), safety (PII detection for SSN/credit card/phone/email, prompt injection patterns, blocklist), and cost (USD threshold, token efficiency)

iparent · Mar 14, 2026 · story in “Show HN: Iris – first MCP-native eval and observability tool for AI agents”

Repo: https://github.com/crasofuentes-hub/deterministic-agent-system Would love feedback from people working on reproducible agents, AI safety evals, or enterprise workflows where non-determinism is a real problem.

Agente2026 · Mar 9, 2026 · comment in “Building reproducible LLM agents with strict determinism guarantees”

Hi HN — I built Trajectly, a tool for deterministic regression testing of AI agents. Problem: agent “evals” are often flaky (network, time, tool nondeterminism, model drift), so it’s hard to tell if a change actually broke behavior.

ashmawy · Mar 5, 2026 · comment in “Trajectly – deterministic regression tests for AI agents”

PMs have context about what the product should do that engineers often don't, and that intuition IS an eval. The problem is it doesn't scale once the system has millions of conversations no one is reading.

laalshaitaan · Mar 4, 2026 · comment in “My spicy take on vibe coding for PMs”

A bit of a rant dressed up as an AI engineering post on LLM eval strategies. In general, we've struggled with how much the LLM eval/observability space centers around pre-baked scorers like ROUGE/BLEU, or off the shelf LLM-as-a-judge setups.

choffer · Feb 27, 2026 · comment in “Better practical evals for real-world LLM agents”

If I go to chatgpt.com it asks me if I'm sure. I wish they had not banned Claude unfortunately when they were evaluating LLMs I wasn't using Claude yet so I couldnt pipe up.

giancarlostoro · Feb 17, 2026 · comment in “Claude Sonnet 4.6”

In my day to day as a Product Manager working in a team that ships AI products, I often found myself wanting to do 'quick and dirty' LLM based evaluation on conversation transcripts and traces. I found myself blocked by 'Gemini in Google Sheets', it was too slow and cumbersome, and it didn't handle eval changes well.

raviisoccupied · Feb 9, 2026 · comment in “Ask HN: What are you working on? (February 2026)”

These days, with LLMs you can sort of build a scaffolding to work around this by asking the agent to break down the assembly in multiple stages and use external libraries to do the partial eval for you, but ... ugh ... what a mess.

ur-whale · Dec 21, 2025 · comment in “OpenSCAD is kinda neat”

On the other hand, while I have the experience to evaluate the output, some of my younger colleagues do not. They are learning, and when I have time to help them, I certainly do, but I wish they just didn't have access to LLMs.

philipwhiuk · Nov 28, 2025 · comment in “A trillion dollars (potentially) wasted on gen-AI”

On the other hand, while I have the experience to evaluate the output, some of my younger colleagues do not. They are learning, and when I have time to help them, I certainly do, but I wish they just didn't have access to LLMs.

BirAdam · Nov 28, 2025 · comment in “A trillion dollars (potentially) wasted on gen-AI”