Taste in the Loop

We’re seeing more and more teams begin to write evals. However, most software engineers are used to working with deterministic code, not probability distributions imposed by non-deterministic systems. Thus, their evals lack statistical rigor — “pass@k” and “I ran the evals 3 times” aren’t sufficient and will invariably lead you to make a “wrong” decision. Your evals need confidence bands. This paper explores ways you can add them effectively.

This post covers three techniques that can be used to hill-climb above frontier-level model performance on a specific benchmark. The first is a form of ensembling, where an open-weight model (GLM-5.1) is given a tool it can call to use an “advisor” (Opus 4.7) — this improved performance and significantly reduced cost (relative to Opus 4.7). Fireworks also applied supervised fine-tuning (SFT) and reinforcement learning (RL) to Kimi 2.6, with both experiments exceeding the performance of Opus 4.7 on the benchmark.

A major challenge with scaling the attention mechanism in LLMs is that it scales quadratically with the number of tokens in context (as every token has to attend to every other token). Subquadratic has released a new model using a new sparse-attention mechanism that they claim scales linearly with context length, while maintaining very high scores on retrieval “needle-in-the-haystack” tasks. Worth a read.

If you feel like you’re babysitting agents all day, you’re not alone. This research article explores a growing form of hidden labor the folks at Glean call botsitting:

87% of digital workers now use AI at work. 75% say it makes them more productive … yet only 13% say their organization is performing significantly better as a result. So where are the gains going? They’re being swallowed by a new, largely invisible form of labor. We call it botsitting.

OpenRouter released a new product offering called Fusion. It ensembles LLMs via sending your prompt to a set of sub-agents, each powered by a different LLM model, and combines their outputs via a separate LLM-powered judge.