Taste in the Loop

Cognition created a new coding agent benchmark this week named FrontierCode, claiming it’s the first benchmark that attempts to measure how mergeable the agent-produced code is. It’s a good read for folks interested in how to create and structure evals for coding benchmarks.

Vincent, a core maintainer of OpenClaw, gives a tech talk on his “dark factory” workflow. Vincent is on the extreme end of AI-pilled adoption. Rather than reviewing code, he watches his agents’ reasoning traces across several terminal swim lanes, à la Cypher in The Matrix. It’s bizarre. Oddly, a lot of his workflow still feels very manual — for example, he runs over a dozen agents locally in swim lanes (but doesn’t seem to leverage the parallelism afforded by async cloud agents).

This research article debunks several myths about the impact of generative AI on software engineering, including “Writing Code Is the Bottleneck” and “AI Will Turn Individual Developers into 10x Developers.” Worth a read.

“Is agentic search (e.g., a grep tool) or vector-based code indexing (a form of RAG) superior?” has become an age-old debate for those building agent harnesses, with Anthropic in the former camp and Cursor in the latter. This research paper brings more rigor to the debate over which is better. Worth a read.

Sylvain has an interesting post on LinkedIn about an “eval” they did that asked an agent to resolve 60+ incidents on a K8s cluster, with mixed results. As Sylvain writes:

There’s a fine line engineering teams will have to walk: automating the SDLC without losing track of what they’ve built.