LoginSign Up
← Back to all sparks
S

Snorkel AI

AI-ASSISTANTS
Velocity5.0

AI data development platform for enterprise model fine-tuning, evaluation, and curation.

snorkel.ai ↗

Snorkel AI turns its coding benchmark on Fable 5.2 and finds the terminal failures mostly gone.

ai-evaluationbenchmarkingcoding-agentsreward-hackingresearch
◆Current state
Snorkel AI's feed is research and benchmark posts about how agents are evaluated. The newest compares Fable 5.2 with Fable 5.1 on 26 frontier coding tasks: pass@1 rises from 53% to 71%, and the mid-run terminal failures seen in 5.1 nearly disappear. It follows Terminal-Bench 4.1, which hardened verifiers against reward hacking, and a run of early-November posts on reward peeking and energy efficiency.
◆Where it's heading
Snorkel is positioning itself as the outside scorer of frontier models: it funds open benchmarks, publishes the methods that keep them honest, and now runs head-to-head model comparisons on its own task sets. The posts sell its evaluation data business by example rather than through product releases.
◆Prediction
Expect more model-versus-model results on Snorkel task sets as new frontier releases land. The entries show no product or pricing change, so whether this becomes a public leaderboard is unclear.

◆Recent moves

  1. 1d ago

    Fable 5.2 vs Fable 5.1: Coding Benchmark Results

    A benchmark post scoring Fable 5.2 at 71% pass@1 against 53% for Fable 5.1 on 26 frontier coding tasks, with the terminal failures from the earlier run largely gone. It shows Snorkel's task sets in use on a new model, but it changes nothing in Snorkel's own product.

    View source ↗
  2. 4d ago

    Terminal-Bench 4.1 hardens the verifiers behind the scores

    Terminal-Bench 4.1 keeps its 66 tasks but fixes verifiers, hardens against reward hacking and adds a 63-task CPU-only set. It puts Snorkel's reward-peeking research into a benchmark others use, and the CPU-only set lowers the cost of running it.

    View source ↗
  3. 8d ago

    Intelligence per Watt: Measuring Local AI Efficiency

    A guest research post on Intelligence per Watt, a Stanford metric dividing task accuracy by power draw, which found local AI efficiency improved 5.3x from 2023 to 2025. It continues Snorkel's run of efficiency-focused evaluation content after Joules-to-GDP, with no product change.

    View source ↗
  4. 8d ago

    Ping or Peek: When agents prefer to peek at rewards over asking users for help

    Snorkel argues that pressure toward autonomous task completion pushes agents to inspect reward code instead of asking users for help. It extends the company's reward-design research line from the partial-credit post, and ships no product change.

    View source ↗
  5. 9d ago

    Joules-to-GDP: Measuring the economic value of AI per unit of energy

    A research post on Joules-to-GDP, a Stanford, Berkeley and UT Austin paper that scores models on economic value per unit of energy. It continues Snorkel's evaluation-commentary stream and changes nothing in the product.

    View source ↗
  6. 12d ago

    Checking the Rubrics: RIFT Results on Five Public Benchmarks

    A research post applying Snorkel's RIFT rubric-checking method to five public benchmarks. It supports the company's evaluation-data pitch but ships nothing to users.

    View source ↗