Back to journal

AI · · 1 min read

Feeling faster is a hypothesis worth measuring

What I took from METR’s early-2025 developer productivity study.

Feeling faster is a hypothesis worth measuring

Expanded from my Telegram note. The English edition is a translation and editorial adaptation.

The detail that interested me in METR’s developer study was the gap between perceived and measured speed. Developers felt that AI helped, while the measured result in that particular setting went the other way.

The experiment concerned experienced open-source contributors using early-2025 tools on repositories they knew well. Its result should be read within that setting, not carried unchanged into every team, model or task.

Count the whole task

When measuring your own work, include writing the request, waiting, reading, correcting and checking the result. A quick first draft can still require a long review.

Quality needs a definition too. For a data task, the answer should be reproducible and consistent with the input. For code, the relevant behavior should work and remain maintainable.

Try a small comparison

Choose comparable tasks and record the total time, review effort and errors you found. Keep notes about task familiarity and difficulty, since those affect the comparison.

This is a practical self-check rather than a controlled experiment. It can still reveal which parts of your workflow benefit from assistance and which need a different approach.

The lesson I keep is to treat my feeling of increased speed as a hypothesis. The useful decision comes from the work that actually reaches a checked result.

Ali Rashedi