What happened? First, the developers burned time prompting genAI. One of the things people praising the technology to high heaven overlook is that writing good programming prompts is hard. As Domenic Denicola, a Google Chrome developer and the jsdom maintainer who was in the study, wrote, he was surprised at just “how bad the models are at implementing web specifications.”
Developers also wound up spending more time than expected reviewing generated code and fixing errors, including — surprise, surprise! — potential security holes.
Why are we so sure genAI makes things faster? Well, never forget that genAI companies are usually selling efficiency and productivity. The faster and more efficient they appear, the more likely you are to buy them. Leaving advertising hype aside, the researchers also noted that “While coding/agentic benchmarks have proven useful for understanding AI capabilities, they typically sacrifice realism for scale and efficiency.” This benchmarking, however, was all about real work.
This story originally appeared on Computerworld
