- Experienced developers actually spent 19% more time on tasks when using AI reports requirements
- Only 44% of AI-Generated Code was accepted by experienced DEVS
- Developers still feel that work is easier when they use AI
Artificial intelligence may not be as beneficial to experienced developers as it can be for new starters, and those who have not yet to develop the right skills have claimed new research.
A new study conducted by model evaluation and threat research (METR) has not only suggested that the developers were less optimistic after the study, but the real results suggest that artificial intelligence actually ended up costing them time.
The study found that Devs with AI spent less time code and search, and more time asking, waiting for and more important, review AI output. Estimated 9% of the time continued to review and clean up AI-generated code, with AI suggestions in general on the right tracks, but lacks details.
AI is actually not the same these developers at any time
Observing 16 experienced developers across 246 genuine tasks on mature open source projects that they were already familiar with, researchers analyzed how the developers interacted with popular tools from Cursor Pro and Claude 3.5/3.7.
Before the study, the 16 experienced developers are expected to reduce the task time by 24%when combining their expertise with artificial intelligence. After the study, they reduced their expectations to only 20%, but analysis after the study reveals that AI actually increased the task’s completion time by 19%.
Fewer than 44% of the proposals were accepted with a lack of contextual knowledge and large, complex storage sites highlighted as contributing factors to the developer’s slowdown. The study also noted that the experienced developers already had great familiarity with the code bases, leaving little space for AI to add some meaningful value.
However, despite the slowdown, many developers continued to use AI tools because the work felt less efforts, which made the work feel more comfortable, even if it wasn’t faster.
“AI capacities in nature may be lower than the results of commonly used benchmarks may imply,” concludes the research document.
