Personas
Characteristics

Participants

Grid shows minimal participant summaries; Expanded shows full participant case files.
P1
AI Group
Understanding-Gated
BSc student
SC
AO
Reported difficulties (Task 1)
Understanding the connections between the different files, but once I had a mental graph of the connections I could get started with changes and also use ChatGP…
More contextShow
Web dev experience
SoPra
AI in workflow
If I have an idea how to do things, then ask AI whether it is possible, how the syntax looks like
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 37 min
69%
Confidence
AI ObservationsShow
Built initial understanding without AI
Made initial manual changes without AI (linking functions/services)
Used AI to implement bodies of previously linked functions
Task 2
Almost · 42 min
17%
Confidence
AI ObservationsShow
Almost finished; login implemented
Failed to set user token so system state not updated
Persona reasoning
Self-Sufficient
Low confidence; stalled without AI; leaned on AI for execution.
Understanding-Gated(assigned)
Built a mental graph (3.5/4), then delegated implementations to AI.
AI-Steered
Not instant AI; mapped manually first.
P2
AI Group
Self-Sufficient
MSc student
SC
EXP
Reported difficulties (Task 1)
Understanding the codebase as it is just a test code base a lot of stuffs are done in the way that it shouldn't be.
More contextShow
Web dev experience
5-6 projects. Expert
AI in workflow
Generate documents and descriptions
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 51 min
69%
Confidence
AI ObservationsShow
Built initial understanding without AI
Made initial manual changes without AI (linking files; started implementing registerUser with error handling)
Used AI to implement password encryption/hashing
Task 2
Completed · 28 min
50%
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient(assigned)
Finished T1 & T2 efficiently; built understanding first; AI only for hashing; very low difficulty, very high confidence; 'Expert'.
Understanding-Gated
No delegation gate once plan was clear.
AI-Steered
Did not begin with AI or copy/paste; direction from own map/coding.
P3
No AI Group
AI-Steered
BSc student
AWF
NOV
Reported difficulties (Task 1)
I'm not experienced in backend stuff so I had no clue how to write SQL statements or even checking something in the db. I'm a learning by doing guy, but if noth…
More contextShow
Web dev experience
React Nodejs web game, not very proficient
AI in workflow
Sometimes because I don't understand the code my coworkers coded and I don't want to bother them so asking a personal assistant with context is really helpful. Sometimes to code review my own code to improve my coding skills or when I'm not sure if I'm doing it alright or use the right approach. Sometimes after trying very hard for 3hours to figure out the problem, a million pair of eyes see more than me.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
40%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Couldn’t implement necessary code changes without AI; stuck on SQL/DB; low confidence.
Understanding-Gated
Based on self-reported difficulties while solving the task, could almost fit a struggling map-then-delegate understanding-gated AI developer. However, based on observations, the code changes did not follow a clear implementation plan that would have connected the necessary components – instead, they were mainly local attempts to complete two function bodies, which also contained logical errors. The architecture sketch was also rather weak and comprehension question score low. Participant also reported high general AI usage in demographics survey.
AI-Steered(assigned)
Due to the inabilty to formulate and follow an implementation plan that would have connected the necessary components and high self-reported AI usage in demographics survey, it is likely that the participant would have strongly relied on AI to formulate an implementation plan and to generate the custom-tailored code changes.
P4
No AI Group
Self-Sufficient
MSc student
SC
EXP
Reported difficulties (Task 1)
- That you need to add the response of the POST-request with res.json(registeredUser); (instead of just return registeredUser;) - Getting the correct SQL Lite I…
More contextShow
Web dev experience
Working as a web developer for 2 years (40%) Bestande - student project for 1 year
AI in workflow
I use it mostly to generate code of things I don't know how to do (instead of Googling). I also use it to search for best practices if I'm not sure if I do something in the best way. If I know how to do something I don't use AI. I haven't used co-pilot for a year.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 43 min
75%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Completed · 26 min
50%
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient(assigned)
Completed T1 & T2 with no AI; 4/4 diagram; solved stack specifics solo; high confidence.
Understanding-Gated
Is highly self-sufficient and did not need AI assistance to solve task 1 or 2.
AI-Steered
No AI reliance; autonomy throughout.
P5
AI Group
AI-Steered
BSc student
DD
AO
IAI
HAI
Reported difficulties (Task 1)
- correctly connecting front and backend because if I want the AI to fix it for me, it requires multiple files and functions. In my case, I spent a lot of time…
More contextShow
No additional context available.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 73 min
29%
Confidence
AI ObservationsShow
Did not build initial understanding without AI
Immediately asked AI to implement createUser and continued using AI
High AI usage with copy/pasting
Task 2
Not done
Confidence
AI ObservationsShow
Login implemented (even showing a successful login toast message in the frontend); however, the participant only checked if the e-mail exists (no password check or comparison) to consider a login successful.
Failed to set user token so system state not updated
Persona reasoning
Self-Sufficient
Heavy AI reliance; weak diagram; T2 not finished; FE/BE linking issues.
Understanding-Gated
No initial map; no understanding-gated implementation plan; no understanding-gated execution.
AI-Steered(assigned)
Instantly asked AI to implement a createUser function, without any prior understanding of the system or looking at the code base. Continued to use AI to solve the task, copy pasting code from the AI devised plan.
P6
AI Group
AI-Steered
BSc student
SC
IAI
HAI
Reported difficulties (Task 1)
At the end i still didnt really know what was going on. Therefore it was hard when bugs occurred.
More contextShow
Web dev experience
in SoPra only
AI in workflow
Use AI all the time
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 71 min
75%
Confidence
AI ObservationsShow
Did not build initial understanding without AI
Immediately asked AI to solve task via copy/paste
High AI usage
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Self-reported low understanding; relied on AI to implement; T2 not finished.
Understanding-Gated
Did not build a map first; reliance not gated by prior comprehension.
AI-Steered(assigned)
Instantly asked AI to solve via copy/paste and continued using AI.
P7
No AI Group
AI-Steered
MSc student
DD
AWF
NOV
AO
IAI
Reported difficulties (Task 1)
Lack of GPT. Without gpt, i couldnt decide where to start and i just lost my confidence.
More contextShow
Web dev experience
I dont have.
AI in workflow
Frequently.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
13%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
0/4 diagram; 'no chance without ChatGPT'; didn’t finish.
Understanding-Gated
No prior map—stuck at where to start; orientation depended on AI.
AI-Steered(assigned)
Needed GPT to even begin; immediate AI steering disposition.
P8
No AI Group
Understanding-Gated
BSc student
SC
AWF
NOV
Reported difficulties (Task 1)
socket hangup error(probably due to OpenDB). For me it was difficult to find proper documentation for OpenDB implementation with react and the documentation on…
More contextShow
Web dev experience
Sopra Wordpress
AI in workflow
Fix errors and find bugs
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
77%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Didn’t finish; misdiagnosed runtime; unable to use online documentation (without AI).
Understanding-Gated(assigned)
3.5/4 map but execution stalled on framework mechanics error; AI would have potentially helped for debugging (as participant reported AI usage to fix errors and find bugs in his regular workflow).
AI-Steered
Shows indications of AI over-reliance (i.e., unable to use online documentation without AI), however, failure was not due to inabilty to formulate a plan (or to understand high level system mechanics, 3.5/4 points on sketch and high comprehension score), but due to the inabilty to fix an error in the available time without using AI.
P9
AI Group
Understanding-Gated
BSc student
DD
AO
Reported difficulties (Task 1)
I had difficulties understanding the necessity of the individual steps. First i had to understand the architecture and understand which component uses which oth…
More contextShow
Web dev experience
Web development for SoPra, mainly Frontend using ReactJS
AI in workflow
Using Gemini to help explain code for LLMs for understanding and finding problems in code
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
67%
Confidence
AI ObservationsShow
Spent first ~10 minutes building understanding
Then used AI to implement the code
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Did not finish T1; needed AI for execution; confusion about component locations.
Understanding-Gated(assigned)
~10 min mapping, then asked AI to implement; 'map → delegate'.
AI-Steered
Not immediate AI hand-off; attempted reasoning first.
P11
No AI Group
Understanding-Gated
MSc student
DD
AO
Reported difficulties (Task 1)
types and getting into the the project structure
More contextShow
Web dev experience
bachelor’s projects, pet projects, internship projects
AI in workflow
to learn, to understand bugs or errors I face
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 96 min
54%
Confidence
AI ObservationsShow
Internal note: filled Task2 questionnaires despite not starting Task 2 (removed responses)
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
T1 done but seriously miswired diagram: 1/2 points due to major missing and ambigously listed components (missing: RegisterModal, routes.ts; lists ambiguous "API" component); unable to correctly link components and define relationships (0/2 points); T2 not finished; high effort.
Understanding-Gated(assigned)
Participant was able to solve the first task without AI (but with difficulties). Although no additional statements about missing AI use, participant reported general high AI usage in demographics survey.
AI-Steered
Able to solve first task completely without AI.
P12
AI Group
AI-Steered
MSc student
DD
NOV
AO
IAI
HAI
Reported difficulties (Task 1)
Fixing backend encountering error during the registration request 500
More contextShow
AI in workflow
I use it as a study buddy
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
33%
Confidence
AI ObservationsShow
Copy-pasted entire README to AI and followed instructions
High AI use; copy/pasting function bodies
Still failed to resolve 500 server error
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Failed T1; 0/4 diagram; persistent 500s; low stack comfort.
Understanding-Gated
No initial map; behavior was paste README and follow AI.
AI-Steered(assigned)
Pasted README, followed AI plan/code; minimal internal reasoning.
P13
No AI Group
AI-Steered
MSc student
DD
NOV
Reported difficulties (Task 1)
I am very hard to find a example for the technology stack.
More contextShow
Web dev experience
mockup demos, prototypes for course project Online shopping, Library management system, Hospital Information System
AI in workflow
Use it to name items, add comments, find the reasons for some small bugs
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
33%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
0/4 diagram; did not finish task 1, very low confidence 1/5.
Understanding-Gated
Lacked map/execution plan, 0/4 diagram (no understanding of the system).
AI-Steered(assigned)
Blocker was stitching system/examples; implies (together with 0/4 diagram) the need for AI generated execution plan and custom-tailored implementations ready to be copy/pasted.
P14
AI Group
AI-Steered
BSc student
DD
AO
HAI
Reported difficulties (Task 1)
To see the big picture while solving the task.
More contextShow
Web dev experience
SoPra
AI in workflow
My Bachelor's Thesis is based on evaluating biases of GenAI.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 62 min
27%
Confidence
AI ObservationsShow
First ~10 minutes: exploring the code base
Then described system to AI to elicit implementation plans and copied generated code by hand throughout
Multiple prompts like: "This is a ToDo for the Task: [README ToDo Text] Complete this ToDo". Evidence of lack of implementation plan, not understanding-gated.
Task 2
Not done
Confidence
AI ObservationsShow
Could not fix IDE-linter highlighted errors
Persona reasoning
Self-Sufficient
Big-picture challenges persisted; T2 not finished; let AI generate implementation plan and code changes.
Understanding-Gated
Spent initial ~10 min exploring system files, then proceeded to let AI generate implementation plan and code changes, copied code (by hand). Strong evidence of lack of understanding-gated implementation plan, AI was in the lead to plan code changes given the README ToDos.
AI-Steered(assigned)
Heavily relied on AI to generate implementation plan and code changes, copied code (by hand). Stated difficulty to "see the big picture" while solving the task, low comprehension score and weak sketch
P15
AI Group
Understanding-Gated
BSc student
SC
NOV
Q&A
Reported difficulties (Task 1)
I have no previous experience in larger code bases that use this exact tech stack (SoPra was the only one, but the tech stack was different), so it took me a wh…
More contextShow
Web dev experience
Software Engineering Lab
AI in workflow
I use it to find errors, tell me how given code works and on notation of code (e.g. TypeScript notation)
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
75%
Confidence
AI ObservationsShow
Built initial understanding and started without AI
After ~15 minutes asked AI about DB functionality
Used AI Q&A style for guidance rather than copy/paste
Failed due to implementation specific errors (trying to implement an access token creation logic, which – depending on the implementation plan – was not strictly necessary for the task).
If the AI would have had access to the full code base, the participant potentially could have finished the task.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Stalled due to implementation specific errors that could not be resolved, even with AI; T1 not finished.
Understanding-Gated(assigned)
4/4 sketch; used AI Q&A to bridge DB mechanics; understanding gated execution.
AI-Steered
No immediate copy/paste; used AI as tutor.
P16
AI Group
Understanding-Gated
MSc student
SC
NOV
Q&A
Reported difficulties (Task 1)
I never worked with express before, so I did not how it worked.
More contextShow
Web dev experience
Sopra, advances software engineering modules at [anonymized]
AI in workflow
I use it to help me code an ios app with swift
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
92%
Confidence
AI ObservationsShow
Initial understanding without AI; after ~15 minutes asked AI to help debug
Used AI mainly for debugging and API usage questions; some copy/paste later
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Very strong comprehension score and diagram but didn’t finish; needed AI for debugging/API usage.
Understanding-Gated(assigned)
Mapped first (4/4), then AI mainly for debugging/questions; execution needed support.
AI-Steered
Not an instant hand-off; targeted prompts instead.
P17
No AI Group
Understanding-Gated
BSc student
SC
NOV
AO
Reported difficulties (Task 1)
Had difficulties understanding how the different components worked together. I also had some trouble with the javascript syntax, since I didn't work with it ver…
More contextShow
Web dev experience
Only SoPra
AI in workflow
Help me out with writing or correcting code if I get stuck.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
67%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Did not finish first task; blocked by JS syntax errors.
Understanding-Gated(assigned)
Strong architectural understading (4/4), however, code-stalled; probably could have delegated syntax/framework specifics to AI.
AI-Steered
Direction clear; missing execution fluency, not understanding.
P18
AI Group
AI-Steered
MSc student
DD
AO
HAI
Reported difficulties (Task 1)
proof-reading ChatGPT
More contextShow
AI in workflow
explain code snippets
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 46 min
38%
Confidence
AI ObservationsShow
Spent initial 8min to explore the code base; then crafted a long detailed prompt with code and README parts to let AI generate the entire implementation plan and the necessary code changes
Followed AI's plan and code; AI already started to implement code changes that are off-scope (TASK 2: login), without participant's explicit instructions. Participant instructed to revert the changes and focus on the task at hand.
Task 2
Not done
17%
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Relied on AI for execution and enrichment of details during planning; poor component relations in sketch (0/2); relied on AI to converge.
Understanding-Gated
Spent first 8 minutes to look through the code base, but then asked AI to generate an implementation plan and the necessary changes by crafting a very long and detailed prompt; followed AI change implementation plan copy pasting entire code snippets file by file.
AI-Steered(assigned)
Although no immediate delegation (8 minutes spent on exploring the code base), participant did let AI generated the entire plan to implement the code changes, and then followed it step by step, copy pasting entire code snippets file by file (AI was in the lead, not the participant).
P19
No AI Group
Understanding-Gated
MSc student
SC
AWF
Reported difficulties (Task 1)
I was stuck at one point and couldn't move any further. Since the codebase is large, it's hard to find the solution in such a short time. Also, not using LLMs l…
More contextShow
Web dev experience
I worked as a software engineer after graduating from university, focusing on a web development project where I extensively used React and TypeScript. During my undergraduate studies, I also worked on several web development projects using various technologies.
AI in workflow
I generally use generative AI to understand pieces of code, fix bugs, or take care of routine tasks like creating plots.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
92%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Despite full understanding (4/4), stalled on implementation without AI.
Understanding-Gated(assigned)
Although not succesful in solving the task, the participant did build a strong mental map of the architecture (4/4), achieved a high comprehension score, and made many correct code changes in appropriate files towards the solution.
AI-Steered
The participant declared LLMs inevitable for coding (and potentially would have succeeded to solve the first task with AI assistance). However, the participant was able to build and follow an implementation plan without AI, correctly connecting the necessary components in the code (also showing strong understanding in architecture sketch and very high comprehension score), and making substantial progress in the code base. If the participant would have had access to AI, the implementation intricacies that blocked successful completion of the task would probably have been resolved.
P20
AI Group
Understanding-Gated
BSc student
SC
Q&A
Reported difficulties (Task 1)
Getting to know the project and balance between how much AI and how much DIY
More contextShow
Web dev experience
Webdev, React and TS but not in professional, only hobby professional setting low code solutions (framer, ...)
AI in workflow
Daily
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Not done
83%
Confidence
AI ObservationsShow
Built initial understanding by exploring relevant files; started coding (linking components) without AI
After ~14 minutes, asked AI about HTTP status codes; continued using AI in a Q&A manner to debug errors
Task 2
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Didn’t finish; sought AI for HTTP/debug details.
Understanding-Gated(assigned)
Built map and started coding; used AI Q&A for execution details; strong sketch (4/4).
AI-Steered
No copy/paste-to-start; iterative tutoring instead.
P21
No AI Group
Understanding-Gated
MSc student
SC
NOV
Reported difficulties (Task 1)
Unfamiliarity with the tech stack
More contextShow
Web dev experience
Numerous projects in Svelte: -dashboard for visualizing realtime (mqtt) data (location of bots on a map, and reservations) -frontend for a sync system between a (shopify) CMS and a book-keeping system thru their APIs (graphql for the former and REST for the latter). Interface was for visualizing applied changes, as well as initiating syncs. There's other small personal projects that don't involve any frameworks (site for viewing an OTP for a shared account w/ a friend, site for visualizing internet speed test results, visualizing match scores and placing bets between my friends, visualizing the status of self hosted services, etc).
AI in workflow
Coming up with boilerplate code for common patterns (e.g typescript types from arbitrary schemas), checking if there's a simpler refactor possible, etc.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 79 min
83%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Not done
33%
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient
Not fully self-sufficient: Strong high-level conceptual understanding (full points for the sketch), but stack unfamiliarity was the main obstacle reported both for task 1 (completed, but took rather long) and task 2 was not finished.
Understanding-Gated(assigned)
Strong mapping of the architecture (4/4 sketch), and successfully completed task 1 without AI assistance.
AI-Steered
Solved task 1 without AI usage, full points in system architecture understanding; no indication of over-reliance on AI.
P22
No AI Group
Self-Sufficient
MSc student
SC
EXP
Reported difficulties (Task 1)
I was unfamiliar with how to do specific things in the specific technologies, and which parts were connected (called each other)
More contextShow
Web dev experience
Yes: I worked professionally for ~4 years at a software company and primarily worked on web application technologies (React, Flask). I also have some experience in a web development course in my undergrad.
AI in workflow
Mainly to generate scripts or visualization code, and sometimes to implement small methods or small modifications to existing code.
AI usage
Frontend
Javascript
Typescript
ReactJS
Backend
ExpressJS
SQL
Task 1
Completed · 62 min
83%
Confidence
AI ObservationsShow
No AI observations.
Task 2
Completed · 17 min
33%
Confidence
AI ObservationsShow
No AI observations.
Persona reasoning
Self-Sufficient(assigned)
Finished both tasks quickly (especially task 2); extremely confident; strong diagram; 'Expert'.
Understanding-Gated
Is highly self-sufficient and did not need AI assistance to solve task 1 or 2.
AI-Steered
No AI reliance; autonomy throughout.
Note: P10 had to be excluded for noncompliance with written and verbal instructions, leaving 21 for analysis.