What are students using GenAI for?

• Many students report using GenAI as a syntax reference - far more than for code generation.
• Many of these were experienced programmers, who just liked not having to remember all syntax all of the time.
• A couple (again, more experienced) had used GenAI to quickly learn new libraries or languages.

Student T

5y, experienced programmer, even before starting UG. Little GenAI use

So I don’t use it very much for programming…

except for trivial lookups…

I do use it for… like one line things so my most frequent one is probably like how to make a 2D NumPy array that’s this many by this many because I just can’t remember like all the brackets and things like that. So I’d say, yeah, I use it quite a lot for silly little lines that I could look up if I wanted to, but… it’s much easier to get that from AI.

Whereas I wouldn’t use it for like big chunks of code. I think I’ve tried it a couple of times for that and … it’s just not always right and it’s, yeah, it confuses me more than just trying to figure it out myself.

Students are routinely going straight to GenAI when standard, non-AI tools could serve them just as well

This makes sense, since they are not taught how to use any editor tools, which have a somewhat high barrier to entry.

However, some students articulate how GenAI is not just more convenient than search, but can be more helpful:

when I ask it a question, I’m able to get almost sort of contextualised documentation. And that’s something I’ve used it for if I’m learning a new like library and I kind of want to get started quickly, but I hate having to like look through documentation, so I’ll just kind of prompt it a bit and I can get started like that.

Student H

3y, novice programmer, extensive GenAI user in many areas of life

oh I don’t know what to do, like, what should I do? I’m going to ask the AI, is something that I’m used to. That thought process is a kind of normal for me now.

They want to enjoy programming, but find it hasn’t yet “clicked” for them.

I feel like if I was good I probably would enjoy it, but it’s just always something that I feel like I’m… I always feel somewhat behind in it. I feel like I should be able to do more than I can quite often.

They find it stressful to be confronted by problems they cannot solve.

To make matters worse, their experiences asking for help from peers and lecturers has not been good.

They were just quite, it’s written there, why are you asking that question? It’s like, oh, no, OK. And then once you have a couple answers like that, you end up just like not really asking anymore.

So Chat-GPT really helps them

but then, I can ask that question to Chat-GPT… and it can rephrase and it will pick out the just the exact piece that I need.

Perceived usefulness, motivation to use GenAI

Most students saw GenAI as error-prone on complicated tasks but useful for documentation, debugging and ideation.

Some students have spent time experimenting with how to get the results they want from GenAI, or testing different GenAI tools.

if I have a problem and I’m wondering if it can do it, I’ll spend quite a lot of time rephrasing and seeing what it actually can do, you know, breaking it apart, see if it can do certain bits, stuff like that, just to see the ability that it has at the minute.

Me: did you ever ask it to just generate some code?

I did that probably every time, but it was more to see what it would do… it was still good I think to see how it went about the problems, just to cross check as well for future reference too.

Others report trying, finding it frustrating or not helpful, and quickly giving up on it.

when it works it’s great, but I think most of the time I’m just kind of frustrated and so I move on very quickly from it.

Not knowing how to use it:

I think right now I struggle to find ways to use it in learning to code

Frustrating experiences have a knock-on effect:

by like the third change it’s already breaking constantly and I need to constantly like look through it and figure out why it’s breaking … and it’s just like, I could probably have gotten like properly good at using matplotlib pyplot… if it wasn’t for the stupid decision to save a bit of time by using ChatGPT

Or just finding low motivation to try:

I think I’m a bit slow to like, pick up these kind of things, like I’m always a bit, I don’t really want to.

• Why are there these differences?
• How can we use these insights to shape practice? Do we want to encourage more tinkering and experimenting?

Who is not using GenAI?

Student F

• felt like they were worse than their peers in 1st year
• But very motivated & strategic, wanted to level up their skills
• Now have a job in software, low-level
• Felt intimidated by peers, worked alone: might think GenAI appealing!
• But no interest in GenAI!

So for example, I stick with like my old phone for like years until it’s literally like, barely working. And, you know, I see like new features and I’m like, oh, this is like, unnecessary and stuff.

I asked if they might use it in future?

But, I feel like it’s not at the stage where, you know, it can replace human creativity and everything, so, I don’t think I would use it for like, actually coming up with new solutions and things like that.

So a mixture of individual traits, low opinion of its abilities.

Student V

And then like for code, I just felt like it was cheating. Erm, (visually contemplating) which I think it… isn’t. I also don’t really know what to ask it to give me the right things, so I think I’m just not, I’m just not that comfortable with it and so I’ve never really tried.

I’m not like completely closed off to it. I just, like I’ve read that it’s quite bad for the environment, is one thing. Like I’ve seen some stuff about how like chat-GPT uses a lot more energy than just like a plain Google search would and, so I’m like if I can just Google it, why not.

I’ve never been like, oh my goodness, like, I just really wish that this were done, like, instantaneously and like I could just, like, get this like robot slave to do it for me.

Ultimately, this student does not feel good about using AI.

Is this being taken into account on the balance sheet? Is it actually worth insisting that a physics student must learn to use GenAI in order to complete a programming course in their physics degree?

Hesitations

Student E is a highly experienced programmer

I think, for a bit of my time I became quite reliant on it. And then I’ve stopped more recently.

Me: “is that because you’ve moved on to CoPilot, or any other reason?”

I don’t really know why actually. Maybe I’ve just got more confident in my abilities

Student G is a novice programmer who both feels that GenAI has made them a better programmer, but intentionally stopped using it after realising they were reliant:

I was using AI for like homework and stuff like that… I shouldn’t have done and it made it harder for me to learn how to code. But when I was actually like learning how to code outside of class, I wasn’t using AI. I was using like human resources I had available to me which was helpful. It was better.

Student U (1st year!) is a very strong programmer who frequently takes on advanced projects independently, and has struggled to find peers to work with at their level. They used GenAI in the past, but…

I think my thoughts around them [AI tools] are you know a little bit tainted by … well the air that surrounds them, that is, you know, it’s like crypto but stupider somehow.

Should it be incorperated into courses?

Almost all of the students reacted “no” reflexively, or immediately expressed concern. Some reflected and came round to “maybe, only if done carefully and not from the beginning”.

Student A:

I think the main thing you would have to be careful about is that you’re not getting to generative AI to completely solve the problem, because for earlier programming courses where it’s very important to develop an intuition, you can just use it to solve the problem without getting any sort of, again, problem solving development yourself.

Student E:

But for people that maybe had done less in the past, they wouldn’t learn anything… Like yeah, AI is great but you still need to know how to code, like fundamentally.

Student N:

even though there is value in talking through your code, there is a big difference between asking … and doing it yourself. Relying entirely on it is a mistake. Having a course that says, sure, use generative AI, is a mistake.

Student V:

you shouldn’t really be given like integral-calculator.com to learn how to integrate.

but immediately went on to say,

if it’s just become a part of how you program in the modern world, then it should be taught, but not as like a all powerful ‘this is gonna solve it for you’

Student I has a different view:

I think it’d be nice to have at least maybe one or two sessions to go over and be like, OK, this is what you can use Chat-GPT for, and to sort of like not feel like you’d be punished for using it.

Clearly banning it isn’t a solution

one of the problems actually I found with the course last year was it was very ‘don’t use AI’, but you could just ask the AI and it would give you the complete run of code.

Superficially similar students have very different experiences

Student B and Student E are superficially similar:

• Both learned to code very young
• Found university courses easy
• Work on coding projects for fun in their free time

Student B uses free Chat-GPT as “contextualised documentation”

Documentation is usually pretty general online, but when I ask it a question, I’m able to get almost sort of contextualised documentation.

but rarely bothers with code generation

when it works it’s great, but I think most of the time I’m just kind of frustrated and so I move on very quickly from it. I don’t continually ask for more help.

when I pointed out that free Chat-GPT was not the cutting edge:

I think I would struggle to get it to generate more than a few things at once just because, then it would, kind of, I would, I’d have to really like, look at it and then understand it, and I might as well write it myself

Student E started using Chat-GPT early and quickly moved onto CoPilot in VSCode

it’s just helpful, isn’t it? Like I don’t remember all syntax at all points. It just speeds up the workflow.

Student E challenged my expectation that “creative coders” would be less interested in GenAI

I like doing things from scratch if I can.

I just find it inherently quite interesting. I like problem solving and it’s… the only place I find myself feeling like actually creative anymore.

Ownership

One of my initial research questions: do students view GenAI as a tool or collaborator? Answer seems to be tool!

I mean, I just see it more as a tool, right? You’re still the controller of this tool, it just makes you more efficient and quick.

This is bolstered by the way students were using GenAI - for relatively simple ‘lookup’ tasks, or perhaps debugging or simple autocompletion.

Interesting to see if this changes as students start using the ‘vibecoding’ tools like Cursor, or Canvas from OpenAI) which seems explicitly designed as a kind of conversational partner.

Student C

Me: Do you think it’d be reasonable to say that ChatGPT is a co-author of your code?

Yeah, I would not hesitate to say that.

But they mostly describe it as a tool.

Student F

Student F (never used GenAI) talks of following an online tutorial

Yeah so I followed that, but I was making sure to actually like do all the coding myself as well, like along with it. And then I was like, I was like branching off, like doing things a bit my way as well.

Suggests that they do not need to start from a blank slate to feel like a project is theirs.

Students struggling with time pressures

Strong message - don’t use GenAI as an excuse to give us more assignments with shorter turnaround. We already feel like we’re not getting enough time to fully engage with material.

Student I

Remember Student I, one of the strongest advocates of GenAI. “How would you sum up your view towards generative AI?”

Timesaver. I think that time saving is the most important thing. Because I can do everything that Chat-GPT would tell me to do, but it can- for me it can take up to 10 times as long. And so with lots of other work going on that isn’t coding based, like quantum mechanics hand-ins or master’s projects, or that senior honours projects going on, to be able to have something that’s just: “I’ve just saved you two hours.”

What do we make of this from an educational point of view?

Really begs the question: how much adoption is driven by the desire to carve out more time for other things (e.g. studying for other modules, or just leisure time), rather than students finding it beneficial for their learning. Programming courses do not have to follow software companies - shouldn’t be cramming more into courses to make them more “productive”!

Student N

Student N’s interview lasted nearly 70 minutes. They were very critical about the university, but ultimately in a constructive way.

in a university physics degree, there is not time for that … you don’t get this chance to practise and try things and experiment and you know, learn in the same way because you have, because … you’re so beholden to the deadlines.

Student D

Student D is an “early adopter” who used various models, but still liked to tweak and tinker with the output.

So in the coursework you don’t have time to … pinpoint these mistakes, never understand to prevent a future mistake causing by this. It just stacks up and you’ll never know.

Student Q

Student Q is motivated to improve their programming, but prioritises their mathematics courses:

that’s always my worry with the with coding courses… You kind of don’t know what kind of coursework load you’re signing up for before you do it.

Student C

5th year, little programming experience before starting their degree.

Their account of a ‘fun’ project was a short (two-evening), self-contained side project that demonstrated a statistical concept.

They enjoy using GenAI when programming:

it’s, helping me in the parts that I don’t enjoy, which is going into the minutiae and the excruciating detail of there’s this library with these methods

I’m still having to do the parts that I enjoy, which is, you know, trying to break something apart, try to analyse something, try to figure out what’s really happening, translating everything into actual words and code.

However, when asked about an “important project”, they talked about a very frustrating experience, an internship where they had to use a large numerical model written by their supervisor.

having to understand someone else’s code that they’d written, and then there was all these things going on. I think that kind of kind of pushed me up…

So I had to just go through it and understand what it was happening … and then figure out how to make my own edits so that it would work in the way that I was supposed to be doing.

Me: were you having fun?

Oh fuck no. No, it was real frustrating. It was very tiring.

Me: why did you not use GenAI to help?

It was such a big repository that it just felt, I couldn’t, you know, how do you even upload the whole thing?

And then, you know, another thing is that it belongs to someone else. I wouldn’t have felt comfortable doing that.

Despite being highly positive about GenAI for programming, they have not fully embraced it

I am cautious about using AI outside of that, like it’s really only programming that, where I’m comfortable using AI.

For them, programming is not a creative or artistic endeavour, but a tool to get at interesting problems, and GenAI is a tool extension.

I wouldn’t use it for anything that involves like creativity or art.

Student I

Student I found learning to code at university extremely challenging:

it was like learning Greek for the first time

They felt intimidated by peers who were already experienced and could complete workshop assignments easily

Any kind of interaction with peers was incredibly stressful

They decided to “tactically fail” the course (Comp Sim) so that they could spend the whole summer holidays focusing on learning to code.

I basically gave up on that course by the end of Week 2

They felt instructions were insufficient, and assessed workshops didn’t give them enough time.

Unsurprisingly, they are one of the strongest advocates for GenAI as a time-saver.

I can do everything that Chat-GPT would tell me to do, but for me it can take up to 10 times as long.

Despite a difficult start, they find programming rewarding (but do not enjoy the process by itself)

I think it’s just I get a great sense of, like, achievement and accomplishment when I’ve spent, let’s say, three or four hours trying to do something and then, you know, you get so many errors … because you know, I can’t necessarily write code that just works straight out the gate. You know, you spend a lot of time doing it and then it works. And you’re like, wow, that’s so cool.

They still find it satisfying to debug code with the help of GenAI

It’s nice enough that I can do that [solve the problem], but in my mind I’m now at a stage where it’s like ehhh (shrugs) I’d rather just be able to go “Mr GPT,what’s going wrong?”

And when I can get it fixed … it’s still satisfying. It’s still nice that I’ve been able to fix the problem, but it just stops me spending hours on it.

However…

I am like presently aware though that it probably would have made me a worse coder in the long run though, because it was a very good thing for me to have had to go out and actually find stuff first. I think it would have made me too lazy.

They insisted that they made the fastest improvements when working on a group project

whenever I did overcome that [social anxiety] and actually asked peers for help, I did get the most benefit from that.

the times when at university that I have become a lot better at coding has been when I’ve been working in a group. And to me it seems like it’s much more realistic to the real world … I do think that coding in groups of three or four… whenever I’ve done that with peers, my understanding and knowledge of what the code is doing and the code that I’ve produced as a result, have just really increased, really improved.