Project Proposal
Experiences of Learning to Code
Joe Marsh Rossney 
Ross Galloway
Britton Smith
This is the proposal submitted to the Principle’s Teaching Award Scheme (PTAS) in March 2024, minus the budget breakdown. Although the project changed shape during the course of its lifetime, this still accurately describes our thought process and intentions.
- Project title
- Incorporating student perspectives towards Generative AI into the redesign of programming courses in the School of Physics and Astronomy.
1 Project summary
The proposed project seeks to understand how the experience of undergraduate students taking programming courses has changed, and is changing, due to the sudden availability of 'Generative Artificial Intelligence' (GenAI) systems such as OpenAI's ChatGPT and GitHub/Microsoft’s CoPilot.
The School of Physics and Astronomy (SoPA) is exploring some possible alterations to its suite of programming courses, which would reflect a change in stance towards encouraging limited use of GenAI in a manner that is purposeful, responsible and critical. The essential aim of this project is to produce primary evidence from students' perspectives to inform planned course redevelopments motivated by similar aims, both within the SoPA and more widely.
The main inquiry will take the form of a series of interviews with ~25 student participants, whose experiences span the periods before and after the advent of GenAI. Additionally, we will take a global snapshot of student attitudes towards GenAI and our programming courses using a survey.
The insights will feed into the development of a set of guidelines and recommendations aimed at Course Organisers. Anticipating a wider relevance than the courses we are directly involved with, we will take steps to disseminate the findings of our study among colleagues in the SoPA, the broader University community, and further afield. The key results will be submitted for publication in a Focused Collection of the peer-reviewed journal Physical Review Physics Education Research. Finally, this study will serve as a baseline and potentially a blueprint for follow-up work as the technological landscape evolves.
2 Project aims
In this project we aim to meet the following key objectives.
1. To gain insight into the shifting perspectives of students towards their development as programmers in the wake of recent advances in GenAI-assisted coding tools.
2. To understand elements of our programming courses that are high-value in terms of positive learning experiences and outcomes.
3. To identify areas where GenAI stands to benefit or threaten these positive learning experiences and outcomes.
4. To develop a set of guidelines and recommendations that will assist Course Organisers in incorporating GenAI into programming courses.
5. To provide a blueprint that can be used or adapted in future for the purposes of monitoring and evaluation of further changes to the programming curriculum, and to develop resources to assist individuals carrying out such work.
The proposed project is standalone, but naturally forms an initial exploratory and baseline-establishment phase of a broader campaign of programme transformation within the School of Physics and Astronomy.
3 Background
3.1 Context
Generative Artificial Intelligence (GenAI) refers to a recent technological innovation which has produced systems that are capable of generating high-fidelity, contextual and multi-modal outputs in response to complex natural language inputs. What makes the GenAI revolution unprecedented is its enormous transformative potential in the hands of ordinary citizens, combined with breakneck speed at which it has been able to realise that potential on a global scale. OpenAI's ChatGPT was the fastest-growing consumer application in history, reaching 100 million monthly active users within two months of its launch in November 2022. All indicators point to GenAI becoming increasingly prevalent in future societies, where it is expected to become deeply integrated in the knowledge economy.
A recent UNESCO report titled Guidance for generative AI in education and research (Fengchun & Holmes, 2023) emphasised the unpreparedness of educational institutions for overseeing validation, integration and regulation of GenAI systems for educational purposes. This is underlined by the many divergent responses among individual educators and educational institutions (Xiao et al., 2023).
Over the last year, each of the programming courses in the SoPA have had different policies towards the use of GenAI, reflecting the different instincts of the Course Organiser(s). Some courses have imposed a strict ban on the use of GenAI to generate code submitted for assessment, treating it as a form of plagiarism and punishable with a mark of zero. Meanwhile other courses have taken a light-touch approach, allowing GenAI while attempting to convey its inherent limitations to students.
Given the lack of empirical evidence to support any given policy, this diversity is not necessarily a bad thing. However, there is consensus among Course Organisers that programming courses must be urgently updated, or at least the emphasis reconfigured, to give students a clarity of purpose and a more consistent experience as they progress through their studies.
3.2 Research Gap
There is unfortunately a dearth of high-quality empirical evidence available to guide educators towards incorporating GenAI into programming courses. Recent contributions to the academic literature have highlighted important considerations (Becker et al., 2023), but guidance aimed at educators is often generic – "AI can produce inaccurate content" – or, more worryingly, produced on behalf of 'EdTech' companies with vested commercial interests (Williamson, 2023).
Even more scarce are studies that focus on the student perspective. In this respect, the composition of current undergraduate students in the School of Physics and Astronomy is uniquely diverse; there are students who took the full set of programming courses without GenAI, others who took introductory courses without GenAI but have used it during advanced courses, and new students who are currently taking introductory courses with GenAI and have never known any different. This opens up a one-time opportunity to gather primary data from the perspectives of students for whom the sudden appearance of GenAI directly impacted their ongoing undergraduate experience, potentially affecting their attitudes, learning practices, and future outlook and career ambitions.
3.3 Theoretical Influences
A pragmatists' perspective on the overarching purpose of higher education is that it should equip the learner with a suite of experiences, technical skills and knowledge, and ill-defined 'higher-order' cognitive abilities that can be drawn on to engage with complex real-world issues and unseen challenges. We take it as axiomatic that these 'abilities' are emergent properties of slow, complex, and frequently self-driven processes loosely labelled 'deep learning' (McGregor, 2020). Our main substantive interest is in understanding where GenAI has the potential to enhance or disrupt deep learning processes, and thereby identify practical measures we can take to meet an optimal balance between deep learning and raw skill acquisition.
The themes we have singled out as likely indicators of deep learning are grounded in cognitive psychology. We furthermore build on a body of educational research concerned with project-based learning, owing to the fact that the majority of the programming courses in SoPA revolve around project work. In particular, empirical studies have identified features associated with a sense of ownership of one's work that positively correlate with hallmarks of deep learning - see Hanauer & Dolan (2014) and references therein. Given the popular framing of GenAI systems not as tools but as 'assistants' and 'collaborators', we aim to establish whether this is reflected in students self-reported feelings of ownership towards their coding projects.
3.4 Wider relevance
Although our study is situated within the SoPA, the difficulties of accommodating GenAI in existing courses with established forms of teaching and assessment are felt across disciplines. Arguably the programming courses offered by the SoPA are a good testing ground for a more general enquiry, being highly attended by students majoring in mathematics, chemistry, informatics, engineering and subjects further afield. This reflects the fact that these have been and continue to be high-value courses for a wide range of students whose interests, prior skills and life goals may be very different, and underscores the importance of a granular approach. Our enquiry is also based around broad themes, to be introduced shortly, that characterise general learning experiences, and are not specific to physics or to programming. We therefore expect any findings from our proposed study to resonate far more widely than one might otherwise expect.
4 Methodology
4.1 Data collection
The understudied nature of the topic demands a flexible and student-oriented approach to information gathering. The main source of primary data will therefore be a series of semi-structured interviews with a sample of approximately 25 student participants from the full range of year-groups. Interviews will be broadly guided towards an exploration of several key themes, identified a priori, while being responsive to new themes emerging from the student participants. A non-exhaustive list of the themes we intend to explore includes ownership (including agency, individuality, aesthetic, pride), affective state (enjoyment, boredom, frustration, epiphany), higher-order thinking (strategic thinking, connectivity), peer group (peer support & competition), and wider context (motivations, aspirations, future planning).
The process of recruiting the first round of student interviewees will originate with a short ~3 minute Jisc survey, sent to students' university email accounts, which will feature a tick-box to indicate that a student is willing to participate in an interview in exchange for a £20 gift voucher. From this initial pool of candidate interviewees we will select approximately 5 students for the first round of interviews, giving preference to those graduating this summer. The survey provides a secondary opportunity for data collection on a larger scale, which we will use to take a snapshot of student attitudes towards GenAI in relation to our programming courses. This will help to contextualise our conclusions from the interviews and establishes a baseline for future longitudinal studies.
The second round of interviewees will be recruited during the first week of the Autumn semester. The RA will stand alongside a temporary display in the James Clerk Maxwell Building foyer, where most students enter and exit. On this display or table there will be information about our project and a QR code for interested students to sign up. On this sign-up page there will be an option to respond to a prompt question, which we hope will help to select a group of interviewees with diverse perspectives.
4.2 Analysis
Interviews will be transcribed and coded: portions of text containing pertinent information will be 'tagged' with one or more discrete codes that correspond to ideas or themes contained therein, with a new code created every time a new theme or idea is encountered. The analysis will be guided by qualitative studies with a similar aim of understanding student attitudes towards their development as learners (Stanley et al., 2015; Thiry et al., 2011). The results of the analysis will be presented, where possible, such as to shed light on the following questions:
1. How do students understand their position/purpose/identity as programmers in light of GenAI?
2. How do students view their past/present/future studies in light of GenAI? In particular, which elements of SoPA's programming courses were most valuable or formative?
3. Are there marked differences in the experiences and perspectives of students who took the same programming courses with and without use and/or awareness of GenAI?
Raw survey data will be processed and represented by visualisations and summary statistics.
4.3 Epistemological standpoint
Interviews will be conducted, transcribed and analysed by a Research Assistant (JMR) with recent, first-hand experience of relevant programming courses as both a student (2014-18) and instructor (2019-23). Their proximity to the student experience is central to the justification of an interpretivist analysis, which aims to 'read between the lines' and tease out meaning from complex responses.
JMR is also supervising two undergraduate students' summer projects which share the theme of generative AI in university teaching. Although these students will focus on their own independent scholarship-funded projects, they have expressed a keen interest in the project proposed here. We are therefore considering the possibility that these students could have some modest involvement, potentially as beta-testers for the interviews and providing 'quality control' on the post-interview analysis.
4.4 Data Privacy and Ethics
The survey will not request information on ethnicity since such information can be used to identify individuals within such a small population. We will instead ask survey respondents to indicate whether they are a UK student or an international student. Students will be asked to sign a consent form to indicate that they are happy for their interview to be recorded and analysed, and for segments of the interview to anonymously quoted in the write-up. The original recordings will be deleted after transcription, and the transcriptions will be redacted to remove identifiable, sensitive or irrelevant information before the coding is performed.
5 Project plan
| Date-date | Activity | Description of activity |
|---|---|---|
| June 2024 | Initial survey and call for final-year participants | Disseminate a short survey to undergraduate students using the mailing lists for SoPA programming courses delivered in the last 4 years. Aim to recruit graduating students for the first round of interviews |
| June 2024 | Background research | Consolidate our knowledge of existing research in this area and bring ourselves up to date with very recent work on GenAI in education. |
| July 2024 | Interview design | Develop an initial interview structure with question prompts and additional guidance |
| July-August 2024 | First round of interviews | Conduct interviews with ~5 third- and fourth-year students using Microsoft Teams |
| August 2024 | Transcription, coding, initial analysis | Transcribe and code first round of interviews using QualCoder, and perform preliminary analysis |
| August 2024 | Review interview design | Review interview design and delivery based on the initial round of interviews |
| August 2024 | Interim report | Write an interim report serving as a blueprint for the second part of the study |
| Late August 2024 | RA hand-over | JMR hands over to the second RA |
| September 2024 | Recruit for second round of interviews | Recruit candidate interviewees by approaching students entering James Clerk Maxwell Building on a day during week 1 |
| September 2024 | Second round of interviews | Conduct interviews with a representative sample of ~20 students using Microsoft Teams or in-person, depending on the student |
| September-October 2024 | Transcription, coding, initial analysis | Transcribe and code second round of interviews using QualCoder, and perform preliminary analysis |
| November 2024 | Main analysis | As described in ‘Method’ section |
| December 2024 | Write up and dissemination | Write a first draft of the article to be submitted for publication, finalise the blog and create a project homepage on the SoPA webpages |
| November-December 2024 | Internal seminar(s) | Seminars with the key staff in SoPA, CSE group in Informatics & potentially others |
| January 2025 | Conference presentation | Present findings at a conference on Generative AI in Education (GenAIEdu 2025), expected to be held at the University of Ulster. |
6 Project outcomes & impact
Earlier we emphasised the urgent need to move beyond general statements and hypotheses about the likely impact of GenAI on learners, towards a nuanced and granular understanding based on empirical evidence. The conclusions of our study will therefore be of substantive interest in and of themselves, which is why we intend to put them forward for wider dissemination via a peer reviewed journal.
At the same time, our primary motivation for this research is the need to update the suite of programming courses in the School of Physics and Astronomy. Drawing on our depth of experience and insights gained from our study, we will create a set of guidelines and recommendations aimed at assisting Course Organisers (ourselves included). Due to the magnitude and ongoing nature of these changes we anticipate the need for an adaptive, multi-stage process of course redevelopment. We feel it is important to ensure that the student voice is listened to from the very start of this process.
The University of Edinburgh is a global leader in education research, and the School of Physics and Astronomy stands among a very small number of UK institutions with a dedicated research group in tertiary physics education. This project will underline our reputation for high-quality, timely research, and we fully expect there to be wider interest in the results of our study among other universities.
8 Updates & response to PTAS panel feedback
8.2 Ethics Approval
“Please confirm that approval has been obtained or will be obtained before the research commences.”
We can confirm that we have obtained ethics approval for this project from the School of Physics & Astronomy, in line with Universities UK Concordat on Integrity, CSE and UKRI policies.
8.3 Timeframe & Budget
“The committee felt that the current timeframe was too short for the scope of the proposed work, especially with an April 2024 start. We would like you to extend the timeframe for project and should consider adding 3-4 months to the current timeframe to enable all work to be completed.”
We agree that the original time-frame was optimistic. We propose that the start date is postponed to June 1st and the project duration is extended by 3 months to cover the Autumn semester 2024. In addition to allowing more time to do the research thoroughly, this has the advantage that the bulk of the interviews may be conducted during the semester where students are present on-campus.
JMR will not be able to continue as RA from September 1st. However, we have found an excellent candidate to take over this role for the remaining four months of the project. This individual, who will graduate with a Bachelors degree this summer, is enthusiastic and highly capable; by the time they take over they will have already completed two research projects on the topic of GenAI and teaching – their Bachelors thesis and a summer research placement under the supervision of JMR. There will be a seamless handover at the end of August, and JMR will remain available to provide support and guidance throughout the remainder of the project duration.
“Revisit and revise the budget. Add additional research assistant costs for extending the timeframe.”
We have added an additional three months of stipend for part-time RA work, on top of the four months originally requested. We have adjusted the pay grade in line with standard pay for PDRAs and PGRAs in SoPA.
“The budget details no costs for interview transcription. Please add these if necessary. If not necessary, please indicate why this isn’t required.”
The RA will do the transcribing so these costs are contained within the RA's stipend. Automated transcription software will be used to create a first approximation of a transcript, which the RA will then check, edit and format while listening to the audio recording. We feel that this approach strikes a reasonable balance between time-efficiency and the analytical benefits of the researcher doing the transcribing themselves. We will only use transcription software for which we are completely satisfied that the input data is secure, i.e. not used for any other purposes and deleted once the transcription is finished.
“Add costs for dissemination, in particular conference attendance is mentioned in the application. Please add costs if this is required.”
We have listed several ways in which we will disseminate our research both within and outside of the University. The majority of these do not come with a cost. However, we would like to send a member of our team to deliver a presentation at a dedicated conference for AI in Higher Education in 2025. There are several such conferences held in Europe each year (e.g. the National Conference on Generative Artificial Intelligence in Education mentioned previously), but at this point dates and conference fees for 2025 have not been publicised. Based on the fees for this years' conferences, we have requested £1000, which includes an estimate of £500 for accommodation and travel. Should we end up requiring less than £1000 we will happily return the remaining money to PTAS.
Our proposed article has passed first review and is currently under second review for the Physical Review Physics Education Research journal. This is a 'Gold' open-access journal and should our article be accepted for publication the cost of publishing will be $1430 (approximately £1126). Should our proposal be rejected we will submit it to another peer-reviewed journal which will likely have similar publishing costs.
8.4 Responses to additional comments
“Would an element of anonymity be useful? Will students who have engaged with courses where AI was banned be honest about their views?”
We agree! The student interviewees will be entirely anonymous from the perspective of teaching staff at the University. Only the RAs, who will not have further teaching duties beyond the end of this project, will know their identity. As stated previously, we will ensure that interviewees fully understand this, and the way in which their data will be anonymised, prior to the interview.
The information gathered in interviews will not be used to ‘check for cheating’ in courses that are in progress. There is little risk of this since none of our team are involved in courses which impose a strict ban on GenAI.
For the survey, we will also keep students anonymous. This means we can ask them to fill in general information such as their year group and preferred gender, but we will refrain from asking for information on ethnicity or any other characteristics that might identify individuals when corroborated.
“The relationship between the survey and the interviews is not clear – will the survey be informed by the interviews?”
Our intention is for the survey to inform the interviews, rather than vice versa. The interviews are really the main focus of this study, but we hope that a survey on a larger scale may draw our attention to areas where we lack granular understanding, and hence help us to design interviews that probe these areas. Other than that we view the survey as a means of cross-validation for the interviews, and as a second line of defence against over-generalisation of our findings.
We originally meant to recruit all interviewees via this initial survey. However, with the extended time-frame we can now recruit by speaking directly to students in the Physics department during the first week of the semester, which we expect to have a higher success rate. We still aim to recruit a small number of students who are graduating this summer from the initial survey.
“How will student interviewees be selected to ensure appropriate representation?”
We have discussed this issue at length in the ‘Methods and Evaluation’ section, although it is fair to observe that we have not detailed a strategy for selection. We will select a cross-section of year groups and genders – that is the relatively easy part. If we were collecting a larger sample, we would use some selection algorithm that pools candidate interviewees into ‘like’ groups and randomly selects a representative number from each group. However, since we are looking at a small sample we feel it is preferable to rely on our judgement in selecting interviewees, seeking a broad range of perspectives based on responses to an initial question prompt when students sign up as candidate interviewees.
9 Budget
Total cost: £1XXXX.XX
The breakdown of costs has been omitted.
9.1 Budget justification
The project proposal is based on a model in which the time-consuming bulk of the research is conducted initially by a PDRA (JMR) who has been heavily involved with the development of this project, and later by a PGRA after a period of working alongside JMR. The funds requested include a stipend for part-time research at standard pay grades for PDRAs and PGRAs in SoPA. We believe that offering incentives to participate in surveys is worth the relatively low cost, since we are more likely to have greater sign-up and hence ability to select a more representative sample. We have budgeted for the cost of a single member of our team attending and presenting our work at a conference, which is a reasonable balance between keeping costs low while effectively drawing attention to our work and the enhancing the University’s reputation in a highly topical area of research.
10 Project team expertise
10.1 Joe Marsh Rossney
I recently completed a PhD in SoPA where my research was on GenAI applications in particle physics. I am an Associate Fellow of the Advanced Higher Education Academy (now Advance HE) and am currently in the final stages of the Postgraduate Certificate in Academic Practice (PgCAP) programme taught by the Institute for Academic Development here in Edinburgh. I have been a teaching assistant on various courses in SoPA since 2020, including two programming courses, and more recently I have been working as a teaching assistant in the Edinburgh Futures Institute on two courses themed around the ethics of data and AI. Before coming to Edinburgh I was a teaching assistant at a secondary school where my primary role was providing additional support for students with special educational needs. My interest in technology in education is longstanding, but I came up with the idea for this particular project while taking the ‘Researching Your Teaching’ course as part of the PgCAP. I do not have prior experience with qualitative research, other than creating and analysing an end-of-course survey, but I am committed to learning and improving ‘on the job’, which will include asking for help where help is needed.
10.2 Ross Galloway
I am a Senior Lecturer in SoPA and also the leader of the school’s Physics Education Research (PER) Group. I have over 20 years’ experience in the teaching of physics and astronomy at the HE level, and 15 years of experience as a disciplinary-based pedagogical researcher. My previous projects in PER have incorporated quantitative, qualitative and mixed methods approaches, so I am well-versed in the methods to be used in this project. As a former Director of Teaching within SoPA, I had oversight of some of the challenges around the rapid influx of GenAI into the undergraduate sphere, and also questions of how to coherently respond at course, programme and school level.
10.3 Britton Smith
I am a Reader in the Institute for Astronomy in SoPA at University of Edinburgh. I am currently the Course Organizer for Computer Simulation, an introductory Python course focusing on object-oriented programming to model physical phenomena. This course has roughly 180 first- and second-year undergraduate students. In the past year, I have witnessed a major shift in the way students undertake programming exercises in this course, away from peer interaction and toward consultation of GenAI. The results of this study will have direct and considerable impact on this course. I am very interested to understand how students perceive the value of learning code development amidst the rise of GenAI and how I can continue to make the course feel relevant to them.
11 Equality impact statement
11.1 Project Team
The diversity of our project team is limited by a combination of staff availability and a more fundamental homogeneity of the people involved in the delivery of our programming courses. It would be very beneficial if the project team were as diverse as the students; we are mindful of the risk of designing a study that reproduces the biases inevitably present in our priors. In particular, with the interview format knowledge is co-created by both the interviewer and interviewee, so having multiple interviewers with different backgrounds and personalities would be helpful. We note that we have had, and intend to continue to have, discussions with interested colleagues and students who are not listed on the project team. That is to say there are established channels from which we can receive input from more diverse perspectives.
11.2 Participation and involvement
We do not expect there to be a barrier that disfavours under-represented groups in terms of their willingness to volunteer as interviewees, other than potentially a reluctance or scepticism about being interviewed (by a white man), which may be felt unequally. In agreeing to be interviewed, or even filling out a survey, the students are entrusting the researchers with their ‘data’, some which may be personal and sensitive. It is important that we are not just vigilant about the privacy aspects of the study, but also transparent so that students feel able to trust us. By holding some interviews over Microsoft Teams we do unfortunately lose some nuances afforded by in-person interviews, but some students will prefer this format. This is particularly true for the initial interviews, which take place outside of term time; a journey onto campus for a single interview may be very inconvenient for e.g. students with disabilities or work commitments.
11.3 Methods or Evaluation
It is highly plausible that the experiences and perspectives of students towards programming and GenAI are differentiated along intersection lines. Both physics and programming are fields that have historically been dominated by white western men, and this historical and cultural fact permeates into the experiences of current students to varying degrees. As an example, it may emerge from our study that non-dominant groups (e.g. women) report different social experiences in the computer labs, which may lead them to appreciate GenAI as a ‘non-judgemental’ assistant.
Having recognised this, it is important to stress that the methods proposed in this study come with substantial risks of drawing conclusions that are systematically biased in a way that potentially harms certain under-represented groups. Any sample-based methods have the potential to suffer from non-generalisability, and the researcher has a duty to guard against over-extrapolating results from small or unrepresentative samples. From a statistical perspective, in the former case the theoretical statistical errors are large and in the latter case results will be biased in a way that is generally impossible to ‘correct’ post-hoc.
We make no claim to guarantee a truly representative sample – indeed it is probably a fallacy to imagine that such a thing exists. It would be ideal if students from a wide range of backgrounds put themselves forward for interview, but cherry-picking individual students comes with its own ethical issues. Even with an apparently representative sample, it would be dangerous to take a very small number of people with a particular characteristic, e.g. religion or ethnicity, as somehow representative of all people sharing that characteristic. This comes back to the fact that, in studies such as ours, one cannot abstain from a subjective, holistic analysis by appealing to statistical arguments. The main tool we have to protect the integrity of our study and the well-being of students is our full engagement with the complexity and intangibility of the questions we have posed.
As discussed in the ‘Method’ section, identifiable information from the interviews and survey will not be included in the analysis. While our intention is to include certain characteristics such as gender in the analysis, we are committed to protecting the anonymity of interviewees and will take necessary precautions.
References
Reuse
Copyright
Citation
@online{MarshRossney2025,
author = {Marsh Rossney, Joe and Hogarth, Sarah and Gabriel Garcia
Elizondo, Polux and Galloway, Ross and Smith, Britton},
title = {Experiences of {Learning} to {Code:} {Perspectives} of
{Undergraduate} {Physics} {Students} in 2024},
date = {2025-08},
url = {https://ExpLrnCode-2024.github.io/},
langid = {en},
abstract = {This site provides access to research materials and
outputs produced during the \_“Experiences of Learning to Code”\_
project, which was run by a staff-student collaboration in the
School of Physics \& Astronomy at the University of Edinburgh from
June-\/-December 2024. The study sought to understand how the
experiences of undergraduate physics students taking programming
courses have been changing due to the sudden availability of
Generative Artificial Intelligence (GenAI) systems. The main inquiry
took the form of a series of semi-structured interviews with 24
student participants, whose experiences span the periods before and
after the advent of GenAI.}
}