Original Paper: https://arxiv.org/abs/2307.16364

By: Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, Brent N. Reeves

Abstract:

With their remarkable ability to generate code, large language models (LLMs) are a transformative technology for computing education practice. They have created an urgent need for educators to rethink pedagogical approaches and teaching strategies for newly emerging skill sets. Traditional approaches to learning programming have focused on frequent and repeated practice at writing code. The ease with which code can now be generated has resulted in a shift in focus towards reading, understanding and evaluating LLM-generated code. In parallel with this shift, a new essential skill is emerging -- the ability to construct good prompts for code-generating models. This paper introduces a novel pedagogical concept known as a `Prompt Problem', designed to help students learn how to craft effective prompts for LLMs. A Prompt Problem challenges a student to create a natural language prompt that leads an LLM to produce the correct code for a specific problem. To support the delivery of Prompt Problems at scale, in this paper we also present a novel tool called Promptly which hosts a repository of Prompt Problems and automates the evaluation of prompt-generated code. We report empirical findings from a field study in which Promptly was deployed in a first-year Python programming course (n=54). We explore student interactions with the tool and their perceptions of the Prompt Problem concept. We found that Promptly was largely well-received by students for its ability to engage their computational thinking skills and expose them to new programming constructs. We also discuss avenues for future work, including variations on the design of Prompt Problems and the need to study their integration into the curriculum and teaching practice.

Summary Notes

Leveraging AI for Improved Programming Education: Exploring Promptly and Prompt Problems

The integration of Large Language Models (LLMs) like GPT-3 into programming education is changing how we approach teaching code. This evolution introduces the need for new skills, particularly in creating effective prompts for AI to generate code.

This development has led to the creation of 'Prompt Problems' and a dedicated tool called "Promptly," aimed at enhancing the ability of AI engineers and students to work with AI.

Changing Landscape of Computing Education

The use of LLMs in educational settings has sparked debates, initially focusing on their potential misuse but gradually acknowledging their benefits in promoting engagement with computational thinking and AI technologies.

Introduction to Prompt Problems

A study at the University of Auckland with graduate students identified difficulties in formulating effective AI prompts for code generation.

This led to the creation of 'Prompt Problems,' a teaching strategy to improve interactions with AI tools.

Meet Promptly: A Revolutionary Educational Tool

Promptly is a web application designed to facilitate and evaluate Prompt Problems. Its key features include:

• Automated Testing: It automatically evaluates the generated code against test cases, providing instant feedback.
• Detailed Analysis: The tool analyzes student interactions, offering insights into prompt effectiveness and the AI's response to different prompts.

Empirical Insights and Findings

Observations from using Promptly with first-year Python students include:

• Engagement and Learning: The tool was effective in teaching programming concepts and enhancing AI interaction skills.
• Prompt Quality: The study highlighted the critical role of prompt quality in determining the usefulness of the AI-generated code.
• Suggestions for Improvement: Students suggested more varied problem types and a better user interface for future versions of Promptly.

The Future Role of AI in Education

The focus on Prompt Problems and the use of tools like Promptly represent progressive steps in computing education, preparing students for a future where AI is ubiquitous in professional environments and promoting critical thinking skills.

However, challenges such as tool accessibility and problem complexity need to be addressed.

Conclusion

The introduction of LLMs and tools like Promptly into programming education opens new paths for teaching and learning.

These developments offer students valuable opportunities to engage with AI meaningfully, setting the stage for a future where AI and human intelligence collaboratively drive innovation and learning.

In essence, the integration of AI through Prompt Problems and tools like Promptly marks a significant advancement in programming education.

As we adapt to this changing landscape, our focus is on equipping students with the necessary skills to excel in an AI-driven future.

The journey is ongoing, and the potential for educational innovation is limitless.