(Credits to Jennifer Teo and Yang Yarong (Kranji Secondary School)
With the emergence of generative AI (GAI), teachers can tap on its potential to redesign our lessons to enhance teaching and learning for all. We can also explore how technologies can support us in shaping new pedagogical approaches and create new learning experiences. Teachers find it challenging to deliver timely and personalised feedback to students, particularly at crucial times in the learning process, due to time and other resource constraints. As feedback is widely regarded as a powerful strategy to improve student achievement, GAI can alleviate teachers’ workload by automating low-level outcome feedback, allowing teachers to focus on higher-order thinking skills during lesson time and reducing tensions between formative assessment and accountability systems.
TeacherGAIA, like other large language models designed for Teaching and Learning, such as Sidekick by schools.ai , gives teachers the ability to monitor and control access to student’s interactions with the GAI.
Teachers are also able to develop and engineer assistants using system prompts to control how students interact with GAI and to accomplish specific tasks, which was the focus of our trial with TeacherGAIA.
Choosing a topic / Description of the lesson
One of the affordances of GAI is the ability to be able to respond to many users at the same time with personalised and timely feedback. On the other hand, there still exist limitations regarding the GAI ability to read and interpret graphical and tabulated data, although this ability is improving.
Given these strengths and weaknesses, we selected experimental planning as an area where GAI would be able to positively contribute to student learning. In the past, students learnt this portion of Physics by writing full experimental plans, and having the teacher grade them by hand. This resulted in slow feedback for the students as teachers were unable to grade such planning very quickly, with each plan being quite lengthy and unique to each student.
To enhance student learning, we needed fast, individualised responses that could quickly guide students towards correct ideas and format of experimental planning, as well as the Ways of Thinking and Doing Science. In addition, the segment of experimental planning had very few calculations, and minimal use of graphs and tables, lending itself well to the affordances of GAI.
With well-designed and well-structured system prompts, it was possible for GAI to take the role of a knowledgeable other to guide students through the thinking processes of planning an experiment, specifically identifying independent, dependent and control variables, as well as how to design an experiment without actually doing it.
In previous years, using the traditional model of teaching, it was physically impossible for teachers to be able to give individualised feedback on student thought processes, and only provide feedback on the final product, which tended to be large in volume and pinpointing multiple areas at once. This made it difficult for students to improve on their planning skills as there were too many things to focus on, and they may not even have had the ability to correct their thinking about how one should approach an experimental question.
We decided to leverage on TeacherGAIA to enhance the teaching and learning of practical planning. However, with the use of GAI as a knowledgeable other, the teaching of experimental planning was now broken into discrete parts– the thinking about how the experiment could be done (with GAI guidance, followed by teacher marking), thinking about how the subsequent experimental data should be analysed (with GAI guidance followed by teacher marking) and finally, how the planned experiment should be presented (with feedback from teacher marking).
The teacher started by introducing TeacherGAIA, giving students basic guidance on prompt literacy and requirements of a physics plan. The teacher introduced some parts of planning, such as graphing and tabulation of data within the planning process as GAI is unable to read non produce a table and/or graph. Students then explored TeacherGAIA to learn about crafting the steps / method for a physics experiment, taking into consideration control variables, repeating of data, etc. Students then refined their plan with the help of TeacherGAIA and teacher then consolidated their learning.
Although we are using TeacherGAIA (a ready made bot by NIE), we would need to craft our prompts for each lesson based on the learning objectives. As we created our prompt (shown in Fig. 1), we considered the Teacher persona and aim of the chat, what the success criteria of the prompt should be, and a checklist of a good experimental plan. We also intentionally made the bot go through several levels of open-ended questions, with our intent being the students who were interacting with the bot to have their thinking triggered using Socratic questioning.
Students’ Feedback
Students found difficulty in crafting the prompts to interact with the chatbot, with the most difficulty about describing the experimental setups, which tended to be novel ones. After the initial learning curve, students became appreciative of the individual guidance.
Below are verbatim responses from two students after using TeacherGAIA:
- At first it was confusing (be)cause TeacherGAIA wouldn’t answer directly to my prompts but after a while I realised I need to clearer and describe out every part of the setup in detail then it finally gave me a step by step procedure of how to carry out the experiment
- The TeacherGAIA was a very interesting and new way for me to learn. It was challenging at first but after a while I got the hang of it and was able to understand how to do planning easily. TeacherGAIA didn’t give me the final answer on the spot but instead guided me and asked me questions along the way to help make my planning better.
Many students also found amusement in exploring planning experiments that were not linked to physics, although in this case while we allowed the students to explore “playing” with the bot to learn how to give more robust prompts, we certainly did not encourage them to do so, and instead gave them practice tasks to learn how to describe novel Physics problems.
Our Verdict – Potential of GAI
Advantage 1: Customised and immediate feedback and prompts to guide each individual student
TeacherGAIA gave customised and personalised responses to each student’s unique plan, offering prompts and questions (as shown in Fig.1) based on the success criteria keyed in as prompts by the teacher. GAI challenged the students to think deeply, allowing them to explore perspectives they may not have considered, engaging them in active learning and helping them connect with material in meaningful ways. In addition, being a one to one chat with a chatbot, there was no worrying about judgement from peers or teachers when asking seemingly silly questions.
In addition, and more importantly, TeacherGAIA encouraged the development of metacognitive skills as students approached the steps of the experimental planning, making them think not just about how to do the question, but also about how to think about thinking. It made them consider what choices needed to be made during experimental planning, and why certain choices were made. This targeted guidance not only reinforced understanding but also built critical thinking skills, making AI a valuable addition to our teaching toolkit.
Advantage 2: Prompt engineering allows for more targeted/ specific responses for students
Another strength of TeacherGAIA lay in the fact that teachers had control over what the knowledge bases the chatbot would pull from, as well as how the bot would respond to student answers. The tool gave teachers the ability to control multiple aspects of the chatbot, from the tone of the message (friendly, firm, encouraging), to the structure of the message (first, ask about independent and dependent variables…, then…).
This enabled the teacher to construct the level of scaffolding required, as well as constrain how long and complicated messages received by students would be. One of the joking comments we had during a discussion also noted that a human teacher who was trying to be encouraging may have gotten angry (and likely not very encouraging) by the third or fourth time a student gave an off-topic answer (whether on purpose or not), but a chatbot would remain encouraging while redirecting the student back to the task.
A well-crafted System Prompt could also confine the discussion topic specific to subject and topic as well, ensuring students did not go too off-topic.
Navigating Challenges
There are, of course, clear limitations on using GAI. For now, there is a lack of ability to read and plot graphs and tables, although this feature is slowly becoming more sophisticated even in the free or commercially available version of GAI. In addition, some information on GAI may be not updated, too complex for secondary school students, or even fictitious information (known as hallucinations). Beyond such common challenges with GAI, teaching experimental planning with teacherGAIA presented its own unique set of challenges.
Challenge 1: Clear prompt engineering and multiple trials required to refine
For both the teacher and the student, some difficulty lay in the accuracy of the prompts. Many people who work in computer science know the principle of “GIGO”, or “Garbage in, Garbage out.”– Poorly designed prompts would result in non-ideal responses, both for students and for teachers.
Students needed to be trained in how to ask questions and how to describe situations to the chatbot, as well as how to respond to the bot’s responses. In this series of Physics lessons, we used the “CLEAR” (eg https://nolongerset.com/clear-framework-for-ai/) framework to teach students how to interact with the bot, which took some time away from the curriculum. In addition, they also needed to learn how to interact with the bot in actuality, meaning even more time was needed for them to even get used to prompting the bot properly.
On the teacher’s side, we needed to ensure there was clear system prompt engineering– such as starting from the success criteria / checklist first before doing the prompts. Although we tried on our own to refine the prompt by acting as “naughty” students trying to trick the system, unexpected issues arose when we actually allowed students to use the system. Some students managed to override some of the preventive measures we put in. In addition, we realised some of our success criteria were also not as clearly spelt out to the bot as we thought it was. For example, students did not realise that they needed to clearly explain which measuring instruments should be used for which variable; when we were testing the system, we naturally (as well trained Physics teachers) put it in while acting as students.
Our conclusion, other than that we were not very good at acting as students, was that it was best if the system prompt writing process was iterative, as each use of the bot with students helped us develop deeper understanding of what the bot should or should not be constrained by on a system level. In addition, each lesson would require a new set of prompts as the lesson objective changed, which would mean more editing work from the teachers end.
Challenge 2: Managing student over-reliance on GAI
As mentioned earlier, some students managed to override some of the protective mechanisms we built into the bot to prevent them from making the bot think for them. Some students managed to cajole the bot into planning the experiment for them (Fig. 2 shows a mockup of one such student conversation) instead of using the GAI as a sounding board (As in Fig 3.).
Images of the avatars in the two chats in Fig 1 and 2 were generated using Large Language Models.
The image of the robot was generated using Gemini by the prompt “ Generate an image of a robot in cartoon style. Crop the picture so it is from shoulders up.”
The image of the boy and girl were generated using ChatGPT by the prompts “Generate a passport photograph of a girl with glasses, a smile and her hair pulled back neatly. She is wearing a neat white blouse. Make the girl Asian and draw it in cartoon style.” and “In the same style, generate a passport photograph of an Asian boy with a cheeky smile in a white button-down shirt.” respectively
We noted that students who made the chatbot write their experiments for them, and thereafter transferred the steps over onto the hardcopy worksheet to be submitted without processing the thinking, did worse in their assessment than students who used the bot as we had planned. They merely copied GAI’s answer and there was little transfer of learning. In another assignment, it was clear that they were uncertain on how to approach the question and had multiple missing steps.
We certainly will want to tighten the system prompt such that students have to be the ones who give the answer, with the system prompt: “do not give any direct answer”, as compared to our current prompt “do not give any direct answer unless the student makes multiple mistakes or asks for help”.
However, it is imperative that even if the system prompt was fairly relaxed, that we encouraged students to ensure they were thinking about their work, instead of using the GAI’s answer without any processing or consideration as part of academic integrity.
After all, the GAI’s purpose in this learning experience is not to take over their thinking, but simply to help augment it.
Our Reflection
Equipping our students with the skills to leverage on AI goes beyond the classroom. It is a soft skill that would place them in good advantage in the workforce. As a baby step, we taught students simple prompt literacy – the ABC framework as well as the CLEAR framework (as mentioned above)
The ABC framework taught students how to interact with the chatbot by
- Asking the bot questions,
- Building on the bot’s answer and
- Critically thinking about the bot’s answer, to seek evidence that supports or refutes the bot’s answer.
Such introductory steps in GAI prompting in a controlled and relatively safer environment sows the seeds of equipping students to learn anything, anytime and anywhere, developing them as self-directed learners.
In addition, with the adjusting work scope for jobs of the future, students can leverage AI as a tutor to learn beyond the school curriculum.
Beyond the academics, moving towards a purpose driven schooling, as students continue to interact with AI well, we develop their competencies in learning new technologies but using them with caution. This skill will be critical for our students, who will be working in a world which continues to experience disruptions from technical advances on higher frequencies than in human history.
Conclusion
As teachers and students become more proficient in using Chatbot, GAI can provide continuous support to under-resourced groups that lack access to sources of academic assistance such as adults, peers or tutors outside of school. They can seek clarification, ask questions and receive guidance whenever they encounter difficulties or have doubts. As teachers develop our students to learn and know how to validate and value add to AI responses, we hope we better prepare them for even more technology driven future.
It is our hope that when the students are comfortable learning from GAI, it can also support our under-resourced students’ access to academic assistance outside of school, being a platform that students can use to clarify their doubts, ask questions and learn on their own.
However, the challenges of GAI do leave questions in our head about what then a teacher’s role is in this new age of GAI Chatbots. Do we become designers of system prompts? How do we teach students to validate and critically assess chatbot responses? How can we equip students to survive in a tumultuous world where technology has advanced to the point that it can, among other things, talk to us like another human? Will teachers be replaced by chatbots in some distant future?
As we stand on the cusp of the GAI revolution, these are possibly questions that we have to ask ourselves.
Written with minimal AI, with the introduction produced with drafting support from ChatGPT.