Timeless principles in the era of generative AI

by Tay Hui Yong and Chue Kah Loong

“Embracing change without principles is like dancing in a hurricane; it might be exhilarating, but it’s probably not going to end well.”  (ChatGPT, 2023)

Assessment systems are in a state of jolt. With the emergence of Generative Artificial Intelligence (GAI) that can seemingly respond to questions with human-like accuracy, assessment has been revolutionised yet again. In a break from traditional artificial intelligence, GAI tools such as ChatGPT are now able to create new content, synthesise data and converse “naturally” with people. Unsurprisingly, this has caused a great amount of concern within the assessment community, as administrators and educators debate measures to tackle the challenges posed by GAI. Only one thing is certain – GAI has taken root and will not disappear in the foreseeable future. Assessment practices thus need to adapt accordingly.

As we embrace GAI, we should not forget the fundamental principles that assessment is based on. These principles are akin to the anchor of a ship. The purpose of an anchor is to ensure that the ship doesn’t drift off course. However, an anchor does not lock the ship in a rigid position. Rather, when an anchor is working properly, it allows the ship to move, albeit not very far from its original location. Additionally, when it comes to anchoring, the most critical factor would be the selection of a suitable seabed. If the seabed is too soft or hard, the anchor may not be able to grab it effectively. In a similar vein, principles remind us that the assessment we design is “fit for purpose” yet provides us with ample room for exploration and adaptation. At the same time, the set of principles should be neither too restrictive nor too lax to allow for flexible adherence. Therefore, a key question to ask would be: What are essential assessment principles to anchor on? At the National Institute of Education, we recommend the following five principles to guide our assessment design and practices (CAE, n.d.).

1. Assessment should be central to effective teaching and learning.
2. Assessment inferences should be valid and reliable.
3. Assessment should be explicit, transparent and accessible.
4. Assessment should promote and improve learners’ capacity for self-directedness.
5. Assessment should take into consideration the impact on learner’s well-being.

The above principles adhere to the notion that the overarching aim of assessment is to promote learning. Regardless of the affordances and challenges brought about by GAI, these principles possess a fundamental enduring quality that educators can refer to when designing or implementing assessment. We highlight two assessments that apply these principles in conjunction with the use of GAI.

The first example makes use of a series of multiple interconnected tasks to assess essay submissions. These interconnected tasks could be in the form of smaller pieces of work that leads to an integrative piece (i.e. patchwork assessment) or a succession of drafts that lead to a final written piece. Students can use GAI to (a) generate ideas for an initial draft, (b) correct their language use (especially if the assignment has a large content component), and (c) obtain feedback on their initial drafts. With reference to the assessment principles, teachers can thus design the assessment so that the focus is on (a) eliciting evidence of higher-order thinking skills through a critique of the initial drafts, (b) empowering and engaging students through self-assessment, and (c) monitoring students’ progress through feedback uptake of subsequent tasks or drafts. In addition, using GAI to check for language errors reduces the amount of construct irrelevance, and may lead to greater reliability in grading. Naturally, in the interest of fairness and transparency, the potential uses of GAI will be made known to all students.

The second example makes use of a multimodal assessment format. Relying on a single method (e.g. once-off written tests), to assess students on knowledge, understanding and application may often result in an incomplete assessment picture. Hence, to increase the validity and reliability of an assessment, teachers can collect evidence of learning through a range of methods. GAI can be used to assist teachers by generating a variety of potential assessment tasks such as scenarios in performance tasks or authentic assessments. As the various tasks typically require a thorough understanding of key ideas, students can then use GAI as a learning partner to break down concepts into simpler terms. Alternatively, they can also make use of GAI to create initial response drafts to work on (e.g. a script for presentation purposes). This approach not only spotlights learning as the main emphasis, but it also provides opportunities and resources for students to self-direct their own learning.

Astute readers would have noticed that there was little reference to the fifth principle pertaining to student well-being. As GAI is a relatively new tool, clear adverse consequences have yet to be seen. Nonetheless, there is always the possibility of negative impacts on different groups of students. For example, learners with dyslexia may encounter more stress if they use text-heavy GAI tools. Teachers should constantly be on the lookout for potential harmful effects when using GAI in their assessment design and practices.

To conclude, using GAI for assessment without considering fundamental principles is analogous to building a house without a blueprint (modified ChatGPT prompt 1, 2023), experimental cooking without knowledge of spices (modified ChatGPT prompt 2, 2023), or prescribing medication without knowing the patient’s medical history (modified ChatGPT prompt 3, 2023). Failure to do so may result in invalid or biased assessments that do not accurately reflect students’ knowledge and skills. Assessment principles can provide enduring bearings for us to navigate through any challenges brought about by technological advancements.

References

Centre for Assessment & Evaluation (n.d.). https://nie.edu.sg/cae