Using P.I. To Manage A.I. pt. 2: Designing Meaningful Assessments by Applying the UDL Framework

Dana Dawson, Ph.D.

Take a moment to think about the last time you worked really hard on something, a time where you focused intently and devoted care and attention to completing a task. It might have been a project related to a hobby, something you did to support a family member or a task required for your job. Consider what prompted you to spend time and effort on the task. Was it because you enjoy the process? Because the outcome was important?

As we think about how to teach in the face of ready access to content generating tools such as ChatGPT, it is important that we reflect on the “why” of the learning activities and assessments we are designing for our classes. Our students are influenced by the same types of motivations that drive us as instructors to make time for certain activities and deprioritize others. Our attention is a valuable resource… and a finite one! We are all constantly making decisions about how to allocate our time and attention. So, if we want our students to invest their time and attention in our coursework and to use AI tools to benefit learning rather than as time-saving shortcuts, we must design activities and assessments that are meaningful and that warrant dedication of effort. Universal Design for Learning provides a valuable framework for identifying strategies to recruit our students’ interest, encourage sustained effort and promote self-regulation.

Recruiting Interest

Here at the CAT, we often encourage faculty to design authentic learning tasks. An authentic learning task is an activity or assessment that connects with our students’ lived reality. Are we designing activities and assessments that encourage the kinds of thinking and problem-solving our students do in their lives outside the classroom or that they might anticipate doing in a future field of work or study? By creating a safe space for students to take risks and test new concepts or skills, we provide a context within which students can experience learning and personal growth.

The first step towards authentic learning is to reflect on whether your assessments are designed to assess learning goals for the course. We often defer to assessments such as essays or exams to gauge student understanding but there may be better mechanisms for making student learning visible. Assessments should be geared to as directly measure student progress toward achieving the learning goal as possible. What can students do to show they have mastered new skills? How can they document their learning? Creating multiple assessment options, multiple paths to success, will help students to feel a sense of agency in their learning and discourage turning to cheating options.

Sustaining Effort & Persistence

Deep learning requires engagement over time rather than in short, disconnected bursts. Our students are more likely to sustain their efforts if we clearly explain the purpose of the activities and assessments we assign, which of course requires clarifying for ourselves what the purpose is in the first place. Take a moment to reflect on the course content, activities and assessments you assigned during the last week of classes you taught. Was there a clear reason for each activity? Did you explain to students the purpose behind what you assigned to them?

We also know that our students are more likely to persist in learning-related tasks if they feel a sense of belonging in your class. Give students opportunities to work with one another to solve problems and share the choices they’ve made relative to course activities. A warm demeanor and supportive stance can help here, too. Students are more likely to persist without taking shortcuts if they see the instructor as a partner in their learning rather than as the course antagonist.

Self-Regulation

While our students will be more motivated to dedicate time and attention to course-related tasks if they understand the purpose, we do not have to be the sole generator of that purpose. Give students opportunities to connect course content with their goals. What do they care about and how might your course content relate? Studies have shown that simply encouraging students to reflect on their values in class promotes motivation and engagement. Provide tools such as rubrics that allow students to monitor their own learning and improvement over time and ensure feedback is targeted and communicates how students can build from where they’re at.

It can also be helpful to explicitly discuss with students the kinds of skills and practices students need to cultivate to succeed in your course. Instructors tend to assume that someone prior in our students’ education taught them how to study or take notes. But for some students, your course may be the first time where a systematic approach is needed to pass the class. Or perhaps your course requires different study skills than any other course they experienced before. We need to be able to articulate to our students what being a good student looks like in our course.

For more information on Universal Design for Learning, visit the CAST website and for help implementing the above ideas, don’t hesitate to reach out to the CAT for a one-on-one consultation.

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Dr. Dana Dawson serves as Associate Director of Temple University’s Center for the Advancement of Teaching.

Using P.I. To Manage A.I. pt. 1: Introduction

Stephanie Laggini Fiore, Ph.D.

We are all teaching in a new reality created by powerful text-generation tools like ChatGPT that allow us and our students to compose text on demand. Lori Salem, Assistant Vice Provost and Director of the Student Success Center, and I wrote an initial post about this last semester. As instructors, we will all need to think hard about how to manage and harness the power of this tool. I use the words “manage” and “harness” intentionally here, as we cannot pretend that we can entirely ban these tools nor can we rely on anti-AI detectors (that I can assure you will not be foolproof). In addition, we have a responsibility as educators to guide our students in the ethical and effective use of AI tools that will be available to them beyond the university in their workplaces and in their daily lives. There is no body of research (yet) that can guide us in using AI for teaching, so we are all feeling our way along by reading, debating, and experimenting with some best ways forward. 

While the teams at the CAT and the Student Success Center are working towards developing a set of guiding principles for managing AI in our classrooms, the way to start right now is by considering how PI can help us manage AI. What is this magical PI, you say? Does it have something to do with Tom Selleck (if you’re my age, you get that joke)? Is it a fancy new counter-AI robot that will solve all of our problems? No, my dear colleagues, it is simply an invitation to examine the fundamentals, the Pedagogical Intelligence that should be the first stop on the road to a set of principles for thinking about teaching in the presence of Artificial Intelligence. In the CAT’s new spring series, Using PI to Manage AI, we will be exploring, both on our blog and on our CAT Tips series on social media, these pedagogical fundamentals as a way to start this conversation. The topics we will explore on our EdVice Exchange blog are all evidence-based ways of designing student assessments of learning in ways that will encourage academic honesty, motivation, and a desire to learn. We will follow each blog post with a CAT Tips video on social media outlining a few concrete ways to implement these assessment strategies in your classes. If you have not recently done a deep dive into evaluating how useful your assessments are for evaluating learning–and also for furthering learning by engaging students in meaningful learning tasks–now is the time! 

We will start the series by exploring how to design assessments that are meaningful for students, allowing them to connect to what we are teaching in ways that help them see the value of engaging in the work. The following blog post in the series will discuss how to use learning assessments to build student self-efficacy in ways that help them to be able to do the work well and  to feel confident in what they are learning. Then we will unpack iterative work that provides feedback and allows for revision along the way. We will subsequently examine summative assessments and strategies for supporting students to think reflectively about how they prepare for these usually higher-stakes assessments. Finally, we will complete the series by introducing some educational technology tools that can assist us in implementing better assessment protocols. 

It will be important to approach this new challenge as an opportunity. It will necessarily push all of us to think deeply about how we are teaching and how we are assessing learning, and in so doing, lead us to more effective practices. We may surprise ourselves by discovering that AI itself can be useful in exciting new ways for learning. In the meantime, know that we at the CAT are on this journey with you, and will be working to support you as you support our students’ learning.  

Note: If you are intentionally using ChatGPT to teach in your classrooms this semester, please email us at cat@temple.edu and tell us about it. Consider also that you can engage in the Scholarship of Teaching and Learning (SoTL) by designing a classroom study to evaluate the impact of ChatGPT on student learning. If you want to learn how to design a study related to your use of ChatGPT in the classroom, contact Benjamin Brock at bbrock@temple.edu for assistance. 

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Stephanie Laggini Fiore, Ph.D., is Associate Vice Provost and Senior Director of Temple’s Center for the Advancement of Teaching.

“Students are using AI to write their papers, because of course they are.”

Lori Salem and Stephanie Fiore

So says the title of a recent article in Vice that has been making the rounds at Temple.  The article describes a new tool called Open Ai Playground, that generates text on demand.  Playground uses GPT-3, a newly developed machine-learning algorithm, to compose the text.  GPT-3 is also the power behind Shortly-Ai, another text-generation tool offering a somewhat different set of features.  The sentences generated by both programs are surprisingly good – they flow, and they have clear and simple prose style.  A student could theoretically type their essay prompt into Playground or Shortly, and the program would generate the essay for them.  And because the sentences produced by GPT-3 are entirely original, the resulting text would not be flagged by a plagiarism detector like Turnitin.

So, is this the end of writing instruction as we know it?  We think not.  But these new programs do have implications for teaching, and that’s our focus in this post.    

We tested both tools to get a sense of what they can do and what it is like to use them.  Both tools make it easy to produce short (paragraph-long) texts that clearly and coherently state a few relevant facts.  It’s possible to imagine a student using them to produce short “blog-post”-type essays, which is exactly what the students in the Vice article say they do. At least for now, neither program would make it easy to produce a longer text, nor to produce a text that was argument-driven, rather than factual. 

But more importantly, these programs don’t—and can’t—help with the real work of writing.  They can create sentences out of sentences that have already been written, but they can’t help writers find the words to express the ideas that they themselves want to express.  If the purpose of writing was simply to fill a page with words, then the AI tools would suffice.  But if the writer wants to communicate something, and therefore cares what ideas and arguments are being expressed, then AI writing tools are not helpful.  

Don’t take our word for this.  In the sidebar, we provide information about how to access and use Playground and Shortly.  Try them and see if you can get them to write something that you can genuinely use.

If you find, as we did, that AI writing tools are not useful when the writer cares about the content of the writing, then we’re halfway to solving the problem of students using AI tools to plagiarize.

The Plagiarism Arms Race

Just because AI generated texts are undetectable right now, doesn’t mean that will always be the case. Someone somewhere is probably already working on a tool that will detect texts written by GPT-3, because of course they are. Students figure out ways to cheat, and companies invent tools to catch them, and then they sell their inventions to us. This is just the latest iteration of that cycle.

To that point, have you seen the YouTube videos instructing students on how to beat Proctorio at its own game? The same Proctorio for which we pay a hefty annual subscription fee?

There has to be a better way, right?

A better way, part I: Encourage Academic Honesty by Creating Better Assignments

This new AI tool is a “threat” to academia only insofar as we ask students to complete purposeless writing assignments, and ones that rely on lower-level thinking skills that ask students to reiterate factual information. The real answer to cheating systems that become more sophisticated is to create better assessments and to create conditions in our classrooms that encourage academic honesty.

There is some very good research on what works to encourage academic honesty. This is a longer discussion than we will take here, but in essence, we should think about what the factors are that lead to cheating behaviors and work to reduce those factors. These include 1) an emphasis on performance (rather than learning); 2) high stakes riding on the outcome; 3) an extrinsic motivation for success; and 4) a low expectation of success. There are very intentional steps that we as instructors can take to reduce these factors, including adjusting our assessment protocols to rely less heavily on high-stakes one-and-done writing assignments, centering writing assignments on issues students care about, and scaffolding writing assignments to allow for feedback and revision.

We also need to look at the kinds of assessments we are using in our courses. The more we move towards authentic assessments and grounded assessments (designed to be unique to the course you are teaching in the moment. They often include time, place, personal, or interdisciplinary elements to make them something not easily replicable), the better off we are. There is a lot of work to be done here, as we often rely on the kinds of assessments we had as students, very few of which were either authentic or grounded. It is much harder to cheat on these kinds of assessments.

Finally, findings from some interesting research on academic honesty suggest that communicating with students about academic honesty works better than you would think, reminding them of their ethical core and focusing on what academic honesty looks like and why it is expected. This is especially effective when timed close to an assessment.

Try it for yourself!
Open Ai Playground
How to try it: Use the link above to open the website and make a free account. From the home screen, click on the “Playground” tab (top right.) Then enter an “instruction” in the main text box. The instruction might be something like “Describe [topic you are writing about.]” Or “Explain [something you are trying to explain.]” Click “submit,” and your results will appear. If you don’t get what you were looking for, you can keep refining and resubmitting your instructions. ShortlyAIHow to use it: Use the link above to open the website and make a free account. Enter a title and a sentence or two and set the output length to “a lot.” Then click the “write for me” button. If you like the way the text is going, you can type another sentence or two and click “write for me.” Or you can refine your original title and first sentence and start over. Please share your results! Copy the text(s) that you “write” and email them to Lori.salem@temple.edu along with any comments you care to offer about the texts or your experience producing them.

A better way, part II:  Adapt instruction to reflect new writing practices

Once upon a time, writing instruction centered around penmanship and spelling.  Those days are gone because developments in the technology of writing (from pens, to typewriters, to word-processors) drove changes in writerly practice, which eventually led to changes in writing instruction. 

Automated text generators are just the latest technological innovation, and they have already changed the practice of writing in journalism, online marketing, and email.  And why not?  There is great value in making certain kinds of writing more efficient. 

Our approach to writing instruction will need to adapt to this new reality.  It’s not hard to imagine a future in which universities teach students how to use AI tools to generate text for some situations, even as they disallow the use of AI tool for others. 

Lori Salem serves as Assistant Vice Provost and Director for the Temple University Student Success Center.  Stephanie Fiore is Assistant Vice Provost and Senior Director of Temple’s Center for the Advancement of Teaching