Using P.I. To Manage A.I. pt. 5: Summative Assessments Can Promote Reflection and Learning, Too!

Cliff Rouder, EdD, Pedagogy and Design Specialist

I have such a vivid memory of a student, upon my returning his exam, quickly looking at the grade, crumpling the paper into a wad, and making a 3-point shot into the wastebasket–right in front of me! Why did I spend so much time providing meaningful feedback on the exam? Even more exasperating than seeing my written feedback ignored was seeing similar errors on his next exam! UGH! It never dawned on me that I could do more than say to my students, “Take some time to review my comments so that you can see where you can improve, and don’t hesitate to come to office hours if you have any questions about the comments.”

Unbeknownst to me at that time, there was a term and theory in cognitive and developmental psychology literature called metacognition. Proposed by John Flavell, a professor at Stanford University at the time, metacognition is often referred to as “thinking about one’s own thinking,” and includes a process of self-monitoring, self-evaluation, and self-regulation that leads to more intentional learning practices. While metacognitive strategies can be used in a number of ways to support our students’ learning, we can use our usually higher-stakes summative assessments (exams, projects, etc.) to help students “think about their thinking.” More specifically, we can help students to explicitly reflect on elements of their exam preparation in order to make a plan to improve on the next high-stakes assessment.

Enter the metacognitive activity of exam wrappers. Dr. Marsha Lovett and her colleagues at Carnegie Mellon University are credited with creating the exam wrapper technique. Wrappers were developed in reaction to their findings that many successful high school students were arriving at college with study habits that are ineffective for higher order learning. Exam wrappers ask students to answer a series of questions that require them to reflect on how they prepared for the exam, whether the results were what they expected, and how they might prepare differently for the next exam.

Saundra McGuire, author of Teach Students How to Learn: Strategies You Can Incorporate Into Any Course to Improve Student Metacognition, Study Skills, and Motivation, states that
that there is a metacognitive equity gap. Often students from under-resourced public schools may not have yet learned how to reflect on their learning and thus continue to employ study strategies that are not as effective as they can be. While all students can benefit from an exam wrapper, the activity may be especially valuable for these students.

The Protocol

Faculty prepare a series of metacognitive questions that help students think about their preparation and performance on a summative assessment. Carnegie Melon’s Eberly Center has some example exam wrappers for different disciplines. What’s nice is you can tailor these questions to the unique nature of your summative assessment: questions getting at the specifics of students’ study/exam preparation strategies, asking them to assess the types of errors they made on the exam, and asking them to develop a concrete plan to improve are typical.

Exams are returned to students wrapped in a piece of paper with the metacognitive questions. Students are then asked to reflect on the questions and develop a plan for the next exam. They submit a copy of the plan to you for homework or extra credit–whichever way you think works best. You review the plan and seek clarification if responses are not specific enough. You can then remind them of their plan and perhaps check in with the class occasionally to make sure they are following the plan as they prepare for the next exam. If you give your exams online or if you’re a fan of reducing paper, the questions can be posed in Canvas.

Lisa Kurz at Indiana University’s Center for Innovative Teaching and Learning points out that exam wrappers can also be given to students before they take an exam. While the name is a little bit of a misnomer for this use, she states, “The exam wrapper provided before the exam might ask students to create exam questions at different levels of Bloom’s taxonomy, predict what major topics will appear on the exam, how much of the exam will be devoted to each topic, and what kinds of questions will be asked. Students can then use this ‘exam blueprint’ as they study.” This works well as a solo, pairs, or small group activity.

If you’d like to learn more about using or adopting one of these exam wrappers or creating your own, feel free to schedule a consultation with the CAT. So, wrap your exam and help your students become metacognitive pros!

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Cliff Rouder serves as Pedagogy and Design Specialist at Temple’s Center for the Advancement of Teaching.

2023 STEM Educators’ Lecture Recap

Cliff Rouder, Ed.D

The CAT’s STEM Educators’ Lecture, held on March 28, featured guest speaker Dr. Cynthia Bauerle (they/them), who engaged faculty from Temple and other regional institutions in this year’s topic, “Utilizing an Ethical Reasoning Framework to Create More Equitable STEM Education.” Dr. Bauerle is a Professor of Biology at James Madison University, formerly holding positions as Dean of the College of Science and Mathematics and Vice Provost for Faculty. Dr. Bauerle is a molecular biologist by training, publishing widely in both scientific and science education journals. Their career interweaves scientific expertise, passion for inclusive teaching, and a commitment to improving STEM education nationally.

In Dr. Bauerle’s interactive presentation, they introduced the ethical reasoning instrument (ERI), a tool that uses a framework developed collaboratively with colleagues Dr. Laura Bottomley from North Carolina State University, Dr. Carrie Hall from the National Science Foundation, and Dr. Lisette Torrres-Gerald from TERC (originally standing for Technical Education Research Centers, but now just known as TERC), a private organization that supports STEM projects and research.

The ERI guides STEM faculty through a series of questions related to eight dimensions of ethical reasoning. Faculty re-center ethical reasoning by using the instrument as they think about the design of their courses, i.e., their pedagogy, classroom activities, and assessments. Important to note is that these questions ask us to consider more than just bringing in course content that addresses ethical issues. Rather, the ERI enables us to reflect on how our course design can be more equitable. To that end, Dr. Bauerle reminded us that this reflective process also requires us to consider the various identities we bring into the classroom, as well as those of the students, with the belief that diverse identities provide richness. It also enables us to explore more intentionally the biases we may hold so that they don’t enter the classroom. 

Users of the ERI should be able to incorporate at least some of the eight characteristics of ethical reasoning (listed below) into their STEM courses. Under each characteristic are sub-questions we can consider in our course design. For example:

  • Fairness – How can I (we) act justly, equitably, and balance legitimate interests?

Here, the sub-questions we can consider are if our courses provide opportunities for students to learn about inequities in science or the consequences of ignoring inequities in the practice of science. We can also consider if our course includes principles of Universal Design for Learning, including access and accommodation.

The other seven dimensions have useful sub-questions as well. Click on each one to see these sub-questions.

  • Outcomes – What possible actions achieve the best short- and long-term outcomes for me and all others?
  • Responsibilities – What duties and/or obligations apply?
  • Character – What actions help me (us) become my (our) ideal self (selves)?
  • Liberty – How do I (we) show respect for personal freedom, autonomy, and consent?
  • Empathy – How would I (we) act if I (we) cared about all involved?
  • Authority – What do legitimate authorities (e.g. experts, law, my religion/god) expect?
  • Rights – What rights, if any, (e.g. innate, legal, social) apply?

Faculty can also use the ERI to evaluate the success of their course implementation. The instrument provides scaffolding for assessment and activity development, as well as examples of each of eight key characteristics of ethical reasoning.

Dr. Bauerle explained that the ERI is being implemented in a variety of STEM courses. Because the ERI is in the beta phase, the research team encourages participants to share any feedback with them regarding its implementation and usefulness. An important reminder was that the ERI is not just a valuable tool for STEM courses–ethical issues that encompass equity are valuable for any course!

If you’d like to keep the ERI discussion going, post a comment on our Faculty Teaching Commons. What are your thoughts about–and experiences with–incorporating one or more of these ethical dimensions? Inquiring minds want to know! As always, if you’d like assistance planning your courses to incorporate elements of the ERI, our CAT staff is ready to help. Make a consultation appointment or email a CAT staff member directly.

Using P.I. To Manage A.I. pt.4: Iterative Work to Strengthen Student Engagement

Jeff Rients

We sometimes tend to imagine education as a linear, cumulative process. As we learn new knowledge and master new skills, we grow towards our future selves. Our education system reinforces this idea as we move from 1st grade to 2nd grade, sophomore to junior, or undergrad to graduate student via a linear progression. However, new learning always exists in the context of our prior learning. What we learned before is not something passive and fixed that we build upon; instead our new learning is in conversation with old learning. The new selves that learning produces don’t exist separate from our old selves, but because of them.

We can create better learning opportunities for our students, building upon the relationships between past and present learning, if we structure our courses not as linear movement through topics or units, but as a series of feedback loops where prior learning is explicitly invoked and connected to present learning. This iterative work takes the form of embedding every new learning opportunity into a context.

Questions to Connect

For example, some instructors start a course in a sequence with an overview of the key topics of the prior course or some sort of review activity. An interactive approach would then take that review and ask students to explicitly connect it to future learning, with a prompt such as “Looking over the key takeaways of the prior semester, which of those topics do you see coming into play with each of the units outlined on the syllabus?”  This can also be done at the individual unit or project level, with prompts such as “What skills or knowledge do you already possess that will be crucial to your success at this task?” Explicitly asking these questions creates new connections between old learning and new, strengthening both in the students’ minds.

Draft and Revision

An iterative approach also provides more opportunities for students to receive feedback. The classic example of this is the semester-long paper, where students hand in pieces (thesis statement, bibliography, conclusion, etc.) and/or drafts prior to the final submission. Each of the earlier submissions receives peer and/or instructor feedback, giving the students the information that they need to improve their approach for the next submission. In this way small mistakes can be pruned away and misunderstandings of the task addressed before the final submission.

Recurring Examples

The use of recurring examples brings earlier content into later contexts. For instance, a case study introduced early in the semester can be redeployed, either with changes to challenge the students’ new knowledge or as an opportunity to reflect on what they have learned since the case originally appeared. Simple prompts like “How would you handle that old case differently, knowing what you know now?” will help make student’s new learning visible. Students can also be asked to critique earlier, intentionally simpler examples, to reveal the complexities those earlier cases glossed over. You could task students to write a more complete version of the case, one that includes the new nuances the students have learned since the original instruction of the case.

Revisiting Prior Work

Many times our students hand things in to us on a “fire-and-forget” model; the ordeal of the task is over and they would just as soon not think about it again. Painful as it might be, we need to help them do that thinking while showing them that their skills and knowledge are growing. For example, you could ask students to do a short write activity describing how they would do an earlier project differently, knowing what they know now. This can even be formalized into a system where students are given the opportunity to revise earlier submissions for a higher grade, provided that they incorporate one or more new skills and concepts introduced after the original assignment.

There are benefits to iterative work on the instructor as well as the student end. If our classes feature more tasks with built-in continuity, it will be easier for us to see our students in a more comprehensive way. Their growth as thinkers and their individual professional or scholarly identities will become clearer to us. That puts us in a better position to offer feedback that makes sense in the context of the individual learner. It also allows us to see when a student’s thinking or writerly voice suddenly changes. Maybe the student has stepped up to the challenge of the course in a new way. Or maybe the shift is due to some sort of distress and the student needs help (possibly including a referral to the CARE Team). Or perhaps the change in student voice is an indicator that the student is getting unacknowledged assistance. Which of these is true and whether the unacknowledged assistance, if any, rises to the level of cheating requires careful consideration and investigation. But without the record that iterative work provides, these issues are much harder to identify.

Note that iterative work is not offered here only an “anti-cheating” technique. Easier detection of changes in student writing is just one benefit of iterative work. The real gain to be made by this approach is that your students will be less likely to see our syllabi as a series of randomly ordered individual tasks and more likely to see the grand trajectory of their learning. We all too often assume that the students can see all the linkages we made between various content and activities, but those linkages are only readily visible to experts such as ourselves. Iterative work helps the students see how the moving parts of the course fit together and how the various things they have learned form a coherent education.

If you’d like assistance in incorporating more iterative work in your class, you can make an appointment with a CAT staff member to talk through your plan. You can also arrange to have someone from the CAT come to your in-person or virtual class to give you feedback.

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series.

Jeff Rients serves as an Assistant Director at Temple’s Center for the Advancement of Teaching.

Using P.I. To Manage A.I. pt. 3: Learning Assessment Techniques That Help Build Students’ Self-efficacy

Cliff Rouder, Ed.D.

“I think I can, I think I can,” goes a well-known line from the classic children’s fairytale, The Little Engine that Could. In this blog post we’re going to answer the question, “How can we, as instructors, help our students to “think they can”? In other words, how can we help develop students’ self-efficacy? 

Let’s start by exploring the concept of self-efficacy. Self-efficacy refers to the level of a person’s confidence in their ability to successfully perform a behavior. Well-known psychologist and researcher Albert Bandura was the first to demonstrate that self-efficacy has an effect on what individuals choose to do, the amount of effort they put into doing it, and the way they feel as they are doing it. Subsequent theorists have also demonstrated that self-efficacy contributes to motivation and is a predictor of intentions to change behavior. Still other researchers have demonstrated the ability of self-efficacy to predict outcomes of student success, such as cumulative GPA, grades, persistence, and stress/strain/coping. 

Clearly, academic self-efficacy matters on a variety of important fronts. Our next task is to examine our role in developing students’ self-efficacy. Here we can get a bit of guidance from Bandura, who posited that there are four sources of our self-efficacy development:

  1. Performance Accomplishments: Previous successes raise mastery expectations, while repeated failures lower them. Here, we can give students opportunities for practice that provide a moderate (neither too easy nor too hard) challenge.
  2. Vicarious Experience: Aka modeling, self-efficacy is gained by observing others perform activities successfully. Here, we can create opportunities for peers (and you) to model productive ways of approaching course requirements.
  3. Social Persuasion: Activities where people are led, through suggestion, into believing that they can cope successfully with specific tasks. Here’s where providing frequent, targeted, and motivating feedback can help. See the CAT’s recent EdVice Exchange blog post for providing feedback that nourishes the mind and heart.
  4. Affective arousal: Some feelings, e.g., stress and anxiety, can lower self-efficacy. Here’s where acknowledging and addressing these feelings can help. 

Below are just some of the many learning assessment techniques that can target and develop students’ sources of self-efficacy. (For a comprehensive collection of learning assessment techniques, check out Barkley and Cross’s recent book available online through Charles Library.)

Think-Pair-Share

Pose a challenging question to students that they must consider alone and then discuss with a neighbor before settling on a final answer. This is a great way to have students model their thinking for their peers. A think-pair-share can take as little as three minutes (quick-response) or longer (extended response), depending on the question/task.

Classroom Polling

Ask students questions throughout your lecture using our newly-licensed polling software, PollEverywhere. Using this method, the learner and instructor can both check understanding anonymously. An alternative low-tech approach is to give students a set of numbered index cards that correspond to the answer choices and ask them to hold up their number at the same time. 

Group Formative Quizzes

Have students complete a quiz individually and then work with a group to compare and discuss answers before submitting a group answer. An added possibility is to have the group use “scratch-off” cards called Immediate Feedback Assessment Technique (IF -AT) sheets. This allows students to assess their understanding as well as practice articulating and explaining concepts to classmates.

One-Minute Paper

Either during class or at the end of class, ask students to produce a written response to a question. This technique can be used to collect feedback on understanding by asking them to identify what they thought the key points of a lecture are, what the most confusing point is, or to voice a question. It can also be used to “take the temperature” of the class and address affective concerns such as stress and anxiety related to the assignment or assessment.

Student-Generated Test Questions

These allow you to assess what students consider to be the most important content, what they understand as useful test questions, and how well they understand the material. They allow students to practice organizing, synthesizing and analyzing large amounts of information in order to prepare for summative assessments.

Rubrics

Sharing the rubrics you have for course assignments and projects, and most importantly, giving them practice using the rubrics to confirm understanding of your expectations, can help lower anxiety about unclear expectations.

You may be asking yourself, “Do I have the self-efficacy to try a new learning assessment technique!?” If so, great! Try it out. Just remember that ultimately your assessments should be chosen based on whether they are helping your students meet the course goals, so use them intentionally. Also, be clear about (and make transparent to your students) the purpose and value to them of engaging in the activity/assessment. 

If your self-efficacy is low, then look to your peers who can model how to successfully implement these learning assessment techniques, help persuade you to try one, and/or talk with them about the apprehensions you might have and how they addressed them. Then, be sure to reflect on the experience and see what you might improve the next time you try it. Getting feedback from your students can also help you assess what might need to be tweaked the next time you implement the technique.

I also invite you to make an appointment with a CAT staff member to talk through your plan. You can also arrange to have someone from the CAT come to your in-person or virtual class to give you feedback. So, let’s help our students go from “I don’t think I can” to “I know I can!

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Cliff Rouder serves as Pedagogy and Design Specialist at Temple’s Center for the Advancement of Teaching.

Using P.I. To Manage A.I. pt. 2: Designing Meaningful Assessments by Applying the UDL Framework

Dana Dawson, Ph.D.

Take a moment to think about the last time you worked really hard on something, a time where you focused intently and devoted care and attention to completing a task. It might have been a project related to a hobby, something you did to support a family member or a task required for your job. Consider what prompted you to spend time and effort on the task. Was it because you enjoy the process? Because the outcome was important?

As we think about how to teach in the face of ready access to content generating tools such as ChatGPT, it is important that we reflect on the “why” of the learning activities and assessments we are designing for our classes. Our students are influenced by the same types of motivations that drive us as instructors to make time for certain activities and deprioritize others. Our attention is a valuable resource… and a finite one! We are all constantly making decisions about how to allocate our time and attention. So, if we want our students to invest their time and attention in our coursework and to use AI tools to benefit learning rather than as time-saving shortcuts, we must design activities and assessments that are meaningful and that warrant dedication of effort. Universal Design for Learning provides a valuable framework for identifying strategies to recruit our students’ interest, encourage sustained effort and promote self-regulation.

Recruiting Interest

Here at the CAT, we often encourage faculty to design authentic learning tasks. An authentic learning task is an activity or assessment that connects with our students’ lived reality. Are we designing activities and assessments that encourage the kinds of thinking and problem-solving our students do in their lives outside the classroom or that they might anticipate doing in a future field of work or study? By creating a safe space for students to take risks and test new concepts or skills, we provide a context within which students can experience learning and personal growth.

The first step towards authentic learning is to reflect on whether your assessments are designed to assess learning goals for the course. We often defer to assessments such as essays or exams to gauge student understanding but there may be better mechanisms for making student learning visible. Assessments should be geared to as directly measure student progress toward achieving the learning goal as possible. What can students do to show they have mastered new skills? How can they document their learning? Creating multiple assessment options, multiple paths to success, will help students to feel a sense of agency in their learning and discourage turning to cheating options.

Sustaining Effort & Persistence

Deep learning requires engagement over time rather than in short, disconnected bursts. Our students are more likely to sustain their efforts if we clearly explain the purpose of the activities and assessments we assign, which of course requires clarifying for ourselves what the purpose is in the first place. Take a moment to reflect on the course content, activities and assessments you assigned during the last week of classes you taught. Was there a clear reason for each activity? Did you explain to students the purpose behind what you assigned to them?

We also know that our students are more likely to persist in learning-related tasks if they feel a sense of belonging in your class. Give students opportunities to work with one another to solve problems and share the choices they’ve made relative to course activities. A warm demeanor and supportive stance can help here, too. Students are more likely to persist without taking shortcuts if they see the instructor as a partner in their learning rather than as the course antagonist.

Self-Regulation

While our students will be more motivated to dedicate time and attention to course-related tasks if they understand the purpose, we do not have to be the sole generator of that purpose. Give students opportunities to connect course content with their goals. What do they care about and how might your course content relate? Studies have shown that simply encouraging students to reflect on their values in class promotes motivation and engagement. Provide tools such as rubrics that allow students to monitor their own learning and improvement over time and ensure feedback is targeted and communicates how students can build from where they’re at.

It can also be helpful to explicitly discuss with students the kinds of skills and practices students need to cultivate to succeed in your course. Instructors tend to assume that someone prior in our students’ education taught them how to study or take notes. But for some students, your course may be the first time where a systematic approach is needed to pass the class. Or perhaps your course requires different study skills than any other course they experienced before. We need to be able to articulate to our students what being a good student looks like in our course.

For more information on Universal Design for Learning, visit the CAST website and for help implementing the above ideas, don’t hesitate to reach out to the CAT for a one-on-one consultation.

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Dr. Dana Dawson serves as Associate Director of Temple University’s Center for the Advancement of Teaching.

Hit the Pause Button: A New Kind of Late Work Policy

Wren Mills, Ph.D.

Like many, I have learned a lot about myself as a teacher and my students as people through the pandemic. We all had to make adaptations, some of which will disappear when (if?) the virus eventually does.  One that I have decided to keep is my new “pause button” approach to late work.  

In March 2020, everything was panicky and confused and, well, not anything any of us had experienced. While I primarily teach online and have no kids that I suddenly found at home as many of my colleagues and students did, I was not immune to their distress their work-life balance shifted so dramatically after spring break.  And with that, I took a deep breath and let go of some control, and boy did it feel good!  This is the message I sent my students about our new “late work” policy: 

If you can’t get work completed on time, don’t panic. I want you to be happy with what you submit, and sometimes life takes over and school needs to be put on pause while we cope.  For my class, hit the pause button if you need to.  Just email me and let me know. You don’t have to share details, just “I need to hit the pause button this week” works. No penalties for late work. This lets me know you’re still out there and trying your best vs just giving up on the class completely.  I hope no one gives up.

I gave this grace to both my undergraduate students and my graduate students. My initial concern about implementing such a policy was that they’d all “hit pause,” and then I’d end up with loads of incomplete students. But that did not happen.  I had a few students per class who had to “hit pause”—some worked in health care, some suddenly had several children at home 24/7, and a few actually got sick, with 2 needing prolonged hospital care.  But you know what? Every single one of them completed their courses.  One of them needed about 10 days beyond the final exam period to complete the final paper, but everyone else finished on time.  

I received many emails thanking me for taking the pressure off and knowing that they had this option, even if they never used it.  The students who did use it said it was the difference in their being able to rest easier knowing that I meant what I said—to focus on themselves and let me know when they were “pushing play” again and what questions they had before they got started. 

In Spring 2020, even I had to “hit pause” when things got to be a little too much as I helped colleagues who had never taught online before shift to that modality way too quickly—and students let me know it was okay if I needed a break, that they completely understood.  When you give grace, you get it in return.  

I have continued this policy every term since then.  And I will keep doing it.  I still have had not even one student take advantage of it.  (I know it will happen eventually, but I’ll deal with it when it does.)

I think the one silver lining we have of this whole pandemic experience is we all had a chance to learn something—not in spare time (!!!), but from the experience of it all. I learned that a little kindness and transparency go a very long way to creating a welcoming learning environment and stronger relationships with my students, and I look forward to continuing to allow this human touch, this little bit of grace, in all of my classes.

Wren Mills is Pedagogical Assistant Professor at Western Kentucky University’s School of Leadership and Professional Studies.

This article is released under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).

Using P.I. To Manage A.I. pt. 1: Introduction

Stephanie Laggini Fiore, Ph.D.

We are all teaching in a new reality created by powerful text-generation tools like ChatGPT that allow us and our students to compose text on demand. Lori Salem, Assistant Vice Provost and Director of the Student Success Center, and I wrote an initial post about this last semester. As instructors, we will all need to think hard about how to manage and harness the power of this tool. I use the words “manage” and “harness” intentionally here, as we cannot pretend that we can entirely ban these tools nor can we rely on anti-AI detectors (that I can assure you will not be foolproof). In addition, we have a responsibility as educators to guide our students in the ethical and effective use of AI tools that will be available to them beyond the university in their workplaces and in their daily lives. There is no body of research (yet) that can guide us in using AI for teaching, so we are all feeling our way along by reading, debating, and experimenting with some best ways forward. 

While the teams at the CAT and the Student Success Center are working towards developing a set of guiding principles for managing AI in our classrooms, the way to start right now is by considering how PI can help us manage AI. What is this magical PI, you say? Does it have something to do with Tom Selleck (if you’re my age, you get that joke)? Is it a fancy new counter-AI robot that will solve all of our problems? No, my dear colleagues, it is simply an invitation to examine the fundamentals, the Pedagogical Intelligence that should be the first stop on the road to a set of principles for thinking about teaching in the presence of Artificial Intelligence. In the CAT’s new spring series, Using PI to Manage AI, we will be exploring, both on our blog and on our CAT Tips series on social media, these pedagogical fundamentals as a way to start this conversation. The topics we will explore on our EdVice Exchange blog are all evidence-based ways of designing student assessments of learning in ways that will encourage academic honesty, motivation, and a desire to learn. We will follow each blog post with a CAT Tips video on social media outlining a few concrete ways to implement these assessment strategies in your classes. If you have not recently done a deep dive into evaluating how useful your assessments are for evaluating learning–and also for furthering learning by engaging students in meaningful learning tasks–now is the time! 

We will start the series by exploring how to design assessments that are meaningful for students, allowing them to connect to what we are teaching in ways that help them see the value of engaging in the work. The following blog post in the series will discuss how to use learning assessments to build student self-efficacy in ways that help them to be able to do the work well and  to feel confident in what they are learning. Then we will unpack iterative work that provides feedback and allows for revision along the way. We will subsequently examine summative assessments and strategies for supporting students to think reflectively about how they prepare for these usually higher-stakes assessments. Finally, we will complete the series by introducing some educational technology tools that can assist us in implementing better assessment protocols. 

It will be important to approach this new challenge as an opportunity. It will necessarily push all of us to think deeply about how we are teaching and how we are assessing learning, and in so doing, lead us to more effective practices. We may surprise ourselves by discovering that AI itself can be useful in exciting new ways for learning. In the meantime, know that we at the CAT are on this journey with you, and will be working to support you as you support our students’ learning.  

Note: If you are intentionally using ChatGPT to teach in your classrooms this semester, please email us at cat@temple.edu and tell us about it. Consider also that you can engage in the Scholarship of Teaching and Learning (SoTL) by designing a classroom study to evaluate the impact of ChatGPT on student learning. If you want to learn how to design a study related to your use of ChatGPT in the classroom, contact Benjamin Brock at bbrock@temple.edu for assistance. 

Follow our Using PI to Manage AI blog series at EdVice Exchange

Follow our companion Using PI to Manage AI CAT Tips Video Series. 

Stephanie Laggini Fiore, Ph.D., is Associate Vice Provost and Senior Director of Temple’s Center for the Advancement of Teaching.

From “Under-Prepared” to “At-Promise”: Reframing Student Preparedness

Jessica Babcock

Throughout the recent semesters, as Temple and the world continued in pursuit of a “return to normal” after the pandemic, many of us found our classrooms to be anything but normal.

Sentiments that years of disrupted learning, increased mental health concerns, social unrest, and various other factors have led to a new and more seriously underprepared population of students radiated through faculty conversations.  “I’m used to students not knowing…, but now they don’t even know…” became a common phrase amongst instructors.  Attempts at improving student ability increasingly yielded lower rates of assignment completion, higher rates of student burnout, and feedback with phrases such as “unreasonable amount of work” and “pointless assignments,” thus creating a seemingly unbreakable cycle of underpreparedness and higher DFW rates.  I know that I, at least, started to wonder: is it the students who are underprepared for my class, or am I underprepared to teach this population of students?

To address this issue and hopefully find reassurance that both the current population of students and I could find success, I joined the Center for the Advancement of Teaching’s Faculty Learning Community (FLC) on Underprepared Students in the Fall 2022 semester.  Joined by twelve other faculty members representing the College of Liberal Arts, College of Public Health, College of Science and Technology, Kornberg School of Dentistry, Fox School of Business, Katz School of Medicine, and the School of Tourism and Hospitality Management, our group met biweekly throughout the semester to discuss research about, experiences with, and potential solutions to underprepared students.

Perhaps the biggest takeaway from our group came in one of our final sessions.  A simple change in terminology, suggested by a participant from CST–based in turn on a conversation with a colleague at another institution–managed to encompass nearly the entirety of our conversations.  When asked if there is a better phrase by which to refer to “underprepared” students, the colleague replied that the current scholarly phrase being used is “at-promise.”  This singular phrase got directly at the heart of what we had spent months discussing.  As described by a participant from CLA, this shift away from the deficit-minded “underprepared” label demonstrated that these students are working toward something rather than working without something. 

Acknowledgement of this fact changes not only the way we view these students, but also one’s view of self and sense of one’s role as an instructor.  Rather than working to “fix” what is missing with these students, our focus can be redirected toward finding ways to support these students in their pursuit of fulfilling their promise.  But to provide this kind of detailed support for each student in our classes seems like a massive undertaking!  How can we possibly take this on, while still teaching the necessary content and maintaining our professional responsibilities?!  It turns out, we had really been answering that exact question all along throughout our FLC sessions – we just didn’t know it at the time.

Our early conversations focused on identifying what was meant by the term “underprepared” and exploring what this looks like in our classrooms.  Our discussions found that while we were from a variety of disciplines, we all saw similar presentations of underpreparedness in our students.  Many of us initially thought being underprepared was largely related to content knowledge, but through deeper exploration of this, we identified that academic underpreparedness is only one piece of a much more complex puzzle.  We then explored the factors that contribute to underpreparedness, and this was perhaps the most eye-opening conversation in our early sessions.  Through research, conversation, and reflection, we identified long lists of personal, academic, economic, social, and institutional barriers to student success, causing them to appear underprepared on the surface.  Simply recognizing the existence of these challenges as an obstacle course of barriers for our students to overcome is a critical component to reframing our thoughts towards our “at-promise” students and their pursuit of success.

The other essential component of our role in helping these students is admittedly more involved than acknowledgement of barriers and was discussed throughout the second half of our FLC sessions – using strategies to create an atmosphere in our classrooms designed for all students to truly learn.  Again, this seems like an enormous task made up of sweeping pedagogical changes, but, actually, success in this venture can be found through a series of small modifications, most of which will improve learning not only for the at-promise students in your class, but for all!  Seeking out these tasks may seem daunting at first, but once you know what can be helpful to these students, there is plenty of information and support!  As stated by an FLC participant from CPH, “If I had known what to look for [before the FLC], this would have been really helpful!”

And our FLC group is here to help YOU with exactly this!  Upon realizing how valuable all of this information was, we wanted to create a resource and means to share with the University at large.  We want all faculty to benefit from the rich discussions, deeper understanding, and lingering questions we have found so important over the past several months.  In service of this goal, we have created this document containing some of the information and strategies we found to be critical in understanding and supporting our at-promise students.  In this document you will find information regarding the barriers students face, tips for classroom modifications, and related articles, blog posts, and other resources we believe to be most helpful.   

Additionally, we wanted to not only continue our conversations, but open them up to the Temple community through round-table discussions.  In these sessions, we hope to share even more insights that we have gained, as well as the questions and concerns we still have, and invite you to join the conversation with your questions and insights as well!

With the ever-changing population of students needing our support, our Faculty Learning Community hopes that these resources will serve to provide you, our colleagues, with the same sense of deeper understanding of and appreciation for our at-promise students that we have developed through these sessions and inspire changes to support these students at all levels and in all disciplines university-wide.

Jessica Babcock is Assistant Professor of Instruction in Temple’s Department of Mathematics and serves as Director of Developmental Mathematics.

Pre-Mortem: Preventing failure and looking forward to ensure success

Deirdre Dingman, DrPH

The premortem is an activity used by companies or their coaches as a strategy to reduce the chances that a project will fail. I first heard about it while listening to psychologist Gary Klein on The Knowledge Project in August 2022 (Ep. #144). The premortem is a twist on preventing ‘bad’ outcomes in that it assumes the project has already failed. That’s right, as Dr. Klein presents it, he has a crystal ball and can see into the future. In that future, the project has failed – it’s true he says – he has seen it. Dr. Klein then takes his participants/trainees through a process where first individually and then collectively they brainstorm all the reasons the project failed. That is part one, and its done like a nominal group process so that all the problems are listed, and the main ones are highlighted. In part two, the participants again generate a list individually and collectively. This time, they find the most likely steps to prevent the adverse outcome. According to Dr. Klein, this activity is effective in reducing failure AND building a shared sense of responsibility for the success of the project.

The premortem can be used in the classroom even when a group project isn’t involved and with similar success. It’s true, I have done this! In this post, I provide two examples of using the premortem with students. And if you’re like me, you will start to see the many ways that you can apply this in your own classes.

At the beginning of each semester, instructors consider ways to set the tone for an engaged classroom (e.g., Ep. #41 of Faculty Focus Live). Many of us ask our students to consider the best and worst classroom experiences they’ve had, or what they think an instructor could do or bring to the classroom and what they, as students, could do or bring to the classroom to make it one of those ‘best’ experiences. The first time I used a premortem in class it was meant to do this. Specifically, I wanted to set the stage for a successful outcome with students in a two-semester course. This particular course is a requirement for the Public Health major. Students must receive a minimum passing grade of C in the first semester in order to progress to the second. If not, they must wait until the following fall to repeat the course. Thus, the stakes are pretty high.

On the first day–after I spoke to the students about the course and we did a few ice breakers to get to know each other–I explained that I put a lot of thought into how I teach the course and I care deeply about each student’s progress. I told them I knew the class was demanding and I wanted to do everything I could to help students succeed. But it doesn’t always work out. And just like Dr. Klein, I told my students I had a crystal ball that could see into the future. It was December 2022, and a student was not going to pass the course. I emphasized that the student was NOT passing and then asked them to first spend a few minutes writing down all the things that went wrong for this student. I told them to think broadly on causes, for example, the students actions or inactions, the instructors actions or inactions, and other life or contextual events. Once the students had time to write their thoughts, we went around the room (about 20 students), with each person stating the first item on their list, or the first item that had not already been stated. The white board was filled with responses such as did not come to class, did not turn work in on time, took too many courses, poor time management, instructor didn’t give clear instructions, had to work full time. We repeated the activity on what the student and instructor could do to keep the student from failing and closed the activity by highlighting the top 3 things students could do to ensure they would earn at least a 73 this semester (i.e., coming to class, asking for help, turning things in on time). I shared what I would do as well (i.e., preparing for class, providing rubrics and feedback and grading their work within days of submission).

My second use of a premortem was embedded in a case study and supported by a Poll Everywhere up vote activity. I created this activity after for a different course on the topic of people who use drugs. Before starting the activity, I got to know the students a bit and highlighted a few things about substance use, dependency, and addiction. The premortem began with a story about a person named Jackson. Jackson is a 30-year-old Black cisgender male. One week ago, Jackson was arrested for possessing 2 grams of heroin. He was released from jail this morning to await trial. A few hours ago, he died from an opioid overdose. 

After pausing for the story to sink in, I asked students, “how did we get here?” and told them to make a list of all the things that might have happened in this man’s life to get us to this point. Some additional prompting was given, for example: think immediately and distally, think about this persons actions and interactions with people, institutions, and society.  Students then wrote their reasons on a poll everywhere thread and upvoted the reasons that were most important. Students then made a list of ways we could prevent deaths from drug overdoses and upvoted the most important ideas. The details about the person in the case study can also be altered to see what different problems students might identify. Because this was the first time I tried this activity, I included a brief exit survey with the following questions.

  1. What do you think was accomplished by the activity we did today?
  2. What suggestions do you for how today’s activity could have been improved?
  3. Is there anything else about today’s activity or topic that you’d like to share?

I was pleased with both premortem activities and believe the activity itself can be adapted for use in any course and any topic. I am happy to brainstorm ideas for you classroom and can be reached at deirdre.dingman@temple.edu.

[And, as always, the CAT is available for one-on-one consultations on this or any other learning activity you are planning for your Temple students. -Ed.]

Deirdre Dingman, DrPH, MPH, CHES is Associate Professor of Instruction in the Department of Social and Behavioral Sciences and Chair of the Collegial Assembly of the Temple University College of Public Health.

The CAT Celebrates 20 Years of Advancing Teaching Excellence!

Cliff Rouder

More than 220 faculty from universities across the region came together with the CAT for the 2023 Annual Faculty Conference on Teaching Excellence co-sponsored by Temple Libraries, the Office of Digital Education, Information Technology Services, and the General Education Program. We were delighted to be back in-person on January 11th and 12th to celebrate the 20-year anniversary of the CAT. The CAT was founded in 2002 as the Teaching and Learning Center (TLC). In 2016, the TLC merged with the Instructional Support Center and was renamed the Center for the Advancement of Teaching. In celebration of the 20-year anniversary, the CAT invited and honored past directors of our center.

This year’s theme was Achieving Rigor Without the Mortis: Keeping High Standards While Rejuvenating Our Students, Ourselves, and Our Communities. Participants grappled with the notion of rigor–its different meanings, how the pandemic may have reshaped our notion of rigor, and how that is being played out in the policies and practices in our courses.

Keynote and Plenary Address

Marcus Johnson and Derek Bruff

The CAT was honored to have Dr. Marcus Johnson, Professor in the Educational Psychology and Educational Research and Evaluation programs at Virginia Tech, as our keynote speaker and Dr. Derick Bruff, author, consultant and Visiting Associate Director of the Center for Excellence in Teaching and Learning at the University of Mississippi, as our plenary speaker. 

Dr. Johnson’s day one keynote was titled, Motivation and Rejuvenation with Rigor + Care. After defining motivation and its characteristics, he shared common misconceptions about what motivates students. He then discussed three constructs from different theories of motivation that could guide us as we consider ways to spark students’ motivation through our assessments, learning activities, and classroom environment: purpose, mastery, and autonomy. By considering our students’ diverse motivations, experiences, and identities, we can target a variety of motivational strategies that can lead to better outcomes for more of our students. Dr. Johnson concluded by encouraging us to help rejuvenate our students by supporting their well-being, helping them to become better self-directed learners, and by giving them opportunities to be inspired and engaged cognitively, behaviorally, affectively, and socially. 

Dr. Bruff’s Plenary on day two was titled, Intentional Tech: Reconnecting Our Students to Learning. He discussed three principles and then asked us to generate and consider a variety of technologies to address these principles, which are to 

  • structure ways for students to learn from and with each other to enhance learning for all and for creating a sense of community;
  • incorporate multiple modalities (audio, visual, embodied) for teaching content that can help more learners succeed;
  • connect our students to authentic learning experiences to raise the value of what they’re learning and to help students see themselves in their desired majors and as future professionals in their field of interest.

He reminded us to base our choice of technologies on whether they help students meet the learning goals for the course. 

Building on the Keynote and Plenary Sessions

In addition to keynote and plenary speakers, the conference featured interactive workshops, breakout sessions, lightning talks, and poster sessions–all designed to generate discussion and share ideas for creating challenging, inspiring, inclusive, and equitable learning experiences. 

New this year was our resource fair, where representatives from a variety of Temple support units explained their services and provided informational materials. Representatives from the Student Success Center, Temple Libraries, The Wellness Resource Center, CARE Team, Disability Resources and Services, and Instructional Technology all joined us for this fair. Additionally, two tables were set up for CAT staff to provide information about the Scholarship of Teaching and Learning (SoTL) scholarship and another table for information regarding CAT’s Teaching in Higher Education Certificate.

This year we invited representatives from a variety of Temple support units to provide resources and explain their services during the luncheon on day 2. Representatives from the Student Success Center, Temple Libraries, The Wellness Resource Center, CARE Team, Disability Resources and Services, and Instructional Technology joined us to share resources. In addition, CAT staff who are leading efforts to expand its outreach for Scholarship of Teaching and Learning (SoTL) scholarship and another table for information regarding CAT’s Teaching in Higher Education Certificate.

Let’s Continue the Conversation

Our annual faculty conference generated many thought-provoking questions and teaching strategies, so let’s continue talking! Here are some ways you can keep the conversation going:

  • Share what you’ve learned with your program/department faculty.
  • Visit our faculty commons to ask questions, pose ideas, and get feedback from the CAT and faculty across disciplines.

On behalf of everyone at the CAT, we wish you a joyous new year and a fullfilling spring semester!