Category: Beyond SFFs

Dana Dawson & Benjamin Brock

The earlier posts in this series discussed how to apply the lenses of self, colleagues, students and scholarly literature to the evaluation of teaching. In the final post in this series, we will reflect on how assessment of student learning at the course and program level allows us to take a step back and ask whether what we’re doing is working. Assessment of student learning in our classes helps us evaluate whether students have met the learning goals for the course: it tells us what our students know and can do, what they have yet to learn and are still working on, and whether our instructional decisions have been effective. Assessment of student learning at the program level helps us evaluate whether students have met the learning outcomes of the program (the curricular requirements, degree or Certificate): it tells us whether the program is designed to deliver the promised outcomes and is structured coherently and in such a way that information needed in later courses is adequately scaffolded in earlier courses. When thoughtfully designed, course and program assessment can foster reflection and dialogue that ultimately benefits the students in our classes and programs.

In this post we discuss student learning goals, learning outcomes, and the relationship between the two. At Temple, learning goals refer to what a student should know or be able to do at the end of a single course, whereas student learning outcomes generally refer to outcomes at the major, minor, certificate or curricular (e.g., GenEd or Writing Intensive) level. Assessment of student learning is most useful when it takes into account both course learning goals and student learning outcomes. For this reason, we encourage you to take into account the following factors when you design assessments. 

Curriculum Alignment

• Ideally, course assessment and program assessment begin with student learning outcomes, or the overall goals of the degree program or curricular sequence your course is embedded within. While you may not have control over course sequence or student learning outcomes, simply knowing where your course fits into the bigger picture can help you thoughtfully design assessments aligned not just with your own course goals, but with the trajectory of student learning both before and after your course. The learning goals specific to your course should be designed to deliver the larger student learning outcomes of the program your course is nested within. 

Identify or Construct Learning Goals and Outcomes

• Course learning goals and program-level student learning outcomes inform your overall course content, class activities and assessments. For this reason, it’s important that they are specific and measurable. By “measurable” we don’t mean that you need to be able to quantify student learning in relation to all of your goals. Rather, a goal should be written in such a way that you can devise a method to determine whether a student is making progress toward it and whether it has been met. For more information on writing course goals, visit our EDvice Exchange post, Learning Goals: Dream Big! 

When Designing Classroom Assessments, Begin with the Goals

• Create assessments and activities that allow students to demonstrate whether they have met the learning goals you have established. Your assessments are an opportunity for your students to highlight their learning and development across the semester, and an indicator of your effectiveness teaching the content you set out to deliver. To learn more, see our post Looking for Evidence in all the Right Places: Aligning Assessments with Goals.

Consider Program Assessment

• Course-based assessments may then be used in program assessments. You may want to work with colleagues in your program to review these artifacts using a rubric that aligns with one or more program-level student learning outcomes. Colleagues may also be called upon to help you interpret your students’ gains across the semester. While reviewing course-based artifacts such as exams, essays and written reflections, aim to identify areas of the course or curriculum in need of revision for future semesters. In other words, be sure to use the results of your assessments.

The Scholarship of Teaching and Learning (SoTL)

• If this is beginning to sound a bit like research, that is the idea . . . once this all becomes more systematic we can move into what is called the Scholarship of Teaching and Learning (SoTL). Systematically inquiring into our pedagogical practices allows us, as instructors, to make evidence-based decisions about our teaching, our classroom activities, and our assessments. Engaging in SoTL ensures our students are learning and developing as best they can in our classroom (Brock & Rouder, in press). 

Where Student Feedback Forms (SFF) are up to our students, peer review requires the input of our colleagues, and assessment of courses and students learning generally falls to the department or program. When we focus on the instructor in the classroom we realize that this systematic, continuous evaluatory process aimed at pedagogical improvement is solely in our own hands as faculty. This process allows us the opportunity to provide evidence that we are performing highly as teachers and that our students are, in fact, learning. It is also an opportunity to consider why we might want to routinely assess our teaching and our students’ learning: are our actions aimed at developing (as opposed to demonstrating) our pedagogical knowledge, competencies and skills, and how might this further our motivation to do so over time? We can think of assessment of student learning as a means to communicate empirical evidence regarding our instructional practices and our students’ experiences. It can be used to demonstrate how we are consistently evolving our pedagogical practices so that our teaching can be as impactful as possible. 

For support with designing assessments, schedule a consultation with a CAT specialist. For help with developing SoTL projects, look for SoTL consultations on our CAT consultations page.

• Brock, B. & Rouder, C. (in press). Celebrating the Scholarship of Teaching and Learning (SoTL). Faculty Herald, Temple University. 
• Linnenbrink-Garcia, L. & Patall, E. A. (2015). Motivation. In L. Corno & E. M. Anderman (Eds.), Handbook of Educational Psychology (3rd ed., pp. 91-103). Routledge. https://doi.org/10.4324/9781315688244 

Dana Dawson and Benjamin Brock work at Temple’s Center for the Advancement of Teaching.

Stephanie Laggini Fiore, Ph.D.

While reflection on one’s teaching, as well as student and colleagues’ feedback, are better-known methods for evaluating teaching, perhaps the most overlooked method is to consider how we use the scholarly literature on teaching and learning to improve our teaching. Instructors who engage with the literature of the scholarship of teaching and learning develop a vocabulary and way of thinking that moves them beyond replication of teaching methods they experienced as a student or ones that were taught to them when they were teaching assistants or junior faculty. Familiarity with this literature allows us to engage in reflection and experimentation that continually evolves our teaching practices. The insights gained from engaging with the extensive body of work on teaching and learning includes both validating effective practices you may already have been using, and of course, opening up new ways of teaching, designing curriculum, assessing learning, and supporting students that we may have never considered. It also clarifies for us why certain methods may work better than others.

It is clear to me why this criterion is often overlooked. When I started working at our teaching center after having taught for over 25 years, I was introduced for the first time to the scholarly literature on teaching and learning. I had dabbled a bit with very specific literature on teaching English as a second language, and I had read a little bit about oral proficiency methods for teaching world languages, but I never moved beyond these limited forays into this kind of scholarship. I don’t think my lack of awareness was unusual. Immersed in my disciplinary research, as most faculty are, I had never had occasion to explore the wealth of scholarship that provides guidance and evidence on how students learn. In my new role at the center, a whole world opened up to me that I never knew existed.

I remember in particular a brand new book that had come out just as I started my role at the center—How Learning Works: 7 Research-Based Principles for Smart Teaching. It was an incredibly good entry point as each chapter pulled together the research on teaching and learning on a variety of topics in coherent form and then suggested strategies we can employ in the classroom. The chapter on student motivation was transformational for me. It validated much of what I had been doing, especially around creating a positive environment for learning, but also provided so many ideas for how to support student learning in more effective ways. When I went back into the classroom, my newfound knowledge really helped me rethink my teaching and implement concrete changes that saw exciting results. If I had been asked to demonstrate how I utilized the literature on teaching and learning to improve student learning as part of a process for evaluating teaching, I could have pointed clearly to the changes I made as a result of this book and the impact those changes had on student engagement and motivation.

So how can you use this lens to evaluate teaching? In particular, you can demonstrate how you have engaged in a process of continual scholarly teaching by taking advantage of professional development opportunities that allow you to delve into the literature on teaching and learning. For instance, have you attended workshops at the CAT, met with an educational development or educational technology consultant at the CAT, or attended other similar programming offered by professional organizations in your discipline? Have you taken a deeper dive by enrolling in longer-term, intensive opportunities focused on particular aspects of teaching and learning? For instance, perhaps you have attended our 12-hour Teaching for Equity series, or you have met monthly with a cross-disciplinary group to explore a teaching topic in a faculty learning community. Maybe you have simply gotten your hands on some excellent literature (the CAT has a lending library available on all kinds of topics!) and have made changes to your teaching based on what you have read. And, of course, taking this a step further, you might contribute to the scholarship on teaching and learning by investigating how teaching or curricular changes you have implemented have impacted student learning, and then presenting or publishing on those findings.

If you have never before considered this particular lens, I urge you to give it a try! Faculty who begin that journey into the scholarship on teaching and learning find it a fascinating and energizing way to evolve their teaching and curricular practices.

Stephanie Fiore is Assistant Vice Provost and Senior Director of Temple’s Center for the Advancement of Teaching.

Stephanie Fiore and Linda Hasunuma

Peer review of teaching gets a bad rap. It conjures up images of being judged, of one’s teaching put under a microscope. Faculty express discomfort and nervousness at being observed in class, and, interestingly, they also resist the idea that they are “qualified” to provide feedback on a colleague’s teaching. That is, of course, if they even give feedback. I have a distinct memory of my chair coming into my class (unannounced), sitting in the back and writing furiously the whole time. Afterwards, I never received any feedback, but I knew that his mysterious impressions of my teaching were written in a report and filed somewhere with my name on it. And, of course, while faculty need a letter written by a peer reviewer for certain summative purposes, such as promotion, merit, or awards, these letters are often little more than a checkbox exercise written by a well-meaning colleague, and certainly aren’t intended to improve teaching.

But it doesn’t have to be this way!

Formative peer review of teaching (and by formative I mean, peer review intended to support continued growth in teaching excellence) should contribute to what Shulman calls making teaching community property. Just as we would never evaluate scholarly research on the basis of offhand comments made around the water cooler, nor should we evaluate teaching in this way. A community of colleagues can provide feedback in both our research and teaching worlds to help us improve the quality of our work. This word—community—is so important here. Done well, peer review should build community in your departments and colleges as you talk to each other about teaching and learning, promote shared educational goals, and of course, create natural support structures when our teaching goes sideways. Within this community of colleagues, a well-designed peer review process helps to encourage reflection and more intentionality in teaching, and energizes us as instructors as we gain more insight into our practices. Note that peer review can take the form of classroom observations, as well as review of a Canvas course, a syllabus, or other teaching artifacts (such as assignments, assessments, and materials). If your department is considering peer review as a professional development practice, the CAT can help you create a protocol that works for your specific department’s needs. 

Well-designed peer classroom observations should be a rewarding collaboration that contributes to the professional development of both the reviewer and the reviewed, as both gain insight into effective teaching practices through this process. There are three stages to an effective peer classroom observation: the pre-observation discussion, the observation, and the post-observation debrief. 

The Pre-Observation Discussion

Before the observation, the colleague conducting the review should try to learn as much as possible about the class goals and other helpful details, and any specific areas of concern the instructor may have about their teaching so that the reviewer can pay special attention to those areas during the observation and provide targeted feedback. 

The Observation

For the observation itself, it is very helpful to use an instrument to guide the reviewer. The CAT has recently created a new comprehensive instrument that may be useful for your peer observations, and there are other models we can share as well. Here are some helpful recommendations for conducting the observation adapted from “Twelve Tips for Peer Observation of Teaching” from Siddiqui et al, 2007):

• Be objective. Focus on specific teaching techniques and methods that were outlined in the instrument. You should communicate your observations, not your judgments.
• Resist the urge to compare with your own teaching style. Being peers does not necessarily mean that the two of you will have the same teaching style. Concentrate on the teaching style of the person and the interactions that you observe.
• Respect confidentiality. Your professionalism and trustworthiness is essential in building a peer review relationship with your partner, so confidentiality is important.
• Make it a learning experience. For the reviewer too, the process of conducting a peer observation is a learning experience, which both builds the reviewer’s skill at providing constructive feedback, and may spark new ideas useful for the reviewer’s teaching.  

The Post-Observation Debrief

Providing supportive and constructive feedback in a timely manner is key to making this experience meaningful to your colleague’s professional development. But this is, of course, the part that worries faculty most. We often advise reviewers to think of the debrief as a discussion between colleagues, focused more on asking questions than telling a colleague what went right or wrong. The guidelines below will help you give helpful feedback in peer observations:

• Give your colleague an opportunity first to self-assess what they did well, what they have questions about, and what they might do differently. 
• Limit the amount of feedback to what the receiver can use rather than the amount you would like to give (we recommend no more than 3 strengths and 3 areas of discussion and improvement)
• Your feedback should be based on observations rather than inference 
• Provide your feedback in descriptive rather than evaluative language, using “I” statements rather than “you” statements. “I saw that some students in the back were disengaged”, rather than “you should have really done something about the disengaged students in the back”.
• Begin with some (genuine) positive comments. 
• Offer constructive ideas, framed as possibilities for consideration. It can help to frame these ideas as questions. “Have you considered trying…?”
• Invite dialogue about your comments and questions. 

Adapted from: 
Ende, J., M.DEnde, J. (1983). Feedback in Clinical Medical Education. JAMA;  250: 777-781; and Oxford Learning Institute.  Giving and Receiving Feedback. http://www.learning.ox.ac.uk/rsv.php?page=319

Peer review can be a rewarding and meaningful part of our professional development if designed with care and transparency and in the spirit of doing our best to support student learning. It can help us build community with our colleagues through a shared sense of responsibility and mentorship about our development as teachers, and encourage personal reflection about our teaching practice. Ultimately, of course, its purpose is to deepen student learning, a goal we share as educators. 

In the next part of this series, we’ll discuss evaluating teaching using outcomes and assessments.

Stephanie Fiore is Assistant Vice Provost of Temple’s Center for the Advancement of Teaching and Linda Hasunuma serves as an Assistant Director at the CAT.

Jeff Rients and Cliff Rouder

In Part I of this series, Stephanie Fiore outlined Brookfield’s four lenses of reflective practice: an autobiographical lens, our students’ lens, our colleagues’ lens, and the lens of theoretical literature. Today we’re going to look at the first lens, our own autobiographical understanding of what is happening in our courses. Reflecting on our own practices and the behaviors of our students is an important component of evaluating our teaching for four key reasons:

• The single instructor model of the classroom sometimes makes teaching a lonely business. We only occasionally have a qualified professional in the room to give us feedback (more on that in the next installment). If we don’t take the time to seriously interrogate our daily practices, there’s simply no one else around to do the job.
• A huge amount of the craft of teaching takes place inside your head! Instructors are constantly evaluating and adapting to the inherently fluid situation that arises when real people wrestle with complex topics. No one else can capture this valuable data, because only you know which thoughts drove your in-the-moment decisions. The only way to make sense of it all after the fact is through reflection.
• Although our students’ opinions and insights are invaluable, if we uncritically accept their thoughts and suggestions then we run the risk of spending our teaching careers incoherently zigzagging from one extreme to another. That does neither us nor our next group of students any good.
• We want our students to be reflective learners, so they can apply their learning in new ways and new situations. Well, we need to practice what we preach! If we are not reflective practitioners then our efforts to teach the principles of reflective learning will come off as inauthentic, because that’s what they will be.

But developing a reflective practice can be hard. For one thing, we might wince a little when we think back on mistakes we’ve made or times when our students just didn’t connect with what we were trying to teach them. For another, we’re all busy and it can seem like a luxury to take the time needed to stop what we’re doing, think about what’s working and what’s not, and revise our future actions. But the only way to understand ourselves and grow as instructors is to invest the time in ourselves that we need to turn our past misadventures into future successes.

The key to a solid reflective practice is to develop a specific regular discipline that works for you. Ideally, you would have a few minutes after every class session to reflect on the events of the immediate past, but a time set aside at the end of each day, or certain days of the week, or even one day a week can work. The longer between the end of the class session and your formal reflection time, the more important it becomes to scribble some notes to yourself during class, so you can remind yourself later what transpired. Additionally, you should consider making an appointment with yourself in your Outlook calendar or whatever scheduling tool you use. Not only will that serve as a reminder to do the reflection, but an appointment with yourself makes the task feel “more real” to a lot of us. If you find yourself regularly canceling or moving the appointment for other things, that may be a signal that you need to choose a different time.

Once you can sit down–preferably alone and in a relatively tranquil space–you will need a reflection method. Here are a few possibilities:

Mark Up Your Lesson

In this technique you add comments directly to your lesson plan and/or slide show. This can be helpful if you teach similar material from semester to semester, provided that you review each lesson well enough in advance that you can implement changes the following semester.

Journaling

We talked about this topic in another EDvice Exchange post. One major advantage of a journal, whether ink-and-paper or electronic, is that it collects all your thoughts together in one place for easy review.

Audio/Video Options

Talking out loud to yourself may sound weird, but it can help you process what is going on in your class. For audio only you can use a voice recorder app on your phone, or something like Audacity. For a video recording, a Zoom room of one and the record feature do the job nicely. Of course, if you’re feeling brave you could publish your ongoing reflections via YouTube or SoundCloud or TikTok! Not enough of us talk publicly about what is happening in our classrooms.

Two other things you’ll want to consider as part of your reflective practice: The first is talking to somebody. A regular debrief with a colleague (or a staff member at the CAT!) can help you put your thoughts into perspective. Even getting together once a month to talk about your teaching can help. The second is that at the end of each semester you should consider a reflection session where you go over everything that has happened in your course and try to synthesize what your big takeaways are. You may even find it useful to write a memo to yourself, with a page or two of ideas of how you want to do things differently next semester.

Whichever options you choose, make sure to go back and review your reflections when you receive your SFFs and when you sit down to revise your course. The former is important because you’ll be able to compare your own insights with those of your students, while the latter ensures that all your reflective work pays off in your future teaching.

In the next installment of this series, we’ll be looking at how our colleagues can assist us in evaluating our teaching.

Cliff Rouder and Jeff Rients both work at Temple’s Center for the Advancement of Teaching.

Stephanie Laggini Fiore, Ph.D.

Evaluation without development is punitive, and development without evaluation is guesswork. (Theall, 2017)

Lee Shulman, past president of the Carnegie Foundation for the Advancement of Teaching and professor emeritus at Stanford University, recounts his surprise that his vision of faculty life as a combination of quiet, solitary scholarly activity and vibrant, collegial interactions with a community of teachers was backward. Says Shulman (1993), “We close the classroom door and experience pedagogical solitude, whereas in our life as scholars, we are members of active communities: communities of conversation, communities of evaluation, communities in which we gather with others in our invisible colleges to exchange our findings, our methods, and our excuses.” In fact, when I speak with faculty about the possibility of implementing new methods of teaching evaluation (such as peer review) that will break down that isolation and begin to develop synergies among faculty for development in teaching and learning, they may fall prey to imposter syndrome, claiming not to be expert enough to provide feedback to colleagues. At the same time, they reveal a sense of vulnerability at the idea of having others observe their teaching.  

But a remarkable thing happened during the shift to remote learning during COVID-19. Faculty began to emerge from their isolation, connect with each other to talk about teaching and brainstorm together solutions to teaching challenges. New Facebook pages dedicated to pedagogy sprang up (the Pandemic Pedagogy group has 31K followers), national disciplinary organizations put information on their websites and circulated it through listservs, department meetings were dedicated to teaching and learning, faculty spoke with students about what worked. In short, because we were pushed into the deep end without a lifejacket, we focused our attention on teaching. And we grew by learning from each other and from our students!

Evaluation of teaching has long been practiced as a mechanism for summative decisions regarding promotion or contract renewal, and faculty will complain (often rightfully so) that it can be either a checkbox exercise devoid of real meaning or based heavily on student feedback. Evaluation of teaching should be so much more! It should create the kind of community that the pandemic briefly afforded us, one in which we as professionals reflect on our own teaching, discuss our practices with colleagues, learn from each other, from our students, and from how well students meet our learning goals, and move towards continual, formative improvement. Stephen Brookfield (2005) suggests that we look at our teaching through four lenses: an autobiographical lens, our students’ lens, our colleagues’ lens, and the lens of theoretical literature. We might also think about how we assess whether our students are reaching the learning goals we’ve set out for them, and what changes we might make to try to improve their ability to succeed in our courses. As Berk (2018) points out, multiple sources can be both more accurate and more comprehensive in evaluating a professional activity as complex as teaching. These multiple sources can be deployed for summative purposes, of course, but more importantly, they can be useful as a holistic tool to help us continue our growth as educators, and our effectiveness in supporting student learning.

We already have a long history of employing the student lens through student feedback forms (SFFs) so this series will not separately discuss this method of evaluation. However, I will mention here how important it is to be mindful of best practices in using SFF data in order for it to provide helpful information towards improvement of teaching. The Temple University Assessment of Instruction Committee has just put out a very helpful guide to using SFF data, Recommendations for the Use of Student Feedback Form (SFF) Data at Temple University. This comprehensive guidance includes a good overview on the purpose of SFFs, what they are and are not, advice for instructors on how to use SFF data, and advice for evaluators on how to use SFFs responsibly and effectively for evaluation purposes. See also How to Read Those SFFs and Flip the Switch: Making the Most of Student Feedback Forms for guidance on the best ways for faculty to use student feedback to improve teaching. And, of course, you can make an appointment with a faculty developer at the CAT to discuss your SFFs.

Remember also that SFFs are not the only way to receive student feedback. I strongly recommend gathering mid-semester feedback as a check-in with your students while there is still time to make changes in the semester. It has the added bonus of having students reflect on their learning and consider changes they may want to make in order to achieve better results. You can also ask the CAT to perform a mid-semester small (or large) group instructional diagnosis. 

This blog series will continue throughout the fall semester with an exploration into the other teaching evaluation methods that can be used to both assess teaching practices and grow teaching excellence. Stay tuned for the following upcoming topics:

Part II: Reflective Practice

Part III: Peer Review of Teaching

Part IV: Assessment of Learning Outcomes

Part V:  Literature on Teaching and Learning

At the end of this series, my sincerest wish for you is that you find new ways to think about your teaching practices, that you engage with your colleagues (and with the CAT!) in productive and enlightening conversations about teaching, that you find a favorite resource on teaching, and that you connect with your students in ways that help them to learn deeply.

Stephanie Fiore serves as Assistant Vice Provost of Temple’s Center for the Advancement of Teaching. 

Kyle Vitale

Read for patterns, ideas, and humor

The Student Feedback Form can be a valuable source of information about your teaching and your courses. As often it can be a site of frustration, confusion, and irrelevant non sequiturs. How do you read your SFFs with an eye to learning about your teaching? We at the CAT would like to share a few approaches that can help you cut through the noise, find that helpful information, and even laugh off those cruder moments.

Read for patterns

The SFF is an inherently subjective object: students typically fill them out with their own personal experiences of your course in mind. While that doesn’t invalidate their observations, it does mean handling them with care. In this context, it makes more sense to look for patterns, where multiple students signal a shared experience about your course. Most of us have experienced that one student who had a terrible time and lets you know about it in the SFF. That single student’s strong language says far less than two, ten, or twenty students arriving at the same conclusion.

So, read for patterns. Do multiple students comment on course organization, or how accessible you are? That probably means a majority of students found you and the course to be clear. This kind of reading is broad, not deep: you resist getting mired in a single comment and instead survey your responses and start picking out keywords or subjects that recur. Make a list of those subjects, and only then return for deeper reading. This approach ensures you maximize your time with material that obviously deserves attention (either for compliment or revision).

Read for ideas

SFFs are not a statement of adequacy. We’ll repeat that so you can read it out loud to yourself and anybody nearby: SFFs are not a statement of adequacy. Research indicates that students sometimes respond to questions for a variety of reasons having little to do with your teaching ability, and of course, SFFs are not built to include evidence and citation. It is therefore best to enter them with an eye toward small ideas for improvement, rather than with your self worth on the line.

Do students share anything they wish they’d seen more of? Do they mention (and remember, look for those patterns) activities they liked? Are there any particularly kind and thoughtful comments that offer suggestions for changes to in-class activities or assignments? Treat these not as personal critiques, but as free advice for you to consider and (maybe, if it makes sense) adopt.

Read for humor

The internet has made us all familiar with the dangers of the “comment section,” and we have seen some crazy things in SFFs: professors accused of knowing nothing, ruining a major, and picking the wrong career. There are only two possible responses to such claims: believe them, or laugh them off. Comments like these prey on our innate insecurities, and almost always come as one-offs, not patterns. While they can hurt and be difficult to forget, keep in mind that they are almost always groundless and operate on an emotional, rather than curricular, level.

So, find a friend and laugh it off! Tell someone “they just have to hear this.” Have a party where you try to beat each other for the funniest, oddest, or worst comment out there. We promise that the moment you hear those statements hanging in the air, they’ll lose their power and you’ll find them easier to dismiss out of hand. If no one comes to mind, reach out to us — we’d love to share a chuckle with you!

Keep it in context 

Remember that SFFs are just one perspective on your teaching. Many others exist including peer observation, your own reflections, more informal conversations with students, or asking a CAT staffer to come visit. SFFs are not comprehensive, and should be digested in concert with other evidence for a fuller picture of your ongoing development as a teacher. As always, the CAT is here to help: if you’d like us to help you decipher your SFFs, feel free to make an appointment and we’d be happy to read them with you.

Kyle Vitale, PhD,  is Associate Director of Temple’s Center for the Advancement of Teaching.