{"id":1628,"date":"2025-04-21T09:47:21","date_gmt":"2025-04-21T13:47:21","guid":{"rendered":"https:\/\/sites.temple.edu\/assessment\/?p=1628"},"modified":"2025-04-21T09:47:21","modified_gmt":"2025-04-21T13:47:21","slug":"partnering-with-chatgpt-to-analyze-user-feedback","status":"publish","type":"post","link":"https:\/\/sites.temple.edu\/assessment\/2025\/04\/21\/partnering-with-chatgpt-to-analyze-user-feedback\/","title":{"rendered":"Partnering with ChatGPT to Analyze User Feedback"},"content":{"rendered":"\n<p>As regular readers of this blog know, I like to explore the use of generative AI as a collaborator in my assessment work &#8211; today I experimented with using ChatGPT to categorize a set of 100 open-ended responses providing feedback on our libraries&#8217; public programs. The analysis includes raw textual data categorization, the production of a graphic, and a &#8220;chain of thought&#8221; exchange around how ChatGPT decided what responses belonged where. &#8220;Decided&#8221; may not be the most accurate word &#8211; it implies a consciousness that AI tools do not have.  But still&#8230;<\/p>\n\n\n\n<p>The feedback form is sent via email to every registrant to our programs &#8211; not all attendees register and the percentage of feedback to actual attendance is only about 5%. Of those, 48% are from community members, 15% from staff, 13% from alumni, 15% from students and 9% from faculty. It&#8217;s interesting to see that friends and community members provide the most feedback on our programs, although our attendance favors undergraduate students.<\/p>\n\n\n\n<p>The identity question is a fixed option response and easily analyzed with Excel. The additional feedback we receive is free text, and that analysis is a little more complicated. We ask, <br \/>&#8220;How did you hear about the program?.&#8221; <\/p>\n\n\n\n<p>This is where I called upon ChatGPT to assist:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><strong><em>Prompt:<\/em><\/strong> These are responses to a question about how people found out about a program. What are themes here?<br \/><br \/>Class<br \/>email<br \/>The Charles Library<br \/>Library Newsletter<br \/>From the artist <br \/>E-mail<br \/>Email<br \/>e-mail blast<br \/>Colleague <br \/><br \/>[edited]<\/pre>\n\n\n\n<p class=\"has-text-align-center\"><\/p>\n\n\n\n<p>Within seconds ChatGPT categorized the 100 responses into categories with an offer to create a visual:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"676\" src=\"https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-1024x676.png\" alt=\"\" class=\"wp-image-1637\" srcset=\"https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-1024x676.png 1024w, https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-300x198.png 300w, https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-768x507.png 768w, https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-1536x1013.png 1536w, https:\/\/sites.temple.edu\/assessment\/files\/2025\/04\/program_attendance_sources_pie_chart_spaced-2048x1351.png 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption class=\"wp-element-caption\">How attendees learn about programs<\/figcaption><\/figure>\n\n\n\n<p>How do we evaluate this output? On the surface (and with a small amount of data) it looks good. And so fast! ChatGPT also shows us how it populated the categories, allowing us to assess the scheme. Digging into the biggest category, Email Communication (42.4%) ChatGPT explains:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<h3 class=\"wp-block-heading\"><strong>1. Email Communication<\/strong><\/h3>\n\n\n\n<p>This is by far the most prominent theme, appearing in many forms.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<ul class=\"wp-block-list\">\n<li>General email: <em>Email, Via Email, E-mail, e-mail blast, Email from Temple<\/em><\/li>\n\n\n\n<li>Specific sources: <em>Temple Libraries email, Library email, Temple email, Becca\u2019s email, Email from Charles L. Blockson Afro-American Collection staff<\/em><\/li>\n\n\n\n<li>Organizational lists: <em>Temple List Serv, Library newsletter \/ AAR newsletter, OVPR, Forecast<\/em><\/li>\n<\/ul>\n<\/blockquote>\n<\/blockquote>\n\n\n\n<p>To get additional clarification, I asked, <\/p>\n\n\n\n<p>Explain why e-blast is categorized as Email Communication not Academic Communication. ChatGPT responds: <\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>Great question! Here&#8217;s the reasoning behind categorizing <em>e-blast<\/em> under <strong>Email Communication<\/strong> rather than <strong>Academic\/Classroom Communication<\/strong>:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Why \u201ce-blast\u201d is considered Email Communication:<\/strong><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Definition of an e-blast<\/strong>: An &#8220;e-blast&#8221; typically refers to a mass email sent out to a broad audience, often through mailing list software or institutional communication tools.<\/li>\n\n\n\n<li><strong>Primary characteristic<\/strong>: The defining feature of an e-blast is its <strong>delivery method<\/strong>\u2014email\u2014not the content or sender\u2019s academic affiliation.<\/li>\n\n\n\n<li><strong>Content-neutral<\/strong>: While it <em>may<\/em> be sent by an academic department or contain academic information, the categorization is based on <strong>how<\/strong> people received the information, not who authored it.<\/li>\n\n\n\n<li><strong>Consistency<\/strong>: Other responses like \u201cEmail from Temple,\u201d \u201cTemple Libraries email,\u201d or \u201cLibrary Newsletter\u201d also fall under this category, regardless of their academic origin, because they share the <strong>same communication channel<\/strong>.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>How Academic Communication differs:<\/strong><\/h3>\n\n\n\n<p>\u201cAcademic\/Classroom Channels\u201d refers more narrowly to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Learning environments<\/strong> (e.g., in-class announcements)<\/li>\n\n\n\n<li><strong>Direct teacher or instructor communication<\/strong> (e.g., \u201cVoice teacher,\u201d \u201cProfessor,\u201d \u201cClass\u201d)<\/li>\n\n\n\n<li><strong>Departmental word-of-mouth or instruction-related messages<\/strong>, where the context is a classroom or academic engagement\u2014not generalized outreach<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<p>So, even if a professor sends out an e-blast, it still fits best under <em>Email Communication<\/em> because it functions more as <strong>broad outreach<\/strong> than <strong>individual academic instruction or class-based communication<\/strong>.<\/p>\n\n\n\n<p>Would you like a brief footnote or caption explaining this added to your chart or presentation slide?<\/p>\n\n\n\n<h6 class=\"wp-block-heading\"><\/h6>\n<\/blockquote>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\n\n\n\n<p><\/p>\n\n\n\n<p>I was impressed with this output on several fronts. If I were to categorize these responses myself, it would have taken up quite some time. I would probably not be able to so clearly articulate my reasoning as to what fit where. The distinction ChatGPT makes between Email Communication and Academic Communication &#8211; based on the <strong>how<\/strong> it is received, seems rather nuanced to me, and not a division I would necessarily have made. Understanding this may lead to improved branding of our communication and outreach. Perhaps our strategies in reaching our different communities should be evaluated. And here we have evidence of personal networks in effectively getting the word out. <\/p>\n\n\n\n<p>ChatGPT never has the last word. There is opportunity for refining the prompts and extending the dialog, and the request to explain through &#8220;chain of thought reasoning&#8221; is informative.  But there are takeaways from this modest example. The analysis starts a conversation with colleagues around these data, what they might mean, and potential actions we might take to respond. The exercise is also a learning experience around AI tools and their increasing sophistication to analyze text, calculate numbers and visualize the results in powerfully convincing ways. <\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>As regular readers of this blog know, I like to explore the use of generative AI as a collaborator in my assessment work &#8211; today I experimented with using ChatGPT to categorize a set of 100 open-ended responses providing feedback &hellip; <a href=\"https:\/\/sites.temple.edu\/assessment\/2025\/04\/21\/partnering-with-chatgpt-to-analyze-user-feedback\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":4680,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[48],"tags":[129,128],"class_list":["post-1628","post","type-post","status-publish","format-standard","hentry","category-surveys","tag-chatgpt","tag-generative-ai"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/posts\/1628","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/users\/4680"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/comments?post=1628"}],"version-history":[{"count":8,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/posts\/1628\/revisions"}],"predecessor-version":[{"id":1639,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/posts\/1628\/revisions\/1639"}],"wp:attachment":[{"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/media?parent=1628"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/categories?post=1628"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.temple.edu\/assessment\/wp-json\/wp\/v2\/tags?post=1628"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}