{"id":399,"date":"2023-07-19T12:51:41","date_gmt":"2023-07-19T16:51:41","guid":{"rendered":"https:\/\/sites.temple.edu\/edvice-test\/?p=399"},"modified":"2023-08-09T09:42:14","modified_gmt":"2023-08-09T13:42:14","slug":"a-survival-guide-to-ai-and-teaching-pt-8-academic-integrity-and-ai-is-detection-the-answer","status":"publish","type":"post","link":"https:\/\/sites.temple.edu\/edvice-test\/2023\/07\/19\/a-survival-guide-to-ai-and-teaching-pt-8-academic-integrity-and-ai-is-detection-the-answer\/","title":{"rendered":"A Survival Guide To AI and Teaching pt. 8: Academic Integrity and AI: Is Detection the Answer?"},"content":{"rendered":"\n

Stephanie Laggini Fiore, Associate Vice Provost<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Even if you\u2019ve done your due diligence in clarifying acceptable use of AI in your course, you may still suspect that students are using these tools in unauthorized ways. While unauthorized AI use is not considered plagiarism, it is still cheating and a violation of the university\u2019s standards on academic honesty<\/a>, as it both uses \u201csources beyond those authorized by the instructor in writing papers, preparing reports, solving problems, or carrying out other assignments\u201d and engages \u201cin any behavior specifically prohibited by a faculty member in the course syllabus, assignment, or class discussion.\u201d The sticky question is, therefore, \u201cHow can I be sure that students have indeed inappropriately used these tools to complete their work?\u201d We may be tempted to lean on detection methods as a solution, but is that the answer to this conundrum?<\/p>\n\n\n\n

Can Humans Detect AI Work Unaided?<\/strong><\/em><\/p>\n\n\n\n

In playing with AI tools, you may have noticed some quirks in the output they provide (based on your prompts): they can be repetitive, go off on tangents unrelated to the topic at hand, or simply produce generic or illogical text. Generative AI can also \u201challucinate\u201d citations or quote text that simply doesn’t exist. These \u201cAI tells<\/a>\u201d can sometimes tip us off to unauthorized AI use by our students. But how good are we at accurately identifying these tells? Our colleagues at the University of Pennsylvania conducted an investigation<\/a> into human ability to detect AI text. They found that participants in their study were significantly better than random chance at detecting AI output but that there was large variability in ability among the participants. The good news is that their findings suggest that detection is a skill that can be developed with training over time (Dugan et al., 2023). At this point, however, few of us have had the targeted training referenced by the authors nor have we been able to dedicate the time necessary to improve. Barring glaring hallucinations or illogical content, most of us are simply not yet familiar enough with the features of AI text to be confident that our hunches are accurate. Try the test<\/a> the researchers used; you may find, like me, that identifying AI text can be pretty darn challenging. And, of course, these tools will continue to evolve and improve, so our ability to detect non-human content may dwindle as generative AI advances.<\/p>\n\n\n\n

Can AI Detectors Do the Job?<\/strong><\/em><\/p>\n\n\n\n

Don\u2019t we all wish that AI Detectors (such as Turnitin, GPTZero, Copyleaks, or Sapling) were the answer to all of our generative AI concerns? Sadly, the simple and definitive answer to whether AI detectors can reliably detect AI-generated writing is \u201cnot at this time.\u201d The reality is that these detector tools are flawed, delivering both false positives and negatives. In addition, unlike plagiarism detection tools, there is no way to verify that the detector\u2019s conclusions are correct as the results do not link to source material in the same way. The CAT and the Student Success Center are conducting an investigation into error rates in a variety of AI detectors; early indications are concerning. In the meantime, others have pointed to the unreliability of the tools in both formal<\/a> and informal<\/a> investigations (here\u2019s another<\/a>), and in explanations<\/a> of why these tools fail. Companies creating AI detectors themselves include disclaimers such as Turnitin\u2019s statement that it \u201cdoes not make a determination of misconduct\u2026rather, we provide data for educators to make an informed decision.\u201d They then go on to advise us to apply our \u201cprofessional judgment\u201d to these situations. That professional judgment, though, can itself be flawed. <\/strong><\/p>\n\n\n\n

Some faculty have been advised to run student work through multiple detectors, but the potential for (both positive and negative) bias may come into play as we make decisions about which detector to believe when they return different results (which, from our experience, they most likely will). My wonderful student couldn\u2019t possibly have used AI so I believe the detector that says it\u2019s human-written. OR I don\u2019t doubt for a minute that this student cheated, so I believe the detector that says it is AI-written.<\/em> Importantly, these detector tools can\u2019t tell us if students have used AI in the ways we have outlined in our syllabi as permissible. Let\u2019s say I am allowing students to use AI for idea generation or for writing an outline, but not for writing full drafts of papers. The detector cannot tell me whether students have used AI in permissible ways. Finally, there are already hacks out there with advice on how to beat the detectors; for example, videos<\/a> that demonstrate how to run AI-generated content through a rephraser in order to fool AI detectors. All this adds up to inconsistent and unreliable results whereby catching those who have engaged in academically dishonest behavior is hit or miss and does not provide incontrovertible proof of misconduct. Most importantly, we have to consider the very real and potentially damaging effects of wrongfully accusing students of cheating when they have not.*<\/p>\n\n\n\n

What\u2019s a Harried Faculty Member To Do<\/em>?<\/strong><\/p>\n\n\n\n

If detectors aren\u2019t reliable and our own skills at detecting AI writing are not mature, what\u2019s the answer? While we will all be adjusting to this new reality for a while, we can keep some fundamental principles in mind to nudge our students towards transparency and academic honesty, the first of which is to give up on a surveillance mentality as it simply won\u2019t be effective (and you don\u2019t want to police students anyway, right?). Instead, think developmentally and pedagogically by taking these steps:<\/p>\n\n\n\n

1.\u00a0<\/strong>Shift from a reactive to a proactive stance. Test your assessments in a generative AI tool to see how vulnerable they are to AI use. Then make some intentional decisions about whether to change assessments or create new ones. In the long run, of course, it is all about our assessments. We may have used these same types of assessments for decades, but they simply may not work in the way we want them to in the age of AI. Review blog posts\u00a0#4<\/a>,\u00a0#5<\/a>\u00a0and\u00a0#6<\/a>\u00a0to think about changes you may make to your assessments, or if you missed our\u00a0Using P.I. to Manage A.I. series<\/a>, see our suggestions there. Remember you can also make an\u00a0appointment<\/a>\u00a0with a CAT developer to help you think this through.<\/p>\n\n\n\n

2. <\/strong>Put a statement about AI <\/a>in your syllabus clarifying acceptable use of AI! I can\u2019t repeat this enough. Our colleagues at The Office of Student Conduct and Community Standards have expressed to us that it is essential to have clear guidelines clarifying what is and isn\u2019t acceptable use of AI in our courses.<\/p>\n\n\n\n

3. <\/strong>Engage your students in a discussion about generative AI and academic integrity, including why you have set the standards you have in your course. Remind them periodically about the ethics of generative AI use. (Look for an upcoming blog post for guidance on how to speak with your students about AI.)<\/p>\n\n\n\n

4. <\/strong>Design courses that reduce the factors that induce students to cheat. James Lang, in his excellent book Cheating Lessons: Learning From Academic Dishonesty<\/a>, reminds us that the literature on cheating points to an emphasis on performance, high stakes riding on the outcome, an extrinsic motivation for success, and a low expectation of success as factors that promote academic dishonesty. The good news is that we know also from the literature on learning that evidence-based teaching practices such as formative assessments, scaffolded assignments, ample opportunity for practice and feedback, development of a positive learning environment, and helping students to find relevance and value in what they are learning will both deter cheating by reducing these factors, and improve learning. Need help in reducing the temptation to cheat? Make an appointment<\/a> with a CAT developer.<\/p>\n\n\n\n

5. <\/strong>Plan thoughtfully for how you will manage situations where you suspect unauthorized use of generative AI, starting with a conversation with the student. (We\u2019ll include advice on how to speak to students in the aforementioned future blog post.)<\/p>\n\n\n\n

There is no doubt that generative AI is a disruptor in the educational space. Our response to that disruption matters for learning and for our relationship with students. Let\u2019s work together thoughtfully towards a productive and forward-looking response. The answer is not detection\u2014it is development<\/p>\n\n\n\n

*<\/strong>Note: If I haven\u2019t convinced you to avoid these flawed detectors in accusing students of cheating, I agree with Sarah Eaton<\/a> that it is essential to transparently state in your syllabus that you will be using detectors. Do not resort to deceptive practices in an effort to \u201ccatch\u201d students. In addition, never use detectors as the sole source of evidence as, of course, the results may not be reliable.<\/p>\n\n\n\n

Stephanie Laggini Fiore serves as Associate Vice Provost at Temple University\u2019s Center for the Advancement of Teaching.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Stephanie Laggini Fiore, Associate Vice Provost Even if you\u2019ve done your due diligence in clarifying acceptable use of AI in your course, you may still suspect that students are using these tools in unauthorized ways. While unauthorized AI use is not considered plagiarism, it is still cheating and a violation of the university\u2019s standards on academic … Continue reading A Survival Guide To AI and Teaching pt. 8: Academic Integrity and AI: Is Detection the Answer?<\/span><\/a><\/p>\n","protected":false},"author":29430,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,7,4],"tags":[],"class_list":["post-399","post","type-post","status-publish","format-standard","hentry","category-a-survival-guide-to-a-i-and-teaching","category-generative-a-i","category-special-topics"],"_links":{"self":[{"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/posts\/399","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/users\/29430"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/comments?post=399"}],"version-history":[{"count":3,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/posts\/399\/revisions"}],"predecessor-version":[{"id":441,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/posts\/399\/revisions\/441"}],"wp:attachment":[{"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/media?parent=399"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/categories?post=399"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.temple.edu\/edvice-test\/wp-json\/wp\/v2\/tags?post=399"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}