Assessment on the Ground

Assessing the Impact of Textbook Purchases

Posted on April 28, 2026 by Nancy Turner

This post was contributed by Karen Kohn and describes her work with the Open Education Group (Kristina De Voe, Lead and Members: Karen, Courtney Eger, Ella Lathan Maia Janssen and Andrew Diamond)

Every semester, Temple University Libraries purchases textbooks for students to use. The textbook purchase process is an important function of our Open Education Group (OEG), which aims to make education more affordable for Temple students by reducing their textbook costs. A 2023 survey of Temple students found half of them were moderately or extremely worried about their ability to pay for course materials. Prioritizing paying for course materials can mean students have less money for basic needs like food or housing, and if they cannot afford textbooks, students perform more poorly in class. The emails that OEG regularly receives from students having difficulty paying for course materials underscore the importance of our work.

The first step in providing access to textbooks is learning what materials are assigned in courses. Staff within OEG receive textbook information in two primary ways: a list from the campus bookstore and course reserve requests from faculty. We check to see which assigned texts are already available through the Libraries and which additional materials are available for purchase. When possible, we prefer to buy ebooks that multiple users can access at a time, though we will also buy single-user ebooks if no multi-user license is available or if it is prohibitively expensive. The reserves staff will also buy print books when no ebook is available.

Estimating Savings

With the university’s current tight budget, it is important for the Libraries to see that our spending is having an impact. OEG regularly reviews data on the usage of textbooks and uses this to estimate how much money students are saving by accessing their textbooks through the Libraries.

A previous blog post explained how we estimate the savings. Student workers collect information from the campus bookstore about textbook prices, and we multiply this by either the number of students enrolled in the class or the number of times the book was used during the semester, or whichever is lower. To understand why the calculation sometimes uses enrollment and sometimes usage, imagine a book that was used fifty times, although the class that assigned it had only twenty students enrolled. At most the Libraries have saved twenty students the cost of buying the book. Now imagine the same book was only used five times, despite there being twenty students required to use it for class. At most only five students used the library copy as an alternative to purchasing the book, so we’d multiply the price by five.

With hundreds of textbooks available through the Libraries each semester, the total estimated savings can be large. In the last three years, the Libraries have delivered between $659,000 and $858,000 in estimated annual student savings from offering electronic copies of their textbooks.

Taking print reserves into account brings the number even higher. While we prefer to provide e-reserves, and faculty sometimes worry that students won’t come to the library to borrow a print book, the data shows that print reserves are still heavily used. The most-used reserve book in fall 2025, College Algebra with Intermediate Algebra, was checked out 1,954 times last semester! With a total enrollment of 1,177 and a price of $327.50, this textbook saved students an estimated $385,467.50 in just one semester.

Return on Investment

The savings estimates make a case that offering textbooks is a worthwhile use of library funds. In the 2024-25 academic year, Charles Library spent $6,291 on new purchases of etextbooks, which comprised only 1.4% of that year’s monograph spending. Estimated savings from these new purchases alone were $19,349. Between 2023 and 2025, the savings to students from new purchases have been three to four times the amount the Libraries has spent. This is a significant return on investment!

As the graph shows, spending in AY2024-25 was lower than the previous two years. In part, this is because the list of assigned texts was shorter than in the past. Temple switched to a new bookstore company in summer 2024, and the company initially found that fewer faculty were reporting their textbook adoptions. Also, the spring 2025 list included a higher than usual percentage of titles had no ebook available for purchase. Though the savings to students due to the Libraries’ purchases of etextbooks were lower, savings were still more than three times what the Libraries spent.

Shortcomings

For a variety of reasons we are not always able to provide electronic access to assigned texts. Traditional textbook publishers often don’t make their books available to libraries as ebooks, and in some cases the ebooks cost so much more than the print that they are not viable options for us. The Libraries are typically able to provide access to approximately a third of assigned texts as ebooks.

Another challenge for the group is the surprising number of library-licensed etextbooks that do not get used. Each semester, between 14% and 24% of etextbooks show no usage. In fall 2022, OEG began using a mail merge to notify each faculty member that the library had an electronic copy of their textbook that students could use for free. This seemed to decrease the number of etextbooks that went unused, which had previously been as high as 68%, but the number is still puzzlingly high. We wonder if students are using pirated copies of their textbooks, or if those who can afford to buy print prefer that format.

Expanding our Impact

Anyone in the library can have a role in amplifying the work of OEG. Many different interactions with faculty could be opportunities to promote the Textbook Affordability Project, a grant award that supports faculty in redesigning courses to have zero textbook costs. We accept applications each year in early spring.

If students have questions about textbook availability, library staff can point them to our webpage of Etextbooks available through the Libraries. When the Libraries don’t have access to an etextbook, OEG encourages our colleagues to share whatever information they can gather about students’ textbook-related needs. We’d be glad to contact these students’ professors to encourage them to put course materials on reserve. The more information the Libraries can share with faculty and students, and the more we can learn about what students need, the better we can help.

Posted in collections, data-driven decision making, process improvement, statistics, user experience | Tagged collections, student experience, textbooks | Leave a comment

12 Years of Special Collections Archival Processing Tracking

Posted on March 23, 2026 by Nancy Turner

Over a decade ago, Assessment on the Ground was launched. The first post was an interview with Katy Rawdon, Coordinator of Technical Services in the Special Collections Research Center. Katy shared with us the many ways she used assessment to improve the management of archival processing, accessioning, and surveying. Katy continues to coordinate technical services for SCRC, and while many things have stayed the same, I expected some changes as well.

Last week we met again, and Katy reflected on the environment in which she started her assessment work – the tracking of special collections processing and cataloging and how that was instrumental in the planning of our move from Paley to Charles. While she now has new tools, like Aeon and ArchivesSpace, that support this work, the fundamentals and value of the work continues – much still conducted manually.

How It All Began

Katy: The day I started at the libraries, Margery Sly (then SCRC Director) asked me to prepare our collections to move out of Paley and into the new library. At Paley, collections were located in many different storage areas. There was a lot of stuff we had no record of, with no location information. We developed a DB/Textworks database to enter collections information. We needed to know, for each collection, the number of boxes and how big they were. We needed to target locations for each collection to move to – in Charles or Kardon. We needed to know what collections needed to be re-boxed for storage in the ASRS.

A big question was, “Will we be able to survey all of these materials in time for the move? So, tracking the quantity of surveying required and the time that would require was essential.

The good news was that we learned we could do it in time. And having that end goal, knowing the project was realistic, was very motivating in getting it done.

Nancy: Since the move, you’ve implemented some new systems, like Aeon and ArchivesSpace. Has that changed the kinds of data you have in making decisions?

Katy: Yes, implementing Aeon (a circulation and request system for special collections) and in particular ArchivesSpace (a collection management system for archives) have really improved our collections management. After the move to Charles, we planned to apply the same process to the survey of materials at Kardon. But Covid hit, so we’re now just getting started with that work. Using ArchivesSpace allows us to generate better reports, but we still use a spreadsheet for year-end numbers. It’s actually the same exact spreadsheet, shared so that everyone adds their own numbers.

Nancy: New reporting tools allow for new questions. What questions do you have for the future?

Katy: What I’d like to start looking at is how much NEW archival material we’ve taking in each year, and how much archival processing are we getting done. Is our archival processing keeping up with what we are acquiring? Ideally, we should be processing more than we are taking in.

We always have had to balance our acquisition of new collections with the need for addressing our backlog. Having this good data about the time required and costs of processing means that we can advocate for new staff. It means that we may take in fewer collections but consider those potential acquisitions in a more mindful way. We are much more discerning.

We may also ask for funding support for the processing. The Philadelphia Folklore Collection is an example, where we were able to hire an archivist specifically for that collection.

Improving the User Experience

Katy: For archival processing, the thoroughness and accuracy with which a collection is processed directly relates to the user experience.

Nancy: That’s interesting. Can you expand on that idea?

Katy: Every patron wants items described at the finest detail possible. We generally describe, or catalog, collections at the folder level. This means that the folder titles need to be very clear, so patrons know what they are looking at. It saves time for patrons when they are searching for something specific, and for staff in retrieving that information for them.

Archives can be challenging for users, because they are physically arranged differently from general collections, and they’re not browsable via open stacks. We’d like more information on how users experience our finding aids, which are the records we create describing our archival collections.

At the end of the day, we want the experience to be as easy as possible.

Nancy: Thanks, Katy. Talking with you is always interesting. And you make a great case for the use of data in making decisions about the management of special collections, demonstrating their use and value, and continually improving the user experience in accessing these unique collections.

Posted in collections, data-driven decision making, statistics | Tagged continuous improvement, special collections, technical services, user experience | Leave a comment

Revisiting the Bento-Style “Everything” Page in Library Search

Posted on February 17, 2026 by Nancy Turner

Note: This post was written by Michelle Mackinsky and Jackie Sipes

Background

Temple University Libraries offers access to books, articles, databases, archival collections and more through Library Search — our primary unified discovery interface. The “Everything” results page uses a bento-style layout to group results by resource type and gives users a high-level snapshot of what’s available across our collections.

In Fall 2025, the Discovery Oversight Group, a team that stewards the ongoing development of Library Search, proposed adding a new resource type to the “Everything” results page: Archival Collections. With our special collections finding aids now searchable in ArchivesSpace, increasing their visibility felt timely and aligned with our broader access goals.

At first this seemed simple: add another bento. But our existing page was already a tightly packed three-column, two-row grid. Adding a seventh box risked turning a scannable snapshot into visual overload. What began as a request to add a new bento, ultimately prompted us to step back and revisit the overall design. If we were changing the page anyway, how might we improve it?

*Screenshot of former Everything page design*

User Research

Joi Waller, Michelle Macinsky, and Jackie Sipes started by reviewing discovery layers from peer institutions and revisiting our own past user research on Library Search. As we compared sites from other libraries, we paid close attention to how they structured their results pages — whether they favored tighter grids or more editorial, column-based layouts; how they handled visual hierarchy; and where key navigation elements like “See all results” were placed.

We were especially interested in whether short, plain-language descriptions could clarify distinctions between resource types. In a unified search interface, categories like “Databases” or “Digital Collections” are not always intuitive. Small cues can make orientation easier — particularly for novice researchers.

This idea wasn’t new. In previous Library Search testing, participants had suggested adding brief descriptive text to clarify what types of content are included in each bento. Research from the University of Michigan Libraries affirmed similar findings.

Google Analytics Data

Before redesigning anything, we looked at Google Analytics click events on bento headers over a two-month period (April 9–June 8, 2025):

Bento Resource Type	Header Clicks (4/9/25-6/8/25)
Books & Media	707
Articles	395
Databases	27
Journals	12
Library Website	10
Research Guides	2

Books & Media and Articles were already positioned at the top of the “Everything” page, and their high engagement re-affirmed this placement. The data also clarified which bentos had lower engagement. If we needed to reposition categories to accommodate a new resource type, Library Website and Research Guides were logical candidates.

One caveat: this data captured header clicks only, not interactions with individual results inside each bento or overall impressions of the page. To understand user expectations more fully, we needed to talk to people directly.

Prototyping & User Feedback

Based on our research, we developed four prototype designs. Each included the new Archival Collections bento and incorporated elements that could improve usability and readability. We intentionally varied the designs to gather feedback on those specific elements with users.

Across the prototypes, we explored different column densities, (two- versus three-columns), compared evenly aligned grid-based structures with layouts that stacked sections in vertical columns for a more newspaper-style layout, tested whether lower-priority categories should be visually grouped or set apart, experimented with boxed styling for clearer separation, and adjusted the placement of Library Website and Research Guides within the hierarchy. By isolating these variables, we were able to ask participants not just which layout they preferred, but why – and which elements meaningfully affected clarity and scannability.

*Four prototype options shared with users for feedback*

We asked participants to compare multiple prototype layouts of the “Everything” page, talk through their immediate reactions, what felt clear or confusing, and indicate which version they preferred. They also completed a ranking exercise to identify which resource types they would want to see first when searching. Nine participants (4 undergraduates, 2 graduate students, 2 staff, and 1 faculty member) took part; most had previously used Library Search (7 of 9), typically to find either books or articles.

Key Findings

*Prototypes set-up in the Charles Library lobby for user feedback*

Prototype Preferences

Layout preference: Prototype #2, newspaper style layout was clearly preferred. Participants described it as “more organized” and noted that the boxes helped sections feel clearly separated.

Number of columns: Preferences were split among participants — some felt three columns were overwhelming; others liked the density on larger screens. Based on comments about focus and clarity, we ultimately selected a two-column layout.

Microcopy: Participants consistently found the short descriptions helpful in confirming what each section contained.

Resource Types Rankings

Participants ranked resource types according to what they would want to see first when searching (lower average = higher priority):

Resource Type	Results
Articles	1.71
Books & e-books	1.71
Journals	4.57
Databases	6.00
Digital Collections	6.00
Library Website (about the library, e.g. hours, study spaces, etc.)	6.14
Media (like streaming video)	6.57
Archival collections	6.60
Research Guides	7.29

Books and Articles were consistently top priorities — reinforcing what we saw in analytics. Archival Collections ranked lower overall, which isn’t surprising: users don’t always anticipate archival materials in a general search. But that’s exactly why visibility matters. If something isn’t surfaced clearly, it’s effectively invisible.

Prototypes and a full description of our findings are available in our final report.

Design Refinements

Because prototype #2 was preferred by most participants, we used it as the basis of the final design. The design was then refined to fully integrate the new Archival Collections bento, and included additions, like archives-specific icons to match our existing iconography.

New icons designed for Archival Collection bento item formats

We also refined the descriptive microcopy through a survey of library staff and student workers. That feedback surfaced an important clarification: archival materials are non-circulating and must be accessed onsite. We incorporated this information directly into the description text to set clear expectations for users about how they can access materials.

Development and Release

*Screenshot of new “Everything” page design*

Development involved integrating Library Search with the ArchivesSpace API and implementing front-end changes to support the new bento and the overall layout refinements. Because the “Everything” page is highly visible and frequently used, we timed the release during winter break to minimize disruption. We implemented a feature flag using Flipflop, allowing development to be completed ahead of release and activated at the appropriate time. The result was a smooth release with greater control over release timing.

Early Impact

Prior to the addition of the Archival Collections bento, the only direct path from Library Search to ArchivesSpace was through finding aid records accessible from the catalog search results. The new bento introduces a more visible and streamlined path to archival collections.

To assess the impact of the new Archival Collections bento, we monitored traffic from Library Search to ArchivesSpace in Google Analytics.

During the first five weeks post-launch, the “Everything” page consistently generated more outbound clicks to ArchivesSpace than catalog results. The table below shows outbound clicks from Library Search to ArchivesSpace by page type over the first five weeks following launch on 1/5/26.

Date Range	Catalog Search Results	Bento “Everything” Page
1/4/26-1/10/26	7	189
1/11/26-1/17/26	5	50
1/18/26-1/24/26	5	36
1/25/26-1/31/26	4	46
2/1/26-2/7/26	9	60

The first week shows a spike in traffic – this is likely due to a communications campaign we launch on January 7. Excluding that first week, the “Everything” page is driving an average of 48 visits per week to ArchivesSpace.

This early data suggests that integrating the new Archival Collections bento is increasing connections to archival materials from Library Search.

This work underscored the value of using both analytics and user feedback to inform decisions. Neither tells the full story alone, and together they provide a more complete picture of the user experience.

The design changes were not dramatic, but they were intentional. Adjustments to layout, hierarchy, and descriptive text improved clarity without altering the core search experience.

We will continue monitoring usage and gathering feedback to better understand how users discover archival materials within Library Search and whether the bento meaningfully supports archival discovery within the broader unified search experience.

Posted in access, usability, user experience | Tagged search, usability | Leave a comment

From Printing to Chat: Visualizing Library Usage Trends over Time

Posted on September 2, 2025 by Nancy Turner

To start off the Fall 2025 academic year, we have a very special post. Dave Lacy, Director of Library Technology and Knowledge Management, shares with us his ongoing project to track the trends of library usage over time. He uses Tableau to create detailed interactive visualizations that allow us to see and explore usage data, combined from multiple sources, in new ways. This post is based on a virtual “interview” with Dave.

First, how would you describe this project?

I would describe it as an attempt to aggregate and visualize all library transactional data under one system. Once normalized under a unified model, we can then enrich this data with demographic details of the patrons so we can see service use distribution across all schools and colleges.

This work focuses exclusively on quantitative data generated by systems and employees at many of our public service points. We often communicate our quantitative data in isolation. Our newsletters and annual reports typically state numbers for a given period: For instance, in FY 24-25 we had over 8,000 reference transactions and circulated over 34,000 books, but we seldom discuss trends over time. Comparing quantitative data over long periods of time often illustrates trends, and it is the trends over time that I was most curious to learn.

What are the various datasets that are included?

Charles Library owlcard swipe data
Ezproxy logs (online database and journal access)
Physical circulation (books, videos)
Inter-library loan
Public printing
Battery / Laptop kiosk transactions
Study room reservations
Instruction sessions
Events and workshops
Reference transactions (in person, telephone, email, chat, LibAnswers)

These datasets are ingested into a data model, using the common attributes from each data source. Those include:

Date/Time
Accessnet ID
Service Name / Data Source
Affiliation (student, staff, faculty, guest)
School / College

I use several quantitative measures to track service utilization over time.

Total Transactions
Total unique Patrons
Service penetration – this is the ratio of total unique patrons and the size of a given community

Finally, I use Tableau Prep for the ingestion and Tableau for the visualizations. The data collection is a combination of manual and automated processes.

Let’s look at one of the visualizations. Walk us through what we are seeing here.

This first dashboard is the main visualization used for all data sources. The top section displays weekly averages across multiple terms. The bottom section show the semester averages across terms, as well as the total size of a given population (enrollment).

In this instance we are viewing Public Printing stats for students from Fall 2018 through Fall 2024.

In Fall 2018 we registered:

9,819 average weekly transactions (print jobs, regardless of # of pages)
3,503 average weekly unique affiliates (students who print something at one of our locations in an average week)
8.4% service penetration with the whole student population (8.4% of students printed something at one or our locations in an average week)
Approximately 42,000 students enrolled at Temple (this is not an official census, nor is it our official FTE)
Approximately 15,000 total unique students over the semester printed at the library
36% of students printed at the library during the semester

In Fall 2024 we registered:

4,161 average weekly transactions (print jobs, regardless of # of pages)
1,517 average weekly unique affiliates (students who print something at one of our locations in an average week)
4.6% service penetration with the whole student population (students who printed something at one or our locations in an average week)
Approximately 33,000 students enrolled at Temple (this is not an official census, nor is it our official FTE)
Approximately 9,000 total unique students over the semester printed at the library
26% of students printed at the library during the semester

We can see a couple of notable trends, here. Printing dropped significantly after we moved to Charles, and dropped again during COVID. Fall 2024 shows a noticeable increase from Spring 2024, but is still down from previous highs experienced in 2018.

We can also look at usage of a given service across all schools and colleges for a given term. The chart below depicts all Charles library swipe access for the Fall 2024 semester. In Fall 2024, the College of Science and Technology (CST) had the highest penetration, highest patron count, and most transactions of all the schools and colleges.

We can see what services are used multiple times by an individual, like the study rooms, versus those that are used just a few times. The chart below shows the use of our online chat service.

Tell us about the actionable insights and changes we might make based on this evidence.

This data has been used to alter location and makeup of battery/laptop kiosks that ensure we are meeting the needs of our patrons. Batteries circulate at a much higher volume than laptops and some locations get higher traffic than others.

We have also used this information to scale down public computing across multiple locations, as we were providing far more options than are necessary.

Thanks!

Thanks so much, Dave, for sharing this work with the Temple staff as well as the broader assessment community. You’ve pointed to some concrete uses for this data.

How students are using the various services offered by the library has changed so much over the years – from the use of printers to how reference is changing.

Combining the data in this way also allows us to see how the different schools and colleges use our services in different ways. For instance, students from the College of Liberal Arts are most likely to check out a book, but students in Science and Technology are more likely to use our study rooms. These differences may inform our promotion of our various services in a more targeted way. One size does not fit all.

Posted in data-driven decision making, statistics | Tagged data visualization, longitudinal analysis | Leave a comment

From Inspiration to Improvement: Reflections on ALA 2025

Posted on July 6, 2025 by Nancy Turner

“Stay awkward, brave and kind”
Brené Brown

Last week I had the privilege of hearing wise words from three remarkable women – all featured speakers at the American Library Association’s annual conference. Thousands gathered to hear from Michigan governor Gretchen Whitmer, ex-Librarian of Congress Carla Hayden, and Brené Brown, academic and leadership guru.

Each has experienced professional and personal setbacks – loss, misunderstanding, unfairness. And yet they persevere, they move forward and seek connection and strength from their community.

Reviewing my jottings from these talks, I’m struck by a theme: Strength and leadership may come from a place of humility, not pomposity or arrogance. Governor Whitmer cited Walt Whitman’s words of advice, “Be curious, not judgmental.” (Although my daughter insisted Ted Lasso first spoke them.) She goes on to describe apology as a strength of leadership – if you never admit to being wrong, you will never learn from your mistakes.

Similar ideas were shared by Brené Brown. Brown, educated as a social worker, is well-known for her research on vulnerability and shame, demonstrating how vulnerability is not a weakness but a strength. Courage is the ability to show up when you don’t know the outcome. Leaders hold themselves responsible for people and processes. They need to be interested more in getting something right than in being right. Leaders are learners not knowers.

It is always an honor to hear from Carla Hayden, here in conversation with author Kwame Alexander. They are good friends, and the crowd of 1000 were friendly as well. So we were privileged to hear Hayden describe the surreal experience of her recent firing — by email. The cited reason was the Library of Congress’ holdings in DEI material made available to children.

In spite of these obstacles and frustrations, Hayden encourages us to “move on.” Get out of your bubble. Keep doing what you are doing. There is strength in numbers.

What does this have to do with library assessment? Assessment, like leadership, is about continuous improvement and learning. It’s about asking questions and listening, rather than assuming, to improve the work we do and the services we provide. It’s having the confidence to know that difficult situations can change if we gather together as a community.

When reflecting on the events of the past year, particularly the de-funding of the Institute of Museum and Library Services (ILMS), ALA president Cindy Hohl began to choke up. The audience cried, “It’s okay.” This was a moving experience for me. The collective support shown at the conference, in the face of many obstacles, renewed my pride in being a librarian professional.

Posted in conference reports, service assessment | Tagged conferences, leadership, library assessment | Leave a comment

Dear ChatGPT. Help me Learn to Make a Pivot Table

Posted on June 16, 2025 by Nancy Turner

In my conversation last week with Karen Kohn, we discussed the necessity of humans interacting with data to achieve the best analytical results. We were exploring statistics from EBSCO on the use of our e-books to understand the top subject areas within this platform.

I shared with Karen the ChatCPT-generated analysis of usage trends and she described her alternative tool of choice – the pivot table function within Excel. Pivots are an easy way of summarizing data – in this case over 5,000 records. But Karen also noted that many library staff are not experienced with Excel.

If the barrier to analysis is skill with Excel, is this something that ChatGPT could remedy? Yes, ChatGPT can help us to create learning materials. In this example, I asked ChatGPT to teach me how to create a pivot table for the same Excel file of e-book usage statistics (download here for Temple Library staff).

Here was my prompt:

Using this excel file, create a step by step tutorial for a novice data librarian on the creation of a pivot table to show the top subject areas by total usage.

The result was pretty bare bones, but here are the first few steps (The full slide deck is available here):

📊 Tutorial: Creating a Pivot Table for Top Subject Areas by Total Usage
🧾 Step 1: Open Your Excel File
Open the Excel file:
eBookTitle_Jun042025_s8483726_20250301_20250531 (1).xlsx
Identify the column labeled “BISAC” (subject area) and the column “Total Access” (usage).

📍 Step 2: Select the Data Range
Click on any cell in the dataset.
Press Ctrl+A to select the entire dataset.
Make sure there are column headers (e.g., “Title,” “BISAC,” “Total Access”).

➕ Step 3: Insert a Pivot Table
Go to the Insert tab on the top menu.
Click on PivotTable.
In the pop-up window:
Ensure your full table range is selected.
Choose to place the pivot table in a New Worksheet.
Click OK.

Note: All formatting provided by ChatCPT

For those who want to gain skills like this, we have many other tools (LinkedIn learning or other training are options). An advantage here is a more customized option, as ChatGPT’s tutorial utilized the file we specified.

What are the other differences? How should this “novice data librarian” select the best tool for a small assessment task like analyzing usage statistics? I asked ChatGPT:

Provide me with a list comparing the use of ChatGPT to analyze this data and the use of Excel and pivot tables to perform that analysis

ChatCPT came up with 14 examples. Here are a few:

📊 Comparison: ChatGPT vs. Excel PivotTables for Data Analysis

Feature / Task	ChatGPT	Excel PivotTables
Ease of Use for Novices	High – conversational guidance, no need to know formulas	Moderate – intuitive GUI, but may require learning steps
Setup Required	Minimal – just upload file and ask questions	Requires opening Excel, selecting data, inserting PivotTable
Speed of Insight	Fast – instant aggregation, summaries, and recommendations	Fast – once PivotTable is set up, updates are automatic
Customization of Output	Very flexible – can sort, filter, summarize with plain text or charts	Moderate – can pivot, filter, and chart but with less automation
Text-Based Summaries	Strong – can explain patterns and generate narratives	Weak – user must interpret data visually or manually write summaries
Visualization Options	Can generate clean, customized charts	Good – built-in chart tools, but with limited customization
Error Checking / Guidance	Yes – explains problems (e.g., wrong column types)	Limited – errors are usually user-detected or implicit

Note: Table formatting provided by ChatCPT. No editing.

It’s interesting that ChatGPT replicates my use of the word “novice” as the first difference. ChatGPT thinks very highly of itself. There’s nothing much it can’t do when compared side-by-side with Excel. It is indeed, easy to use, it’s fast, and no special software is required. ChatCPT is always willing to suggest patterns and recommendations.

Unlike Excel, where the user must interpret the data, ChatCPT is happy to provide an answer. And this is where some skepticism and critical thinking – the essential human element – are required.

Thanks again to Karen for prompting me to take my example one step further to share an additional use case for generative AI tools in support of assessment.

Posted in process improvement, technology use | Tagged ai, artificial intelligence, data analysis, Excel | 1 Comment

Deepening the ChatGPT Conversation with an Expert

Posted on June 11, 2025 by Nancy Turner

The library provides access to over 3 million e-books (based on our 23-24 reporting to ARL). Most of that access is acquired through e-book packages and EBSCO is a big one, with over 370,000 titles in a range of subject areas. EBSCO sends us quarterly reports on the usage of those titles, with statistics on views and downloads by title, purchase type, publisher and subject area.

Last week I received our report for March through May of 2025. The csv file was 5620 rows, so I asked ChatCPT to help me make sense of it. I uploaded the data and used this prompt:

Analyze this file please. It is an excel spreadsheet that lists the usage of electronic books at an academic library. I would like you to provide top level insights into the usage by numbers and content trends.

Within seconds, ChatGPT provided me with this:

📊 Overall Usage Statistics

Total Accesses: 15,892
Abstract Requests: 15,061 – indicates strong exploratory or research use
Online Views: 13,363 – the dominant method of engagement
Chapter Downloads: 1,984
Full Downloads: 545 – suggests some selective reading patterns
Turnaways: 483 – may indicate unmet demand or access limitations
Print & Email Requests: minimal (Print: 239, Email: 10)

📚 Top Subject Areas by LC Subclass

These reflect areas with the highest interest:
E – U.S. History
LA – History of Education
PN – Literature (General)
ML – Literature on Music
DT – African History
HQ – Women’s Studies, Sexuality, Family
HV – Social Pathology, Criminology, Social Work
RC – Internal Medicine
PS – American Literature
LB – Theory and Practice of Education

These trends suggest:
Strong interdisciplinary interest spanning humanities, health, and education.
Particularly high engagement with history, literature, education, and social sciences.

It was interesting that the use of abstracts was so high, and I was surprised that U.S. history was the top subject. I was impressed with this output, and the speed with which the data was analyzed. But of course, there are experts here at the library who are in a far better position to evaluate the analysis – its accuracy and its value. I asked Karen Kohn, Collections Analysis Librarian, to review ChatGPT’s output with me.

Karen was patient. Typically she looks at usage data for an entire year rather than a quarter. That aligns with our renewal cycle, and of course we report these figures to organizations like ARL on an annual basis.

Let’s also keep in mind that EBSCO is one of dozens of e-book packages and platforms that we license. While EBSCO covers all subject areas, a trend within this package is not necessarily what we’d see in JSTOR or ACM (Association of Computing Machinery).

WIth those caveats, Karen and I continued our conversation with ChatGPT.

I asked to see an analysis of “turnaways.”

📉 Turnaways Analysis

Turnaways occur when users attempt to access a title but are denied, often due to license restrictions (e.g., limited user seats).

🚩 Top Turnaway Subject Areas (LC Subclass)
HF – Commerce / Business Administration (257 turnaways)
BL – Religion / Mythology (105)
DT – African History (33)
E – U.S. History (27)
RA – Public Health (15)

This suggests a notable unmet demand in business, religion, and global history/public health content, which may merit review of current licensing models or potential acquisition of multi-user or unlimited access licenses.
Let me know if you’d like a visual of subject turnaways or recommendations on mitigating high-turnaway titles.

Karen says,

I can see places where it’s especially important to have the person writing the prompts think carefully about what to ask and look closely at the results.

There are a handful of serious outliers in this dataset, it doesn’t always know when to check for them. At one point ChatGPT points out that What the Best College Students Do accounts for 90% of Belknap Press’s total usage. Later the report notes that there are 257 turnaways in the HF section, which “suggests a notable unmet demand,” but it doesn’t point out that 256 of these are for a single book.

She’s right. There needs to be a human who is examining the data, looking for inconsistencies and prompting ChatGPT when there are results that need clarification. One of the ironies of using AI tools is this: we ask an AI tool because we don’t know an answer, but to best evaluate the output, we need some knowledge of the subject. Karen continues, explaining that:

This is analysis that I could do in Excel using pivot tables, but I recognize that not everyone has the same Excel knowledge, even people who look at collections data as part of their jobs. As someone who is comfortable in Excel, I really like to do things manually so I have more control over how the data is displayed and can look closer at specific details when I have questions, but not everyone wants to or can make their own Excel tables.

Fair point. Look out for Part II of this post, in which I ask ChatCPT to draft training materials on creating a Excel Pivot chart. And a big thank you to Karen Kohn who helped to remind us of the need for human intervention and a critical eye when incorporating AI tools into our work with library assessment.

Posted in collections, digital collections, statistics | Tagged ai, artificial intelligence, ChatGPT, collection use, generative ai, statistics | Leave a comment

Students Asking Students, “How Do You Use the Library?”

Posted on May 12, 2025 by Nancy Turner

The Student Library Advisory Board program provides a natural opportunity for library staff to learn from students and to gather feedback about their experience with the libraries’ spaces and services. Board members, paid a stipend each semester, are regular users and through tours and meetings with staff, have above-average knowledge of library operations. As their spring 2025 project, this year’s SLAB conducted an independent project – interviewing fellow students, friends and roommates about their experience with the library. We anticipated that this peer-to-peer approach would yield responses that would be different, perhaps more honest, than the feedback library staff hear directly. The exercise proved useful to students as well – providing them an expanded perspective on the library and its use.

How Did We Organize the Project?

Students were given a set of detailed instructions for conducting semi-structured interviews, including a “script.” We provided tips on how to be neutral in asking questions as well as suggestions for probes. Students were each tasked with interviewing three peers and to document responses on a shared slide deck. Our meetings for the spring were structured around this project – with test interviews, debrief, and opportunities to gather and share findings together. Our aim was the identification of themes and from this, potential recommendations to library administration for improving student experience with the library spaces and services.

5 out of the 6 SLAB members participated, conducting a total of 16 interviews. Thanks go to these students:

Joseph Lendacky, Fox School of Business (Marketing)
Lana Lerner, College of Science and technology (Computer Science)
Katelyn Man Freshman, College of Science and technology (Biochemistry)
Deyana Tabatabaei, College of Science and Technology (Genomic Medicine)
Sehaj Venugopal Varma, College of Liberal Arts (Neuroscience)

What Did We Learn about Library Space Use?

Goes to the library to print stuff out and check out any textbooks needed for class

Does not use the library and doesn’t feel like going

Uses the library almost every day and goes in between classes

To summarize the findings about space use:

The library is valued for its quiet spaces to study, both for individual work and as part of a group. Study rooms and individual carrels are most popular. One student expressed discomfort with large tables requiring them to sit with people they don’t know. And students also report socializing at the library. Some use the space every day, others have no interest in the library at all. The library is used for also for practical necessities – the printers are popular, as are the charging stations, and for items placed on course reserve.

Students asked their peers about other favorite places to do school work. Dorms were cited, particularly if a printer is available, as well as Alter Hall, the J&H basement, and the Tech Center. There may be an appeal for study spaces provided in classroom buildings, where students in similar classes can gather. The Founder’s Garden was mentioned twice, in addition to the rooftop of Mazur Hall, indicating that students enjoy studying outdoors if the weather is fine. Coffee shops and the cafeteria provide a “change of scenery” and a positive “vibe.”

As noted above, study rooms at Charles and the 24/7 space are cited as facilitating group work, with white boards mentioned as a useful room amenity.

What Did We Learn about Collections Use?

Found a paper through Google Scholar and then logged in through Temple which redirected to the Temple library website.
- Primarily uses Google and isn’t aware of library resources
- 1st step they usually take is to type keywords into the library catalog and then going to sources cited in good papers. Problem: sometimes papers aren’t actually available through the catalog.
  - Mainly uses online resources, finds physical resources time-consuming and at times difficult.

Students asked their peers about the use of the library catalog, the library web site, and how they located materials when doing research. To summarize:

Google or Google Scholar are the typical starting place for research. Respondents did not seem to distinguish between the library catalog and the library website more generally. There is wide variability around awareness of the library’s resources. One student describes following a citation trail but expressed frustration when the full text article was unavailable through the library catalog. Another seems oblivious to the availability of online resources entirely.

Students are divided on format, some express preference of print, others enjoy the accessibility of online resources. Some find it easy to locate materials through the catalog, others find it frustrating to locate physical materials.

Instructors are frequently the key initiators of library materials use by providing a direct link to the catalog for course materials or placing an item on reserve.

What Did We Learn about Library Assessment?

We did our best to ensure quality data by providing students with clear instructions and a script to guide them in conducting interviews. But our own vested interest in the project idea and its potential for wider data collection may have clouded our expectations regarding the depth of the interviews, the documentation, and students’ enthusiasm for sharing. We imagined a lively final meeting with our students, excited to discuss their findings and potential recommendations to library administration.

We learned, of course, that students are not assessment librarians. And not all data gathering projects need to be tightly controlled assessment projects, rigorous in method and analysis. We, as library staff, know more than we knew before. We have some interesting insights to consider and we have learned from our own experience with the project and how students engaged with it. And that can be enough.

Thanks to co-leads for SLAB 24-25, Justin Hill and Evan Weinstein.

Posted in library spaces, qualitative research, research work practice, user experience | Tagged peer-to-peer interviews, students, user experience | Leave a comment

Partnering with ChatGPT to Analyze User Feedback

Posted on April 21, 2025 by Nancy Turner

As regular readers of this blog know, I like to explore the use of generative AI as a collaborator in my assessment work – today I experimented with using ChatGPT to categorize a set of 100 open-ended responses providing feedback on our libraries’ public programs. The analysis includes raw textual data categorization, the production of a graphic, and a “chain of thought” exchange around how ChatGPT decided what responses belonged where. “Decided” may not be the most accurate word – it implies a consciousness that AI tools do not have. But still…

The feedback form is sent via email to every registrant to our programs – not all attendees register and the percentage of feedback to actual attendance is only about 5%. Of those, 48% are from community members, 15% from staff, 13% from alumni, 15% from students and 9% from faculty. It’s interesting to see that friends and community members provide the most feedback on our programs, although our attendance favors undergraduate students.

The identity question is a fixed option response and easily analyzed with Excel. The additional feedback we receive is free text, and that analysis is a little more complicated. We ask,
“How did you hear about the program?.”

This is where I called upon ChatGPT to assist:

Prompt: These are responses to a question about how people found out about a program. What are themes here?

Class
email
The Charles Library
Library Newsletter
From the artist 
E-mail
Email
e-mail blast
Colleague 

[edited]

Within seconds ChatGPT categorized the 100 responses into categories with an offer to create a visual:

How do we evaluate this output? On the surface (and with a small amount of data) it looks good. And so fast! ChatGPT also shows us how it populated the categories, allowing us to assess the scheme. Digging into the biggest category, Email Communication (42.4%) ChatGPT explains:

1. Email Communication

This is by far the most prominent theme, appearing in many forms.

General email: Email, Via Email, E-mail, e-mail blast, Email from Temple

Specific sources: Temple Libraries email, Library email, Temple email, Becca’s email, Email from Charles L. Blockson Afro-American Collection staff

Organizational lists: Temple List Serv, Library newsletter / AAR newsletter, OVPR, Forecast

To get additional clarification, I asked,

Explain why e-blast is categorized as Email Communication not Academic Communication. ChatGPT responds:

Great question! Here’s the reasoning behind categorizing e-blast under Email Communication rather than Academic/Classroom Communication:

Why “e-blast” is considered Email Communication:

Definition of an e-blast: An “e-blast” typically refers to a mass email sent out to a broad audience, often through mailing list software or institutional communication tools.

Primary characteristic: The defining feature of an e-blast is its delivery method—email—not the content or sender’s academic affiliation.

Content-neutral: While it may be sent by an academic department or contain academic information, the categorization is based on how people received the information, not who authored it.

Consistency: Other responses like “Email from Temple,” “Temple Libraries email,” or “Library Newsletter” also fall under this category, regardless of their academic origin, because they share the same communication channel.

How Academic Communication differs:

“Academic/Classroom Channels” refers more narrowly to:

Learning environments (e.g., in-class announcements)

Direct teacher or instructor communication (e.g., “Voice teacher,” “Professor,” “Class”)

Departmental word-of-mouth or instruction-related messages, where the context is a classroom or academic engagement—not generalized outreach

So, even if a professor sends out an e-blast, it still fits best under Email Communication because it functions more as broad outreach than individual academic instruction or class-based communication.

Would you like a brief footnote or caption explaining this added to your chart or presentation slide?

I was impressed with this output on several fronts. If I were to categorize these responses myself, it would have taken up quite some time. I would probably not be able to so clearly articulate my reasoning as to what fit where. The distinction ChatGPT makes between Email Communication and Academic Communication – based on the how it is received, seems rather nuanced to me, and not a division I would necessarily have made. Understanding this may lead to improved branding of our communication and outreach. Perhaps our strategies in reaching our different communities should be evaluated. And here we have evidence of personal networks in effectively getting the word out.

ChatGPT never has the last word. There is opportunity for refining the prompts and extending the dialog, and the request to explain through “chain of thought reasoning” is informative. But there are takeaways from this modest example. The analysis starts a conversation with colleagues around these data, what they might mean, and potential actions we might take to respond. The exercise is also a learning experience around AI tools and their increasing sophistication to analyze text, calculate numbers and visualize the results in powerfully convincing ways.

Posted in surveys | Tagged ChatGPT, generative ai | Leave a comment

Quick Poll Leads to Rapid Improvement

Posted on February 24, 2025 by Nancy Turner

The University’s office of Institutional Research and Assessment launched a new assessment tool last fall, and the libraries was one of the first departments to try it out. It’s called a Quick Poll, and we used it to gather data on the use of study rooms at Charles.

These polls are short 1-2 question surveys that appear when students, faculty or staff log onto their Temple portal. The visibility and ease of completion mean that participation rates are very high. Our poll was fielded between November 8 and November 22 2025, and in that brief period, we received 4954 responses! As those responses roll in, they are available immediately to the research team (myself, Caitlin Shanley, Jackie Sipes, Steven Bell, Justin Hill and Evan Weinstein).

The expectation for quick polls is that we take quick action too – our team has already put together some next steps to improve the student experience with study room

Our poll was just three questions. We first asked whether or not students use the study rooms at Charles.

Of the 4954 students responding to the poll, about 2/3 do use the study rooms at Charles. That’s good news.

Do you use the study rooms?	Count	%
Yes	3246	65.52%
No	1708	34.48%
Grand Total	4954	100.00%

For those that do not use the study rooms, we asked why not. This was a fixed response question, and provides some insights towards improvements we might make.

Why not?	Count	%
Do not use the library	393	28.46%
Other study spaces are sufficient for my needs	374	27.08%
Did not know about them	217	15.71%
Other	199	14.41%
The reservation system is too confusing	198	14.34%
Grand Total	1381	100.00%

For instance, over 15% of the students responding say they don’t use the rooms because they don’t know about them. There’s a messaging opportunity. About 14% don’t use the study rooms because the system is too confusing. Here’s an oppotunity for Jackie Sipes, our User Experience Librarian to collaborate with Justin Hill in Access Services to streamline the process – to the extent possible in Springshare’s reservation software system.

Our third question had students tell us how they used the rooms. This information aligns with what we observe. About 43% use the rooms as part of a group, and a large portion also use the spaces for quiet study.

Activity in Study Room (Up to 3)	Count	%
Studying alone in a quiet space	1745	27.65%
Working on class assignments with others	1496	23.70%
Working independently alongside friends (co-working)	1211	19.19%
Using special features in the rooms (for example, whiteboard, large monitor, electrical outlet)	794	12.58%
Participating in online classes or meetings (for example, via Zoom)	644	10.20%
Meeting with a student organization	207	3.28%
Eating	191	3.03%
Other	23	0.36%
	6311	100.00%

As noted, the poll has provided us with solid numbers to guide us in tweaking our reservation system. In addition to communication and clarifying the reservation process, we will tighten up the reservation system by reminding students of upcoming room reservations and use QR codes in the individual study rooms to make reservations easy.

Concrete changes that should maximize our space utilization and enhance the student experience at the same time.

Posted in library spaces, service assessment, statistics, surveys | Tagged library space, student experience, study rooms, surveys | Leave a comment