Learning about Our Users: The Website Use Survey

Posted on December 20, 2017 by Nancy Turner

Cynthia Schwarz, project manager for the Libraries’ Web Environment Redesign Project, contributed this post. She reports on the results and next steps of the team’s user survey conducted last month, providing us with a healthy return of 460 user responses.

What did you set out to learn with this survey?

The Website Redesign and Blacklight Project Team wanted to gather feedback early on in the project about what is important to our patrons with regards to the website. The online survey contained these questions:

  • When you last visited the library website, what did you do while you were there?
  • What library resources or collections are valuable to you?
  • Please rate in order of importance: (Library Search, Borrowing from another library, Library Events, Individual and group study space, Printing and Computing, Library Hours, My Library Account, Information, Blogs, News and Social Media, Contacting a Librarian. )
  • What other tasks do you visit the library website to complete?

Can you tell us about your survey results? 

Library search is clearly of primary importance, as indicated with an average ranking of 4.18 on a scale of 1-5.  In the “heat map” below, green indicates high importance to users, while red is lesser importance (Blogs, news, social media averaged 2.10).

The results demonstrate what library staff already suspected, that patrons primarily come to the website to access to books, articles, journals and databases. These are all discovered primarily through the library search interface on the website’s homepage. The results also demonstrate that the computers, printers and study spaces are used and of high value to respondents. While these services are not directly related to the website, the information may useful as we plan the physical space in the new library building. Additionally, we’ll want to make sure that information about our computing, printing and study spaces is highlighted on the new website.

Were there any challenges in terms of interpreting the results of the survey?

Well, it seemed clear that all respondents didn’t read the questions carefully, since many cite printing and computing as very important – although they don’t do that through the website. Since most of our surveys were launched through a “pop-up” on library computers, the majority of our responses (over 55%) were undergraduates. And we don’t know if this response is representative, whether undergraduates are also the primary users of our website. The results show that faculty and graduate students use the website to locate articles first, then books. Undergraduates say that books, research, databases and homework/study are most important.

What Do You Plan To Do Next? 

This survey is one of several methods we’ll use to understand how our website is used – and what resources and services are most important to our community. We may use this information to highlight events and collections in a different way. We may offer different navigation paths to different user types. But as importantly,  it assures us that our work in developing a robust discovery tool is what will be most valued by our users.

Thanks, Cynthia. Great project. Thanks for sharing it with us. 

 

 

 

 

 

 

Posted in surveys, usability | Tagged , , | Leave a comment

Notes from the Field: Collections and Collective Action

Posted on December 6, 2017 by Nancy Turner

Last week I participated in a symposium on the changing mission of academic libraries. While not directly related to assessment or strategic planning, those practices are implicit as we explored together “the mission of academic and research libraries in the 21st century information environment.”

The event was sponsored by the New York chapter of the Association of College and Research Libraries (ACRL). The organizers suggested that the library’s “traditional” mission – stewardship and guidance in the exploration of recorded knowledge and experience valuable for higher learning, may be due for some adaptation as research, teaching, publishing and mass communication are changing.

So how are we changing? A theme throughout the day was collaboration and collective action. Libraries are rethinking the traditional model of collection development, where the research library needs to own everything, maybe in duplicate. Eloquently speaking to this shift was David Magier,  Princeton’s Associate University Librarian of Collection Development. Magier’s work includes the coordination of (yes!) 55 subject specialists, all engaged with collection development (and supported by a healthy collections budget).  In his talk on “Collecting, Collaborating, Facilitating: New Dynamics in the Role of Content in the Research Library’s Evolving Mission” Magier described changes from his perspective (with a nod to Lorcan Dempsey’s notion of a facilitated collection). 

For example, Princeton is collaborating with Columbia and the New York Public Library to share storage space. They started with a “condominium” model – where libraries engage common processing but collections are maintained separately. This model led to lots of duplication. The libraries are re-conceptualizing the approach to create a more “collective” collection. In the emerging universe of shared print archiving, libraries commit to retention of a title allowing other libraries to let it go. It’s not easy. The model involves MOU (Memorandum of Understanding) about retention, as well as borrowing agreements between the institutions. But,  for the end user, the access to content is greatly expanded.

For those users, it does not matter how the content gets into their hands. Our job is to reduce the “friction” patrons experience in getting to the resource they discover. This friction can be more or less.  To click on a title in the library catalog and be able to read its entirety,  that’s no friction. To access distantly held materials by acquiring a travel grant to visiting an archives  – that’s a lot of friction. Magier thinks this balance can be expressed as a mathematical metaphor, with content being all the resources, and friction the pain in accessing that content:

Content ÷ Friction = Happy Spot

How does this relate to assessment? Or strategic planning? In order to do any of this well, the library needs to consider its core communities and their needs; both in the short and long term. The majority of attendees at this symposium were not from Princetons, but rather city branch libraries or community colleges. Their  patrons and the collection support they require are quite different. Their collections strategy look different as well. 

Figuring out the best balance of friction and access is also an assessment issue. What are the expectations of our users? What is the user experience we need to provide to them? How can we make the access to content as seamless as possible? 

And to understand the possibilities and costs, we need to collect and analyze the data. At what point should we be subscribing to a journal rather than using interlibrary loan? At what point are we better off buying the e-book rather than paying the usage fee? How few simultaneous users licenses can we get away with to best serve our patrons without frustrating them? 

These questions require collaboration within our library organization (collections, technology, access, subject specialists, administration) as well as without (our user community and partnering institutions).  It is exciting and inspiring to hear how other libraries are taking collective action.

Sunrise over Secaucus before a day of meetings

 

Posted in access, collections | Tagged , , | Leave a comment

The Pain of Numbers

Posted on November 16, 2017 by Nancy Turner

 

I have just completed the Libraries’ submission of annual statistics to Temple University’s Office of Institutional Research & Assessment. I’ve been doing this kind of work for years, and it still pains me. This in spite of tremendous support from colleagues.

Why is it challenging? Well, this year we were in the midst of migrating data from Millennium to Alma at the end of the fiscal year (June 30, 2017). Mark Darby had to get his statistics in quickly (volume and title count, number of e-books). Extracting collections expenditures data is not too difficult, but Temple’s multiple fiscal close dates means that we may not have final numbers until months after the actual end of the fiscal year. I count on the expertise of Christine Jones and Brian Schoolar for that.

Reference and instruction statistics are pulled from multiple sources: department annual reports, LibAnalytics, various Google sheets. From the outside it seems inefficient (and it is), but Special Collections Research Center, Research & Instruction, Blockson, Digital Scholarship Center, Ginsburg, Public Programs – they all have somewhat different needs for documenting what they do. So we accommodate that and standardize the numbers as best we can.

I use Google Analytics for statistics related to use of the the library’s web site and digital collections. This is relatively straightforward, except that digital collections are spread out over several domains. These need to be aggregated.

Counts of the number of staff members seems a straightforward metric, but we need to decide on who should be counted as a “professional” and who not. Archivists, information technology staff, editors? The library has many, many roles that our outside traditional MLS-degreed librarians. And each library counts it differently. ARL (Association of Research Libraries) has us count professionals. ACRL (Association of College and Research Libraries) has us separate professionals into two groups: librarians and non-librarians. The definition of a librarian reads, Staff members doing work that requires professional education (the master’s degree or its equivalent) in the theoretical and scientific aspects of librarianship. That’s kind of vague!

Circulation of physical materials should be straightforward, but ARL is only interested in books and media – no reserves, no computers. Our coding systems don’t always make it easy for us to pull those kinds of materials out of the reports. Study keys – those numbers are  important  to us locally, but not to ARL or ACRL.

The electronic resources statistics. This is a huge effort, mainly on the part of Darina Skuba and Karen Kohn – who go to each vendor site to pull statistics on e-book usage, article downloads and database searches. Then we combine two sets of files (The Counter standard is based on calendar year but we report for the fiscal year), to determine a total number.

Finally, I rely on Royce Sargent (HSL)  and Carla Wale (Law), to provide me with the same sets of numbers for Health Sciences and Law.

Why do we do all of this? Several reasons: ARL and ACRL provide useful metrics for benchmarking our activity and institutional support compared to our peers. The numbers can provide important information about trends in academic libraries – what’s going up (e-resource usage, collections expenditures) what’s going down (circulation of physical materials, reference). Finally, the University relies on these numbers as they contribute to rankings, ultimately attracting the best faculty and students (and alumni support) to Temple. So I endure the pain and promise myself it will get easier next year. 

 

 

Posted in statistics | Tagged , | Leave a comment

Assessment Community of Practice Focus: Faculty Survey on Undergraduate Instruction

Posted on October 25, 2017 by Nancy Turner

Last week’s Assessment Community of Practice meeting focused on findings from the Ithaka S+R faculty survey, particularly those related to Undergraduate Instruction. Twelve librarians participated and we were lucky to be joined research project team members  (Rebecca Lloyd, Annie Johnson, Fred Rowland and Kristina De Voe). The full summary report is linked here:  Ithaka Survey Undergraduate Instruction_Final

Finding: A high percentage of faculty (72%) show video and their classroom and almost half (47%) assign the creation audiovisual or digital media projects.

Multimedia is used a good deal in classrooms, and increasingly, student assignments include a multimedia component. This finding prompts some further questions:

  • How do instructors find out about media offerings at the Libraries? Are they included in regular notifications of new library materials? 
  • Is it easy for instructors to embed library-licensed video or other media into course management systems? As easy as incorporating YouTube?
  • Do we, as librarians, need to be more savvy about the kinds of functionality available for use of multimedia in the classroom? Copyright restrictions? Technical barriers? 

Students are also required to create  “multimedia” projects as part of their course work.

  • What do these “audiovisual or digital media projects” look like? Are they Powerpoint presentations or podcasts?
  • Does the library provide sufficient support (technology-equipped spaces and services) for students working on these projects?
  • Do we have the information we need to make appropriate referrals to other services on campus that do provide support, and how might we partner with them?
  • What kinds of services and support will there be in the new library building for students as they develop digital technology projects? 

Finding: Materials that are openly available on the web are of interest for course content. 79% of faculty often or occasionally “give preference to assigning course text or materials that are freely available”.

This is a finding of real interest to the Libraries as we promote Open Educational Resources.

  • How do faculty understand “freely available”? Might this mean YouTube videos, or open access scholarly journals?
  • Are faculty aware of the Library’s Textbook Affordability Project?
  • How should we encourage faculty to use librarians help them locate quality resources for use in their courses?

Finding: A good percentage (50%) of faculty are strongly positive about the adoption of “new pedagogies or instructional approaches that take advantage of opportunities offered by digital technology.” For instructional support when introducing these new pedagogies, they rely most on their own ideas 72%, then on scholars in their personal network (52%), and then on the library (40%).

Faculty  tend to rely on their own devices when learning about new technologies to incorporate into the classroom and in spite of technology challenges in the physical classrooms,  many are interested adopting new approaches.

  • What support is currently available to faculty interested in learning about digital technologies for use in the classroom?
  • Are there technologies available to faculty through their schools that we should be aware of?
  • How might the Libraries provide more support for faculty in need of instructional support in these areas?

These Assessment Community of Practice meetings generate many good questions, and allow us to consider how research into practice can be challenging.  Surveys, no matter how well designed, do not always lead to clear answers or implications for library services. Did the respondent understand the question and the way that we, as survey designers, meant it?   How do we best write up our findings in a compelling, yet neutral and balanced way? As the research team can attest to, turning raw numbers into a meaningful and compelling story took hard work and a willingness to creatively translate findings into  action steps.

 

 

Posted in instruction and student learning, surveys | Tagged , , , | Leave a comment

Communities of Practice: People, Spaces, Services

Posted on September 26, 2017 by Nancy Turner

Last Wednesday 20+ library staff members came together to talk and learn about usability testing conducted here at Temple University Libraries. This discussion, part of our Assessment Community of Practice,  covered a range of topics, from how usability testing is defined,  how it is being used, what kinds of data we collect from this method,  particular challenges, and changes made based on the assessment.  The Community of Practice conversations facilitate us learning from one another, and also model the process of research leading to real changes in how we do library work. Assessment isn’t just evaluative – it leads to action, and new questions.

For me, the meeting fell on the heels the Designing Libraries conference held at the Hunt Library on the campus of North Carolina State University. The building is designed by Snohetta, and it was thrilling to see how the space and services come together to inspire and facilitate all kinds of work – I watched a student in the Hill Library makerspace designing and building an engine with a 3-D printer, and other students deep into reading while ensconced in some very colorful furniture.

Community of practice, to me, connotes mutual learning, shared values, and the sense of engagement and participation between members that is supportive and non-hierarchical. So can spaces foster a community of practice?  There were many examples at NC State — makerspaces, virtual reality labs, digital scholarship centers and music rooms, all serve  to bring people together for scholarly work that is creative, innovative, and often interdisciplinary.

Lauren Di Monte (Data and Research Impact Librarian, University of Rochester) spoke of using programming to discover user needs, and using programming to drive the development of spaces. Versatile spaces allow for multiple uses, from study to exhibition space to presentation space. She suggests that people are drawn to spaces where something is going on, even when they just want to study.

Jane Alexander (Chief Information/Digital Officer at the Cleveland Museum of Art) provided vivid examples of how digital and physical objects create engagement and community as users are encouraged to experience art in new ways. An example is Gallery One , a 40-foot interactive MicroTile wall that displays over 4000 images of the collections, and users can organize these in multiple ways. (I’m trying to think how we could use this technology for browsing books!)

The conference generated lots of talk of intersections and boundaries, and how we mix pedagogy with fun, how we engage users with the physical through digital, interactive experiences. Spaces, technologies, and services can foster this community.  But ultimately, it is the people who share a sense of purpose that move these innovative programs forward.

Hunt Library, NC State University

 

 

 

 

 

 

Posted in conference reports, organization culture and assessment, usability | Tagged , , | Leave a comment

It’s All About Continuous Improvement

Posted on September 14, 2017 by Nancy Turner

Developing an organization-wide culture of assessment is a slow, incremental process. I  get frustrated by this fact at times, particularly when I imagine a perfect world of systematic, routine plans put into place for assessing and demonstrating the value of all we do here in the library and press – supporting faculty, students, and community. But there is more going on at Temple than first meets the eye; examples of assessment and program evaluation that taken together,  are pushing us towards a culture of continuous improvement.

For example, we’ve been using a paper form for collecting feedback on our public programs for years. This week I met with Kaitlyn Semborski, Sara Wilson and Steven Bell to discuss other ways of gathering feedback – creating an online form and tweaking our questions to get a better sense of how well the programs meet the needs of our many patron types – from students to friends in the community.  We talked of “tracking back” via email with instructors who send their students to these programs to get feedback on the effectiveness of the programs for their curricular needs.  This will be essential information as we shape the programs going forward.

I met with Ginsburg Instruction Librarian Natalie Tagge last week to brainstorm the assessment of instruction provided to first year medical students. The information literacy component utilizes flipped classroom, video, and a pre-test, and the Dean was enthusiastic about last year’s efforts. This year Natalie intends to build on that success by using a rubric to evaluate student presentations as students (we hope) demonstrate their grasp of the material. It’s a good method because it allows us to assess learning outcomes in an unobtrusive way, placing no undue burdens on the faculty or students.

RIS librarians are also having conversations about more systematically documenting the work they do with instruction and research consultations. Tracking back to faculty, recording repeat customers, insuring that we have more detailed information about research consultations. We know that numbers do not tell the whole story. Anne Harlow points out that real “effectiveness” is when students are able to locate quality information on their own, without intervention of library staff. This means that we may be supporting students even more, yet our transaction numbers may go down. But engaging in the conversation about meaningful metrics is a good start.

A final example. At Temple we collaborate across departments in many ways – planning the orientation of student workers for public services desks is no exception. Staff from Access & Media Services, Library Administration, Reference & Instruction, Library Technology Services, coordinated by Katerina Montaniel, worked hard to organize this year’s orientation. The effort was strengthened even more by the inclusion of 3 students in the planning. As it turned out, the student-produced video providing “tips” for working a desk was a highlight – it was informative, clever and funny. Another change to orientation this year was a simple assessment, an evaluation form entitled: “Please Help Us Continue to Improve”. On a scale of 1-5 (5 is the best), 30 students weighed in – with an average score of 4.48. Well done!

Including students in the planning of the orientation made a huge difference in developing a sound, but also fun, training session for our student workers. What else did we learn? Many students didn’t realize they have ready access to a panic button when they are working a public desk. Important information. And going forward, staff will know even more about what topics to emphasize to engage student workers in the essential aspects of work on the public service desks.

So although much of the work of assessment goes under the radar, it’s actually happening in a big way. Even more importantly, a growing number of staff members, from throughout the organization, are thinking about assessment and engaging in the process. Thanks for letting me brag about it!

Posted in organization culture and assessment | Tagged , , | Leave a comment

Meet Mary Marissen: Swarthmore’s Emerging Library Assessment Professional

Posted on August 29, 2017 by Nancy Turner

The lovely McCabe Library at Swarthmore College

This summer, on a sweltering July morning, I had a Saxby’s coffee with Mary Marissen, librarian at Swarthmore College. She’s taking on some new roles in assessment at the library and we met to talk about emerging trends in this area. The “interview” below is based on our conversation, and pursuant email exchanges. It runs the gamut from overcoming resistance from faculty regarding “assessment”, to connecting initiatives to the strategic planning process.

NBT: How did you come to take on your new role of assessment?

I have been working in libraries as a paraprofessional for a number of years, first in the Boston area, and then at Swarthmore, starting in 2002. I eventually began working toward an LIS degree and graduated from the Drexel iSchool in 2012 while continuing to work in the technical services department in McCabe Library at Swarthmore.

In the summer of 2014, I was offered the opportunity to work on the first user experience research at Swarthmore for our online catalog, Tripod. I found myself really interested in and engaged with user research and I’ve become a bit of a proselytizer here ever since then. My work on the project helped launch me to a professional position and assessment was added to my new title and responsibilities.

NBT: Did you bring any special background, or education, to this – or is it an ongoing interest?

In a word, no. But responsibility for user research and assessment seem often to go together in professional responsibilities elsewhere and so it made sense when applying for the promotion to put them together. Swarthmore has always done assessment, but I began to take a more active role and have become the convener for our standing committee.

One of the many wonderful things about Swarthmore is the deep level of support for professional development. I’ve taken the six courses required for the certificate in UX from the Library Juice Academy and I’m currently enrolled in an excellent LITA course on Contextual Inquiry. I’ve received support to attend the Design 4 Digital conference for Library UX professionals and also the Library Assessment Conference in Washington last fall. I’m grateful for the time I’m provided for learning and thinking in addition to the expectations there are for planning and doing.

I think it’s fair to say that there is some faculty resistance to the word assessment here, so one of my interests is in how to begin to shift the perception from assessment being an imposed compliance responsibility to an appreciation for understanding and value — learning from meaningful assessments and making productive changes as a result.

NBT: How is the work of assessment organized at Swarthmore? Do you have a committee, or is it just you?

Swarthmore has had a standing Assessment Committee for a long while. When I began to take responsibility, I had the benefit of an excellent strategic plan already in place to provide a framework and orientation for what to do. Although that planning cycle is coming to a close, it has provided a great template for informing new work.

As someone who has taken on assessment or user experience studies as a new part of their job scope, how do you prioritize your work?

This has been one of my biggest challenges and steepest learning curve! I moved from very task oriented library work — highly detailed and interesting– but task oriented. I knew at the end of the day exactly what I had accomplished and what I should begin working on the next day. This is an entirely different world for me, albeit a more interesting one, in which there are big-idea projects I need to approach, involving much more thinking and strategic planning. I love that, but I sometimes struggle with setting priorities for myself. I’ve tried different organizational tools, Trello, a simple calendar, a bullet journal. None of these are magic. I’d love to hear how others answer this question!

NBT: What are the advantages of doing assessment at a small college?

I love working at a small college because I know everybody and can reach out to almost anyone for help or to talk through an idea. We’re a relatively flat organization and people are usually very willing to cross traditional departmental lines to work together on projects of interest. There is a lot of flexibility here as a result and it’s great.

NBT: Are there particular challenges?

I suppose the advantage of knowing everyone can also be disadvantageous. Many of us have been here a long while, and despite the flexibility, we may bump into some preconceived notions of what work ought to be done and by whom. But we’re lucky to be incredibly well resourced and supported. It’s difficult to think of disadvantages.

NBT: You’ve mentioned a couple of times how the strategic planning process connects up with assessment. Could you expand on that?

Swarthmore Library’s strategic plan was largely shepherded by a good strategic thinker at Swarthmore, Kate Carter, the Head of Digital Initiatives and Strategies. She pulled a team together with the Assessment Committee at the time. The plan was substantive and aspirational. I think the goals they identified are timeless in nature. In fact, we could probably just keep them all and substitute newer objectives and different approaches that continue to guide us and keep us current.

After looking back at the plan cycle coming to an end, we realized that there is a tension between the aspirational nature of it and finding ways to actually accomplish the things that are in it. We were getting bogged down keeping track of things and almost making things up, by simply doing some things and then going back to look for how they might fit the plan instead of the other way around, having the plan guide what we do. So we started to think of something shorter and more accomplish-able. We also consciously tied it to the priorities of Swarthmore College for the first time. This was made easier by the College also identifying three strategic areas for planning. But now that we’re finishing up work on the plan, I wonder if we are setting the bar high enough. I think so? But I’m not positive.

NBT: What are you most looking forward to in your new role?

I’m eager to launch our new strategic plan. The conversations we’ve had to identify priorities and articulate goals has been inspiring. I’ve also received approval to hire some student user experience workers beginning in the fall, and I’m really excited about training them, getting them to work and implementing changes as a result.

On that positive note – we’ll wrap up. Thanks very much for being so open with sharing your thoughts on assessment and planning. Good luck, and I’ll look forward to talking with you again next year.

 

 

Posted in organization culture and assessment | Tagged , , | Leave a comment

Assessment & Analytics Community of Practice “Kick Off” : A Recap

Posted on August 17, 2017 by Nancy Turner

Yesterday here at Temple University Libraries we kicked off a new approach to developing our culture of assessment, an Assessment & Analytics Community of Practice discussion to which all staff were invited.

The Kick Off  focused on two recent research projects conducted here: a qualitative interview project with religious studies faculty and the Ithaka S+R /Temple faculty survey. What I want to facilitate at these sessions is the opportunity for staff to think together about how we connect user research to strategic library practice. To that end, I asked the Ithaka Research Team members to consider these questions, provided here with some key points:

In terms of the work you do with faculty, what was the most significant finding?

  • Faculty (religious studies) are using social media to create informal communities to a greater extent than we thought. Many use Twitter to keep up with other researchers in their field.
  • Faculty are incredibly busy.
  • While faculty do want a wide audience for their scholarship, they perceive “open access” as less targeted than publication in more traditional vehicles.

In understanding faculty work practice, and how the library can support that, what were the particular benefits of the research method? What was problematic?

Interviews

  • Interviews provide robust data and good “story-telling” potential.
  • Processing that data take time; we were fortunate in having our audio interviews professionally transcribed.
  • Yet even with a small sample (12) we were able to discern themes.
  • In the end, a narrative actually was easier to write with the qualitative data compared to the quantitative.

Survey

  • Surveys are hard to design. While they provide a way of gathering from a broad group, the question format can be limiting, and we never know if respondents understand the question in the same way.
  • We worried about skewed results, i.e. respondents were mostly faculty who like and want to support the library.
  • Survey data takes time to analyze. We spent lots of time massaging the numbers, and it was not easy to turn those numbers into a good (and even-handed) narrative.
  • That said, the “hard numbers” may be more usable for “talking points” and advocating for the library’s value to the University, particularly as we compare our results with other research institutions.

In using the findings to consider next steps (actions, not additional research), provide an example. Connect something you learned to something you will do.

  • Services like document delivery to faculty offices are important as they save the time of scholars.
  • There is plenty of room for educational outreach. Areas like open access, open educational resources, institutional repositories  are not commonly understood. There is also room for educating faculty about how we can support their needs in scholarly communication, e.g. negotiating a license with a publisher
  • Providing liaison librarians with “talking points” related to survey findings would be helpful, particularly those that point to the librarians’ value towards improving student critical thinking and research skills.
  • There are implications here related to collection development as well, particularly as we see the extent to which faculty use media in their classrooms and assign media-related assignments to students.

Some thoughts in general:

No method provides ready “next steps” for action. And much of what we learned was not surprising, nor “ground-breaking”. Our faculty are, for the most part, like faculty at other research institutions. So it behooves us to pay attention to the literature – to other research. An example was the recent Ithaka S+R report on Rethinking Library Liaison Programs for the Humanities, and its insights into changing paths for library work with faculty.

Next steps:

We had a great turnout yesterday, with staff from Health Sciences, Research & Instruction, Outreach & Communications, Access & Media Services, Special Collections Research Center, Administration, Library Technology, Scholarly Communication, and Digital Library Initiatives.

We’re already planning the next few months’ session – there is lots to talk about and lots of interest in assessment activity – usability and undergraduate education are upcoming topics, as well as a working group for Alma Analytics skills-sharing.

Thanks especially to members of the Ithaka Research teams : Rebecca Lloyd, Justin Hill, Kristina DeVoe, Annie Johnson, and Lauri Fennell for sharing their thoughts to make this first program a success. (And Fred Rowland, present in spirit)

Posted in organization culture and assessment, qualitative research | Tagged , , , | Leave a comment

Improving our Understanding of Faculty Work: Two Research Projects

Posted on August 15, 2017 by Nancy Turner

Over the last couple of years, Temple librarians have engaged in two significant research projects with potential for informing our work with faculty. This week’s kickoff for the Libraries/Press Assessment Community of Practice provides an opportunity to discuss these projects with members of the research team, focusing on how assessment research can be turned into strategic “next steps”.

This post excerpts highlights from the summary research reports and I hope, provides a “teaser” for Wednesday’s session, when we’ll focus our attention on methods and implications for the research. That session is scheduled for Wednesday, August 16 at 3:00 in the Paley Lecture Hall. All are welcome to attend!

Religious Studies Scholarship at Temple
In October 2015 Temple University Libraries joined 40 other libraries to explore the research practices of faculty in religious studies. Coordinated by Ithaka S+R, the project sought to understand the resources and services these scholars need to be productive and successful in their research. Here at Temple, we conducted 12 in-depth interviews. Members of the team included Fred Rowland, Justin Hill and Rebecca Lloyd. This “ethnographic” method of listening to faculty, offered us new and different insights into the perceptions of our users. It deepened our understanding of the many challenges scholars face in conducting research and establishing a secure place in the academic world. Some takeaways:

• The Temple University Libraries and academic libraries in general are strong advocates and supporters of open access publishing. However, librarians need to be sensitive to the institutional pressures that faculty face in the areas of tenure, promotion, and merit-based pay increases. The current academic reward system prioritizes selecting a publisher based on prestige and frequently on the speed of publication. In contrast to the sciences and social sciences, concerns about metrics related to citation count or social media impact were less evident in faculty decision-making. Selecting the correct target audience is very important.

• One inference we draw from our interviews is that faculty perceive open access publications as offering no meaningfully defined audience. It is easy to imagine the open web as a vast ocean of undifferentiated content. Traditional publishing serves as a locus of attention, a node on a network, in which a definable audience can be imagined. This is an area where education may come into play. As our outreach efforts foster a greater awareness of the open access institutional structures that promote discoverability and coherent research communities these options may become more attractive.

Faculty Survey
In the Fall of 2016 we also conducted a localized version of the national Ithaka S+R Faculty Survey , deployed to 3,678 faculty members with a response rate of 15% (548 responses). In many ways our survey overlapped with the content areas of the interviews, with questions related to scholarly communication and publication practices and attitudes towards open access in addition to perceptions of student research skills and undergraduate instruction practice. Members of the analysis team included: Fred Rowland, Kristina DeVoe, Rebecca Lloyd, Annie Johnson and Lauri Fennell.

Key Survey Findings
• 80% of faculty strongly agree in the importance of the library as it helps undergraduates develop research, critical analysis and information literacy skills. This figure is 10% higher than other research institutions responding to the national survey. Over half of faculty strongly believe that undergraduate students have poor skills related to locating and evaluating scholarly information, although improving those research skills is very important to the majority (70%).

• The roles librarians play are also highly valued. Less than 12% of faculty believe that “with easy access to academic content online”, the role of librarians at Temple University is less important.

• 90% of faculty respondents strongly agree that it is important the library pays for materials they need for their research and teaching. And while the library’s collections are still of prime importance for faculty research and teaching (89%), 71% are supplementing this with freely available online resources.

• Over 80% prefer the print format of a scholarly monograph for reading in depth. A smaller majority (65%) prefer the digital format for searching. This preference of format is significantly different for journals. Most faculty (70%) are “completely comfortable” with the cancellation of a print journal if the electronic format is available.

• And of interest as we consider our own institutional repository, more Temple faculty say that in the past five years they have shared their research via a preprint (65%) than via a blog post or social media post (50%). In addition, a majority (72%) of faculty support mandatory publicly funded research be made freely available online.

• 46% of Temple faculty respondents have received or are receiving government or other external funding for their research. This is of interest for our library in planning research data management services, when federal funding typically requires a plan for making research data more openly accessible. 83% of faculty report storing their research data on their local computer.

Final Thoughts on Survey
The survey data provides us with a head start in understanding faculty needs, as well as potential “gaps” in their awareness of library services related to open access, data management, as well as our rich multimedia collections. But a survey like this does not provide us with specifics or clear cut future directions. For those insights and ideas, we will continue to rely on our close relationships with faculty and understanding of changing work practice. While the survey provides us with a broad view across all disciplines, the survey method must be mixed with a more in-depth, nuanced, qualitative approach to understanding the research and teaching practices of our users.

Posted in qualitative research, surveys | Tagged , , , | Leave a comment

Library Assessment Repository Project

Posted on July 26, 2017 by Nancy Turner

Over the last several years, the American Library Association’s LLAMA (Library Leadership & Management Association) Assessment Community of Practice has been exploring community interest in a repository for library assessment resources. The repository would provide open access to survey instruments, case studies, raw data – providing the community with a single space for locating and sharing materials.

This spring I conducted a survey on behalf of LLAMA to gauge interest –  379 responses.The majority, 86.8% (n=329) of respondents were from academic libraries, and 7.7% from public.

I spent much of my ALA conference in June reporting on the survey findings, but I think they are of general interest and worth relating here as well. Particularly because some of the issues and concerns raised might apply to establishing other types of open access repositories – whether institutional or disciplinary.

The survey was short – just 11 questions. This may be a reason we received lots of comments from librarians who wanted to tell us even more!

Question: Please indicate how likely you would be to use these types of resources if they were part of a freely available online Library Assessment Repository:

By a large margin, the types of resource most likely to be used are assessment instruments (surveys, questionnaires, rubrics). Least likely to be used are raw data.  Raw data was also least likely to be deposited.

Scale Green=High; Red =Low

Question: What factors would contribute to your willingness to use these resources?

Prospective users are looking for quality resources that are relevant to their immediate research need. Out of 272 free-text comments, themes included:  quality, relevance to the work, peer-review, ease of access and free availability, robust search functionality, metadata and overall organization of the resources.

Question: How likely would you be to deposit the following types of materials?

Survey responses indicate somewhat less likelihood of deposit of assessment resources than the likelihood of use.  264 respondents would be likely or most likely to deposit assessment instruments.  This is highest ranking type of material for deposit, with case studies also ranking high for potential deposit.

Scale Green=High; Red =Low

Question: What factors would contribute to your willingness to share your resources?

Prospective contributors to the repository are also looking for ease of use and accessibility. Other themes from the 244 free-text comments: concerns that one’s own work would not be “worthy” of sharing, lack of permission from library administration or IRB (Institutional Review Board), and privacy concerns. There is fear that submitting raw data would result in misunderstanding or misusing it. How would that data be “framed” or annotated so that its analysis made sense?  Another concern:  “Why would I submit my work to a repository when that doesn’t count towards my tenure/promotion process?”

Next Steps:

What started as a great idea has taken on some “legs”, but has also turned into a complex project with high expectations. Our assessment community members would like lots of rich, quality, peer-reviewed content that meets their particular research needs for their particular library type. The repository must be easy to access and deposit must also be easy, yet resources must be described thoroughly with rich metadata to insure discoverability.

The need seems greatest for small libraries who may not have many opportunities for professional development. Librarians who are seeking support and information as they take on new responsibilities in assessment expressed a particular need for the resources and networking opportunities made available through a repository.

LLAMA Assessment is now looking at how best to more forward, particularly as it seeks partners for building technical infrastructure and soliciting deposits.  The survey confirmed the value and need for a repository of library assessment resources, but is just an impetus. Now the real work begins.

Note: To read the report: AssessmentRespositoryNeeds_SummaryReport

Posted in surveys | Tagged , , | Leave a comment