Assessment on the Ground

Notes from the Field (and Lake) : ALA 2017 Chicago

Posted on July 5, 2017 by Nancy Turner

Photo: D. Lacy

This June many of us headed off to Chicago for the annual American Library Association conference. The weather was perfect – warm temperatures with great views of the sky and lake, enjoyed off hours, of course. I asked for contributions to this post – for colleagues to share a program that was particularly useful or provocative. I’ve excerpted from those accounts, but please do feel free to contact folks individually with questions, follow-ups.

Brian Boling reports on the session, How Are Our Instructors Truly Using Media? A Multifaceted Approach to Developing Departmental Course Media Use Profiles by Scott Spicer from University of Minnesota. “Scott has developed a workflow for ingesting course syllabi into NVivo [a textual analysis tool] and coding the syllabi for mentions of required viewing and media production assignments. He shares his analysis with subject liaisons with an aim towards informing their outreach to faculty – particularly important as faculty may not be aware of the rich offerings at the library (both collections and services).”

Steven Bell also reported on different ways librarians can reach out to faculty. He describes an ARL-sponsored program on changing liaison roles, Talk So Faculty Will Listen; Listen So Faculty Will Talk. “It addressed the common frustration liaisons face when it seems that faculty ignore efforts to build bridges that could lead to enhanced use of library services and resources for students and faculty. The gist of the program, and this might not come as a surprise, is that it takes more than informational email blasts and participating in orientations (good but low impact activities) to build productive relationships with faculty.” Four strategies for talking with faculty, suggested by panelists:

Mistakes to avoid: 1) You do most of the talking 2)Focusing on “We have a resource to solve that…” 3) Ignoring emotions/feelings
If these mistakes sound like you…you need to talk less and listen more. Start by developing open ended questions to engage faculty in a conversation where they do most of the talking.
Four enabling questions were suggested: 1) What are your challenges as a researcher and teacher? 2)How is your field changing? 3)What are the emerging areas of emphasis for the department? 4)What are your hopes for library services?
Be concise in messages to faculty that are informational. Emphasize time-saving and productivity aspects of resources/services.

Adam Shambaugh also cites a program on emerging roles for librarians, ALCTS’ Creating the Future of Digital Scholarship Together. “Laurie Allen spoke about the digital initiative that began at the Penn Libraries to preserve federal climate data in the wake of the 2016 presidential election. This initiative—known as Data Refuge—would eventually include librarians, preservationists, and scholars from across the country. Laurie’s presentation, showcasing a variety of emerging roles for librarians and information professionals, was one of the most interesting and insightful talks I attended at ALA this year. “

Justin Hill attended a meeting of the SHARES group to discuss interlibrary loan practices and concerns. “Among the topics discussed was the investigation of the viability of allowing on-site access to visiting scholars from member institutions. This conversation is indicative of the types of things being explored by consortia to reinvent what’s possible within groups of like-minded institutions. Where activities within consortia has traditionally been limited to just supplying ILL to members, the discussion has now shifted to begin discussing other possibilities like on-site access, shared programming, consortial collection building activities, and group purchasing powers.”

David Lacy attended pre-conference focusing on Search Engine Optimization with the “ambitious” theme: “If someone is searching Google for something that our Library holds, it should be at the top of the list”. Dave says, “the session provided me with an abundance of ideas that we can apply to our new online environment.”

Joe Lucia attended a session presented by Chicago YWCA Chief Executive Officer Dorrie McWhorter, [formerly a CPA and management consultant for a Big Ten accounting firm. “Her theme was ‘the business of social impact’ and her focus was on how she’s worked over the past four years to build a workplace environment for the YWCA staff that is emotionally healthy, supportive & positive. Her intention in fostering that environment is to enable people to deal with the stresses and demands involved in bringing services to women and girls who are often in situations of critical need. Much of her presentation focused on interpersonal dynamics, relationship-building, and the core perception that people who feel valued, recognized, and respected are likely to carry those traits into the field in their daily work. She talked about a commitment to self-care for her staff and the benefit of regular (weekly) check-in meetings between managers and their team members that addressed current challenges, recent successes, continuing problems, and emotional well-being. Sounds touchy-feely, of course, but she reported a fundamental change in workplace culture in the direction of continuous connection and continuous professional growth that has allowed the organization to completely eliminate performance reviews.”

That would be nice.

I asked for some “fun facts” as well. Justin gets the prize for most weird hotel, rumored to be haunted by the ghost of Al Capone, and Dave’s meal sounds most interesting: at Maude’s Liquor Bar, he enjoyed “ liver pate, steak tartar, roast duck salad, and Sazarac.” Me, I had the mixed popcorn special at the airport – half caramel and half cheese – sounds gross but it was quite good!

Thanks to everyone for contributing.

Photo: N. Turner

Posted in conference reports | Tagged conferences | Leave a comment

Assessment for Planning: Special Collections Research Center Uses Data to Drive Location Decisions, and More

Posted on June 20, 2017 by Nancy Turner

Planning to move the Special Collections Research Center from its current space to the new library building takes careful planning, with different challenges than the general collections. For one, there are manuscripts AND books to be stored, with collections that vary in size from a few pages to many, many boxes. Not every item is described at the same level. Much is very rare, and much is fragile. And yet, through our robust discovery systems and extensive finding aids, an item could be requested at any time.

Fortunately, the Special Collections Research Center (SCRC) staff including Jessica Lydon (a member of the Libraries’ Assessment Group), and the rest of the “Mapping the Move” team, have already started to collect and analyze data that will inform the decision making as to what materials go where.

For instance, SCRC collections will be housed the new library in two separate compact shelving installations, open shelving in the reading room, and in a dedicated aisle within the Automated Storage Retrieval System (ASRS), as well as off-site at the library depository. Each of these locations has strengths and challenges in terms of proximity, accessibility, access, and time to retrieval.

Jessica and the team are using reports from AEON (the SCRC registration, circulation, and tracking system) and DB/TextWorks (the collection management database), as well as other reference and cataloging statistics to help map material to appropriate storage. A collection that is frequently used, but not too large, might go to onsite storage. A collection that is extensive and housed in many, many boxes, may go to the depository where contents can be searched by staff prior to delivery, or to the ASRS. Any material heading to the ASRS will need to be listed at the box level, with each box added as an item to the collection’s catalog record.

While data is useful, it also must be reviewed with a human eye to make meaningful decisions. For instance, the Albert Einstein Medical Center records received lots of use this last year, but that demand was influenced by their 150th anniversary and planning that went into an online exhibit and a history of the center. That collection may now be stored in a less accessible space. The Philadelphia Special Investigation Commission (MOVE) Records receive constant use from many types of patrons (students, community, media) so this will be housed onsite. The Science Fiction book collection, which is used less frequently and cataloged at the item level will go to the off-site vault.

Data also will inform decisions about public services. The move to the new library may necessitate closing the reading room for some periods. Data shows us that the type of patrons using special collections in the summer (visiting scholars) have a different use pattern than the heavy afternoon use when classes are in session (Temple undergraduates). This kind of information will be useful as well when the communication plan is put into place.

Staff in SCRC are asking questions of other services as well. In October 2016, they conducted an analysis of their reference transactions, applying a READ (Reference Effort Assessment Data) scale to understand the complexity of the reference work they were doing. This month they are doing that exercise again, in order to compare the two time frames, when different types of patrons use SCRC. While the READ scale was not difficult for staff to include in their record-keeping, they admitted that a generic tool like READ does not adequately reflect transactions in special collections — reference requiring a combination of expert knowledge of the collections as well as complex strategies for addressing questions, and specialized interview skills.

Using special collections data to guide decision-making is never a straight path, and seemingly straight-forward questions may require multiple data sources. “What is the time frame between the creation of a finding aid, or the cataloging of a rare book, to its subsequent use?” We have anecdotal evidence, but not concrete numbers. To get at this answer, SCRC staff members consulted reference statistics and in-house use documented in the AEON system, technical services processing data, and looked at Google Analytics for web use information. And there may be no discrete dates to compare: the processing of a collection can take months.

This kind of fuzziness in addressing questions would make most of us throw up our hands in frustration and despair! Fortunately, in SCRC, we have dedicated folks who are willing to handle the complexities as we consider the special nature of their collections move.

LACUNY 2017: What’s Next for the Academic Librarian?

Posted on May 25, 2017 by Nancy Turner

Last week I had the opportunity to attend the LACUNY 2017 Institute, a small conference sponsored by the Library Association of the City of New York at LaGuardia Community College. The theme was The Future of Librarianship: Exploring What’s Next for the Academic Librarian. I was drawn to the conference anticipating several lively panel discussions of interest to me: the application of organizational development towards more inclusive environments at work, egalitarian teams for grassroots organizing, and building the post-collection library (i.e. no physical books at all).

What a wonderful professional development opportunity for librarians.

In this post I want to share the themes from Barbara Rockenbach’s rich keynote address, Thinking about IMPACT on Research, Teaching and Learning. Rockenbach currently serves as Interim Associate Director for Collections and Services at Columbia; her talk drew upon broad experience, from computational methods applied to humanities scholarship to undergraduate research and instruction. I especially liked how she connected the themes of space, collections, partnership and context to both current research and real life examples from her home institution.

The library can be a space that puts the user front and center, preparing spaces that are flexible, that facilitate conversations, that are moving from passive to active space in support of student success. The space may be quiet, social or collaborative. For instance, Studio @ Butler is “co-owned and co-programmed by faculty” and provides open labs, instruction in tools like R and Python, and “talk shops” that incorporate both formal presentation and experiential workshop. A great example of David Lankes’ “library as conversation” with the community.

Rockenbach used Lorcan Dempsey ‘s ideas to talk of the future of library collections – valuing less what a library “owns” to the service a library facilitates. We will be thinking less about “our collections, our systems, our buildings” to considering the communities we serve. Thinking less about the life of the library towards thinking about the library in the life of the user.

The library is a partner in the work of scholars. Columbia participated in a collaborative Librarian Liaison Institute pilot with Cornell, ARL, and the University of Toronto. Its much discussed report (here at Temple too!) on the evolving role of liaison librarians had this as one conclusion: librarians need to focus away from the work of librarians to that of scholars, and to develop engagement strategies based on their needs and success factors.

The April 2017 US Library Survey suggests a disconnect between university administration and library leadership, in terms of how the library contributes to the teaching and research mission of the institution.

Library directors are pursuing strategic directions with a decreasing sense of support from their institutions. There is evidence across the survey that library directors feel increasingly less valued by, involved with, and aligned strategically with their supervisors and other senior academic leadership.

Pointing to Columbia’s efforts, Rockenbach says that libraries engaged with strategic planning efforts are less likely to suffer this fate. Libraries need to commit its values as well as its objectives to the larger institutional context. For instance, what does it mean to be a diverse and inclusive library?

The library provides space for learning, but also serves as an alternative to the formal classroom. The idea was captured eloquently by Ta-Nehisi Coates’ in his Between the World and Me :

I wanted to pursue things, to know things, but I could not match the means of knowing that came naturally to me with the expectations of professors. The pursuit of knowing was freedom to me, the right to declare your own curiosities and follow them through all manner of books. I was made for the library, not the classroom.

And, a big thank you to Barbara Rockenbach for allowing me to re-cycle her perspective on the future of academic libraries.

Posted in conference reports | Tagged conferences, organizational change, strategic planning | Leave a comment

Improving How We Support New Professionals: The Resident Librarian Program

Posted on April 24, 2017 by Nancy Turner

Temple’s Resident Librarian program is now in its third year, and we have started the recruiting process for our 2nd cohort of resident librarians. I sat down with the program administrators Richie Holland, Director of Administration, and Sandi Thompson, Head of the Ambler Branch library, to learn more about how they incorporate an assessment process to insure that this important initiative is successful.

NT: Tell me a little about the resident librarian program here at Temple. What was the impetus for initiating the program?

RH: Well, this has been on my back burner for a long time. The University of Delaware had a long standing program named in honor of civil rights leader and historian, Pauline A. Young. Here at Temple Libraries, our overall goal is to provide mentoring and support the diversification of librarians in the profession.

As the library’s Human Resources person, I’ve attended several meetings at ALA and I participate in the ACRL Residency Interest Group. I reached out to colleagues at other institutions with programs like this, and also talked with participants in resident programs. There is a bit in the formal literature, but most of my “research” was talking with others with experience.

ST: And then we also had the Temple/Drexel/Penn staff development day, with Jon Cawthorne. He’s also part of the ACRL Diversity Alliance.

NT: What does a successful program look like? What are you aiming to accomplish?

ST: We want to instill in our resident librarians confidence as they move into a career position. They will have experience in several areas of librarianship, and have solid accomplishments, or projects that they’ve completed. They know what it takes to be a professional. And of course we want them to thrive in their next professional job.

We had to think about how best to accomplish this. For instance, we deliberately organized the program to fit the interests and needs of the resident. They take responsibility for developing a program tailored to their interests and the experience they want. Then we provide them with lots of support – regular meetings with us, with their mentors, with site supervisors, and healthy travel support. We picked the brains of residents at other libraries on how to structure our program to make it really attractive to prospective applicants. One thing that came out loud and clear was to hire more two, providing the librarian with a ready cohort.

Once our resident librarians were in the program we set up different ways of “checking in” with them to insure all was going smoothly. For each rotation, the librarians wrote up their experiences, and we talked with site supervisors about what was working; the strengths and weaknesses for each experience.

NT: I know you have been carefully tracking on the program over the last two years. Tell me about some changes you are making this next cycle, based on that experience.

RH: Yes, we asked the current residents to provide us with feedback, as well as others involved with Temple’s program. We made a couple changes to the job description. For instance, candidates need to know they will be working with diverse management styles, and be amenable to that. Of course, that flexibility is a life-skill for all of us. Since we’re building a culture of assessment here at Temple, we added language about decision-making informed by data.

We want to recruit candidates who are a good fit for these positions, and having the right job description gives a more accurate sense of what the resident experience is like.

We’re also evaluating the timing of the different rotations – we did four, some libraries do as many as five. We want the librarians to have adequate time to complete a substantive project in their rotations. Then of course in their second year they stay in one department.

With each step of the process, from recruitment to the program structure itself, we’re continually tweaking, to make sure it works for everyone.

NT: We don’t always think of “assessment” taking place in the HR area, but you both have modeled a process that fits that definition – you identified a goal, or something you wanted to accomplish, you conducted research into other programs, you set up the program and sought feedback continuously and from multiple sources on how well the program was working and how it could be improved, and then finally, you are making adjustments to improve the program going forward.

So thanks for sharing the process.

Posted in organization culture and assessment | Leave a comment

ACRL Conference Points the Way: Reports from the Field

Posted on March 30, 2017 by Nancy Turner

Last week many of us headed down to Baltimore (along with 3400 other librarians) to attend the ACRL (Association of College and Research Libraries) biennial conference. By all reports, it was great learning experience, plus it’s always fun to catch up with colleagues from here and other institutions. Since there was so much going on, I asked my Temple colleagues to share their highlights.

We were all inspired by the keynotes – an exceptionally diverse and stimulating set of speakers included data visualization “rock star” David McCandless (Information is Beautiful), writer and feminist Roxane Gay on the meaning of diversity, and our amazing Librarian of Congress, Carla Hayden.

Steven Bell reports that he was asked about Temple’s new library about 100 times. He attributes this to his participation in the panel discussion Ready or Not? Pressing Trends, Challenges, and Tech for Libraries – about the NMC Horizon Report 2017 Library Edition.

Fred Rowland found Andrea Brooks’ Shifting the Discourse: Information Literacy as an Opportunity to Address Intellectual Virtues of interest, the concept that intellectual virtues like open-mindedness, self-awareness, respectfulness, curiosity, and flexibility are embedded in the ACRL Framework. Citing Jason Baehr, Brooks argued that instruction librarians should invest more energy in instilling these intellectual virtues in the students they are addressing. Also provocative for Fred was Addicted to the Brand?: Brand Loyalty Theory as a Means of Understanding Academics’ Scholarly Communication Practices Cara Bradley applies the perspective of branding theory from marketing to understand the role “brand” plays when scholars and scientists choose to submit their work. Open access journals might compete for effectively if attention was paid to this kind of branding.

The panel Rebecca Lloyd found particularly interesting was Reference: The New Dirty Word. She reports, “The topics addressed were very similar to the discussions we’ve had here about shifting librarian roles, student worker training, librarian visibility, etc. Even though it wasn’t new territory, it was reassuring to hear that other libraries are also struggling with these questions and finding that no matter which approach they choose, there are trade-offs and aspects of the service model that need further improvements. The main take-away was that there is no “right or wrong” on the future of the reference desk. It’s individual to each institution and there are variety of viable paths forward.”

A presentation that stood out for Natalie Tagge was Nicole Cooke’s invited talk, “How would you like to be remembered? Expanding your pedagogy and professional practice”. The presentation described Univ of Illinois’ School of Information Sciences classes dedicated to issues of diversity, social justice, and race, gender and sexually, attempting to infuse these concepts throughout the curriculum.

A useful session for Erin Finnerty was, ‘Everything you wanted to know about predatory publishing but were afraid to ask.’ providing a comprehensive snapshot of the current state of predatory publishing, as it relates to the role of librarians and researchers. Erin reports that “the Q&A session was particularly fruitful, and she gave useful recommendations for educating the different members of our user community – undergrads, pre-PhD students, faculty, etc.”

I asked my colleagues what they thought they might try here at home. Natalie says this: “I may consider experimenting with making course guides organized by research process instead of organized by type of source. I saw a poster (Pathfinder or Pedagogical: Transforming Course Guides for Student Success) about a research study indicating that students retained more from guides organized around the research process. I think this could actually work well for any instruction sessions focused on evidence based practice.”

Noting how many academic libraries are involving students for peer support, Steven would like to continue our own conversations about how work and train students in making such a program a success here.

Assessment continues to be of interest. Several of us attended Metrics Selection across the Research Life Cycle, featuring Chris Belter, the Bibliometrics Informationist at the National Institutes of Health Library. She pointed out (perhaps this is obvious) that metrics need to be question centered, and that having “lots and lots” of metrics, though each imperfect, allows us to triangulate to establish a better view of reality. Erin notes that NIH has an “incredibly informative LibGuide that outlines their major data aggregation and visualization tools, and provides article recommendations and examples of their work.“ Take a look at: http://nihlibrary.campusguides.com/bibliometrics

Annie Johnson’s take on the conference: “The best part of ACRL was meeting people from other libraries who are working on similar initiatives and projects.” Rebecca also enjoyed networking: ” I am still quite new to being a ‘history librarian,’ and it is very helpful to chat with librarians at other institutions who are dealing with similar responsibilities and challenges.”

Congratulations to Annie on her poster presentation, as well to Steven Bell (a panelist and a poster-presenter). And thanks to everyone who shared their ACRL stories.

Finally, the food highlights: Jack & Zach’s lunch counter veggie burger, falafel, beets and an ancient grains salad from Cava Mezze, broiled rockfish, and for me, a sausage, peppers & provolone sandwich to take out and eat on a park bench, once it hit the 70s.

Note: The ACRL conference proceedings are freely available online at: http://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2017/ACRL2017_A.pdf All 778 pages.

Posted in conference reports | Tagged ACRL, conferences | Leave a comment

Grounded or Toppling Over? The Three-Legged Stool of Assessment Culture within the Organization

Posted on March 1, 2017 by Nancy Turner

Today marks my third year at Temple University Libraries – a good time for reflection on how the organization has developed a culture of assessment. One thing I have learned. An assessment librarian does not a culture of assessment make. The practice of continuously asking questions of ourselves, our value to the institution, considering the metrics and research methods appropriate to those questions and values – it takes time to embed that practice into our work.

When I interviewed for this position, I began my presentation to the staff with this slide, composed of a word cloud based on definitions of the word culture:

I asked, “What would a culture of assessment look like?”, suggesting that it meant having a shared value system among staff about the benefits of assessment practice in how we do our work – that practice then becomes a pattern of behavior and is incorporated into the thinking about our work and improving our service to users. I think this concept still holds true.

Organizational Development

A culture of assessment must be more than the “doing” of assessment. It requires us to analyze, reflect on, and act upon our findings. For instance,

We must consider changes based on our focus group data, our surveys, our usability testing, our interviews with faculty.
We must then share our findings with staff and external audiences, and make explicit the changes we are making (or not) based on research findings.

This feedback loop is essential, and is continuous.

How do we empower our staff with the skills to do that research? It’s easy to gather data, but we need methods that insure the data and their interpretation are sound. This constitutes the organizational development aspect of building culture – in which assessment is not limited to a few but all staff are empowered with the skills to participate.

Strategic Planning

In 2014 here at Temple Libraries/Press we established an annual strategic action planning process that included an expectation for assessment with each objective. As new initiatives and ongoing ones are considered, we ask ourselves:

What does success look like? Ideally, that answer will be more than a count, but a measure of impact or growing reach into our communities.

Is our instruction program effective? How do we know that our students are incorporating critical thinking skills into their research and coursework?
Are we reaching new audiences? How are those communities engaged in our programs and how might they support the library in other ways?

These measures of impact (and return on investment) are much harder to determine. Gathering that evidence requires a commitment to planning and sustained support for a robust technical infrastructure for data collection, analysis and access.

Finally, a culture of assessment requires that these questions of effectiveness, self-examination and advocacy are systematically integrated into the fiber of the organization.

Are we reviewing our operations in light of our values for transparency, developing staff and collaborating across departments?
At budget time, do we review our spending and our work flow efficiencies as we justify new funding requests?
When a position goes vacant, do we reconsider that position within the organizational direction and new priorities?

In my three years I’ve learned (and as Lakos and Phipps discussed 13 years ago*) that organizational development, strategic planning and assessment serve together to balance that three-legged stool, and only with alignment in these areas will the stool stand firmly on the ground. Equally essential are the staff, the participants in this culture, that bring assessment alive as a true shared value for the library.

Three years in and we are definitely on the ground.

*Lakos and Phipps, Creating a Culture of Assessment: A Catalyst for Organizational Change (portal: Libraries and the Academic, Volume 4, Number 3).

Posted in organization culture and assessment | Tagged culture of assessment, organizational change | Leave a comment

From Assessment to Leading Change in Windy, Wet Atlanta

Posted on January 30, 2017 by Nancy Turner

Coke is king here in Atlanta, so I was surprised to see comedian Paula Poundstone with a Pepsi for refreshment during her hysterical stand-up routine in front of 500+ librarians at the American Library Association’s midwinter meeting. She made much fun of our love for meetings, meetings to discuss meetings, and meetings to discuss those discussions. I’m more fortunate than some, with only four official meetings to attend, from the ARL Survey Coordinators meeting to the editorial board meeting of ACRL Library Trends & Statistics, LLAMA Assessment Organizational Practice and ACRL’s Assessment Discussion Group. Thank goodness for the comedic relief of Poundstone!

But it was a productive conference, and I had the opportunity to learn quite a bit as well as meet up with former colleagues. That’s one of the highlights of conferences for me. We learn from these personal networks as well, of course. At the ACRL Assessment Discussion Group, one of our topics was what libraries are doing to support student retention. Anne Moore (former colleague at New Mexico State, now Dean of Libraries, UNC-Charlotte), updated us on what’s going on in her library.

At UNC Charlotte card readers are installed throughout the library – outside study and instruction rooms, at the circulation desk, outside the building, and at the Speaker Center. (This space is run by Communication Studies and is more than a presentation room. It is staffed with graduate students who can coach and critique practice presentations). Space user data is connected to not only Banner data (for demographics) but also responses to the NSSE (National Survey of Student Engagement). This is all pulled into a SQL database to which Tableau points, allowing UNC Charlotte to create compelling data visualizations that demonstrate relationships between use of the library spaces (accessible to all, 24 hours a day) and students’ grades.

An ongoing programmatic theme for ALA is the future of research libraries, changing roles of library staff and strategies for supporting that change.

One of the best was Joseph Zolner’s Managing Change & Fostering Innovation. Hofner is Senior Director, Harvard Institutes for Higher Education. He led the ACRL Leadership group through an activity where our table came up with an initiative (ours was to increase diversity in ACRL membership, and then identify various types of change resistance: from loss of control to fear of uncertainty, to recognition of real threats. To learn more, take a look at Rosabeth Kanter’s article, Managing the Human Side of Change in Management Review.

Kathleen deLong (University of Alberta Libraries) spoke as part of an ACRL program on Leadership for New Roles. Referencing the Transition Leadership Wheel, she spoke of the need for balancing opposing “forces.” Capitalizing Strengths but Going again the Grain; Instilling a Sense of Urgency yet Being Realistic and Patient. Key to getting the right balance is trust.

Good message, and one that resonated with me. Change is hard for us all, but trust and balance are key. And I suppose, being true to our core values. Like Paula and her Pepsi.

Posted in conference reports | Leave a comment

Assessment Reflections 2016

Posted on January 10, 2017 by Nancy Turner

January is a time of reflection – this post is just that, some ideas that sparked my interest last year, with hopes of delving into them more deeply in 2017.

Sunday morning’s radio listening was doubly-rewarding, as I heard two of my favorite woman in media: Krista Tippett, host of On Being: The Big Questions of Meaning, interviewed Maria Popova, the creator of Brain Pickings, an amazing weekly compilation of her reflections on vast and deep reading in a range of literature, from science and philosophy to poetry and children’s books. Popova covers deep territory in the interview, from the perpetual process of identifying self, to the balance of acquiring new information (easy) to thinking and knowing (hard). She has real skepticism about our pursuit of productivity, or the illusion of busyness as real productivity. But back to assessment.

Are We Using the Right Data?

Towards the end of the conversation Tippett asks Popova how she measures success; what external success might look like. My ears perked up. This is a question I am continually asking of my colleagues, as a way of considering appropriate measures for assessment. Popova describes how she used to pay attention, and “hang her sanity” on metrics such as Facebook likes and retweets. They are “so tempting and so easy because they’re concrete. They’re concrete substitutes for things that are inherently nebulous.“

But she says now, the “one thing that I’ve done for myself, which is probably the most sanity-inducing thing that I’ve done in the last few years, is to never look at statistics and such sort of externalities. But I do read all of the emails and letters — I also get letters from readers. And to me, that really is the metric of what we mean to one another and how we connect and that aspect of communion.”

Popova’s words eloquently express thoughts I’ve had this year related to data and meaningful metrics. We talk of data-driven decision making, but is the data we are using the right data? Can numbers alone measure success?

If we are going to make changes based on evidence, whether qualitative or quantitative data, we need to agree on those measures of success. Decision making for organizational change comes about through a collaborative negotiation of shared program goals and agreement as to how success will be evaluated.

Assessment and Organizational Structures

I’ve also been thinking a good deal about how assessment and organizational structure are connected. This year I participated in many teams, (Single Service Desk Design, Physical Collections, the Ithaka Religious Studies Faculty project, the Data Dashboard group). It makes sense, as much of our work in assessment necessitates a team organization.

I think these projects work well, are exciting and promote mutual learning – because of some factors:

there is a common goal – sometimes there is a formal charge, but not always.
they bring together interested staff members who bring expertise, but also a motivation and belief in the project at hand
they also allow for a less departmentalized, “silo’ed”, work towards innovation and problem solving. Teams work best if there is partnership and collegiality rather than hierarchy.
and ideally, teams engaged with research and assessment use their findings to promote organizational change

But that organizational change only comes about with agreed upon evidence for those changes.

So in my own role as assessment librarian, I battle with these two, almost contradictory things, all the time. How do we balance our value for data-driven decision making, while recognizing that these measures are imperfect in describing the complexity of real life and what is truly meaningful? Your thoughts, welcome!

Posted in organization culture and assessment | Tagged culture of assessment, data-driven decision-making, organizational change | Leave a comment

When Numbers Fail Us

Posted on November 28, 2016 by Nancy Turner

The recent election demonstrated in a powerful way the limits of data, in this case a multitude of polling numbers, towards understanding, or planning, for our future.

New York Times. http://www.nytimes.com/interactive/2016/upshot/presidential-polls-forecast.html

As an assessment librarian who counts on numbers to tell a story, I could not help but take this “failure” to heart. In our talk of data-driven decision making – what are we missing? Are we not asking the right questions? Or do our lenses (rose-colored glasses?) prevent us from seeing the whole picture?

I touched on this topic at the recent Charleston conference, where I participated in the panel Rolling the Dice and Playing with Numbers: Statistical Realities and Responses.

I discussed balancing the collection of standard library data elements over time, in order to discern trends, with the changing nature of metrics required to provide a meaningful reflection of the 21st century library’s activities and resources.

Last year, the Association of Research Libraries (ARL) and the Association of College & Research Libraries (ACRL, ALA) formed a joint advisory task force to suggest changes to the current definitions and instructions accompanying the Integrated Postsecondary Education Data System (IPEDS) Academic Libraries (AL) Component.

For example, the IPEDS instructions for counting e-books originally said to “Count e-books in terms of the number of simultaneous users” – a problem if we have a license with no access restrictions. Another example is IPEDS’ request that libraries NOT include open access resources, including those available through the library’s discovery system. Not only can this be a difficult number to collect, but counting only the resources “we pay for” goes against the library’s value of making available quality, open access resources to its community.

A continuing discussion on library liservs is related to whether our traditional metrics were “meaningful”. The question was prompted by the publisher of Peterson’s College Guides requesting we report a count of a library’s microforms. We must ask ourselves, “What sort of high school student selects a school based on the library’s collection of microfiche?”

Increasingly, I am frustrated by the “thin-ness” of our metrics, the data that we use to measure ourselves and our success. Not just that it doesn’t tell a robust story. But it also seems to pigeon-hole us with an out-dated notion of what the library does and the service it provides.

Usage metrics are proxies, but are not measures of success. We need to dig deeper. Our instruction statistics demonstrate growth in sessions and students served. Yes, we reach out to faculty and we may be asked back into the classroom. But are we able to demonstrate real learning? How are we demonstrating effectiveness? For instance, we might be more deliberate and systematic in collecting data related to our partnerships with teaching faculty – developing better course assignments; end of year feedback loops on student learning. These are harder, a little fuzzier, but arguably more important, measures of our library work.

Posted in conference reports, instruction and student learning | Tagged conferences, instruction, metrics | 1 Comment

Lighting the Path with Assessment

Posted on November 10, 2016 by Nancy Turner

Every two years librarians engaged with assessment gather together to share stories, methods, and research findings. We inspire one another as we work toward creating a culture of assessment at our institutions. This year 600+ of us met in Crystal City (between Arlington and Pentagon City) at the Library Assessment Conference sponsored by the University of Washington and the Association of Research Libraries. Conference organizers invited keynote speakers with the expectation of providing us with provocative food for thought — these two did not disappoint.

Lisa Hinchliffe (University of Illinois at Urbana-Champaign) spoke of Sensemaking and Decision-Making.

Hinchliffe noted that higher education is experiencing increasing competition and financial pressures. This environment requires libraries to “pivot” – to re-consider what is central to our work and what we can leave behind.

What is the new work we can do?
Are we prepared to step up fast enough so that our funders can see our value fast enough?

The value the library creates is not just economic, although shared services and collections DO create economic value, as important are the values of equality-building (i.e. inclusion, equity and social justice).

And assessment can serve as a map, or compass, towards the future – a kind of strategic guide. While an assessment program allows us to see different directions that are possible, it can not tell us which path to choose. The path must be selected based on how best we align our resources to our goals. How best to demonstrate, with evidence, our outcomes and value. Yes.

The next day we heard from Brian Nosek, University of Virginia, on Promoting an Open Research Culture. Nosek also directs the Center for Open Science (COS).

Through several participatory activities, Brian demonstrated that we can not help but experience the world through our own mind. Once we see a picture, it can be hard to see it another way. We all looked at the Horse and Frog Illusion, and while half the room saw a horse, others saw a frog. Try it out here.

In the research world, this idea relates to open access to data. Crowdsourcing the analysis of data makes for a more accurate and neutral picture of reality. Silberzahn and Uhlmann reported on an experiment with 29 teams of researchers, all answering the same research question with the same data set.

They found that the overall group consensus was “much more tentative than would be expected from a single-team analysis.” Crowdsourcing research, or bringing together many teams of researchers can “balance discussions, validate scientific findings and better inform policymakers.” (See the article in Nature)

Nosek went on to describe the Open Science Framework as an infrastructure for creating more open workflows that increase process transparency, accountability, reproducibility, collaboration, inclusivity and innovation. Exciting and important work.

So how does this apply to assessment? If nothing else, perhaps it will make us more humble as we talk about decision-making with data. We need to recognize that the data can tell many stories, and if we are to be honest and diligent in our work, we need to be open to the many ways in which those data can be interpreted and used.