2005

Using Usage Data

This had to be the single longest program offered at ALA this year, short of the all-day preconferences. FOUR hours! But for someone genuinely interested in the topic, such as myself, the program quite amazingly sustained interest throughout. But it would be absurd to even TRY to blog the program in any great detail. So I’ll just try to hit a few high points here and there. The program coordinator promised to post all of the presentation slides on the ALCTS website eventually, within a few weeks after the convention.

Use Measures for Electronic Resources: Theory and Practice
Monday, June 27, 2005 1:30 – 5:30 PM
Collection Management and Development Section, Association for Library Collections and Technical Services

Speakers (in the order they spoke):
Martha Kyrillidou, Director, ARL Statistics Program
Dr. Peter T. Shepherd, Project Director, COUNTER
Oliver Pesch, Chief Strategist for Electronic Resources, EBSCO Information Services
Daviess Menefee, Director of Public Relations, Elsevier
Todd Carpenter, Business Development Director, BioOne
Brinley Franklin, Director of Libraries, University of Connecticut
Joe Zucca, University of Pennsylvania Library

The program was organized into three large segments with 2 or 3 speakers representing each:

  1. Standards
  2. Vendors
  3. Universities

Standards

Martha Kyrillidou began by discussing what she described as a draft report, titled “Strategies for Benchmarking Usage of Electronic Resources across Publishers and Vendors.” A preprint of this white paper is available online. The paper describes the ARL E-Metrics project, describing the history of attempts to evaluate usage of networked electronic resources, then analyzes results of a surveys of ARL members compiled in 2000 and again in 2004.

Ultimately what you want, said Kyrillidou, is not to have to deal with each vendors statistical reporting system separately, trying to combine all those numbers. She suggested three possible approaches to a solution. The first would create a combined database for sharing data across institutions, kind of a new OCLC for statistical data. In the second proposed model, multiple databases would be used, but the databases would be capable of talking to each other. In the third approach, there would be no databases at all, but rather a standard, with everyone using the same XML DTD, or some equivalent type of technology.

Use is not everything. Focus on the user. DigiQUALâ„¢ is an attempt to focus on digital library service quality. This project is funded via the NSF and NSDL. Institutions can use DigiQUAL to create a user survey for evaluating their web sites.

Kyrillidou’s final slide showed a dress with the text “Does this make me look fat?” written across it. Everyone wants the statistics they collect to make them look good.

Peter Shepherd is the COUNTER (Counting Online Usage of Networked Electronic Resources) project director. He began by providing an update on current COUNTER activities and progress. Release 2 of the COUNTER Code of Practice for Journals and Databases was released in April, 2005.

Dr. Shepherd provided the following principles for usage statistics:

Usage statistics:

  • Should be practical
  • Should be reliable
  • Only tell part of the story
  • Should be used in context

How can usage statistics help us measure success?

Both libraries AND vendors need usages statistics.

COUNTER Release 2 includes specifications for consortia-level reports, although only 2 of the 5 reports must be available at the consortial level.

Dr. Shepherd put in a plug for COUNTER membership. Libraries can join for only $375/year, and consortial membership is $500.

Vendors

Oliver Pesch provided an overview of EBSCO’s statistical strategies, and their stats management interface. He made the important point that libraries need to isolate federated search sessions and searches (via IP address or by usergroup/profile) so that these are counted differently than normal searches. He illustrated how a single user search can create multiple searches across various vendor statistical reporting systems. NISO is developing a standard which will allow metasearch activity to identify itself as such to databases.

He also suggested that we take a look at ERUS as a stats consolidator. ILS vendors often provide options as well.

Daviess Menefree provided similar background information on Elsevier’s statistical reporting activities.

Todd Carpenter spoke on behalf of smaller publishers. Now that BioOne allows full-text crawling of its journals by the search engines, 96% of its traffic comes from Google.

Universities

Brinley Franklin presented a summary of three in-house university unit cost studies which analyzed all aspects of journal costs, and compared print with electronic. Typically the non-subscription costs: staffing, housing of print journals, etc. were substantially higher than the subscription costs.

In a Drexel University study from 2002, the cost per use for print journals was $17.50, while per use costs for e-journals was a mere $1.85. A similar study in Muenster, Germany the following year had much the same results: 16.68 Euros print per unit cost, and 3.47 Euros for e-journal per unit cost. Not to mention that in both studies the e-journal use was much higher than the print use.

A 2003 University of Virginia study calculated a cost per article downloaded of $1.64 and a per search cost of $0.52. A University of Connecticut study found per search costs of $1.15 and $1.20 in 2002 and 2003, respectively. In a CARL study, Alliance libraries realized a per search cost of $0.25.

Unit cost data can become a very powerful tool for management and collection development decisions. One conclusion that can be easily drawn from these studies is that universities should work cooperatively to substantially reduce bound journal collections. There is no reason for every institution to house and service the same enormous backfile print collections.

MINES (Measuring the Impact of Networked Electronic Services) provides a totally different approach to evaluating e-journal usage. MINES uses a web-based survey form and sampling plan to measure who is using which resources, from where, and for what. These brief (3 or 4 question) surveys pop up during the authentication process, and are answered by selected users before they gain access to the resource.

E-Use Measurement: A Detour around the Publishers

To say that Joe Zucca’s work is impressive is a major understatement. My reaction was “I want this guy doing MY stats!” Basically he and his people are bypassing vendor generated statistics entirely, and are generating incredibly granular statistics using web metrics. He showed us graphs and charts measuring usage of electronic resources by student housing locations: a “party” house vs. an academically oriented “house” or an average upper division house.

One interesting byproduct of his statistical studies was a very high degree of correlation between checkout of print items with login access to electronic resources over time. The total numbers for e-resource use were an order of magnitude larger than the print checkout numbers, but when one went up or down, so did the other, proportionally.

My personal conclusion: we (me, in my job as statewide database licensing project manager for the State of Washington) should be doing a lot more with usage statistics than we are.