The Institute of Museum and Library Services (IMLS) is a federal agency that fosters leadership, innovation, and a lifetime of learning through grants to museums and libraries.
If outcome-based and other formal program evaluation methods are new to your institution, many helpful publications are available to introduce them. IMLS offers this list as a resource; it is not intended to be limiting or exclusive, and it is not an endorsement of any particular publication or web site.
These resources draw most of their examples from educational and social service settings, but many are readily applicable to typical goals of library and museum programs. URLs change frequently and it is very difficult to keep pace. If you find a closed or incorrect link, try entering the full title of the publication or resource in a good search engine. This will often direct you to the new URL.
Manuals, Guides, and Other Resources
Many titles below are available at no cost online. While terminology differs from publication to publication, basic concepts are very similar. All are designed for use by organizations who want to know the results of their programs in terms of human benefits, whether those are called "impacts," "results," or "outcomes."
Administration on Children, Youth, and Families, Department of Health and Human Services (nd). The Program Manager’s Guide to Evaluation. Washington, DC:DHHS. This excellent introduction was developed for grantees of this program and provides very concrete, practical explanations. It is accompanied by additional guides for specific kinds of human services programs funded by the agency. Available as of 1/16/08.
Bond, Sally L., Boyd, Sally E., and Rapp, Kathleen A. (1997). Taking Stock: A Practical Guide to Evaluating your own Programs (PDF, 248 KB). Chapel Hill, N.C.: Horizon Research, Inc. This manual was developed for community-based science education initiatives through funding from the DeWitt Wallace-Readers Digest Fund. Participating advisors included the Association of Science-Technology Centers and the National Science Foundation. Available as of 1/16/08.
California State Library (nd). Outcome Measurement Information includes an excellent bibliography referencing a number of journal articles about outcomes-based evaluation in the library world (see Evaluation of the Use of Library Services and Technology Act Funding in California (PDF, 4.3 MB)). Available as of 1/16/08.
Diamond, Judy (1999). Practical Evaluation Guide: Tools for Museums and Other Informal Educational Settings. Walnut Creek, CA: Alta Mira Press, 192 pp. Available from American Association of State and Local History, 1717 Church Street, Nashville, TN 37203-2991. This guide provides a concrete guide to tools and approaches for understanding how well programs and exhibits communicate the intended messages to museum audiences. It includes samples of numerous strategies for collecting information on museum learning, and describes how to construct and use them.
Joan C. Durrance and Fisher, Karen E. with Marion Bouch Hinton (2005). How Libraries and Librarians Help: A Guide to Identifying User-Centered Outcomes. Chicago: American Library Association, 183 pp. Chapters include Introduction and Overview; Organizational Readiness Survey; Preparing for Outcomes; Logic Model; and Data Collection Plan. See the closely related Information Behavior in Everyday Contexts (IBEC), below.
Hernon, Peter and Robert E. Dugan (2002). Action Plan for Outcomes Assessment in Your Library.Chicago, IL: American Library Association, 192 pp. This recent title provides data collection tools to measure learning and research outcomes linked to user satisfaction. This practical, how-to manual, with detailed case studies from actual outcomes assessment programs, is intended to help academic librarians: plan an outcomes assessment strategy in line with accrediting guidelines; identify user needs, collect and analyze data; present findings; and measure the value of services, and identify areas for improvement. The Plan includes a chapter designed to help public libraries apply its concepts.
Informalscience.org, is a resource and online community for informal learning projects, research and evaluation funded by the National Science Foundation and hosted by UPCLOSE at the University of Pittsburgh.
Information Behavior in Everyday Contexts (IBEC), University of Washington Information School and University of Michigan School of Information joint venture (2004). A simple, flexible effective methodology for evaluating outcomes, targeting libraries and community-focused services. It includes worksheets and examples, and the method is in the process of being piloted by a group of public libraries from large to small for a variety of typical library programs. Available as of 1/16/08.
Innovation Network provides an online "workstation" approach to program development and evaluation planning that allows users to build a program plan, an evaluation plan, and a budget, essentially most of the pieces important to a competitive proposal or good program management. You must register (no charge) to access the tool. It fully integrates the program planning and evaluation planning pieces and provides tools (under resources) for developing data collection strategies and instruments. Available as of 1/16/08.
Korn, Randi, and Laurie Sowd, (1999). Visitor Surveys: A User’s Manual, Professional Practice Series Nichols, Susan K. (Compiler); Roxana Adams (Series Editor), Washington, DC: American Association of Museums, 164 pp. Available from AAM Technical Information Service, 1575 Eye Street, NW, Suite 100, Washington, DC 20005. This thorough introduction to visitor surveys in museums may also be useful in library and evaluation contexts; includes worksheets and samples.
Korn, Randi and Minda Borun (1999), Introduction to Museum Evaluation, American Association of Museums, Washington, DC.
Joseph R. Matthews (2004). Measuring for Results: The Dimensions of Public Library Effectiveness.Westport, CT: Libraries Unlimited, 240 pp.
National Science Foundation (2008), Framework for Evaluating Impacts of Informal Science Education Projects. Report from a National Science Foundation Workshop, A. Friedman (editor).
National Science Foundation (2002), The User-Friendly Handbook for Project Evaluation, prepared by Joy Frechtling.
New York State Library (2006). Outcome-Based Evaluation. Resources developed as part of a multi-year training plan to develop outcomes-oriented planning and measurement capacity for New York’s library personnel. Available as of 1/16/08
Rubin, Rhea (2005). Demonstrating Results: Using Outcome Measurement in Your Library. Chicago, IL: ALA, 176 pp. This latest addition to the PLA-sponsored Results Series uses familiar task breakdowns along with key terms in a step-by-step, service-oriented format so that readers can master the outcome measurement process. Applying these concepts in examples and in two running case studies, an Internet class for seniors, and a teen mother-tutoring program brings the model to life. The "Toolkit" includes tips on creating evaluations, coding data, and selecting a sample. Rubin suggests that by assessing the impact of services in users’ lives, public library directors and program managers demonstrate to funding bodies their accountability and the effectiveness of programs, thus positioning their libraries to secure maximum funding.
State Library and Archives of Florida (2004). Library Services and Technology Act Outcome-Based Training Toolkit (LSTA Toolkit). Includes an Outcomes Plan "wizard" and data report forms for Florida LSTA grant reports; point-of-need instruction in outcome-based evaluation, instruction on data collection, tools for data analysis, guidance in reporting project progress, and strategies for reporting project successes and results
Taylor-Powell, Ellen, Sara Steele, and Mohammed Douglah (1996), Planning a Program Evaluation. University of Wisconsin-Extension, Madison, WI. This is an excellent suite of text-based resources for program development and evaluation that allows individual "what I want to know, when I need to know" approaches. While examples come from the community service context, this is well constructed and well written, and provides a significant amount of detail, including report examples, sampling protocols, and other elements of a genuinely comprehensive all-text "toolkit." Available as of 1/16/08.
Texas State Library and Archives Commission (nd), Austin, TX, Outcomes. An excellent, wide-ranging resource developed primarily for local library users, this includes a bibliography, examples, and an online forum. Includes many resources also identified here. Available as of 1/16/08.
W.K. Kellogg Foundation Evaluation Handbook (January 1998). An excellent and tested guide for social service and educational programs, from a foundation that provides significant funding for not-for-profit organizations.
Sage Publications, Inc., 2455 Teller Road, Thousand Oaks, CA 91320, 805-499-0721 is a commercial publisher that specializes in publications on evaluation and related subjects. They offer many titles that cover aspects of evaluation in detail.
Finding and Making Good Use of a Consultant
Outcome Based Evaluation is still new to library and museum management, so as yet very few consultants combine knowledge of program evaluation, outcomes measurement, and the library or museum context. The best source of contacts is others in your field. If you cannot identify someone with all the experience you need, it is probably good strategy to hire a well-recommended generalist and to invest in orienting the consultant to your institution and its culture.
The Nonprofit Resource Center. Management Resources and Consultants for Nonprofits. This source is designed for administrators, board members and volunteers of nonprofits; provides a listing of individuals and firms that offer consulting services to 501(c)(3) organizations. Click on "Consultants" under the Support Organizations menu heading. Available as of 1/16/08.
The Foundation Center Learning Lab. Working with a Consultant or Technical Assistance Provider: A Resource List. Available as of 1/16/08.
TechSoup.org. Managing a Consultant. Short and pithy tips for non-profits, specific to technology consultants, but applicable to program evaluation and many other needs. Available as of 1/16/08.
Langford, Linda, and William DeJong. How to Select a Program Evaluator (2001). The Higher Education Center for Alcohol and Other Drug Prevention. Available as of 1/16/08.
Department of Health and Human Services. How Do You Hire and Manage an Outside Evaluator? Tips for finding and managing program consultants, Chapter 4 from the DHHS The Program Manager’s Guide to Evaluation. Available as of 1/16/08
Baselines, Outcome Indicators, Data Collection Tools, and Related Resources
A number of web sites offer additional guidance for outcome-oriented evaluation of community service and educational programs. You may find the following useful:
American Evaluation Society. Links, publications, and information about evaluation training. Links are provided to consultants offering evaluation services; none of these have been assessed by IMLS and no endorsements are implied. Available as of 1/16/08.
Arts Education Partnership, the President's Committee on the Arts and the Humanities and others have produced Champions of Change: The Impact of the Arts on Learning (2000), a publication edited by Edward B. Fiske that reports findings of a number of research projects designed to measure concrete impacts on educational achievement from sustained activities in the arts. It provides examples of significant indicators and a model of how evaluation (in this case systematic research) can demonstrate impact. Available as of 1/16/08.
Association of College and Research Libraries. Sources of Information on Performance and Outcome, assembled and annotated by the Assessment Standards and Accreditation Committee. Available as of 1/16/08.
The Finance Project is a non-profit organization formed in 1994 to "support decision making that produces and sustains good results by developing and disseminating information, knowledge, tools and technical assistance for improved policies, programs, and financing strategies." See in particular its publications, as an example Evaluation and Performance Measurement Capabilities for Internet Technologies in Human Services (Jim Frech, January 2003),Available as of 1/16/08.
Free Management Library provides extensive information about evaluation including materials developed by Carter McNamara for The Management Assistance Program for Nonprofits, plus links to manuals for evaluation and other resources Available as of 1/16/08.
The Harvard Family Research Project publishes a free newsletter for evaluation professionals (The Evaluation Exchange) and has a helpful Web site that includes full-text publications on evaluation. Available as of 1/16/08.
Independent Sector offers pointers to resources and guides. Available as of 1/16/08.
Library Research Service, Colorado Department of Education. This invaluable resource provides social and economic statistics, technology statistics, data about libraries and communities, information about research methods, and tools for statistical analysis. Designed primarily for library planning, but provides baselines against which outcomes can be measured for indicators appropriate to many library and museum educational programs. See especially Counting on Results, a report of IMLS-funded research to develop common library outcomes measurement tools. Available as of 1/16/08.
National Assessment of Educational Progress (NAEP), provides access to statistics applicable to measuring the educational impact of library and museum programs. It provides research data on educational indicators in the arts, science, math, and history among other topics, which suggest useful indicators and baselines against which program outcomes can be measured. Available as of 1/16/08.
National Center for Educational Statistics National Education Longitudinal Study of 1988 (NELS:88), reports findings of major longitudinal effort designed to provide trend data about critical transitions experienced by students as they leave elementary school and progress through high school and into post-secondary institutions or the work force. Data includes indicators of student learning, early and late predictors of dropping out, and school effects on students' access to programs and equal opportunity to learn. These can provide baselines applicable to library and museum educational and outreach programs. Available as of 1/16/08.
National Science Foundation Directorate for Education and Human Resources. The Online Evaluation Resource Library (OERL) includes plans, instruments, and reports that have been used to conduct evaluations of projects funded by the Directorate for Education and Human Resources of the National Science Foundation. OERL also contains glossaries of evaluation terminology, criteria for best practices, and scenarios illustrating how evaluation resources can be used or adapted. Examples and tools are in the context of science and related applications, but are transferable. Available as of 1/16/08.
The Staff Room for Ontario’s Teachers, Rubrics, provides a broad array of sample rubrics (tools for comparing progress towards common outcomes) with links to several no-cost online programs for creating your own rubrics. Available as of 1/16/08.
United Way Outcome Measurement Resource Network, provides a broad array of readings, resources, and methods applicable to evaluation. Available as of 1/16/08
University of Michigan Institute for Social Research Survey Research Center, Monitoring the Future, reports findings of an ongoing study of the behaviors, attitudes, and values of American secondary school students, college students, and young adults. It provides baseline data for numerous indicators of program impact on education, behavior, and attitudes for a large sample of these groups. Available as of 1/16/08.
Western Michigan University Evaluation Center. Principal activities are research, development, dissemination, service, instruction, and leadership for advancing program, personnel, and student/constituent evaluation, applied primarily to education and human services. This site includes a directory of evaluators that can be searched by country, region, state, organization, or area of specialty. Note that the listed professionals represent many approaches to evaluation, and do not all have expertise in outcomes measurement or in library or museum contexts. Available as of 1/16/08.
Automated Online Survey Tools
The number of commercial products and services for this purpose is increasing steadily, and those below are given as examples only, with no implied recommendation. Others can be found by searching for "Online Survey Software" using Internet search engines. All are designed to make the job of creating and using online surveys simple for users. They typically provide a variety of templates and standard surveys, as well as tips for building effective surveys. Information is automatically tabulated as it is entered by the responder, with the data automatically compiled by the product/service for the user to interpret. Costs and fee structure vary from product to product. A reminder–surveys are not the best tool for all outcomes.
WebSurveyor. The fee structure can be fee-per-survey or a multi-survey license arrangement.
SurveyMonkey. You can try this site at no charge for up to 100 responses to up to 10 questions. Easy to use. Full access is available by monthly or annual subscription.
Zoomerang. The fee structure is an annual subscription.
1/16/08
Links to Web sites outside the IMLS Web space are offered for your convenience. IMLS does not control the Web sites listed above and takes no responsibility for the views, content or accuracy of the information you may find there. Providing a link to this site does not in any way constitute an endorsement of the site or its content on the part of IMLS. Once you leave the IMLS Web site, the IMLS privacy policy no longer applies.