Evaluation of Research, Technology, and Development Programs
AEA 2007: Evaluation and Learning
November 7 – 10, 2007
Baltimore, Maryland
RTD Topical Interest Group Sessions

For more information:
Gretchen Jordan, gbjorda@sandia.gov
George Teather, gteather@sympatico.ca
Brian Zuckerman, mbzuckerm@ida.org

 

Wednesday, November 7

1.  Closing the Loop: Mapping Value to Inform Research Management

Chair:Neville Reeve, European Commission

 

A Framework for Evaluating Large Scale AIDS Clinical Research Networks

Brendan Cole and Jonathan Kagan, National Institutes of Health, Melissa Burns, Mary Kane, and Kathleen M. Quinlan Concept Systems Inc, William Trochim, Cornell University, Daniel Montoya, Hill and Knowlton

 

Analysis of Follow-up Evaluation Results of Research and Development (R&D) Projects Applying Logic Model to Elucidate the Process of Innovation

Kazuki Ogasahara,  Ministry of Economy, Trade and Industry, Osamu Nakamura,  National Institute of Advanced Industrial Science and Technology, Kazuyuki Inahashi,  Ministry of Economy, Trade and Industry,  Chikahiro Miyokawa,  Ministry of Economy, Trade and Industry, Yoshitaka Kimura,  Ministry of Economy, Trade and Industry

 

Contribution of Evaluation to Management of Research and Development (R&D) in the Process of Technology Transfer: A Knowledge Value Mapping Approach

Juan Rogers, Georgia Institute of Technology

 

Thursday, November 8 

2. Hard Cases: Measuring and Facilitating Interdisciplinarity and Inter-Organizational Interactions

Chair: Isabelle Collins, Technopolis

 

University-Industry Collaboration: An issue for Ireland as an Economy With a High Dependence on Academic Research

James Ryan, CIRCA Group Europe Ltd

 

Measuring the Interdisciplinarity of a Body of Research

David Roessner, SRI International, Anne Heberger, Alex Cohen, Marty Perreault, The National Academies, Alan Porter, Georgia Institute of Technology,

 

Wikis in Evaluation: Evaluating Wikis for Theory Development in a Multi-disciplinary Center

P Craig Boardman, Nathaniel Deshmukh Towery, Brian Zuckerman, Science and Technology Policy Institute

 

3.  Research Evaluation of the Upcoming Europeans Union’s Framework Programme

Chair: Peter Fisch, European Commission

 

Networks of Innovation in Information Technology: Technology Development and Deployment in Europe

Nicholas Vonortas, George Washington University, Franco Malerba, Nicoletta Corrocher,

Lorenzo Cassi, Luigi Bocconi University

 

A New System for Research Evaluation Under the European Union's Seventh Framework Programme

Neville Reeve, European Commission

 

The European Union's Seventh Framework Programme and the Role of Evaluation

Peter Fisch, European Commission

 

4.  Looking Inside the ResearchCenter Black Box: Using Evaluation Research to Promote Organizational Effectiveness of Scientific Research Centers

Chair: Denis Gray, North CarolinaStateUniversity

Discussant: Gretchen Jordan, Sandia National Laboratories

 

Evaluating Leadership Development in an Research and Development (R&D) Context: Assessing Alpha, Beta, and Gamma Change

Bart Craig, North CarolinaStateUniversity

 

A Multi-variate Study of Graduate Student Satisfaction and Other Outcomes Within Cooperative Research Centers

Jennifer Schneider, North CarolinaStateUniversity

 

Enhancing Collaboration Between Historically BlackColleges and Universities (HBCU) and Research Extensive Universities

Andrea Lloyd, North CarolinaStateUniversity

 

Predictors of CooperativeResearchCenter Post-Graduation Survival and Success

Lindsey McGowen,  North CarolinaStateUniversity

 

5.  Tools for Useful Performance Assessment of Science and Technology Programs: An Example

Chair: Jerald Hage, University of Maryland

Discussant: Alfred Powell, National Oceanic and Atmospheric Administration

 

Prologue    Jerald Hage

 

A Strategic Balanced Scorecard for Publicly Funded Science and Technology Programs

Gretchen Jordan, Sandia National Laboratories

 

Perceptions of the Research Environment: Kinds of Networks, Research, and Projects

Jerald Hage, University of Maryland

 

Measuring the Immeasurable: Innovation and the Economics Benefits of Satellite Data

Jonathon Mote, University of Maryland

 

6.  Identifying Challenges to Using Evaluation to Inform Program Management and Public Policy

Chair: Connie Chang, United States Department of Commerce

Discussants: Connie Chang, United States Department of Commerce and Rosalie Ruegg, TIA Consulting Inc

Think Tank

 

7.  Research, Technology, and Development Evaluation TIG Business Meeting

2006 TIG Leaders: Gretchen Jordan, Sandia National Laboratories, George Teather, Independent Consultant, Brian Zuckerman, Science and Technology Policy Institute

 

Friday, November 9

 

8.  Evaluation of Community-based Participatory Research and Community Mobilization Strategies to Prevent Chronic Disease and Youth Violence: Advances and Lessons Learned by Two Research Center Programs at the Centers for Disease Control and Prevention

Chair: Alicia Norris, Centers for Disease Control and Prevention

Using Document Review and Data Abstraction to Inform Management of a Federal Research Program: Lessons, Benefits, and Challenges Found by the Centers for Disease Control and Prevention's Prevention Research Centers Program

Demia S Wright, Centers for Disease Control and Prevention

 

An Evaluation of Community Based Participatory Research and Community Mobilization: Formative Research Results From the National Academic Centers of Excellence on Youth Violence Prevention

Nancy Stroupe, Centers for Disease Control and Prevention

 

9.  Strategic Evaluation in a Public Research Institute to Contribute to Innovation

Chair: Osamu Nakamura, National Institute of Advanced Industrial Science and Technology

Discussant: Naoto Kobayashi, National Institute of Advanced Industrial Science and Technology

 

Strategic Evaluation of Research Units Towards Innovation in a Public Research Institute

Osamu Nakamura, Shin Kosaka, Michiko Takagi Sawada, Isao Matsunaga, Masao Koyanagi, Koichi Mizuno, Naoto Kobayashi - National Institute of Advanced Industrial Science and Technology

 

Evaluation System with PDCA Cycle in the Management of National Institute of Advanced Industrial Science and Technology

Sunao Kunimatsu, Osamu Nakamura, Yoshikazu Arai, Hiroshi Sato, Shinichi Kikuchi, Suzuko Nakatsu, Naoto Kobayashi -  National Institute of Advanced Industrial Science and Technology

 

10.  Peer Review and Learning: New Uses

Chair: David Roessner, SRI International

 

Peer Review of Transformative Research: Strategies and Challenges in Identifying Innovation in Ex Ante Evaluation

Elmer Yglesias, Science and Technology Policy Institute, David Kaplan, CaseWestern ReserveUniversity

 

Peer Reviews or Peers Reviewing? Peer Review as Policy Learning in Innovation, Research and Education

Erik Arnold, Isabelle Collins, Technopolis, UK

 

11.  Ex Ante Evaluation: Methods for Estimating Innovation and Other Research Outcomes

Chair: George Teather, Independent Consultant

 

Ex Ante Portfolio Analysis of Public R&D Programs for Industrial Technologies in Korea: Practices at the Korea Institute of Industrial Technology Evaluation and Planning

Yongsuk Jang, George Washington University, Jongman Park, Korea Institute of Industrial Technology Evaluation and Planning

 

Impact Evaluation in Preliminary Feasibility Analysis of National R&D Programs

Jiyoung Park, Korea Institute of Industrial Technology Evaluation and Planning

 

12.  A Directory of Evaluation Methods for Managers of Public Research and Technology Programs

Rosalie Ruegg, TIA Consulting Inc

 

Saturday, November 10

 

13.  Ex Ante Evaluation: Evaluation as an Agent of Program Change: An Example From Austria

Chair: Klaus Zinoecker, Vienna Science and Technology Fund

 

The Evaluation of Genome Research Austria (GEN-AU): Overview of the Study's Aims, Structure, Methods, Results, Implications, and Impacts

Klaus Zinoecker, Vienna Science and Technology Fund, Alfred Radauer, Iris Fischl, Roald Steiner - Austrian Institute for SME Research,  Brigitte Tempelmaier, Austrian Economic Service, Rosalie Ruegg, TIA Consulting Inc

 

Developing a Plan for Future Monitoring and Impact Analysis of Genome research Austria

Rosalie Ruegg, TIA Consulting Inc

 

14.  A Roadmap for Developing a Public Health Research Portfolio Evaluation Program

Chair: Robin Wagner, Centers for Disease Control and Prevention

 

Overview of Methodology Used to Develop a Research Evaluation Program

Jerald O'Hara, John Araujo, Mona Choi, Catherine Pepper, Robin Wagner, Guijing Wang, Trevor Woollery, Centers for Disease Control and Prevention

 

MEDLINE Search Strategies vs. Relevant Retrieval: How Closely do They Match for a Research Evaluation Topic?

Catherine Pepper, Jerald O'Hara, John Araujo, Mona Choi, Robin Wagner, Guijing Wang, Trevor Woollery, Centers for Disease Control and Prevention

 

Evaluating Public Research Investment: A Literature Review

Jerald O'Hara, John Araujo, Mona Choi, Catherine Pepper, Robin Wagner, Guijing Wang, Trevor Woollery, Centers for Disease Control and Prevention

 

Extending the Pay-Back Model to Incorporate Costs as Well as Benefits to Measure the Net Impacts of Organizational Expenditures on Public Health Research

Guijing Wang, Jerald O'Hara, John Araujo, Mona Choi, Catherine Pepper, Robin Wagner, Trevor Woollery, Centers for Disease Control and Prevention

 

A Bibliometric Methodology to Inform a Logic Model for Evaluating a Public Health Research Portfolio

John Araujo, Jerald O'Hara, Mona Choi, Catherine Pepper, Robin Wagner, Guijing Wang, Trevor Woollery, Centers for Disease Control and Prevention

 

 

15.  Using Logic Models to Evaluate Research and Technology Diffusion Results: Two Cases

Chair: Jeff Dowd, United States Department of Energy

 

Cutting Edge Logic Models for Research and Technology Programs

Gretchen Jordan, Sandia National Laboratories

 

Linking Projects to Program Outcomes in Metrics for Technology Development Programs

John Mortensen, Energetics, Inc.

 

The Logic of Indirect Programs to Diffuse Technologies: The Example of Training

John Reed, Innovologie

 

Getting From Training to Credible Energy Savings: An Evaluation Template

Harley Barnes, LM Business process Solutions

 

16.  The Follow-up Monitoring and Outcome Survey for National Research and Development Projects in New Energy and Industrial Technology Development Organization (NEDO)

Chair: Takahisa Yano, New Energy and Industrial Technology Development Organization

 

Study of the Correlation Between Ex-post Evaluation and Follow-up Monitoring of National Research and Development (Part I)

Hiroyuki Usuda and Momoko Okada, New Energy and Industrial Technology Development Organization

 

Study for the Important Management Factors Based on Follow-up Monitoring Data (Part II)

Setsuko Wakabayashi, Tsutomu Kitagawa, Takahisa Yano, Kazuaki Komoto, New Energy and Industrial Technology Development Organization

 

Approach for the Understanding of Outcomes Derived from National Research and Development of Energy Conservation Project (Part III)

Kazuaki Komoto, Tsutomu Kitagawa, Takahisa Yano, Setsuko Wakabayashi, New Energy and Industrial Technology Development Organization

 

17.  Forging a Strong Link Between Research and Science Policy for Air Quality Decisions

Chair: Dale Pahl, United States Environmental Protection Agency

 

An Overview of National Ambient Air Quality Standards

Ron Evans, United States Environmental Protection Agency

 

A Paradigm for Federal Particulate Matter Research

James Vickery, United States Environmental Protection Agency

 

Relationships Among Atmospheric Contaminants, Air Quality, Human Exposure, and Health

Rochelle Araujo, United States Environmental Protection Agency

 

Synthesis and Evaluation of New Scientific Knowledge

William Wilson, United States Environmental Protection Agency

18.  National Performance Evaluation System of Research and Development Programs in Korea: System and Applications

Chair: Sang-Youb Lee, Korea Institute of Science and Technology Evaluation and Planning

Discussant: Jiyoung Park, Korea Institute of Science and Technology Evaluation and Planning

 

Method and Application of the Survey and Analysis of National Research and Development (R&D) Programs for the Performance Evaluation in Korea

Keun-Ha Chung, Hyejung Joo, Herin Ahn, Korea Institute of Science and Technology Evaluation and Planning

 

The Performance Evaluation of Research and Development Programs in Korea

Seong-jin Kim, Soon Cheon Byeon, Korea Institute of Science and Technology Evaluation and Planning

 

Design of Metaevaluation Model for National Research and Development Programs in Korea

Young Soo Ryu and Soon Cheon Byeon, KoreaInstitute of Science and Technology Evaluation and Planning, Byung Dae Choi, HanyangUniversity

 

Development of the Evaluation Methodology for the Basic Research in Korea

Hyeyoung Yang and Sangki Jeong, Korea Institute of Science and Technology Evaluation and Planning

 

Performance Evaluation of Agriculture Research and Development Programs in Korea

Hoijong Jung, Korea Institute of Science and Technology Evaluation and Planning

 

19.  Centralized E-Tool for Organizational Performance Management: National Institutes of Health (NIH) Government Performance Result Acts (GPRA) & Program Assessment Rating Tool (PART) Assessments

Deborah Duran, National Institutes of Health