Evaluation Capacity Project

Enhancing Evaluation Capacity

With support from an NIGMS supplement (3T32GM008336-33W2 in response to NOT-GM-21-024) our project had the specific aim of developing new evaluation tools to measure (a) student-centered, (b) formative, and (c) quantitative dimensions of evaluation, to complement existing program assessments.

  • Our surveys, developed in collaboration with WSU’s SESRC, includes language from the NIGMS T32 funding opportunity announcement NOFO with the goal of creating value for similar training programs.
  • We also experiment with the “Net Promoter Score” (which is of research interest due to uncertain utility in academic settings).
  • Building these tools helped us develop enduring capacity (necessary training) to independently create new tools and analyze evaluation data.

Evaluation Capacity Diagram

New Surveys Developed

Below are pdf copies of our developed instruments. Please email Matt Peck, our project coordinator, to attain Word copies.

Annual Survey (trainee version)

Annual Survey (faculty version)

Protein Biotechnology Class Survey

Exit Interview Survey

Winter_Biotech_Rotation_Report

How our surveys help to improve our program:

  1. Make sure trainees have access to safe and supportive research environments.
  2. Assess needs and impact of mentor training on the research environment.
  3. Ensure program activities are valued by trainees and improving their skills and career preparation.
  4. Ensure the program creates opportunities for trainees to provide feedback and assess changes made on these suggestions.

Initial Outcomes and Future Plans:

  1. The capacity to create surveys has grown and principles of survey writing and data evaluation are being disseminated to our trainees and faculty in other program activities.
  2. Our annual survey has been shared with partner doctoral programs to scaffold their information gathering.
  3. The initial data have helped us evaluate a revised course and establish a baseline for monitoring mentoring climate, program inclusivity, and professional development.
  4. Despite the attractive potential of the “Net Promoter Score” to be a summative score in a single measure, feedback from our users is that it creates more confusion than standard Likert scales. However, it does seem to pick up more signal between groups and track with other ways of measuring similar concepts.
  5. The long-term plan is to collect data annually to monitor longitudinal changes in trainee and trainer attitudes and have data to correlate with more traditional measures of training outcomes (job placement, time-to-degree, etc.)

Acknowledgments:

We had a great collaboration experience with WSU’s Social and Economic Sciences Research Center. We acknowledge Adam McKee, Lauren Scott, and James Mike Schiwart for their expert assistance. We thank Trymaine Gaither for suggesting an investigation of Net Promoter Score.

SESRC