Data gathered during the formative and summative evaluation will focus on process and outcomes indicators that evaluate the HHMS AGEP Sequential and Cognitive-Apprentice model. That is, the formative evaluation will examine the inner workings of the model within and across the various institutions (e.g., what is actually taking place) and determine the extent to which the model was implemented with fidelity (i.e., adherence to the program timelines, meeting programmatic benchmarks and the quality of delivery). The formative evaluation will consist of post-course, seminar, and mentorship evaluation surveys and questionnaires to gather data from program participants, community college faculty mentors, and program leads about their experiences, successes, and challenges in the AGEP HHMS program.
Program participants will be surveyed to measure short- and long-term outcomes during the program years to assess students’ interest in an academic career, pedagogical knowledge and skill, knowledge of culturally-responsive pedagogy and diversity issues in higher education, and development of teaching self-efficacy. Moreover, brief assessments (i.e., evaluation forms) will be used to measure the Level I students’ perceived gains from webinars, seminars, and workshops. Level II assessments of students’ pedagogical knowledge and understanding of faculty roles and responsibilities will be embedded in the courses. At Level III, the mentored teaching experience, mentors and student instructors will be interviewed and/or surveyed about their experience and the impact their experience has on student instructors’ career decision-making. Additionally, program stakeholders will be interviewed about their work with implementing the program, their mentoring experience, as well as research experience testing the program model.
Program documentation, participation records, and self-study findings will be systematically collected and analyzed to detail the ongoing implementation of academic activities. The AGEP-Tracking System will provide data that will be used to examine and highlight academic trends and track participants’ applications for post-doctoral and teaching positions. Formative data will be regularly provided to the AGEP HHMS project so that mid-course corrections can be made as warranted.
The summative evaluation will investigate the extent to which the model achieved its intended aims of enhancing the level of pedagogical preparation of URM doctoral STEM students in the AGEP HHMS Alliance, increasing the number of URMs who are well prepared to teach in postsecondary institutions, and adding to the theoretical research and knowledge base on URM doctoral students’ pathways to the professoriate, as there is little empirical evidence on how teaching during one’s time as a graduate student affects long-run outcomes, including subsequent labor market prospects, research productivity, and other academic and professional outcomes. Ongoing collection and analysis of formative data collected nearing the end of the program will address larger summative impact questions on changes in attitudes, knowledge, or skills and practices that extend beyond the period of program participation and into the daily work and professional plans of the program participants. Such long-term changes can lead to institutional change over time. Additionally, the summative evaluation will examine sustainability of the AGEP HHMS program by examining, for instance, any practice or policy changes within and/or across the participating institutions as a result of AGEP grant, course work that can continue across campuses for STEM doctoral students, and the ability for continued mentorship and institutional collaborations that will support the advancement of underrepresented STEM students.
