Powered by Publics Learning Memo: Using Improvement Science to Guide Cluster Work

By Tia Freelove Kirk

Led by University of Texas at Austin, the Powered by Publics Evidence Improvement cluster is convening over 30 campus leaders and subject-matter experts at high-enrollment institutions across the country to increase student success in the entry year by improving practices in advising and first-year experiences.

The Evidence Improvement cluster is using improvement science as the framework for organizing its efforts. Improvement science stems from quality improvement approaches in industry and healthcare, and it incorporates networked learning and collaboration to address specific “problems of practice.”1 The cluster is committed to directing improvement efforts in pursuit of educational equity for low-income, first-generation, and students of color, as stated in the Powered by Publics strategic priorities.

Using Improvement Science as a Guide for Cluster Work
Improvement science provides helpful guidance for answering two critical questions this cluster raised at the beginning of their collaboration: 1) how to filter a lot of information about student success drivers to identify shared barriers that the cluster can address together; and 2) how to see the system that created those problems.

In 2019, cluster partners identified student-centered advising and first-year experiences as target areas to address through an inventory of their campuses’ highest priority student success initiatives focused on the entry year. They then formed working groups modeled after “Networked Improvement Communities”2 that engage campus experts, practitioners, and key sources of information to continue developing a process for improvement. Both working groups developed driver diagrams to visually describe their theory of what contributes to improved student-centered advising and first-year experiences and drafted problem statements to focus their efforts (For more information on cluster progress in 2019, see this cluster’s previous learning memo).

This year, as campuses began addressing the effects of COVID-19 on their campuses, the cluster’s efforts to develop a process for assessing, testing, and implementing evidence‐based practices in advising and first‐year experiences became even more critical.

After pausing collaboration so partners could focus on addressing urgent matters on their campuses, cluster leaders reassessed drivers and identified newly emerging barriers to connecting first-year students to the university through advising and first-year experiences. Through empathy interviews and facilitated brainstorming, campuses identified the following priority barriers to success for the least well-served students:

• Student academic preparation
• Student well-being
• Student sense of belonging
• Student awareness of and engagement with support services
• Student access to technology
• Coordination and communication among faculty

Working groups agreed to collectively address two major barriers given COVID-19 and shifts in campus priorities: students’ sense of belonging and quality of engagement with support services.

Pursuing Educational Equity through Focused Learning and Improvement
The Evidence Improvement cluster is focusing learning and improvement efforts around pursuing educational equity for low-income, first-generation, and students of color. This takes shape in the cluster’s fall action steps, which began with determining high-leverage processes to address (described above) and includes:

• Sharing existing ways campuses measure the effectiveness of engagement with advisors and sense of belonging created via faculty, peer mentors, and peer-to-peer connections;
• Reviewing data collected to identify who is experiencing inequitable outcomes;
• Creating a hypothesis for the cause of identified outcomes gaps;
• Brainstorming indicators that signal an improvement; and
• Planning equity-focused small tests of change.

The improvement science framework is equipping cluster partners to quickly look at the system driving students’ connections to the university, use disciplined inquiry to identify where that system is breaking down, and carry out a process for partners to try small changes on their campuses that will help the entire team learn quickly.

Learning Highlights
Attending to variability across campuses: While first-year learning communities exist on all campuses, they vary in the level of centralization and scope. Because of this, working group members are now working to map programs to quickly compare program similarities and differences.

Measuring student sense of belonging: Multiple cluster campuses measure student sense of belonging, whether through feedback from students during first-year programs or through campus-wide student experience surveys such as the Student Experience at the Research University (SERU) survey and the University of California Undergraduate Experience Survey (UCUES).

Shifting advising and orientation during COVID-19: Across cluster campuses, advising programs shifted into high gear to support students as they faced daily programmatic and campus environmental shifts from the spring through the fall. Cluster partners have led impressively quick changes in services to accommodate students, including, but not limited to:

• Transitioning in-person advising to exclusively offering remote advising appointments;
• Creating a remote orientation experience coupled with in-person campus tours for students; and
• Shifting two-day advising and orientation sessions to live virtual workshops with pre-recorded videos followed by live Q&A sessions with advisors and group registration workshops

Key Resources:

• “Using Improvement Science to Accelerate Learning and Address Problems of Practice” from the Carnegie Foundation for the Advancement of Teaching
• “Why a NIC?” from the Carnegie Foundation for the Advancement of Teaching
• Harvard Education Review’s comprehensive summary of Learning to Improve: How America’s Schools Can Get Better at Getting Better (Bryk, Gomez, Grunow, and LeMahieu, 2015).

Institutions in the cluster:

• Texas A&M
• University of California, Davis
• University of Texas at Austin – Cluster Lead
• University of Florida
• University of Oregon
• University of Utah
• University of Washington