Unpacking the impacts of programmatic approach to assessment system in a medical programme using critical realist perspectives
Executive Summary
This briefing document examines the implementation of a programmatic approach to assessment within a large-scale medical program, utilizing critical realist perspectives to unpack underlying causal mechanisms. Traditional medical assessment, rooted in positivism, relies on high-stakes, single-point-in-time examinations. In contrast, programmatic assessment (PA) emphasizes the longitudinal aggregation of data points to support learning and facilitate holistic progression decisions.
Critical Takeaways:
Paradigm Shift: PA transitions assessment from something "done to students" (empiricist) to something "done with students" (constructivist-interpretivist).
Implementation Tensions: Significant discordance exists between pedagogical theory and student experience, specifically regarding the interpretation of "stakes," "weighting," and the purpose of e-portfolios.
Structural and Cultural Constraints: Deep-seated norms favoring "objective" psychometrics often clash with the subjective, narrative-based nature of PA, potentially leading to a regression toward traditional methods (morphostasis).
The Hidden Curriculum: Perceived ambiguity in official structures often drives students toward unofficial channels (e.g., social media, senior students) for validation and clarity, creating a parallel "hidden" curriculum.
Disruptive States: Failure to align structure, culture, and agency can result in states of Anomie (student alienation), Agnosis (institutional ignorance/apathy), and Agenesis (failure of the practice to fully develop).
1. Overview of the Assessment Transition
The move from traditional to programmatic assessment represents a fundamental shift in how clinical competence is defined and measured.
Traditional Assessment Paradigm
Basis: Rigid, quantifiable, "objective" final assessments.
Formative Use: Limited; primarily used as practice for summative hurdles.
Critique: Fails to capture multifaceted competencies; over-relies on numbers to represent complex human behaviors.
Programmatic Assessment (PA) Paradigm
Basis: Longitudinal aggregation of diverse data points (vivas, clinical tests, reflections, group tasks).
Function: Continuous assessment where every data point is optimized for feedback and learning.
Decision-Making: Progression is determined by a committee (e.g., Portfolio Advisory Group) through a holistic review of a "Student Progress Record."
2. Empirical Analysis: Student and Faculty Experiences
The "empirical" domain of reality—what was observed and reported—reveals significant friction during the navigation of the new system.
Navigation of Assessment Structures
Students frequently reported "angst, confusion, and uncertainty" regarding the structural changes.
Ambiguity of "Weighting": Students struggled with the lack of percentage-based grades. One student noted: "I was legitimately gobsmacked... I never heard of not informing your students about what their assessments are worth."
Interpretation of "Stakes": Tensions arose over the definition of formative vs. summative. Students expressed fear that minor tasks could unexpectedly impact their progression: "You could be sweating on a minor thing to know if you have passed the whole year."
The Role of Proctored Exams: Students tended to judge the "worth" of an assessment by its level of surveillance (proctoring), rather than its pedagogical intent.
Interactions and Culture
Faculty-Student Tension: A "blame culture" was identified, where students perceived faculty as defensive when issues were raised.
The Role of Learning Advisors (LAs): Clinicians assigned to 4–6 students.
Value: Students valued LAs for reassurance, career advice, and lifestyle mentoring.
Conflict: Dissonance occurred when LAs were viewed as assessors. Effective PA requires delinking the advisory role from the assessment role to foster trust.
Advisor Insight: LAs generally found the role motivating, viewing it as a form of "mentoring and a bit of parenting" rather than formal decision-making.
3. Causal Explanations: The "Actual" and "Real" Domains
Using Bhaskar’s ontological stratification and Archer’s morphogenetic approach, the document identifies hidden mechanisms driving these experiences.
The Domain of the "Actual" (Events and Hidden Mechanisms)
The Hidden Curriculum: Due to perceived ambiguity in official channels, students turned to private social media groups and senior students. This "secret garden of curriculum" became the primary source of trust and validation, rendering some official university systems redundant.
The Conflation of Learning and Assessment: The attempt to make all assessment "formative" often resulted in students feeling that everything was high-stakes, leading to an increased "assessment burden."
The Domain of the "Real" (Enduring Structures and Agency)
Archer’s framework explains whether a reform flourishes (Morphogenesis) or returns to status quo (Morphostasis).
Structural Conditioning (T1): Pre-existing university rules and a culture of psychometric "objectivity" offer temporal resistance to PA.
Socio-Cultural Interaction (T2-T3): Agents (students and staff) use their reflexivity to navigate the change. If the structure is too stiff, anomie (normlessness) occurs.
Morphostasis (Regression): If progression decisions revert to being based solely on marks/scores rather than narrative portfolio data, the system regresses to the traditional mode.
Morphogenesis (Elaboration): This occurs when the community accepts and adapts to the new features, such as the successful integration of the Learning Advisor system.
4. Disruptive States in Programmatic Assessment
The analysis identifies three primary disruptive states that emerge when the theory-practice gap is not addressed:
Anomie (Person Level): A state of "normlessness" for students, resulting in alienation, dysfunctionality, or inactivity due to the mismatch between intended purpose and lived experience.
Agnosis (Program Level): Institutional "not knowing" caused by structural stiffness or cultural apathy. This happens when pre-existing beliefs regarding "objective" testing are not proactively addressed.
Agenesis (Practice Level): The failure of the PA model to fully develop in its functional form. This results in "naïve, incomplete, or eclectic narratives" and "incompletely hybrid" versions of the reform.
5. Conclusion: Conditions for Sustainability
For programmatic assessment to flourish as a radical reform, it must be viewed as a complex social change rather than a mere educational design problem.
Key Requirements for Growth:
Structural Agility: Rules and policies must support, rather than hinder, narrative and holistic decision-making.
Cultural Alignment: Long-standing beliefs in the superiority of "objective" numbers over "subjective" human judgment must be challenged.
Agency Optimization: Both students and faculty must be empowered to use reflexivity and internal conversation to navigate the medical program, rather than feeling surveilled by it.
Clear Definitions: Terms like "stakes," "weighting," and "remediation" must be transparently defined to prevent the emergence of parallel, unofficial curricula.
