Evaluating Impact: Your Approach to Monitoring and Reviewing Evidence Following TEF 2023
Melissa Turner is Head of Strategic Projects and Change at Newcastle University but is writing in a personal capacity.
TEF and Quality of Teaching
There are undoubtably clear benefits to shining a spotlight on the educational experience of our students. The TEF forces us, whether we are doing it as part of business as usual or not, to holistically evaluate the impact of what we are doing. Analysing benchmarked data and reviewing what we are doing across our institutions is a helpful process (although very labour intensive) to support us in improving our provision. The question is whether the sector would be doing this regardless of the OfS mandating the TEF. The devolved nations opting out sends a clear message about how much value their regulating bodies feels it brings.
So What?
The TEF process has brought into sharper focus the need for the sector to get better at evaluating impact, in project management speak ‘the benefits realisation’ piece. When crafting the TEF we are aware we need to be asking ourselves ‘so what?’ following describing an intervention, but sometimes we find ourselves with little or nothing to say in response. This indicates that from the concept phase of an initiative we need to consider what evaluative techniques we are going to use and what benefits we want to see.
Importantly, we need to be brave as institutions to stop something we have started or modify how we are delivering it if it is not going to deliver the planned benefits, which we can only judge through ongoing evaluation. It might be time wasted but we will have learnt lessons from it to inform our next steps, and better that than to proceed with something we know will fail. It sounds obvious but if we cannot articulate the strategic rationale for doing something as an institution in relation to the benefits it will bring reputationally, operationally, and/or financially and we are unable to establish a way of measuring its impact, we need to ask ourselves ‘why are we doing this?’. Continuous improvement (not just for TEF) requires us to regularly evaluate quantitative and qualitative evidence to make judgements and subsequent change.
TEF Workshop Insights
During a TEF workshop I recently delivered with around 50 institutions, we explored where in their qualitative and quantitative evidence, they felt their biggest gaps were in demonstrating impact, and how they planned to ensure the gaps didn’t exist in future. This led to discussion about what would put them in the strongest position for TEF 2027. Below are the key themes discussed.
Gaps in evidence and strategies for addressing them
One theme discussed was how institutions often lack evaluation of interventions or, as some put it, ‘evaluation is not baked into projects at the start’ and it can be challenging to demonstrate how a discrete project has impact on institutional change. This linked to discussions around the scale and diversity of institutions, in which many departments are doing different things to address the same or different issues, making it difficult to evidence across the student body.
There is no silver bullet, but if institutions have university-wide strategic programmes, for example to improve assessment and feedback, this should impact the whole student body, despite the approach being nuanced for each discipline. Success requires us to plan and manage the change effectively to bring people along and ensure the impact is felt university wide.
It was felt the purpose of the institution was not always clear and became crystalised through the process of writing the TEF narrative. Again, this could relate to the large and diverse nature of institutions, with multiple priorities and stakeholders, such that our strategic imperatives are not as clearly understood as we had intended. This is perpetuated by external agencies placing higher and higher expectations on HEIs, with limited resource.
Solutions raised for addressing the gaps included:
Developing an internally accessible dashboard for data transparency and keeping the TEF (indicators) live in the institution
Building in project evaluation at the start and linking to a wider ‘Theory of Change’ approach
Establishing an internal repository of information for TEF narratives, including capturing faculty-level and cross-University-level good practice. Indeed, sharing of good practice across an institution was indicated by the TEF Panel as being evident in the stronger submissions.
Preparing for TEF 2027
One of the main themes that came through the discussions was cultivating a TEF culture and fostering shared or community ownership. Examples given included:
Identifying TEF champions
Briefing colleagues to understand how the TEF works and what evidence is needed
Identifying what we could amplify across the institution
Building into course enhancement plans, flagging of potential TEF examples (adding to a central repository)
Delivering training/awareness sessions to reflect upon outcomes and consider examples from Gold award institutions.
Other ideas included:
Following up on initiatives planned/referred to in TEF 2023, demonstrating how pilots have been scaled up and delivered impact/benefits
Having a focus on strong, authentic strategic direction for continuous improvement (regardless of the TEF)
Looking to plug the data gaps, specifically with regards to Educational Gain so we can build a more meaningful narrative with evidence
Developing a collaborative framework for working with and engaging students.
Final reflections
There are more points for us as individual institutions and as a sector to consider, including: will educational gain remain and how we will embed and measure this? What learning are we taking forward from the student submission both in our own institutions and others? And how will we learn from each other to improve the quality of the student educational experience across the sector?
As most indicators are already set for TEF 2027, should we be thinking even further ahead to TEF 2031, knowing what we do today will impact those indicators? We know for TEF 2023 the indicators contributed no more than 50% of the final rating, and hopefully this will remain for future TEF. By focussing on the things that matter to our students, through planned and evaluated interventions, the educational experience and outcomes of our students will improve … and good TEF results will follow.
