Formative assessment does not improve learning

Education in the UK is susceptible to passing fads and fashions, which often results in a knee-jerk reaction from those headteachers who fall into the trap of believing it is important to have in place every new shiny bell and whistle before an inspector rocks up at the gates. The truth of managing a school well is quite different; it is not the job of a headteacher to impose a never-ending cycle of initiatives, rather being a headteacher is about being the intelligent, humble custodian of an educational institution for a period of time. Doing the job well means being well informed, careful, and strategic. Our knowledge of how learning occurs continues to increase and mature – headteachers should, of course, be cognisant of this new knowledge and should plan to bring those advantages to their pupils in a well thought through approach with the support of their well-trained staff. This is a far cry from forcing intelligent teachers to implement unproven fads and fashions at the drop of a hat.

A major problem arising from the UK’s tendency to take the scattergun, shiny new toy approach is the meaning of highly effective approaches, with firm findings from research to support them, is lost or misconstrued. Instead of careful integration of new educational knowledge, all too often we see well defined and well-designed pedagogical improvements being reduced to meaningless, mind-numbing initiatives.

Take, for example, the knowledge discovered and refined over many decades by educators, cognitive scientists and psychologists, there is clear benefit to pupils when they are asked to recall previous ideas and this benefit is even present when pupils are forced to retrieve memories simply by encountering questions without the need to answer them correctly – a ‘testing effect’ arises, where learning is improved as pupils necessarily have to re-member, which is to say they must create a new memory. It is an interesting finding and is reliably replicable in the laboratory and in the classroom. Headteachers and teachers should be aware of these benefits and plan for occasions when pupils will be required to retrieve prior knowledge. There is much more to it, of course – the literature on ‘testing effect’, ‘retrieval practice’, ‘spaced learning’, ‘method selection’ and myriad other intriguing consequences of recalling earlier learning is extensive and goes back many decades. Teachers have known about these effects for a very long time indeed and, for many expert teachers, it is an embedded stratagem in their repertoire of pedagogic choices. All well and good. But then…

Well, then along comes a fresh packaging of these old ideas and a PR campaign to promote them. However, contained within the fresh, shiny new packaging is not a detailed, nuanced, complex exploration of a profound idea, but a significantly diminished version – one designed to fit with today’s need for pithy explanations and easy implementation. In our example, the result is a diktat handed down by the headteacher, who does not wish to engage with the literature and is content with a pithy summary, demanding all teachers must now bolt on to the beginning of every lesson a 10 minute ‘retrieval practice starter activity’. And so, once again, the profound becomes the mundane, the impactful becomes the time sink, the years of research and development becomes the easy to roll out initiative. Box ticked, inspector happy. Or so the unthinking headteacher smugly assumes.

There is, of course, nothing new about this problem. The dumbing down of educational discourse has mirrored the very same dumbing down in all public debate since the mid 1990s. Though, given education should be in the very business of defending knowledge and truth, it is particularly saddening schools have given in to the modern pressure to do away with nuance and conviction to be replaced with anodyne, simplistic consensus.

I recall attending a training day in the late 90s at which a colleague I had hitherto regarded as smart and diligent presented a session on Paul Black and Dylan William’s rather excellent assessment summary, ‘Inside the Black Box – Raising Standards through Classroom Assessment’, which I had already read with interest. My colleague rushed through a few slides and ended with an instruction to all teachers their lessons must now contain ‘formative assessment’ because ‘formative assessment improves learning’. This was curious for two reasons. Firstly, how odd it was to assume any teacher sitting in the hall that day did not already frequently use formative assessment approaches as a normal part of their teaching. Secondly, it is incorrect.

After about the fifth or sixth time of my colleague saying the line, ‘formative assessment improves learning’, I raised my hand to ask to speak. A groan from those teachers who just wanted all of this to end. And then I said, ‘it simply isn’t true to say formative assessment improves learning. It does not. It is the actions one takes based upon this formative assessment which might possibly improve learning’.

I didn’t think this a controversial thing to point out, but his presentation contained absolutely no discussion of what pedagogic actions teachers could take based on different outcomes from the formative assessment activities he was promoting (which, by the way, appeared to be colouring in pupils’ names in either red, amber or green on a spreadsheet for no discernible reason).

So, there are two outcomes from the recent trend to dumb everything down; really bad ideas make it into the classroom and cause teacher burnout from initiative overload, and really good ideas are so diminished they become easy to dismiss or to demonise by those who do not act in good faith.

By way of example, consider the debate around Deliberate Practice. Deliberate Practice is a long-established approach which, although variation exists in its implementation depending on circumstances, has a set of core elements defining its use. It has its proponents and opponents. And, because so many people refuse to properly engage with the literature, it has been easy for its opponents to claim Deliberate Practice ineffective and, therefore, to steer teachers away from its use. Perhaps the most egregious example of this was the 2014 meta-analysis by Macnamara et al. This study set out to diminish the importance of Deliberate Practice and concluded it ‘not as important as has been argued’. Many people quickly pulled together pithy reports (or even Tweets) to announce the nail in the coffin for deliberate practice. Yet few (perhaps none) bothered to read and stress-test the report. In fact, Macnamara et al made this claim even though their own report found an effect size of 0.38 for the influence Deliberate Practice has on performance. You may well ask, is an effect size of 0.38 worth changing policy over? Well, to put that figure in a more everyday context, here are the effect sizes some certain behaviours have on mortality: obesity (0.08), excessive alcohol consumption (0.13) and smoking (0.21). These are much lower, but we all agree are not to be ignored (indeed, consider how much public money is spent trying to change these behaviours and you’ll have a good sense of how important society thinks these effect sizes are).

So Macnamara et al find a significant effect size and yet the wording of the report is used by many to claim Deliberate Practice is not important. That’s odd. But nowhere near as odd as the truth.

Of the 88 studies Macnamara et al used in their meta-analysis, 18 were not even about Deliberate Practice. This enabled the reporters to manufacture a lower effect size. When those 18 studies (which contribute 45 effect sizes) are removed from the calculation, the effect size for Deliberate Practice increases further still. Why include 45 effect sizes in a report about Deliberate Practice when those effect sizes are completely unrelated to Deliberate Practice? Could it be because those who wish to oppose (or support) a specific approach (perhaps because of their own ideological beliefs) know the teaching profession has succumbed to the soundbite? Thankfully, we have the likes of SD Miller et al who are not willing to simply accept a pithy line – this group re-examined Macnamara et al (2014) in their 2020 paper ‘To be or not to be (an expert)’ and highlighted the flaws in its methodology.

So, should teachers take time to plan for Deliberate Practice and formative assessment and a whole host of other approaches painstakingly evaluated for efficacy over decades? Yes. But…

As I said to my colleague all those years ago, formative assessment does not improve learning, it is the actions one takes based upon the formative assessment that count.

Learning is complex. There is not a single method to be deployed and all will be well. Rather, it is about the combining of tactics and choices, in real time, with real teachers and real pupils in a responsive dialogue with each other. Deliberate Practice does have a good impact on performance, but it’s just one of many ways of increasing the chances of successfully bringing about long-term, durable learning.

We should, therefore, deploy as many of these proven approaches as we can as pupils ascend from novice to expert with any new idea. There is a cumulative effect. In much the same way taking steps to avoid obesity (ES = 0.08) does not really shift the needle in terms of how long one is likely to live, combined with avoiding excessive alcohol consumption (0.13) and smoking (0.21) and other known impactful behaviours, the cumulative effect really starts to make all the difference. As a pupil moves from a novice appreciation of a new idea, teachers can draw on proven methods of checking and securing prerequisite knowledge, bringing about awareness of new knowledge by making connections, moving the knowledge from an initially inflexible state to a flexible one, such that motivation to push ahead can be achieved and teacher and pupil can continue the dialogue in a responsive cycle of teaching and learning until the pupil gains an automaticity with the new idea and can use this success to move through naive, purposeful and finally deliberate practice, which, in turn, gives the pupil such secure foundations and a fluency of understanding that they might make wider connections in knowledge and behave as a domain expert might behave to achieve a mature appreciation of the new idea. Moving through these phases of learning, the teacher deploys numerous proven tactics, each with their own effect. Standing alone, these individual effect sizes may well be small, but combined the overall impact is great.

In my 2019 book, Teaching for Mastery, I described the journey through these phases of learning in a single diagram of the mastery cycle and emphasised the importance of not breaking the cycle. The elements are of little utility when used in isolation – the power comes from combining them in an expert teacher repertoire. Do not allow a consultant to tell you things like ‘formative assessment improves learning’ without insisting the conversation goes further to explain all educational interventions, such as formative assessment, are but one part of a whole.

Differentiation

During a recent visit to a school, a head of mathematics explained the school's vision of differentiation as simply pupils ending up with different outcomes - some pupils, for example, would complete the red exercise of problems, some the amber, some the green.  The problems in one set were, in the classrooms visited, unrelated to the problems in another.  I asked the HoD, 'so, differentiation is basically the view some pupils can learn the stuff you wanted them to learn and others can't?'

I think 'differentiation by outcome' is a dreadful, reductive view of education.  A view that promotes the entirely wrong idea some pupils can learn well and others pupils cannot.

Instead, I view differentiation as the set of actions a teacher takes in order to guarantee all pupils learn well.

It is a big topic to do justice in a very short blog, but here's a quick stab at describing a more useful view of differentiation...

If we are working with more than one pupil, then it is always the case there will be variation in the experiences the pupils have had to date, in their understanding of ideas, in the maturity of their knowledge schemata, in how quickly they can make sense of a new idea, and in how keen they are to do so.

Differentiation is simply a teacher’s response to all of the variations existing within a class. Understanding the class is made up of individual human beings with unique and different lives means teachers can appreciate the burden upon them to ensure all pupils have a successful experience of learning whatever new idea the teacher is planning for them to understand.

So, differentiation is just a way of saying how the teacher reacts to the pupils in front of them. This is a continual process and changes from class to class, idea to idea, even day to day. Perhaps it is helpful to consider the phases a teacher and class progress through as they work together on a new idea.

To begin with, the teacher will seek to establish the pupils’ ‘readiness’ for learning a new idea – this could be through some sort of diagnostic activity or through discussion or through detailed prior knowledge of the pupils. Clearly, pupils will have differing levels of readiness – some will have forgotten things, some will have missed key moments, some will have independently prepared more than others, etc. The first stage in differentiation, then, is the actions the teacher takes based on an individual pupil’s readiness. For some pupils, the teacher may react by providing corrective instruction, working carefully to undo and overcome a misconception, for example. For other pupils, perhaps some pre-teaching will help them to connect partially forgotten ideas. Other pupils will be perfectly well equipped to proceed with new learning having demonstrated their readiness by mastery of the diagnostic activity – the teacher might react here by extending the pupil’s domain expertise in a prerequisite idea by asking them to work on an unfamiliar problem or they might simply allow the pupil to progress to the new learning. This will depend on the teacher’s plan for classroom management and whether or not they wish all pupils to receive the introduction to the new idea together.

When pupils are ready to learn a new idea, the next step is instruction. We know understanding new ideas relies on understanding earlier, pre-requisite ideas. This is how we construct new knowledge – by linking it to what is already understood and using this understanding to ‘bridge’ to new meaning. The teacher can do this by using story-telling and metaphor. To enable metaphors to come to life and have mathematical meaning, the teacher uses models. The models are explored in examples and these examples form the way of narrating the instruction.

The second step in differentiation is, therefore, when teachers react to how readily (or not) pupils are making sense of the instruction. They do this by changing the examples, the models and the metaphors they are using to animate their instruction. The order in which these changes are made is really important. I have previously written about how to react during the instruction phase in this blog, Models, Metaphors, Examples and Instruction

All pupils (all people, in fact), grip new ideas at different speeds. The purpose of instructing pupils is to bridge from a mathematical idea in my head and understood by me to one that the pupil is able to make meaning of. Working out whether or not the individual pupils in front of us are making appropriate meaning is best achieved through dialogue – as we narrate an example, we then ask them to work on a similar problem and narrate back at us their thinking. In other words, we are using the to-and-fro of examples and problems as a conversation between teacher and pupil – the pupil is forced to articulate their meaning.

The next step in differentiation is, therefore, to react to the pupils in front of us by varying the number of examples they are asked to respond to until each individual is communicating the meaning the teacher is aiming for. This is just a way of checking that the meaning is being received. We should not be fooled into thinking their ability to articulate the correct meaning is an indication that any learning has taken place. At this stage, it hasn’t. But we do now know that we are able to ask the pupils to work independently on problems. We can now ask them to do some mathematics.

Doing mathematics is an absolutely vital step in learning mathematics – it is through doing mathematics pupils begin to truly learn mathematics.

It is important we do not stop at the point of them simply knowing – the point they were able to give the correct articulation. Imagine a pupil learning to play piano, for example. The teacher could tell them the keys they need to press and the order in which they must be pressed, with what pressure and at what pace so as to produce a certain tune. And the pupil could articulate back at the teacher the precise instruction – they know how to play the tune. But that doesn’t mean they can play the tune.

A teacher could explain, through the use of several examples and problems, how to multiply over a bracket, say, and a pupil might articulate back at the teacher the precise instructions – they know how to do it. But that does not mean they can multiply over a bracket. This is why we now give the pupils ample opportunity to actually do the mathematics. We want pupils to be so competent in doing the new mathematics they achieve a fluency in doing so. That is to say, they can perform precisely without the need to give attention.

The next step in differentiation is clearly the amount of doing we ask of individual pupils – they will all achieve fluency at different rates.  Initially, this doing might be thought of as naive practice. Once the new skill is something pupils are comfortable with, it is time to start learning.

This might sound a trifle odd and some people will argue, surely, if the pupils are fluent, they have learnt what they need to. But this is just the first step. Learning only occurs at the boundary of our current ability. All pupils have pretty much unlimited potential, but they only continue towards expertise if they continue to operate at their limits. Automaticity is a poor aim for any lesson – it represents a pupil who is no longer learning.

To ensure learning is becomes long term and durable, we now ask the pupils to engage in practice.

Effective practice occurs in phases too. Firstly, teachers should create opportunities both in the classroom and beyond, for pupils to engage in purposeful practice – this type of practice is goal driven. Considering the mathematical skill the pupil has been working on and now has automaticity with, teacher and pupil examine carefully the common errors the pupil is making.

For instance, the pupil who can fluently multiply over a bracket may well forget to multiply the second term in the bracket two times in every, say, ten questions. We now have a goal – it is highly specific to the pupil and, through dialogue with the teacher, the pupil can set about undertaking more practice with an awareness of that goal – they can be looking out for the common mistake they make and can try to reduce the number of times they falter to, say, just two times in every forty questions. Purposeful practice can be carried out independently at home because the pupil has a success metric to give them continual feedback and spur them on.

Purposeful practice keeps the pupil at the limit of their competence and, therefore, creates the cognitive conditions for learning to occur. So, the next step in differentiation is how the teacher reacts to the pupil’s need for purposeful practice – varying the amount of practice, the goals and the feedback to best realise the individual pupil’s limitless potential to learn. A pupil can significantly improve their mathematical skill through purposeful practice. But it does have its limitations, since purposeful practice leaves the pupil to determine how best to overcome their common mistakes.

The next stage in differentiation is, therefore, how the teacher responds to the pupil’s progress with their personal purposeful practice by deciding what type of deliberate practice to provide to the individual pupil. Deliberate practice is also goal driven, but draws upon what is already known in a domain to improve performance. With the pupil above, who has been forgetting to multiply the second term, the teacher can coach them in overcoming the problem by telling them about tried and tested ways for doing so. In other words, in the deliberate practice phase, the teacher trains the pupil in the approaches that experts in the domain have developed and used to overcome the very specific problem they are facing.

The final stage of practice is designed to help further assimilate the new learning with the pupil’s developing schema of knowledge. Now, practice problems are randomly mixed with problems of earlier learnt ideas – this removes recency and cue from the pupil’s practice exercise and forces them to retrieve previously learnt skills and to identify when to select certain mathematical tools.

The final stage in differentiation is, therefore, the teacher’s reaction to a pupil’s agility in selecting appropriate methods in mixed problems – all pupils will improve their method selection at different rates, so the teacher carefully judges the amount of practice required and supports the individual pupil as required.

This view of differentiation can be thought of as the oft quoted idea of learning being like building an enormous edifice. Constructing a mighty building requires very careful placement and gradual levels of scaffolding. Here, the teacher is the scaffold, providing all the necessary support and rigour needed for the pupil to fulfil their potential.

And just like the construction of an edifice, it is key that the scaffolding is removed at the right moment to let the building shine.