Here we go again: on Tuesday the largest teaching union denounced Nigel Farage’s Reform UK as a “far right and racist” party. The National Teaching Union (NTU) also pledged funds to oppose Reform candidates, and its leader, David Kebede, dismissed Farage as a “poundshop Donald Trump”. One would have thought his members would more profitably concentrate on their core responsibility, which is to teach and improve attainment levels and enhance the well-being of our children in schools.

Rather than grappling with the forces that Farage has unlocked and continues to channel as Reform prepares to make sweeping gains on 1 May in the local elections in England, much of the left would rather condemn him as a racist and, by implication, all those who would vote for his party, which according to recent polls is 25 per cent of the electorate. But the left has long misunderstood Farage – or lazily tried not to understand him at all – and failed to think seriously about what lies behind the mass disaffection that has powered the rise of the contemporary radical right.

Farage is a public-school educated former City metals trader and yet he has broad cross-class appeal. I’ve interviewed him on several occasions over the years and followed him on the campaign trail. I’ve observed his relentlessness and opportunism but also his pragmatism: he believes he goes as far as he can with his anti-immigration rhetoric while remaining within the mainstream. “No one did more to beat the far right in this country than me,” he told me last summer, adding: “If you think I’m bad enough imagine what comes after me. But while I am here that person will not arrive.”

As Conservative prime minister, David Cameron tried something like the NTU approach when he insouciantly dismissed Ukip supporters as “loonies, fruitcakes and closet racists”. Farage and his so-called people’s army had their ultimate revenge on Cameron when Remain lost the 2016 referendum. In retrospect, Brexit broke the Conservative Party, now reduced to a rump of 121 MPs, the lowest number in its history.

Cameron was not alone in underestimating Farage. During his ill-fated leadership of the Labour Party, Ed Miliband said to me on several occasions that be believed Ukip posed a greater threat to the Conservatives than Labour. He was wrong (a third of Labour voters chose Brexit in 2016) just as he was about the nationalist insurgency led by Alex Salmond in Scotland.

The Tories no longer underestimate Farage: they fear his destructive potential and his willingness to destroy them. Kemi Badenoch is bracing herself for heavy loses to Reform and the Liberal Democrats at the local elections. Meanwhile, Robert Jenrick, whom she defeated for the leadership, aspires to unite the right and, therefore, become prime minister. But he must know that Farage, as he agitates, campaigns and provokes, stands in his way and that there can be no path back to power for the Tories that does not also include Reform. But any alliance with Reform could split the party.

If you talk to Labour MPs who represent Red Wall seats, they consider Farage to be the de facto leader of the opposition and are deeply anxious about the threat from Reform, which has professionalised since the general election when it was little more than a pseudo party centred on a cult of personality. Farage did not even know who most of the party’s candidates were in July and the process to vet them was non-existent, but Reform still won over 4 million votes.

The progressive mind struggles to understand populism, which as the philosopher John Gray has written has no clear meaning but “is used by liberals to refer to political blowback against the social disruption produced by their own policies”.

When I interviewed him last summer, Farage likened himself to the American evangelist Billy Graham, and his shtick is increasingly part secular preacher, part game show host, part political rabble-rouser. Like Bruce Wayne’s adversary the Joker, he remains, in whatever guise he adopts, an agent of chaos. Yet influenced by Giorgi Meloni, the post-fascist prime minister of Italy, he is seeking to attract more traditional conservatives and always has a defining phrase: Take Back Control; We need to get our country back; Britain needs Reform.

Farage claims to be contemptuous of the whole Westminster jamboree but loves the political game. That sense of being an insider-outsider and scourge of what he dismisses as the “political establishment” are part of his appeal along with his direct, straight-talking style and fearlessness. Reform’s policy programme is incoherent, and it has begun to embrace a fusion of both left and right economic populism; the party is pro-reindustrialisation, militantly opposed to Labour’s net zero targets and demanded the nationalisation of British Steel before Labour acted.

Farage thrives on hyperbole and exaggeration. Note his response to the British Steel debacle when he accused Jingye, the Chinese owners of the imperilled Scunthorpe steel plant, of “lying about everything”. Or his response to Hashem Abedi’s horrific attack on three prison officers in a separation centre at HMP Full Sutton. “This monster should never have been in a position to hurt our brave prison officers,” Farage said of Abedi, who was sentenced to a minimum of 55 years in prison after being found guilty of mass murder for his part in the Manchester Arena attack. “It is worth remembering that this family came to Britain as asylum seekers. The system is broken.”

In three short sentences on social media, which he has mastered, Farage distilled the essence of his politics: the system is broken, immigration is out of control, honest working-class people are being betrayed.

Shabana Mahmood, the justice secretary, is one of Starmer’s most able senior ministers, but inherited a prison system in crisis. She is preparing for her departmental budget to be cut in real terms by as much as 11% in June’s Spending Review, although the prison estate is under extreme strain: prisons are 98.9% full, according to the latest government figures. As she surveys a prison system close to collapse, Mahmood has good reason to agree with Morgan McSweeney, Keir Starmer’s chief of staff with whom she was aligned in the Labour Together network, when he says, “people have had the hope beaten out of them” and believe “nothing can change”.

That sense of fatalism is dangerous, and Farage seeks to exploit it. But Labour believes he has two significant vulnerabilities: 1) his friendship with Donald Trump, who is widely loathed in Britian, and 2) his equivocations on Putin’s Russia and conviction that the West provoked the full-scale invasion of Ukraine. Farage has populist instincts, but these are not popular positions in the country.

A defining theme of this parliament will be the fragmentation of the voting system, with no one party able to command majority support. If Labour is to retain power as something more than what McSweeney has called a “winner among losers” it will not be the party’s ardent progressives but the exponents of “hard Labour” - Mahmood, McSweeney, John Healey, the defence secretary – who will lead the strategic fightback against Reform. They at least understand that denouncing Farage, and anyone who would vote for his anti-system party, as “far right and racist” would be to repeat the mistakes made by Cameron and Miliband and, in a different context, Hillary Clinton and Kamala Harris in their struggles against Trump in America. It would serve only to fire up and mobilise the people’s army as they demand change, any change, at any cost.