Labour’s historic defeat in the Caerphilly by-election was both a rejection of the party in Wales, where it has been dominant for a century, and of the national leadership of Sir Keir Starmer. The result confirmed the national trend: no ruling party has ever fallen faster in the polls than Starmer’s Labour.

Inside No 10, on Friday, they were keen to explain the defeat as just another “midterm protest vote” against an incumbent, but it was much more significant than that. As we report today, Wes Streeting, the health secretary, compares the Caerphilly loss to Plaid Cymru in a Labour heartland to the Hartlepool by-election defeat in 2021 after which Starmer almost resigned as leader and Tony Blair called for the “total deconstruction and reconstruction” of the party.

Since the Caerphilly defeat the p-word is being used repeatedly by Starmer’s allies and opponents alike: progressive. Labour needs to be more “progressive” and stop trying to “out-Reform Reform” (an implicit attack on Starmer’s chief of staff, Morgan McSweeney), Lucy Powell said after she was elected as the new deputy leader. This is code for moving left and, one suspects, for avoiding the kind of tough decisions on immigration and welfare taken by Mette Frederiksen’s Social Democrats in Denmark, Europe’s only successful centre-left government, but which Labour’s progressive majority cannot countenance.

Powell, sacked from the cabinet last month, now has her own powerbase as the leader of the soft left faction inside parliament. She is closely allied to Andy Burnham and Ed Miliband and was backed by Neil Kinnock, usually a loyalist but who recently warned that Starmer had “only months” to turn things round. Kinnock had “reservations about the capability” of Starmer’s senior advisors in No 10 (presumably he meant McSweeney) and said the prime minister needed to “demonstrate that he is authentically Labour”. Or should that be authentically progressive!

For Reform, as one senior party source told me, the result was clarifying. “Thursday night showed what we are up against. There was an amazing surge of support for Reform but when there is coordinated tactical voting in Labour heartlands – and you also probably get Tories voting Plaid in this case as well to keep us out – then we have a problem.”

Farage “personally invested a lot” in the contest, I was told, but the leadership believes expectations were allowed to “get out of control”. A lesson learned. What is clear is that Reform, a party that barely existed before last year’s general election, remains the insurgent force in British politics and will take on all-comers, in England, Scotland and Wales, but in doing so may well overreach. Starmer’s aides were keen to stress Reform’s support “has a ceiling”. But then so does Labour’s. Whose is the higher?

To stop Farage, progressives will contemplate anything in the years ahead, even forming a kind of “popular front” of resistance as favoured by the new Burnham-aligned Mainstream campaign group: Labour, the Liberal Democrats, the Greens, sectarian leftist MPs, and perhaps even some Europhile liberal Conservatives. Desperation, after all, makes unlikely allies.

In the 1930s, as fascism spread across Europe, Stafford Cripps, who would later serve as Clement Attlee’s chancellor from 1947 to 1950, urged Labour to form an anti-fascist “popular front” with the Communists and the Liberal Party. Attlee resisted and instead joined Winston Churchill in the wartime coalition.

Farage is no Oswald Mosley, and a contemporary “progressive” alliance assumes that there is the potential for unity among socialists, social democrats, and liberals. There wasn’t in the late 1930s and there isn’t today. If anything unites the left today it is negatives - fear (of popular opinion on immigration and identity issues), loathing (of Farage and Reform) and a kind of mutual contempt (among different factions). In any event, a new Labour leader would be required to lead an anti-Reform popular front.

If that leader is neither Burnham nor Angela Rayner, could it be Ed Miliband? Despite having led his party to abject defeat at the 2015 general election, Miliband remains popular among party members and his net zero messianism fits neatly within the party’s comfort zone. That he is even being mentioned as a possible leadership candidate reveals how much Labour activists are in thrall to the politics of a dead era.

But these are new, turbulent times, and Labour’s struggles are deepening divisions. Consider the fragmentation of the electoral map as Plaid Cymru rises, the SNP is emboldened, Reform UK consolidates its national poll lead, and support for the Greens, under the leadership of Zak Polanski, the self-styled eco-populist (whose new podcast is already one of the most downloaded on the political charts), surges. If these trends continue, Labour and the Conservatives will be crushed in next May’s English local and devolved elections. That would bring us closer to an outcome I have long seen as inevitable: the replacement, even under first-past-the-post, of the old two-party system with a seven or (if a radical Corbynite left outfit is established) eight-party system. Call it the “Europeanisation” of British politics.

Across Europe, the old parties of the centre-left and centre-right are in retreat. Even when they win, they do so with much reduced vote shares. In France’s 2022 presidential elections, the Socialist candidate and the Gaullist Republican together won less than seven per cent of the total vote, a humiliation unimaginable a generation ago. In Germany, the Social Democrat (SPD)-led three-party coalition, which included the Greens and Free Democrats, that had promised renewal was routed in February’s federal elections. As the old order crumbles in Europe something new and more volatile replaces it.

In opposition, Labour strategists often invoked Olaf Scholz, the lawyer-technocrat SPD chancellor who preached “respect” for working people. It was a comforting comparison – until Scholz won only 16 per cent of the vote, and now many in Labour fear a similar reckoning as their own support falls to historic lows.

Empowered by a Commons majority of 174, Starmer entered office vowing to restore stability and to lead the British people on a “rediscovery of who we are”. But he now governs from a personal position of extreme weakness. His opponents on the soft left believe he is trapped: he may now have the cabinet he wants after the recent reshuffle, but he has lost the trust of a restive parliamentary party. And much of the country as well. They are confident Labour has only one way to turn: left.