THE ANT MILL — how theoretical high-energy physics descended into groupthink, tribalism, and mass production of research

I have written a new book about the sociology of hep-th. The field is in crisis: no major breakthrough in 50 years and yet a remarkable absence of conceptually new ideas. This calls for an explanation

Here is a paradox: the research field of hep-th has on the one hand not produced any major breakthrough for half a century while it on the other hand displays some very remarkable sociological features: the field has coalesced into a small set of large research communities centred around decade-old ideas with a striking absence of conceptually new ideas. This is the exact opposite of what one would expect from a research field that has come up against a wall. You would expect to see an explosion in creativity with researchers taking off in many different directions. This odd paradox calls for an explanation. In ‘The Ant Mill’ I offer one.

How is it possible that a theory like string theory can dominate hep-th for more than three decades without having produced a thread of falsifiable empirical evidence? And why are there so few alternatives to string theory? Sure, there is loop quantum gravity, but that theory has also not produced any empirical evidence. There exist a small number of other approaches, but these are almost entirely based on very old ideas. Where is the explosion in creativity and innovation that one would expect to observe in a research field like hep-th that is searching for a new direction?

And especially: where are all the young ideas? When the field was in a similar situation more than a century ago — when it was on the verge of discovering quantum mechanics and general relativity — there was a creative and innovative outburst lead by very young researchers. Where are all the young rebels today?

Why do researchers keep punching the same old ideas instead of trying something new? And why do they cluster so much? Historically the field has always been revolutionised by lone researchers, but these are almost entirely absent today. How come?

In ‘The Ant Mill’ I attempt to answer these questions. Based on 25 years of experience as an insider in several of the leading communities in hep-th — especially loop quantum gravity and noncommutative geometry — combined with my own sociological research, I analyse in ‘The Ant Mill’ the historical changes that the field has gone though in the past century and chart the social forces and incentive structures that these changes have generated. This analysis shows that a massive social force towards mainstream research has emerged, a force that pushes the field towards tribalism and groupthink — and prevents young researchers from pursuing their own, independent research interests.

This explains why the field has lost its creative powers; there is today very little room for new, innovative ideas that depart from mainstream research, and the reason for that is that the incentive structures generated by the sociological changes reward obedience, conformity, and submissiveness, while they punish innovation, creativity, and independent thinking.

The key sociological drivers are first of all the massive growth of the community over the past century combined with increased levels of competition, which began to grow very significantly sometime around the 1980s, as well as changes in the value system, which has shifted away from expert evaluation and towards quantifiable measures such as publication and citation counts. As I show in my book, it is relatively easy to understand why these three factors combined with the near-complete absence of new empirical data will necessarily push a research community like hep-th towards mainstream research and the formation of large, hierarchical communities.

In ‘The Ant Mill’ I show that these developments are ongoing and accelerating. The level of competition continues to grow at an accelerating rate and as a consequence the field is becoming more homogeneous and increasingly unlikely to be able to produce what could be its salvation: a new major scientific discovery.

So things are getting worse, and with that the notion of a scientific community is gradually fading like a photograph left in the sun — like a photograph from the 1927 Solvay conference where all the old legends silently observe us up through the past century. Like a jury.

the incentive structures reward obedience, conformity, and submissiveness, while they punish innovation, creativity, and independent thinking

‘The Ant Mill’ is a personal book. I have based my analysis on more than 25 years of experience in the field, and thus for me this is also a deeply personal affair. As a young researcher I was one of the young rebels. Straight after having completed my Ph.D. I launched a new research project in an unexplored direction, a move that gave me much unwanted insight into the sociology of the field. In ‘The Ant Mill’ I combine my analysis with personal anecdotes to show readers what the culture in hep-th is like. A culture where career-thinking often trumps scientific integrity, where tribalism appears to have become the norm, and where being a scientist for many has become a job rather than a calling.

It is a culture where compromise has become the norm, and where being relentless and unwilling to compromise has become an outlier behaviour, especially among young researchers, that takes you out of academia.

One thing that struck me as I wrote ‘The Ant Mill’ is how predictable all this is. It is not difficult to dissect the sociological makeup of a research field like hep-th, and it is not hard to see that the four key sociological drivers that I mentioned in the above will inevitably skew the field towards groupthink and tribalism. However, what is hard to understand is that no countermeasures have been put in place. After all, we are talking about some of the supposedly most intelligent people on the planet. The truly interesting question, which I discuss in ‘The Ant Mill’, is why theoretical physicists have not been able to resist the underlying sociological forces that have lead their research field into the desert.

A key aspect of the social changes discussed in ‘The Ant Mill’ has to do with the demographics of hep-th. A central question is what a winning strategy looks like in the new academic reality that has emerged. Who wins the battle of mainstream research, and who loses it? A key point in my book is that it is the technicians — i.e., those who prefer to focus on a narrow subject area and hard analysis — who have a clear advantage, compared to the more visionary researchers — those, who think more broadly and creatively but are less skilled when it comes to hard analysis — and therefore it is the technicians, who win. I believe this explains why the big conceptual ideas are absent in hep-th today.

Another key aspect discussed in ‘The Ant Mill’ is that a winning strategy involves membership of a large research community. Chances of survival for a young researchers are significantly higher if he or she belongs to one of the large communities. This explains why the solo researcher, and especially why the young solo researcher, has almost entirely disappeared from hep-th, something that half a century ago was the norm.

I wrote ‘The Ant Mill’ because I needed to understand what I have seen during my years in academia. It made no sense to me and yet, it is perfectly sensible. The mechanics underlying this schism is what my book is all about.

And I wrote ‘The Ant Mill’ because I sincerely believe that in order to move forward we must examine not only the evidence in front of us but also the cultural bagage that we bring with us. Without a serious analysis of our research community we risk ending up in an intellectual hamster wheel, where all we accomplish is to make incremental improvements on other peoples failed ideas. I believe we can do better than that, and I believe that we owe it to society.

You can buy ‘The Ant Mill’ on Amazon as a paperback, a hardcover, and an e-book. The book has a foreword written by Peter Woit.

If you would like to help please share this post as well as the link to ‘The Ant Mill’. Thank you.

Kind regards, Jesper