In preparing the previous post on "Measuring Gravity" I skimmed a thousand or so physics journal articles. Like most lay people interested in physics, I'm interested in the fundamental questions of how we are going to solve the great unsolved mysteries of quantum mechanics, general relativity, and cosmology, the existence of which is universally accepted in the field.
The physicists who concern themselves with these questions at times like the present, when all the old billion dollar toys have been milked for all that they are worth, and none of the new billion dollar toys have started spewing out intereting new data yet, are theoretical physicists who publish mostly in the broad subfields of high energy physics and general relativity. And publish they do, thousands of them who collectively publish hundreds of articles a day under the intense publish or perish pressures of academe.
Theoretical physicists are overwhelmingly legitimate professors at respected institution of higher education with PhDs from accredited universities who do work on these papers for essentially all of their their work time when they are not teaching or attending faculty meetings, for most, 20-30 hours of week during the semester, and 40-60 hours a week during summer vacation. They have mortgages, spouses, and children. They read typical daily newspapers, collect Dilbert cartoons, drink coffee and play Sudoku, when they aren't taking their kids to soccer or teaching acne faced kids about Maxwell's Law, dot products, and torque. Living in university towns from the time I was seven until the time I was twenty-four, with only a one year interlude in between, and having taken college level physics classes in two different universities, I know these people.
I preface what I am about to say with the ad hominem comments above, because most theoretical physics papers (maybe 90%-95%) are truly wacky by the standards of almost any other discipline in the sciences.
A theoretical biology paper might propose that New World fauna hung out on the Bering Land Bridge for twenty-thousand years before dispersing, rather than dispersing immediately upon crossing it.
In contrast, in theoretical physics, far more than a majority propose new dimensions (several numbers are popular but there is no consensus), new kinds of particles (often dozens) and/or new fundamental forces for which there is virtually no empirical evidence, the vast majority of which have no significant support outside a small clutch of fellow travelers in the theoretical physical community who are a vanishingly small minority within the whole. Many argue from the "antropomorphic principle" which basically says that the world is the way that it is because we live in it, so it must be that way. A small percentage of the more grounded articles discuss experiments which could be used to prove one or the other of these theories, or "no go" theorems that rule out entire classes of these ideas, which are generally ignored by the people who propound them in the first place. Sometimes, several journal articles will engage in dialog which basically consists of each side explaining to the other why their proposal is baloney due to its lack of rigorous mathematics or contradiction of existing experiments, which usually receives a response in kind along the lines of a statement that the critics didn't actually understand the paper that they are criticizing.
Unlike articles by lay people, the vast majority of these wacky theories are (1) theoretically possible given current experimental knowledge, (2) toy models that illustrate a concept to be used later that don't purport to have physical reality, or (3) incapable of being applied in a sufficiently definite matter to make predictions that can be proven or disproven at this time.
Also unlike articles written by lay people, most of them use the accepted mathematical language of physics, cite to existing work in the field, and are variations on a theme of one of dozens of recurring approaches to reconciling the unsolved problems of fundamental physics. For example, many use some form of Lie algebra to establish a set of equations similar to, but not identical to the core equations of quantum mechanics used in connection with the standard model of particle physics.
Furthermore, unlike lay people, the authors generally understand, although they don't usually admit it in print, that it is highly likely that the precise wacky theory that they are presenting is probably not the way that the universe really works, and are simply offering trial balloons to explore various approaches to getting a final solution that could go somewhere when additional data starts to favor that general approach.
It is half theology, half role playing games for people who are really, really good at math and physics, and half whimsy. Somewhere buried in there is a sincere and widely shared desire to get to the bottom of the question, but that desire is stymied, among all but the very most prestigious among them who may be brilliant but may simply have delusions of grandeur, by the tacit recognition that we are stuck in the process of solving the puzzle and can't get unstuck until we have more very expensive to obtain data to clear out the ticket of unworkable theories.
This isn't restricted to the oft maligned string theorists. Essentially, it is a disease of the entire endeavor. And, a break though would be an amazing wonderful thing. But, up close in this part of science, it is hard to herald the actual process by which new scientific knowledge is generated as anything much more sophisticated than the room full of monkeys with typewriters trying to write Shakespear.