Over the last few years, I’ve seen lots of work dismissed if not peer-reviewed. It is touted as the gold standard of quality research. Work that has not been peer-reviewed or ‘failed’ the peer review process (for whatever reason) is dismissed as irrelevant. Is that warranted? What is peer review – and what is it not?
Broadly, peer-review is the concept by which science is supposed to be validated. When a researcher has an hypothesis or has conducted an experiment, it is, in theory, submitted to his peers – other scientists of equal, lessor, or greater expertise in the same field of endeavor – for review prior to being unleashed on the scientific community or public at large.
Peer review as generally understood today is a relatively recent phenomenon, really only being formalized in the 70s. In the past, the scientific community tended to be smaller and it’s practitioners more known to each other; lots of institutions would self-publish rather than submit to large external journals. In a sense, an informal peer review would occur via personal communications followed by a public review after publishing (or response/rebuttal papers in the literature). Peer review as it generally operates today – conduct research/theorize, write-up, submit to external journal, journal distributes to anonymous ‘referees’ (peer reviewers) who judge the work worthy and recommend that the journal publish (or not) – was only really codified as part of the process of scientific publication in the late 60s/early 70s.
In the abstract, what are the pros and cons.
Pros:
- In principle, guarantees the quality of the research, verifies it was well done and ‘truthy’ – e.g. maybe wrong, but basically well done and grounded in solid scientific principles. It has, essentially, a gatekeeper roll, keeping egregiously wrong (in a process sense) and potentially malevolent research from being adopted.
That’s really it for pros we might generally consider ‘pure’ (i.e. some people involved might considers cons below to be pros…)
Cons
- Slows down the release of research. Depending on the field, can take up to a year to publish – this is not only peer review, but post peer review prior to publication as well. Pre-print servers mitigate the publication delays since many people now submit articles to the pre-print server as soon as accepted and often before.
- Can be abused – e.g. reviewer can delay review if they are working on something similar (or steal the idea), or reject downgrade research that competes with the reviewers or has a different/opposite position. Can be done consciously or sub-consciously (i.e. steeped so much in their view that contrary ideas are outside of valid inquiry – smart scientists are not generally terribly humble, though the brilliant ones often are).
- Provides a bottleneck where interested institutions can prevent the airing of certain ideas – not that I would suggest certain government institutions did so during Covid, no sir.
- Main one, at least to me, is that it is used as an imprimatur of ‘correctness’ – i.e. “that’s not peer-reviewed, you can dismiss it”. Coupled with 2 and 3, it can be used to steer scientific research and ‘consensus’ in a preferred direction, regardless of correctness.
I haven’t really seen 2, but maybe it’s more prevalent in fields that are more financially important – e.g. medical. Of course, in the age of heavily state subsidized scientific inquiry everyone has a financial interest, even if just welfare for the overly credentialed class.
So given the ‘single’ pro, at least in my formulation, does it fulfill at least that? Clearly it’s not perfect (e.g. the deliberate fabrication of the amyloid plaque research – I assure you that happens if not always quite that impactful), but we really can’t require that any human endeavor be flawless. The question is does it fulfill it in general better than an alternative? My position is no, largely due to the Cons, normally expected to be considered flaws to be corrected, becoming the norm. This is due in large part to the enormous amounts of money that permeate scientific fields in the age of federal funding of science. There are a lot of people making a lot of money, not only in the mere act of publishing (think editors, printing, advertising – at least in market-lucrative fields), but also for researchers.
To keep this shorter (yeah right), I’ll not go off into things like the medical literature where I think there’s a lot of deliberate manipulation of the process to make sure state and industry funding continues to flow the right people – industry, federal bureaucrats, and universities. With the later, the researchers/professoriate at the university know what’s important – keep the money flowing to the university. There’s an army of mid-wit bureaucrats (that in general far outnumber actual no-shit scientists) to keep employed and prestige to be mined.
In smaller, less lucrative fields – at least less lucrative in general as there aren’t the main source of filthy lucre for large companies, they are very lucrative for individual researchers and institutions – the failure of peer review is largely manifest in the opposite way. Too much sub-par research is published, owing to the publish-or-perish sort of mentality. Rigorous peer review imparted a degree of gravitas to papers and the authors of those papers – sort of like smaller versions of dissertations. That meant that hiring and tenure decisions started becoming based on how many ‘peer-reviewed’ papers you have. Whether one is awarded grants will depend on how many ‘peer-reviewed’ papers you have (and peer review is also used in the awarding of grants directly).
On the face of it, these are good things(tm) – if rigorous peer review is weeding out lower caliber scientists, that’s who you want to hire. You want to continue funding people who are producing that high quality work, not ‘wasting’ it. But that also puts the wrong incentives in place. For an individual researcher tagged with reviewing a manuscript, the might not want to hurt a young researchers prospects; and their students – and hence they themselves – might benefit from making sure things are published. The journal editors don’t benefit from restricting publishing – they make their money from lots of papers being published. Universities certainly don’t want to limit the publishing or grant awarding. Their reputation and, more importantly funding, depend on having lots of grants flowing to the institution and producing lots of newly minted scientists generates its own demands for more funding. So everyone along the whole chain of the conduct of research has an incentive to insure that things get published as that will ensure the continued flow of funding and the expansion of their bureaucracy. Over time, peer-review is no longer ensuring research quality, but often times quite the opposite. The prestige and respect afforded to rigorous, high quality peer review incentivizes making it, intentionally or otherwise, the exact opposite of what gave it that prestige and respect in the first place.
Is there a solution? Personally, I think the ubiquity of the internet and cheap storage will eventually destroy the current model of peer-review. The barriers to publication are too low to continue the gate-keeping role of peer review for the benefit of journals, institutions, and established researchers (and the armies of bureaucrats ‘necessary’ to keep it all running). Not quickly or easily as there’s way too much money to be laundered for everyone involved, but it will happen, especially as the it becomes more apparent that peer review no longer functions the way the average person things it does, giving an unearned imprimatur of rigor. People can publish what ever they want on the myriad of pre-print servers these days – lots of garbage will get out as it does under the current system, but there are also armies of individuals ready to tear the garbage down, a scientific ‘community-notes’ of sorts. The current gate-keepers and their armies of bureaucrats will not go away quietly or concede, but will, I hope anyway, become increasingly irrelevant.
Summary – stop looking for the “peer-reviewed” label. If someone dismisses research because is it not peer-reviewed, or advances other research because it is, be cautious. The words ‘peer review’ no longer, if they every truly did, have the meaning you think they do.
In principle, guarantees the quality of the research, verifies it was well done and ‘truthy’ – e.g. maybe wrong, but basically well done and grounded in solid scientific principles. It has, essentially, a gatekeeper roll, keeping egregiously wrong (in a process sense) and potentially malevolent research from being adopted.
I don’t see “show the results can be reproduced” in there.
I don’t think that’s really within the purview of peer review. It would be virtually impossible for anything beyond the most trivial research for peer review to demonstrate reproducibility. Ideally, verifying ‘truthiness’ should ensure that the research is reproducible or falsifiable – reproducibility should be a downstream outcome of the peer review process.
The fact that, when checked, such a large fraction of research is not reproducible, means that a lot of research is poorly done and peer review is not catching the low quality process – largely, IMHO, due to the incentives touched on below.
“‘truthiness’ should ensure that the research is reproducible or falsifiable”
I cannot warn strongly enough that ‘truthiness’ in itself is a very good reason to set aside everything and start again from scratch. It is a warning that something aint right. I might here alter the iron law ‘if it seems too good to be true, it is.”
The reason for that give me an idea for another brain fart ‘article’.
“Experts agree!” should be a giant red flag, at this point.
^^
Yes.
At the very least, it should occasion the question, “Who are these experts, and more importantly, why are they experts?”
Off the horoscope thread: AAA arrived at 4:30 yesterday. Even with the powerpack plugged in starter still cranked over slowly and wouldn’t start engine although voltmeter read 12.3 volts. Road mechanic got in and pumped the gas while cranking and Brrmm! with large cloud of white vapor coming out the tailpipe, which continued to billow out. Mechanic said water got into the gas lines, had seen it several times recently, and would burn out of the fuel in 10-20 minutes. Not a problem as I had a short road trip to go on, which I promptly started out on.
I last fueled at Wawa (my regular fuel stop) in DE just across the state line. I hope they don’t have leaky tanks.
All’s well that ends well?
Sauruman: You wouldnt happen so know if gas tanks draw from the top or bottom would you? The tanks at the station I mean. I suddenly realize that is something I dont know.
For some reason, the Great Barrington Declaration and the fallout therefrom comes to mind…
In any such case I just mentally flip in “priests” for “experts” and then finish evaluation. Probably not surprising how little passes that filter.
Relatedly, you can keep the same truth value by substituting in “God” for “Society” or “The Greater Good.” Any abstract entity that nevertheless has needs/desires/health actually.
“Any abstract entity that nevertheless has needs/desires/health ”
I alone know what those needs/desires/health are. See, I am an expertologist myself. I bet y’all didnt know that.
Peer review seems like a way to institutionalize the dynamic that science advances one funeral at a time. “Let’s give the old guys a blocking position on new thinking! That’ll work out just great.”
Thanks Mr. Meat. An excellent essay.
People can publish what ever they want on the myriad of pre-print servers these days – lots of garbage will get out as it does under the current system, but there are also armies of individuals ready to tear the garbage down, a scientific ‘community-notes’ of sorts.
This happens pretty frequently already in the health and nutrition space. Most people realize that the science is garbage and it’s quite literally back to the drawing board time. It is interesting how quickly dumb ideas get exposed and challenged. A very good thing.
I can’t tell you how much I sensibly chuckled at you calling him ‘Mr. Meat’.
Mr. Meat
That was, after all, my nickname in high school amongst the ladies.
Not really (as is that disclaimer is really necessary), but leave an old man to his false memories of past greatness.
Hopefully not ‘Mr. PutridMeat’ though. That would be a tough one to live with.
Was that image from Despair.com?
Yes. I should have linked it and credited the original source. As a competent peer-reviewer, you have pointed out that lack of scientific rigor. The oversight will be corrected in proofs prior to publication.
I’ll bet the food pyramid was peer reviewed.
Excellent article, it’s amazing how much rot the internet has exposed in everything.
The fact that, when checked, such a large fraction of research is not reproducible, means that a lot of research is poorly done and peer review is not catching the low quality process – largely, IMHO, due to the incentives touched on below.
You obviously know vastly more than I. I get the impression a lot of “scientific revelations” are based on intentionally opaque methodology.
intentionally opaque methodology
I think that happens quite often, but the reason for ‘intentionally’ can be very different.
There’s the obvious financial incentive in certain fields – all really, but just at a different scale. There’s often very well conducted, rigorous studies that, with the right metrics and/or methodology, will give you the ‘right’ answer. But you can obfuscate the ‘right’ answer behind methodology and hiding the data so the right metrics and methodology can’t be applied. Not saying covid and statins and various nutritional research fits that, no sir.
Somewhat financially related too – with huge amounts of federal funding available for research, institutions obviously expanded the amount of research being done to attract more of those dollars. That naturally exploded the number of people – students, faculty, researchers – in all kinds of disciplines. As a result, you sample deeper in the bulk of the distribution of peoples abilities – that will inevitably dilute the quality of the work. Opaque methodology can obscure the lack of rigor, both intentionally and subconsciously.
Some of the practitioners suffer, deservedly, “imposter syndrome”, and cover that with intentionally obtuse analysis and writing.
Some of the practitioners don’t understand the black box they are applying and whether a give method is even appropriate for their case. But can write down Bayes’ equation in a paper, produce a bunch of corner plots, and assign sciency error bars that account for your priors.
With all the incentives in place, both will get through the peer review process.
Another worthwhile substack.
Thanks, that looks very interesting, if only in another creative way to avoid the work that, in theory, I should be doing for this whole paycheck thing.
PS – I giggled more than I should at the “My correlation is causation because my P is wee”.
One that could short cut some of this would be to provide the data set used in the research. I realize that sometimes isn’t possible for a variety of reasons including confidentiality.
Speaking of peer reviewed research that has an agenda…
Journal that published faulty black plastic study removed from science index
https://arstechnica.com/health/2024/12/journal-that-published-faulty-black-plastic-study-removed-from-science-index/
I find it amazing that the sacred “peer review” process managed to miss a calculation error that is central to the conclusion.
If your dataset isn’t big enough to obscure the individuals in the data set, then your likely dataset isn’t big enough to support whatever conclusions you are drawing.
The author of the arstechnica piece is obviously racist. Criticizing the conclusions of a POC! The horror!
The author of the arstechnica piece is obviously racist. Criticizing the conclusions of a POC! The horror!
A Plastic of Color?
So, all government edicts can be dismissed out of hand? Today’s Biden dump of off shore drilling is because “some people don’t like it”
Well, that’s good enough for me.
Same as the NIMBY response here to any sort of mining.
Great article. Spot on about the peer review process. It’s been interesting to see how the internet has opened up the publishing space now. There’s a lot of junk publishers out there, but also some solid startups that are shifting the industry.
I haven’t really seen 2, but maybe it’s more prevalent in fields that are more financially important – e.g. medical
Not often, but I run into this every now and then with smaller medical niche areas. I remember one unhinged review from about a decade ago where it was clear the reviewer must have worked for a competitor. That’s not necessarily bad if their expertise can provide a helpful review, but this one let their conflict of interest get the better of them.
Peer review as generally understood today is a relatively recent phenomenon, really only being formalized in the 70s. In the past, the scientific community tended to be smaller and it’s practitioners more known to each other; lots of institutions would self-publish rather than submit to large external journals.
Along these lines, the Royal Society of London posted their proceedings online starting from the 1830s.
https://royalsocietypublishing.org/loi/rspl/group/c1800.d1830.y1832
You can directly read submissions from Tesla, Faraday, Darwin, et al. A different time when scientists were also philosophers.
You can’t fool me. There wasn’t an internet till at least 1850.
When the automobile was first introduced, there were any number of car makers, most of which didn’t survive.
We are seeing something similar with online publishing. My best guess is there will be some sort of consolidation, if only by editor, to allow people to make educated guesses as too the quality of the writing.
So, all government edicts can be dismissed out of hand? Today’s Biden dump of off shore drilling is because “some people don’t like it”
Those decisions were based on rigorous scientific analysis!
Speaking “studies” and bogus math and science.
Sugary drinks may cause more than 330,000 deaths a year, study suggests
https://www.cnn.com/2025/01/06/health/sugary-drinks-deaths-wellness/index.html
Now let’s see how Congress deals with corn subsidies in the face of this incontrovertible new science!
I’m too lazy to look it up, but I think HFCS is used mostly because the US punishes sucrose imports. There may well be actual corn subsidies on that as well, but I think the biggest issue is with sugar imports.
I think the corn subsidies are the culprit. Subsidies simply make it super cheap for manufacturers to use it, thereby creating the deadly combo of hyperpalatability and low cost.
There is also a price floor for sugar that makes it more expensive to use American sugar.
Google confirms that corn is heavily subsidized. Which makes HFCS heavily subsidized. And sugar imports are covered by tariffs.
It’s almost like corn producers control congress.
Why else would we burn the shit in our cars?
Google confirms that corn is heavily subsidized. Which makes HFCS heavily subsidized.
It’s like ultra-processed tax payer funds.
Or they might not.
In re: sugary drinks, if it’s acidic, you’re not going to have disaccharides in it, regardless of the sweetener originally used.
https://www.youtube.com/watch?v=NY66qpMFOYo
Incredibly well-done and much appreciated. Not *scientific* peer review, but I wish folk could identify(!!) themselves as ‘subjects’ to try out meds, etc on their own. The laws restricting people from ‘testing’ (still) experimental meds have prevented millions of people from getting treatments that would’ve ‘saved’ them from various maladies. I wish we could accurately quantify it. (Can/have we?)
Shingrix: The ‘vaccine’ for shingles. “Only for those above 50.” I wasn’t 50 when *I* got it. I’d STRONGLY prefer to not have it again. Why can’t I just try it out? Caveat emptor, and all. Outright poisonings, etc – those are already crimes. Just prosecute accordingly. Top-Down Authority ruling over individuals’ health doesn’t spark the anger it should, mostly IMO cuz the numerous, opaque layers between patient and actual treatment makes folk blind. Dammit.
(Shingles sucks. Desperately. I strongly recommend avoiding it! Apparently I can get it *again* and I couldn’t get Shingrix even if I wanted to cuz I’m 13 years too young. Shingles legit angers me.)
I agree with you, the same thing happened to me. On the bright side, there’s usually a multi-decade gap between outbreaks.
Eee! I *got* it when I was 18 and I *turn* 38 in four months! My multi-decadal cherry’s gonna pop soon!!
*swoon*
O/T vent/rant: Most of the counties here in SW OH are under a Level 2 snow emergency (out of 3,) so my employer put out the word that we would delay opening until noon, with employees to report at 11:30 a.m. Thus I bundled up, shoveled and de-iced the front walk for the mail carrier’s sake, then cleaned the mountain of snow off my eternally-parked-outdoors car as best I could (while the snow continued to fall.) I rousted TT out of a comfortable sleep to start clearing the driveway with our snow blower while I cleaned up and got ready for work.
It usually takes me about half an hour to drive across town to my office, and assuming I’d need extra time, I left at about 10:45 a.m….just as a follow-up announcement was sent saying we were staying closed for the day. As the roads were still quite treacherous, I, of course, directed all my attention to my driving and none to my phone, as one does (or should do.) After great difficulty, I arrived at the office to find no one there. I stopped the car and THEN found the message about the closing. 😒
To make this rant marginally on topic, I’ll just say this: I believe that under the circumstances, were I to kill my boss for taking so long to announce that we were closed after all, I would not be convicted by a jury of my peers.\vent/rant
I’m in my second year of retirement and I’m still getting DOD notices about the status of Aberdeen Proving Grounds.
You can disable those messages or get out of the notification system.
Can I disable the notification system?
You can disable those messages or get out of the notification system.
No CAC, no access.
Call the helpdesk and complain until they send your ticket to the wrong department.
I’m waiting for the lingering part of the storm so I can go dig out for tomorrow morning.
Looks to be just over a foot in total. Little dog hates it. Older dog ventured the backyard.
I don’t want to jinx it but damn we have mild winters here. There are times I miss the insane amounts of snow and ridiculous temps.
Oh wait, no I don’t. I’m officially a weather pussy.
We got a different storm than you.
It’s been *very* underwhelming here in SE PA.
All the snow got left here.
Some of it landed in Maryland.
I’m hearing complaint about how their snowless paradise got a foot of snow when Albany NY is dry.
It was definitely worse to the south of here – more snow and areas that got ice. (OBE, did the ice make it up to NKY?)
KC got hammered: https://weather.com/storms/winter/video/kansas-city-buried-by-heaviest-snowstorm-in-years
We got a pointless dusting. The wind was unpleasant though.
I had some excellent shotgun runs though. If I’m ever assaulted by four dudes standing should to shoulder I can neutralize them in under 2 seconds (assuming I have my loaded shotgun at hand).
Speaking of “experts”…
But how do you classify flaming homeless people in your year over year statistics? Arson, murder, vandalism or something else?
Disorderly conduct.
In this website, we believe the Science™️.
SCIENCE!
https://www.youtube.com/watch?v=V83JR2IoI8k&ab_channel=ThomasDolby
there are also armies of individuals ready to tear the garbage down, a scientific ‘community-notes’ of sorts.
Until we get an echo chamber.
Get an echo chamber.
Echo chamber.
Very true. In fact, one might think of the current state of peer review as the evolution of a decent system into an echo chamber controlled by, if not top down/centrally, then at least the incentive structures.
I think that devolution is mitigated the larger the chamber, hence hope for a (free) internet. Of course there-in lies the rub – even some hosting in the pre-print space started getting pressure to remove material during covid. If peer-review became more effective in a more open free-for-all process, those pressures would mount and the re-capture would proceed apace.
Eternal vigilance and all that.
Of course publication is most important to academics because it is easier than pulling in grant money.
And what will break publication isn’t just the alternative venues (upstarts versus establishment) but the decline of the institutions that live on publication, i.e. research universities.
Of course publication is most important to academics because it is easier than pulling in grant money.
“because it is easier” -> “because it is the route to”
For the researcher (and key to getting the next job/promotion) and the institution. Institutions hire people who publish because they are bringing in grant money. Not sure it’s a separable problem.
One that could short cut some of this would be to provide the data set used in the research. I realize that sometimes isn’t possible for a variety of reasons including confidentiality.
“Climate Science” makes perfect sense as long as you use their data and fudge factors without questioning them.
In many fields, there is a push to release all the data in raw form to, if not prevent the reproducibility problem, at least make it correctable.
Of course, they tried to make the covid gene-therapy studies private for 75 years. Make what you will of that.
Of course, generally climate research doesn’t provide raw data – it needs to be corrected, otherwise you might not draw the right conclusions. In some sense, that’s very true. Raw data can be, at best useless, often misleading. But the answer is not to only release your processed data but to also release the raw data and the exact methodology you used to produce your processed data.
At this point, I doubt there’s much uncorrupted climate data left.
RC – Not with how the temperature monitoring stations ahve been sited.
But how do you classify flaming homeless people in your year over year statistics? Arson, murder, vandalism or something else?
“Didn’t you see the NO SMOKING sign?”
Electoral vote certified. Congress failed to protect Democracy. Let the wailing continue.
Don’t recall who else here is a John Ringo fan. I subbed to his substack a while back, but didn’t notice he’d started a new Troy novel. He posts one chapter at a time every week or so (maybe a little faster). (I need to go back and see what his last series was…I’ve been a little distracted).
This is basically Troy Rising book 4 – starting immediately where book 3 ended. Pretty gripping so far, and great to see him return to the setting.
I am. Thanks! I’ll be checking it out.
Still waiting for more in the Posleen universe.
I s he actually actively writing it? When he posted the initial excerpt he said he had found an old file he had lost back when he was writing Troy and was posting that. I was hoping it would inspire him to write more. I am rereading Kratman’s Carreraverse novels, because I needed a little crucifying the media porn.
Appears to be. It’s like the 4th or 5th project I’ve seen him post since I subbed, and it definitely seems like an active project the way he described it in ch1.
Foreword/prologue was Dec 11 and he’s up to Ch 9 this morning.
Safety netting for our most vulnerable
President Joe Biden on Sunday signed the Social Security Fairness Act, bipartisan legislation that clears the way for teachers, firefighters, policeman and other public sector workers who also receive pension income to receive increases in their Social Security benefits.
The benefit boost comes as the new law repeals two provisions — the Windfall Elimination Provision, or WEP, and the Government Pension Offset, or GPO — that have been in place for more than four decades.
The WEP reduces Social Security benefits for individuals who receive pension or disability benefits from employment where Social Security payroll taxes were not withheld. As of December 2023, that provision affected about 2 million Social Security beneficiaries.
Reward their service. Let them live with dignity.
Am I reading that right?
Their money wasn’t stolen for SS, but now they are going to get paid SS?
Can I stop paying in?
At this point, I could not care less about them trickling some money back to me in 12 years.
You are reading it correctly… They didn’t pay in (asked for an exemption since they had a GOVT pension and they wouldn’t have to rely on SS) That “not paid in money” wasn’t counted towards their SS. and now, that is somehow wrong. That this got through is a travesty.. perhaps the calculations were poor and could be updated, but removing them entirely?.. No.
The details are always, well you worked with SS deducted for Y years.. but had Z years of a govt pension… how do you calculate the SS payout.. that was always biased to boost “poor payees” without making that obvious.
We discussed it a bit a few weeks ago. At the time they didn’t want it…but now that would be unfair….
https://www.nationalreview.com/2024/12/both-parties-are-driving-the-runaway-social-security-insolvency-train/
Ideas so good, they have to be mandatory.
https://www.teepublic.com/t-shirt/4795419-the-government-ideas-so-good-they-have-to-be-manda
No, it’s the other way around. They’re no longer penalized in their earned SS benefits due to calculating non-SS benefits.
The quality of their shirts and printing has gone way down hill.
I blame Covid. And the commies.
Make them take a covid booster every month to get those benefits.
Yay for double-dippers and Democrats flipping on their own law against that.
Double dippers in the sense that they had to earn twice.
SS is a crock of shit. Arguing over what shit stinks less.
“With the repeal of WEP and GPO, federal retirees, along with so many others, will finally receive the full Social Security benefits they’ve earned,” William Shackelford, president of the National Active and Retired Federal Employees Association, said in a statement.
Those poor government employees, working for poverty wages all those years.
Ummm…
This is really cool. I just don’t have any place to keep one.
https://x.com/gunsnrosesgirl3/status/1874022458992582810
Remember, a chinese knockoff air carrier is a forever pet, not just for christmas.
Get the mini version.
Oh Lord. That looks like tedium and frustration all rolled into one.
Not just no, but hell no.
No worries. The tweezers are included!
How about the free glasses for myopia?
don’t have any place to keep one
You don’t have a moat?
Speaking about peers getting a spanking. Even The Nation is unimpressed.
Support for Sanders is massively overstated. He was the only non-Hillary candidate that anybody had heard of on the Team Blue side, and a lot of Dems tossed him a primary vote as a middle finger to the party.
Damn you. I read “Evan The Nation” and got the giddies. Then I was equally unimpressed to discover “Joe Biden Loves Awarding Participation Trophies to the Failed Establishment.”
Evan’s nation has no participation awards (especially “I VOTED” stickers); Coca-Cola again has a tenth of a gram of purity; Shave-and-a-Haircut is the national anthem; and a slew of other bullet points you can all unlock through middling (to generous) donations to my righteousness. So there.
I had some excellent shotgun runs though. If I’m ever assaulted by four dudes standing should to shoulder I can neutralize them in under 2 seconds (assuming I have my loaded shotgun at hand).
As long as they don’t run away (or shoot back).
Or have body armor.
These assailants were made of steel. The shot just dusted off.
Not Adahn is unprepared for the Robot uprising.
The plates are face-sized.
“Or have body armor.”
That’s why you practice headshots.
How far can they get in 1.74 seconds?
My apologies. This is exactly the sort of articles we need more of and of intense interest to me. I missed it due to unforeseen circumstances. goddammit.
Thank you PUTRIDMEAT. You are a credit to our company.
My comment: Everything you said is correct but none of it would matter if not for this one factor – “…in the age of heavily state subsidized scientific inquiry everyone has a financial interest…” and “…the enormous amounts of money that permeate scientific fields in the age of federal funding of science.”
That instantly turns everything political and ‘not science’. Scientists, like lawyers and news commentators, say what they are paid to say by people who have agendas unrelated to their respective fields. It is poison.
*disclaimer: lawyers saying what they are paid to say is not a bad thing. They are key to a functioning adversarial system. Science is supposed to be an adversarial process. Paying scientists has the opposite effect as it does with lawyers.
Mr. Meat, thank you for the article on Peer Review, but I’m going to be “that guy” and completely disagree with a central premise of your article.
“Broadly, peer-review is the concept by which science is supposed to be validated.”
This is incorrect. Science is validated – and differentiated from non-science – solely by the predictive power of its models. Scientific models are graded by the strength and scope of their predictive power: from conjecture, to hypothesis, to theory, and then to Law. Predictive power is the sole determinant of a model’s validity, not whether or not your paper was viewed favorably by a committee and deemed publication worthy. The demarcation b/w astronomy and astrology is not “peer review” – it’s the difference between Jeanne Dixon saying Aries make better lovers and Einstein saying “there will be an eclipse at X time on X date” and besting the prior Newtonian model in accuracy.
FWIW, someone linked William Briggs’ substack above and his book “Uncertainty” is worth a read by everyone, alleged scientists and lay people alike (additionally, I can assure you because I know him in meatspace that Briggs would tell you the same thing I just wrote above). Publication, in its truest sense, is nothing more than the act of “making public” one’s claims. The scientific publication industry has turned itself into the gatekeepers of what is deemed “science” by their monopoly over scientific journals and in my day job I’ve been approached by more than one scientist about suing the big publishers, who are as criminal as any other cartel.
The replication crisis and demarcation problem are both a direct result of Karl Popper and his intellectual heirs (Kuhn, Lakatos, and Feyerabend) and their denial of induction, which is at the heart of science. Hume’s inductive skepticism reared its ugly head and led to Popper’s (and Fisher’s) insistence that science be deductive, which is (IMO) a form of mental illness. It is out of this muck that the useless “p-values” came, where Fisher (also a deductivist) claimed that the “probability of the data” was what mattered, rather than the probability of the hypothesis. If you want to read a proof of the error of Hume’s inductive skepticism, it’s spelled out brilliantly by David Stove in “Popper and After: Four Modern Irrationalists.”
I could talk about this ceaselessly, but I had to correct you on peer review as being how science is validated. If this were true, then how do we explain advances in (classified) weapons systems or Elon catching a return rocket? SpaceX published nothing for “peer review” because Elon knows that science is validated when the rocketship returns on course, on time, and has a successful catch. THAT’S validation, not the opinions of academics beholden to a corrupt publishing industry.
Ozy, you are talking about models as if the (sole?) purpose of science is to produce predictive models. I had thought science was supposed to be explanatory, with predictive power a possible bonus, depending.
The classic scientific process isn’t model building, it is experimental testing of hypotheses. A hypothesis is a proposed explanation of an observed phenomenon. The experiment gathers data (inherently backward looking), and a successful experiment validates the hypothesis and should be replicable. The first experiment isn’t really even a prediction of what the replication will find. Now, eventually experiments may cohere into a Theory of X, which may show some ability to predict, but even if they don’t, they are still science.
thought science was supposed to be explanatory, with predictive power a possible bonus
I think it has to be more than a ‘bonus’. Without predictive power, a hypothesis remains a hypothesis. Think something like string theory – makes no falsifiable predictions but can, post-hoc, provide a bunch of explanatory power. In the old days, the same could be said about epicycles. With every new observation that was not predicted, even if the hypothesis can be retro-fitted to incorporate it, it is less likely to be a correct description. With every observation it predicts a priori, it becomes more likely to be a correct description.
At some level it’s a matter of not being a hard and fast rule. Newtonian dynamics was a ‘validated’ theory, making predictions about the motion of objects to very high accuracy. Until it didn’t. Does that make it not science? It’s still makes the engineering feat of SpaceX capturing its rocket possible (I don’t think they include general relativistic equations of motion in engineering the capture). I don’t think science should be thought of as an answer, but rather a process. And the process that generates tests of a hypothesis or theory is, to me, just as much science as the process that generates the hypothesis in the first place – your explanatory point. But without prediction and falsifiability, you can’t treat an idea as a scientific hypothesis or theory.
RC – Explanation is in the eye of the beholder. It is a “hunch” about causation, and can provide *some* useful information for subsequent scientists, but no, explanation is NOT the domain of objective science.
PM – I know what people are getting at when they point to peer review, but “peer review” itself proves and validates nothing. It is, in my opinion, nothing more than the academe’s attempt to (yet again) claim that what they do in the halls of the theoretical IS “the science.” Peer review (like p-values) is another swing and miss. Peer review tells us nothing about how much closer to the “truth” we are – probability theory (and a proper philosophy of science) do.
In a dead thread, but it will help me collect my thoughts at least!
This is incorrect
I agree. That was very imprecise language on my part – which can be part of the problem. Imprecision that led to the concept being incorrect. I don’t think it invalidates the rest of the discussion, but it is an incorrect definition of peer-review. And I wouldn’t call it the central premise of the article either.
1) I think what I was trying to get across is that peer-review, in theory, is supposed to validate the process used and presentation of, the research being reviewed. Have the authors laid out their data clearly, how it was collected, sources of error? Have they thoroughly laid out their analysis approach and justified it? Have the quantified the uncertainties in both the data and analysis? Do their conclusions follow from the data and the analysis? Could a competent peer, given the data, apply the same analysis and arrive at the same conclusion? I think this idea is better captured in the single ‘pro’ statement as opposed to the statement you point out. The ‘pro’ makes no assertion (quite the opposite) that the research itself has been validated, only that it was done with a appropriate rigor, and deserves to be considered within the body of knowledge in that field. The actual validity will ONLY be established, hopefully to better and better precision, with repeated consistency in the face of new data and especially predictive power.
2) The main premise was that, if that is peer-review in theory, it is not that in practice, for the reasons cited in the article. And even further, most people do not even recognize the limitations of the pure, rigorous implementation of peer review. Rather, they take, not without some fault on the part of scientists themselves along with science reporters, peer-review to confer the status of ‘true’ to the conclusions. Additionally, the incentives around the institutions of most scientific inquiry essentially guarantee that the rigorous application of peer review outlined in 1 will not happen.