[_ Old Earth _] Peer Review in Science

[Mike]

A bit of a side bar conversation was being had in another thread that I'd like to discuss - the validity of Peer Reviews in science. The supporter(s) of it gave a vehement argument that this is a trustworthy method of testing research, but my experience has taught me otherwise.

After a degree in math and an MBA, I wound up in bio-tech research. Go figure... After 12 years of witnessing the fraudulent side of research by respected people in their chosen field of science, I had to get out of Dodge, and I'm glad I did. The weaknesses in peer reviews are as vast as the number of such "trustworthy" processes that have resulted in hidden emails, destroyed data and humiliated heads of research. It's not only that scientists can be motivated by the need for results to re-up their funding or the agenda of the financial sources. It's that research can be twisted. And the people who do the peer review are at the mercy of the research in the first place.

It is WAY too easy to hide entire segments of a study. The notion that uninvolved scientists can fully assess the processes and results of a study is laughable. I've seen this system break down and the owners of the research never own up. I've read about it a lot more. In fact, they proactively go lengths to obscure the truth. For science to hide behind this practice as proof that they have checks and balances in place to ensure the integrity of their research is to have no integrity in the first place. So much for "pure science" that attempts to be intellectually superior to theology.

 

[jasoncran]

mike, i for the life of my cant figure that out. i mean the belief that scientists as a whole are perfect and flawless. i worked with a former biologist and she distrusted medical labs. it was so easy to corrupt samples and data.

go figure.

 

[Mike]

Jason, I'm not saying they all are corrupt, and I'm sure you aren't either. But some are, and many are without even being conscious of it. Research does not pay well, but if your research is a success, there are often times great financial rewards. So what could possibly be the motivation of a researcher to skew the results in his own favor and have a study that supports his theory?

And if anyone doubts this, I say... read a news paper!! Pharmaceuticals alone! How many drugs have been pulled off the market when they found out they were dangerous. They pulled Vioxx off the market 10 years ago, and then it was revealed that certain cardiac events somehow didn't make the final print. Merck withheld data!... and then it was peer reviewed!... and then it was approved by the FDA... And these weren't corporate lackeys. The scientist in charge of these multinational arms of this research were supposedly some of the most sought after people in their field!

Now, that's a trustworthy system!

 

[reba]

Peer review is when the reporters/scientists/teacher/lawyers/ the blue collar guys at the bar,set around and tell each other how great they are.....


Christians get bashed all the time because we do disagree i believe we have a pretty good peer review maybe a model for some other groups.

 

[Barbarian]

A bit of a side bar conversation was being had in another thread that I'd like to discuss - the validity of Peer Reviews in science. The supporter(s) of it gave a vehement argument that this is a trustworthy method of testing research, but my experience has taught me otherwise.

After a degree in math and an MBA, I wound up in bio-tech research.

Odd, I never heard of a biotech researcher with an MBA. When I was doing my work in systems, from time to time, an MBA candidate would take a course on decision-making. We loved them, because they brought the curve down. It was always interesting listening to them. They argued that a good manager can manage anything, without knowing anything about it.

Much of the damage done to American business has been due to MBA programs at universities, turning out people who think they actually know something of value to business. Forty years ago, car companies were headed by engineers who actually knew what they were doing. Then MBAs moved in.

How's that working out? Read Lee Iacocca's Where Have All the Leaders Gone? to get the details.

Go figure... After 12 years of witnessing the fraudulent side of research by respected people in their chosen field of science, I had to get out of Dodge, and I'm glad I did.

So, how about naming some names and the circumstances? My guess is there isn't any.

The weaknesses in peer reviews are as vast as the number of such "trustworthy" processes that have resulted in hidden emails, destroyed data and humiliated heads of research.

E-mails are not part of research. You should know that. No one bases any work on emails.

It's not only that scientists can be motivated by the need for results to re-up their funding or the agenda of the financial sources. It's that research can be twisted. And the people who do the peer review are at the mercy of the research in the first place.

Give us an example of that. We know you want us to believe you. Give us some reason to do that.

 

[jasoncran]

Jason, I'm not saying they all are corrupt, and I'm sure you aren't either. But some are, and many are without even being conscious of it. Research does not pay well, but if your research is a success, there are often times great financial rewards. So what could possibly be the motivation of a researcher to skew the results in his own favor and have a study that supports his theory?

And if anyone doubts this, I say... read a news paper!! Pharmaceuticals alone! How many drugs have been pulled off the market when they found out they were dangerous. They pulled Vioxx off the market 10 years ago, and then it was revealed that certain cardiac events somehow didn't make the final print. Merck withheld data!... and then it was peer reviewed!... and then it was approved by the FDA... And these weren't corporate lackeys. The scientist in charge of these multinational arms of this research were supposedly some of the most sought after people in their field!

Now, that's a trustworthy system!

i am not, but yes i know that reaserch pays little. what barbarian doesnt know is that the person i mentioned was a REASEARCHER! she fish farmed and studied ways to bred bigger and better seafood for consumption.

these go out of business alot . she loved the work but theres no job security in it.

another friend who works for the state has said reasearch is about sales.

 

[Mike]

Odd, I never heard of a biotech researcher with an MBA. When I was doing my work in systems, from time to time, an MBA candidate would take a course on decision-making. We loved them, because they brought the curve down. It was always interesting listening to them. They argued that a good manager can manage anything, without knowing anything about it.

I worked in a research division alongside with the researchers to support statistical analysis with data reports. If your point here was to discredit people with MBA's, that's fine. But you don't have to convince me. Earning one is not difficult, nor is it evidence that one is any more capable of solving problems than when one starts. I will complete my process of becoming useless when I earn my black belt in Six Sigma.

My reason for giving these credentials was to point out the irony that I would find myself in the area of research. I don't make anything of it, and I think in my 18 months here, I've mentioned it one other time.

So, how about naming some names and the circumstances? My guess is there isn't any.

I'm not anxious to drop names. I've managed to respect the confidentiality forms that I signed when I left for the past 6 years. So, you can accept what I say or believe that I completely fabricated it. It sounds like you've already have your opinion. No matter. I'm over it.

The point of the OP was to focus on whether Peer Reviews are actually an effective way to eliminate fraud or research processes. Peer Review was said to be evidence that scientists can wean out biased studies, whether intentional or not. I'm quite sure it isn't.

Take the Vioxx scandal for instance, since it was pretty much the tsunami of its industry. The Cox-2 class of NSAIDS was supposed to usher in a new and wonderful day for arthritis sufferers. Finally, the Cox-1 could do its job in relieving inflammation while the Cox-2 would be blocked and patients would suffer less ulcers. This was to be the turning point for NSAIDS. Of course time would reveal that this class had an insidious effect on the heart. At higher doses, especially, Vioxx showed evidence of cardiovascular effects. Participants in the study had an inordinate rate of events.

But no one ever knew it, because something curious happened on the way to paradise for Merck. Many of the cardiovascular events didn't make the final reports that their peers studied and the FDA reviewed. Now, if Peer Review is so effective in eliminating things like this, why was this not caught? I don't blame the people who reviewed the trials, because how could they have known? But this example exposes the limitations of Peer Review.

E-mails are not part of research. You should know that. No one bases any work on emails.

I realize emails aren't research. My point was that this was the response of heads of companies after their unethical decisions were exposed.

Give us an example of that. We know you want us to believe you. Give us some reason to do that.

Done. If the breakdown of the process in Vioxx-Gate doesn't convince you, nothing I say will. We know Merck is not the only pharma company that incurred huge fines for dishonest data that wasn't exposed with PR.

Regards, Barbarian.

 

[Barbarian]

Clearly, that was dishonest on the part of Merck. And I was unnecessarily dismissive of MBAs.

But they did get caught. Outright fraud in data often won't get caught by peer review, but there's the next step. When subsequent studies don't match the claims of the original paper, someone is in trouble.

In this case Merck had enough money invested in the drug that someone up the line decided that maybe the data wasn't as bad as the researchers said it was, and changed it. And the subsequent exposure cost a lot of money.

Hard lesson. This happened with climate reports under a previous administration, where managers were permitted to go in and change the reports of scientists. Some people lost jobs over that, too.

 

[Rhea]

The point of the OP was to focus on whether Peer Reviews are actually an effective way to eliminate fraud or research processes. Peer Review was said to be evidence that scientists can wean out biased studies, whether intentional or not. I'm quite sure it isn't.

I would respond that it is not an effective way to eliminate fraud, but it is an effective way to minimize it, followed by the other parts of the process which were in the original thread, namely competitive re-tests, and consumer report re-tests.

I personally would add that for consumer protection risky industries, such as food and drug, I would like to see some government oversight that helps to minimize corporate greed fraud, such as government-sponsored test labs where a company can take their internal trial and apply for a gov't agency to complete a "public trial" to see if the product is safe.

When we know the potential biases, we can create steps to significantly mitigate them if we have the public will. Science does know HOW to do this, assuming we don't run into "close the FDA!" type lobbies.

The overall point being to separate the effects of the process from the opportunities for people to not do the process. The process of public trial, peer review and competitor retesting can have a significant effect on data integrity. The fact that businesses will lie and cheat does not impugn the review process, it impugns the oversight process. In other words, we know where to go to fix the fraud, and it's not by stopping peer review, but rather increasing it and putting in barriers to hidden data.

I would like to see the bill where a company can take a product that they have tested internally and are confident in its safety and efficacy, and submit it for public trial where the opportunity for fraud is greatly diminished. Current business lobbies HATE the idea of government safety oversight and they fight it tooth and nail, resulting in the Merck-type dangers that you have seen.

Again, it was not the peer-review that was the problem, it was the lack of it.

I realize it's tempting to say, "peer review is the biggest joke ever, and therefore we can throw out all scientific conclusions that we don't like (while trusting completely the ones that we do) and call ourselves logical." But that's not really logical, not repeatably reliable, not a safe avenue.

Are you all reallyarguing that peer review should be abandoned because it's the biggest joke ever? Think about what you're saying in this thread, "peer review is not realiable". The obvious conclusion then is, "let's not do it any more." Are you REALLY ready for that world? I'm not, I would completely lose faith in products without it at all.

 

[Barbarian]

Because peer review works, scientists would do it, regardless of who disapproved. It's going to happen, like it or not.

Other controls for things that affect the public are perfectly fine. Indeed, after peer review, the study still has to be reproducible, and if not, the researcher is in trouble.

 

[Mike]

Are you all reallyarguing that peer review should be abandoned because it's the biggest joke ever? Think about what you're saying in this thread, "peer review is not realiable". The obvious conclusion then is, "let's not do it any more." Are you REALLY ready for that world? I'm not, I would completely lose faith in products without it at all.

No, I was arguing that Peer Review does not eliminate bias from a trial or its administrators. This was offered as an example of what science does to govern itself with checks and balances.

They can do PR or not, but biased scientists and the tainted results they produce won't be impacted by it. Oh, and Vioxx had plenty of peer reviewed data. I'm not sure if you meant to imply that it didn't, but of course, any new drug facing the FDA is required to have it. The FDA doesn't determine if a drug can be sold in the public. Techinically, the only thing the FDA decides is if Medicare will reimburse for it. If it doesn't, no drug will survive. All commercial payors follow Medicare guidelines.

 

[jasoncran]

common sense, he is saying thats it not a perfect system.

sheesh, is it that hard to see that like all mens instution it isnt perfect, or does some of you believe that scientists dont die. lie, get thrown in jail and or have halos.

i believe mike would like to see reform. that would be good starting place.

 

[Rhea]

Are we arguing about the difference between "biggest joke ever" and "doesn't completely eliminate fraud"? or are we actually talking about the vast broad space between those two? The thread was started as an offshoot off a post by Reba that said peer review was "the biggest joke ever". And when people objected to that as hyperbolic and fanciful, the reply was, "well it doesn't completely eliminate fraud, now, does it!!"

'Cause I think no one is objecting to "doesn't completely eliminate fraud".
And quite a few are objecting to useless, biggest joke ever, doesn't prove anything.
But why jump straight to, "doesn't completely eliminate fraud"? It's a good system, one that is very effective. <== nothing in that statement claims that it is flawless.

I would love to see some reform in some areas, too, as I outlined above.


I don't care so much in the field of "effectiveness of deoderant" or "gas mileage of auto", those are satisfactory with the current peer review because the real contest is between competitors who will absolutely test each others' products.

But with food and Drug, I would very much like reform in the area of INCREASED peer review by, for example, mandatory external testing site before marketing to public.

Then the peer review of those tests would be more transparent and hence meaningful.

AS I said above.

 

[Rhea]

On reading Mike's claim that the FDA cannot prevent the marketing of a drug, I was surprised and went to see support for this statement. It does not seem accuarate, as the FDA seems to have the authority to "pull drugs from shelves" when they are found to be unsafe, as well as to prevent their sale if they have not yet been found to be safe.

FDA Basics

FDA is responsible for

• Protecting the public health by assuring that foods are safe, wholesome, sanitary and properly labeled; human and veterinary drugs, and vaccines and other biological products and medical devices intended for human use are safe and effective
• Protecting the public from electronic product radiation
• Assuring cosmetics and dietary supplements are safe and properly labeled
• Regulating tobacco products
• Advancing the public health by helping to speed product innovations
• Helping the public get the accurate science-based information they need to use medicines, devices, and foods to improve their health

 

[jasoncran]

mike correct me. i believe that you stated way back that some members of the fda also had conflicts of interest meaning they own stock in merck etc..

is this correct?

 

[Rhea]

If that's true, they should be prevented from owning stock to any significant value in any individual drug company. I believe there are blind funds that public officials can invest in to take part in the stock market without being aware of the particular companies they own.

 

[Mike]

On reading Mike's claim that the FDA cannot prevent the marketing of a drug, I was surprised and went to see support for this statement. It does not seem accuarate, as the FDA seems to have the authority to "pull drugs from shelves" when they are found to be unsafe, as well as to prevent their sale if they have not yet been found to be safe.

FDA Basics

The link didn't work when I went to click on it. I could be wrong, but this has always been my understanding. The approval of new drugs and the decision to pull them from the market are two different things. When the FDA approves a drug, they are approving that Medicare will reimburse for them. It's hopeless for a company that is not reimbursed by Medicare, because it's not only Medicare that pays or doesn't pay, but as I said, no commercial insurance company will pay for a drug that Medicare isn't. In a sense, they're off the hook.

Pulling a drug off the market is different, because this is a drastic decision to guard against the safety of patients. Like I said, I could be wrong...

Rhea, I didn't have in mind the two points that you did. Somewhere between "PR is a reliable methodology to guard against fraud or bias" (which I believe was your your point in the other thread) and "the biggest joke ever", which was reba's is...

It shouldn't be given nearly as much value as it is by the science community. It seems to be the standard that needs to be met. The FDA requires two parallel studies, and they have to be Peer Reviewed, as if this makes them legitimate. Studies can take place at sites throughout the world, and surely these are not all regulated as well as some. They are also multi-national. The variance is just too great and leaves for too much to question.

mike correct me. i believe that you stated way back that some members of the fda also had conflicts of interest meaning they own stock in merck etc..

is this correct?

I guess it wouldn't surprise me if they did, but I never said that. Just about everyone I know believes the FDA needs to be scrapped and replaced by a whole different system. Depending on the pressure they are under, they are either ultra-protective, or too anxious to usher something through. Vioxx was a landmark case that will be in text books for a long time to come. They rolled out the red carpet for the cox-2 class of NSAID's. Pfizer and Merck were the two gorillas of the industry, both in a race to be the first to market. Right or wrong, the FDA went into hyper-protect mode after their faulty processes were put in the spotlight.

 

[jasoncran]

rhea dont get me started on the sae and mechinical engineers, cars are built to fall apart and so you and i have to fix them and replace them.

so either its done on purpose or the pr is a joke(the former is what i believe) i can go on and on on the purposeful poor desings and placement of parts i have seen. ie rubber bushings near the exhaust manifold.

 

[Rhea]

rhea dont get me started on the sae and mechinical engineers, cars are built to fall apart and so you and i have to fix them and replace them.

so either its done on purpose or the pr is a joke(the former is what i believe) i can go on and on on the purposeful poor desings and placement of parts i have seen. ie rubber bushings near the exhaust manifold.

Jason, I've been a member of the SAE for 15 years, and working on cars for twice that. And I call "false" on your statement. It's just a fear and persecution theory.

It is true that the nature of the market makes replacement cheaper than repair in many ways, but cars are about the last item to fall prey to this.

Cars have improved so significantly in recent years it is astonishing. You probably don't remember seeing what cars looked like in the 1960s and 1970s but I can testify that smoke pouring out the tailpipe was a regular thing and oil drips below the engine were equally common. It used to be expected that you would check your oil every time you filled up because of the likelihood that you would be down a quart from leaks. Nowadays, you just get it changed every 5-8 thousand. You used to have to replace parts much more often, and rebuild on a regular schedule due to warping and intrusion of oil into the cylinders or coolant into the block.

Cars require significantly less repair than they used to (although the repairs have gotten more expensive) because of better machining and tolerances.

The idea that SAE or any auto manufacturer would deliberately make cars with planned repairs is ludicrous. Why? Because the evidence shows that the cars are getting more miles with fewer repairs than ever before.

I have owned cars built in 5 different decades and there is absolutely no reduction in quality - quite the opposite. Sure they're getting more complex and harder to work on, but they are very clearly NOT being designed to fail.

 

[jasoncran]

Jason, I've been a member of the SAE for 15 years, and working on cars a lot longer than that. And I call "false" on your statement.

It is true that the nature of the market makes replacement cheaper than repair in many ways, but cars are about the last item to fall prey to this.

Cars have improved so significantly in recent years it is astonishing. You probably don't remember seeing what cars looked like in the 1960s and 1970s but I can testify that smoke pouring out the tailpipe was a regular thing and oil drips below the engine were equally common. It used to be expected that you would check your oil every time you filled up because of the likelihood that you would be down a quart from leaks. Nowadays, you just get it changed every 5-8 thousand. You used to have to replace parts much more often, and rebuild on a regular schedule due to warping and intrusion of oil into the cylinders or coolant into the block.

Cars require significantly less repair than they used to (although the repairs have gotten more expensive) because of better machining and tolerances.

The idea that SAE or any auto manufacturer would deliberately make cars with planned repairs is ludicrous. Why? Because the evidence shows that the cars are getting more miles with fewer repairs than ever before.

I have owned cars built in 5 different decades and there is absolutely no reduction in quality - quite the opposite. Sure they're getting more complex and harder to work on, but they are very clearly NOT being designed to fail.

oh, i know of that as i have older employees older then you that talk of those days

but what and why would the chevy 1500 have common problems with wheel bearings that have been known to fail at approx 30k miles. we have had five trucks do that.

out of warranty,the checy s-10 doesnt get more then 30k miles on the upper control arm bushings as they are placed unprotected near the exhaust we replace them with polyurethane and the problems go away.

sure the american cars have improved but why is it that you get more life from hondas and foreign cars. if i took and did my paper route with ford ranger it wouldnt last as long as my nissan. i should have stated american cars not foreign. how many older hondas and foreign are on the road these days vs american cars. the hondas and toyatas win hands down.

sorry i fix more american cars then japenese cars, the old timers say that made aton off american cars back in the 80s and 90s vs the japanese. we have three toyata trucks and they dont have the issues the fords and chevy do and are used the same.

we get them from the factory and we have to add oil etc, they are often at the lowest level not topped off.i dont have confidence in american cars, sorry. when you have spent more on a truck that you maintain (ie flush brake system and change oil etc) then a truck that is used way more then why would you have any confidence

it cost me 165 usd for a thermostat on my ranger, i replace that on a yr ago as i always do when i flush the coolant. it only had 9k miles on it. ford says that is a common problem for them.both on the ranger 2.3l and also the 4.0 explorer.