Reviewing Efficacy of Marketed Drugs and 'Medical Devices'

Hello Bargain hunters,

As part of my work I regularly perform literature reviews and I've found that looking in to the claims that some products make about being 'clinically shown to…' and 'scientifically proven to…' helps get my science juices flowing. I've started applying this to posts I see on OzBargain, particularly those that I think may be misrepresenting 'scientific facts' or presenting flawed articles as peer-reviewed proof of efficacy in their product.

I'm going to continue with this even if nobody reads them, but I was wondering if there would be any interest within the community in learning how to go about determining whether or not a bargain is actually the miracle cure for this, that and the other that it claims, or if it's just snake oil. If anybody would like a brief description of the review process I go through, let me know here. I'm also keen on hearing any criticisms people have with my reviewing as it helps me get better at my job.

Here are the two examples of products I've put effort in to examining for efficacy:

Flexiseq Joint Lubrication Gel 50g Free Sample Worth $49.95

SpaTone 100% Natural Iron Supplement 28 Sachets $17.99 @ Direct Chemist Outlet [VIC/QLD/NSW] ($27.99 @ Chemist Warehouse)

My thoughts are outlined in the comments, though the first product (Flexiseq) began to 'rub' me the wrong way (if you'll graciously accept that awesome pun), and I ended up doing a full review and submitting it to ABC's The Checkout 'Tip-Offs' section. Haven't heard back.

Cheers guys

EDIT: Here we go then! Currently writing up my process and I'll update again soon.

Okay - let’s start from the top; and as a precursor I’m too lazy to cite all of this information, but if it’s beyond a cursory Google or it is requested I’ll do my best.

INTRO:
You have a doubt about a product that has been advertised to you but you’re not sure if you should believe what they’re selling you. While it’s important to be critical about the legitimacy of a product’s claims, it is also equally important to not let a ‘need’ to prove a claim wrong overwhelm you and lead you down a rabbit hole of bad science for the sake of feeling right. This is, in my opinion, one of the biggest problems plaguing clinical research when money is involved – ultimately, scientific accuracy takes a backseat to economic viability (I’ll come back to this in a bit).

HOW TO READ A SCIENTIFIC REPORT:
I’m going to start with how you should read a peer-reviewed report, beyond the abstract. Things that are of importance when judging the paper are the journal it was published in, whether or not it has been cited, the presence of an ‘Author’s statement’ or similar ‘Acknowledgements’ section, the methods section and the discussion section. Obviously results are relevant but can be determined from the conclusion, and we primarily want to decide whether or not to give the paper the time of day. First thing I do is read the abstract as it contains the essential information as the author wanted it to be digested. This typically highlights the main findings of the paper, and mentions briefly the methodology used. For a clinical study performed in a human population, ideally this is where a sample size would be mentioned. A low sample size will provide less meaningful results. How large should a sample size be? Large enough that the results are a valid and statistically significant representation of an even larger population, as reflected by the p value, the standard deviation and standard error of the mean. A p value reflects the probability that the observed results could be obtained if the null hypothesis is true. The null hypothesis essentially indicates that there is no correlation or relationship between two variables/measured groups. It is what you disprove during a test, rather than proving your hypothesis (given that to do so would be to discount all unknown variables that may impact the results). So a p value of 0.05 is in essence the probability that two related variables when compared in some way have a relationship to one another in terms of the experiment. If a study found that in 99 out of 100 cases of “a ball moving”, there was “a hand pushing the ball” and reported a p value of <0.001, there would be almost no chance that a ball moving under the context of the experiment did not have a hand pushing it.

INTERPRETING REPORTED DATA:
Standard deviation is expressed, usually, as a ‘plus-or-minus’ variable in brackets next to a cited population mean. This indicates the values above and below the mean of a population cited that include 68% of the data, or ‘one standard deviation’ from the mean. This is based on the assumption that a population follows a Gaussian distribution, in which 68% of the population falls within 1 standard deviation of the mean, and 95% of the population falls within 2 standard deviations from the mean. It is important to keep in mind when interpreting standard deviation that if the population does not follow a Gaussian curve i.e. a bell curve model (such as having a far outlier, or two distinct groupings), the standard deviation will not have an accurate meaning.

Standard error of the mean is a calculation that takes in to account the standard deviation, as well as the population size, and uses it to interpret whether or not the sample size offers an accurate representational mean of the true population mean; i.e. if your group of 100 people is an accurate representation of the population they represent. It gets smaller as the sample size increases, given that there is less error between the mean of the population and the mean of the sample the closer the sample approaches the population in number.

REPORTING BIAS:
Final point on reading a paper; the ‘Acknowledgements’ section will include any personal biases the researchers or people who were in charge of editing and approving the study prior to submission may have had. Ideally this will read ‘The authors of this paper recognise no bias in the publication of this paper’ or something along those lines, but often in clinical trials of drugs you will find some short-hand notes wherein people’s names are reduced to initials. This is where you will see things like ‘At time of publication, X was working for Y company as a paid consultant’, etc. It’s not always bad but it does give you an idea of how the publication came to be.

EXAMPLE PRODUCT & RESOURCES FOR INVESTIGATING:
For now, back to your questionable product. If you’re at the pharmacy you may like to ask the pharmacist what the benefits are of the product over others in the same market. If they can’t help you or you would like to verify that the pharmacy doesn’t hold a bias (which they can, and do – bottom line is that they are a business and will have a vested interest in companies willing to promote their drug/product over alternatives), the next step would be to take to the web. Start off by finding the company, their product, and then searching for clinical studies cited by the company through which they can confirm the accuracy of their claims.

I’ll try it now with an example; I want to buy paracetamol but I’m not sure if Homebrand is fine or if I should shell out a few extra bucks and buy one of the name brand alternatives. Panadol looks nice – they have nicer packaging and more options – so maybe I should buy Panadol instead? I have a pretty gnarly headache and I want it to go away ASAP, so maybe one of the ‘Fast Pain Relief’ products would be best, such as Panadol Rapid Caplets? Let’s look at regular Panadol 20 pack vs. Coles branded Paracetamol 20 pack. Both products have 20 capsules per pack, each capsule has 500mg of paracetamol as the active ingredient. The difference is that while the Coles brand paracetamol is $0.70 per pack, the Panadol product is $3.85 per pack – that’s a $3.05 mark up for no added value, aside from packaging. So what about the Rapid Caplets? Also 20 pack, also 500 mg per caplet, only this time the retail price is $6.05 (they’ll likely be on sale for slightly less though – still more than the regular Panadol). $5.35 extra for packaging and – what exactly? What is it in the Panadol Rapid that causes it to be “Absorbed twice as fast as regular Panadol tablets”?

The labelling won’t answer this question because they’re not required to disclose anything other than the active ingredient. The ‘active ingredient’, it is important to note, is the ingredient of a pharmaceutical drug that has biological significance, i.e. the part that isn’t the excipient (the drug vehicle – the substance in which the active ingredient is suspended or otherwise delivered). The form in which the active ingredient is held does however effect the overall activity of the drug, so the inactive components shouldn’t be entirely overlooked when making a judgement on drugs (for instance, if your paracetamol came in a metal capsule, the efficacy of the active ingredient would be ‘not at all effective’).

So we take to the Panadol.com.au website where, sadly, they do not provide supporting research on any of their products. They do offer a toll-free Infoline however, through which I am sure you could request that they send you supporting research for the product you are interested. Whether or not that actually works, I can’t say for sure. End of the road? Nope.

The next step is the one I take, but in this example it is not the one that worked, so if you want to skip it be my guest:
Next up is the company that owns Panadol, GSK. Somewhere there must be evidence of the clinical validity of this product, otherwise t wouldn’t be on shelves as it is. The au.gsk.com site has much more information available, and as luck would have it they have a fair bit more ‘transparency disclosure’ related information on their site than most pharmaceutical companies. This might have something to do with the USD$3 billion settlement they had to pay after pleading guilty to promotion of drugs for unapproved uses and a failure to report safety data and kickbacks to physicians for their promotion of GSK related products in the United States in 2012 – but is neither here nor there. Most drug companies with a primary listed stock will have a Wikipedia page that is at least half devoted to controversies (conjecture, but in my opinion accurate). Anyway.

Helpfully GSK include a ‘Clinical Study Register’ that can be accessed by clicking on ‘Healthcare professionals’ and then ‘Clinical Study Register’, or by going straight to www.gsk-clinicalstudyregister.com. From here you can observe reports and publications from the pharmaceutical company relating to any GSK-sponsored clinical study. A report does not equal a publication, however; nor does it mean that the findings are accepted as an accurate representation of the data. These are primarily the reports provided by the pharmaceutical company through which they can offer transparency in their funding and actions. In this instance, I couldn’t find what I was looking for in terms of support for Panadol Rapid (the closest I could find in terms of supporting documentation: http://www.gsk-clinicalstudyregister.com/files2/096dee33-c6e…, a comparison of ‘Fast Dissolving’ Paracetamol with regular paracetamol, at two different dosages. It is confusing to read but the takeaway is that 1000mg of ‘FD’ paracetamol slightly outperformed 650mg of standard paracetamol over 6 hours (unsurprisingly?) in terms of pain relief. It doesn’t go in to why ‘FD’ paracetamol is actually ‘FD’), so I didn’t continue with this method. I still think it’s worth mentioning as continuing on from here we are opening up to reports from outside the manufacturer and provider of a product – which is still perfectly fine as long as you know where to look.

As a side note, if you’re looking for clinical trials, https://clinicaltrials.gov/ is a service of the US National Institutes of Health, acting as a registry and results database of clinical studies with human participants. If a drug went through clinical trials to be sold in the US, it’s probably mentioned here. Australia and New Zealand have an online clinical registry too (http://www.anzctr.org.au/Default.aspx), though it also sources information from clinicaltrials.gov, so there’s some cross over there.

The next step; (In this case, also not the best next step to take, although this was because I am now looking for non-pharmacological specifics of a paracetamol product):

NCBI; I’m sure you’ve heard of the site. The National Center for Biotechnology Information is an organisation run through the US National Institutes of Health put in place to amalgamate data online for research purposes, and has many facets. These incorporate the fields of animal and plant biological sciences, and the site contains many useful tools for researchers in biotechnology and medical studies – including many services in genetic research, data analysis, research methods and learning tools. The one you are probably most familiar with, however indirectly, is PubMed, an amalgamation of “over 26 million citations for biomedical literature from MEDLINE [the U.S. National Library of Medicine biomedical bibliographic database], life science journals, and online books.”

PubMed is every researchers’ best friend and trusted companion (conjecture, IMO accurate), and is usually my first stop when looking to prove a point. It is also the first stop for most people looking to win an argument on the internet; Google ‘PubMed’, search for your topic, find the first article that supports your opinion and look no further. This is one problem university students, including myself, tend to fall into when they aren’t putting effort into a research paper or essay and just need to meet a quota on their reference list. It’s all well and good making a citation to a paper you found on PubMed, but if the person marking your paper it at all worth their salt, they’re going to see right through it if: the paper you cite is behind a pay wall and you make abstract-level commentary without reviewing results and methods; you fail to observe other papers that may have cited it later on with different findings; you only cite a paper that is ‘going against the grain’ as it were in terms of findings in relation to other studies that find the opposite of what it claims; etc. etc. etc…

PubMed found me many paracetamol-related papers detailing the different mechanisms that are in play when paracetamol is absorbed by the body in its oral tablet form. These would be enough to make a judgement on how efficient Panadol Rapid is, however I’m looking to find out exactly what makes it ‘twice as quick’ as regular Panadol when being absorbed.

I’m going to cut short my foray into Panadol Rapid as it is stressing me out. The issue I ran in to was discerning the ‘inactive’ ingredients that form the drug-delivery mechanism, and if you Google the product then you’ll find sites like Australia’s https://www.tga.gov.au/ (Therapeutic Goods Administration) and New Zealand’s (very helpful) http://www.medsafe.govt.nz/index.asp (NZ’s Medicines and Medical Devices Safety Authority), who will often include the constituent components of the non-active ingredient. These tend to be components you can find out more about by looking them up in Wikipedia and discerning their use from there; if a component strikes you as interesting, search for it on PubMed to see if there are any papers investigating the specific findings related to it. Searching for something specific like a molecule name will likely give more relevant results. This can be immensely time consuming however, depending on how deep down that rabbit hole you go.

I’m exhausted so I’ll leave it there. A few other points I couldn’t decide where to fit in:

Depending on the research group, and most often when a drug or medical product under trial is involved, the onus is on the researchers to produce a paper that supports the efficacy of the product they are testing. This doesn’t always reflect in the papers that are published – although the wording surrounding the results of a drug trial that hasn’t yielded positive and statistically significant results will often be your first clue as to whether or not the authors/editors of the paper have a bias. However, even if a research group with the funding of a drug company publishes an unbiased paper that shows a given drug does not work as intended, it doesn’t mean that they won’t be pushed to continue research on that same drug if there is an indication that it may be more financially beneficial for a drug company to market as opposed to related drugs, e.g. it may be a cheaper alternative that they want to be able to show, even once, has worked as well as its alternatives.
Feel free to message me if you have any questions or comments.