Reading a study a clinic quotes at you
You will sometimes be handed a paper, a summary, or a slide with a citation on it. Here is what to look at, in what order, and what each finding means.
Read a quoted study in this order: what molecule was studied, in what species, in what tissue, for what condition, in how many people, against what comparison, assessed by whom, over how long, funded by whom. Nine questions, all answerable from the abstract and the declarations, most of them in under two minutes.
The result comes last and matters least, because a result from a design that cannot support it tells you nothing. Most of the studies quoted in this category fail at question one, two, three or four, before the design is even reached.
2.4.1The order is the method
Most people asked to assess a study go straight to the finding, because the finding is what they want to know. That is the wrong end. A finding is a product of a design, and if the design cannot support the inference being drawn, the number is decoration.
Working through the nine questions in order has a useful property: you will usually stop early. A great many citations offered in support of cosmetic polynucleotide treatment concern a different molecule, a different species, a different tissue or a different clinical problem, and once you have established that, nothing further about the study matters for the purpose it is being used for.
2.4.2The nine questions
1. What molecule
PN or PDRN, and which preparation. If the study concerned a shorter fragment preparation and you are being sold a longer chain product, a bridge is being crossed silently.
2. What species
People, animals or cells in a dish. Cell culture work tells you what a cell can do in a dish. It does not tell you what happens in a face, and the gap between the two is where most failed treatments in medical history have lived.
3. What tissue
Facial dermis, or something else. Wound healing in a different tissue is a different problem with different biology.
4. What condition
Cosmetic skin quality, or a clinical problem such as a wound, an ulcer or a joint. Efficacy in repairing damaged tissue does not transfer to improving undamaged tissue, and the mechanism reasons for that are not subtle.
5. How many people
You do not need to calculate power. You need to notice whether the number is small enough that a handful of participants could have driven the result.
6. Against what comparison
No comparison, before and after, a sham, or an active comparator. This is the single most informative question after the first four. In a treatment with an immediate physical effect, an uncontrolled before and after design cannot distinguish the claimed mechanism from swelling.
7. Assessed by whom
Blinded independent assessors, unblinded investigators, or the participants themselves. Facial assessment is exceptionally susceptible to expectation, in both the person being assessed and the person assessing.
8. Over how long
Weeks or months, and whether the final measurement falls inside or outside the window in which the material could still be present.
9. Funded by whom
Look at the declarations at the end. Manufacturer funding, product supply, and author consultancy relationships are all disclosed in properly reported papers, and their absence from a paper that plainly had industry involvement is itself a finding.
| Question | What a weak answer looks like | What it means for the claim |
|---|---|---|
| What molecule | A different preparation from the one you are being sold | The bridge has to be argued, and usually is not |
| What species | Cells in a dish, or rodents | It tells you what is possible, not what happens |
| What tissue | Not skin | Different biology, different problem |
| What condition | Wounds, ulcers, a joint | Repairing damage is not the same as improving intact skin |
| How many people | A handful | Unstable estimate, easily driven by a few participants |
| What comparison | None, or before and after | Cannot separate the effect from swelling or expectation |
| Assessed by whom | The participants, or unblinded investigators | Facial assessment is highly expectation sensitive |
| How long | Weeks | Cannot address persistence, which is the central claim |
| Funded by whom | Manufacturer, undeclared | Known directional bias, and a reporting failure |
A framework written by this publication to organise a decision. It is not a measurement, it is not drawn from any study, and no figure in it should be quoted as a finding.
2.4.3The moves to watch for
None of these are unique to this field, and recognising them is a general skill.
- Citation by association. A general review of nucleotide biology is cited to support a specific cosmetic claim. The citation is real and the claim is not in it.
- The promising conclusion. A paper concluding that results are promising and larger trials are warranted is quoted as evidence that the treatment works. The words larger trials are warranted mean the authors do not think it has been shown.
- The relative improvement. A large sounding relative change on a scale whose absolute range and clinical meaning are not stated.
- Satisfaction as an outcome. Patient satisfaction after a paid cosmetic treatment is measuring something, but what it measures includes the desire not to have wasted money and the effect of having been looked after attentively.
- The unnamed study. Clinical studies show, with no citation. There is nothing to read, so there is nothing to assess.
2.4.4Where to check, free, in five minutes
PubMed indexes the biomedical literature and is free. Searching the title of a paper you have been shown will usually find the abstract, the author list and the declarations. The Cochrane Library holds systematic reviews. Trial registries, including ISRCTN and ClinicalTrials.gov, let you see whether a trial was registered before it started and whether the outcomes reported match the outcomes it said it would measure, which is one of the most revealing checks available to a non specialist.
Reporting guidelines collected by the EQUATOR Network describe what a properly reported trial contains. You do not need to read them in full to benefit from knowing they exist, because a paper that departs conspicuously from them is telling you something.
2.4.5How to ask for a citation without a scene
The wording that works is simple and neutral. That is interesting. Is there a paper I could look at?
What follows is informative whichever way it goes. Some practitioners will send you something within the hour. Some will say candidly that they are relying on supplier material and their own experience, which is a respectable answer and considerably better than a fabricated citation. Some will treat the request as a challenge. You have learned something useful in all three cases, and you have learned it politely.
2.4.6The limits of this method
This will not make you a methodologist, and it is not meant to. What it does is stop you being persuaded by a study that cannot support the claim attached to it, which is the failure mode that actually occurs in consultations. It also gives you a defence against the opposite error, of dismissing a treatment because someone on the internet was confident. The nine questions apply equally to material arguing that a treatment is useless.
Questions
I have no scientific training. Can I really assess a study?
You can assess whether a study is capable of supporting the claim attached to it, which is the part that goes wrong in consultations. That requires reading an abstract and a declarations section, not understanding the statistics.
What if the clinic will not give me a citation?
Then there is nothing to read and nothing to assess, and the claim should be treated as an assertion. Asking is not rude, and the way the request is received tells you a good deal about the practice.
Is a paper in a journal automatically reliable?
No. Journals vary enormously in what they will publish and how they review it, and even good journals publish work whose design cannot support the use later made of it. The nine questions apply regardless of where a paper appeared.
What does it mean when authors say larger trials are warranted?
It means they do not consider the question answered. That sentence appears in the conclusion of a great many papers quoted in marketing as proof, and it is the authors saying the opposite of what the marketing says they said.
Should I be suspicious of every industry funded study?
Suspicious is the wrong frame. Industry funding is a declared, measurable source of bias towards positive findings that belongs in your assessment. A registered, independently analysed, well designed industry funded trial is better evidence than a sloppy independent one.
- PubMed, US National Library of Medicine
- Cochrane Handbook for Systematic Reviews of Interventions
- EQUATOR Network, reporting guidelines for health research
- CONSORT statement for reporting randomised trials
- ISRCTN registry
- ClinicalTrials.gov
Links to regulators, professional bodies, legislation and research indexes. They are cited because they are public and checkable, not as endorsement of this publication. No source listed here has any commercial relationship with us.
Revisions to this document
This reference is revised when the evidence changes. Subscribe and you will be told what was revised, what the verdict was before, and what it is now.
One email when this document is revised, with what changed and why. Any sponsor line is labelled as such and can never influence an evidence verdict. Rates are published on the newsletter page. We do not sell, share or rent the list. Unsubscribe from any email.