# Statistically significant evidence and clinically significant evidence

Significance is defined as the quality of being important.

- One hypothetical example would be a study to test a new drug for arterial hypertension that has a statistically significant P-value, e;
- From the notes on the logic of experimentation , you will recall that this depends on the sample size the bigger the sample, the more confident you will be that it produces trustworthy results and the size of the difference observed;
- Note that you would almost certainly not alter your approach if the study results were not statistically significant i.

In medicine, we distinguish between statistical significance and clinical importance. Medical studies are carried out on selected samples of people, but the goal is to apply the findings to another population e.

### Subscribe to our newsletter

Naturally, a concern is that the sample used in the study could provide misleading results. Perhaps it was a very small sample; perhaps it was a biased sample that is not equivalent to the people you are treating; perhaps the sample was large enough, but by chance or bad luck it contained people who gave wacky results.

Statistical significance considers the first and third of these concerns. The middle one, bias, cannot be detected by mathematical deductive logic: This is dealt with in the notes on bias.

- There are some tricks and pitfalls that could explain this;
- N Engl J Med;
- Statistical significance versus clinical significance;
- However, the blood pressure reduction is only 5 mmHg, which in clinical practice does not justify adopting the new drug;
- This method consists of summarizing the effect of treatment in terms of the number of patients that need to be treated with the therapy in order to expect to prevent one adverse event;
- December 27, 2013; Revised:

Consider a study that shows a new therapy to be superior to the existing therapy. Statistical significance calculates the probability that the results observed in a study may have been merely a chance finding, and would not be repeated if the study were re-done.

## Statistical significance vs. clinical significance

From the notes on the logic of experimentationyou will recall that this depends on the sample size the bigger the sample, the more confident you will be that it produces trustworthy results and the size of the difference observed. If the study showed a huge difference between new and old therapies, the result is more likely to be real. In other words, you have drawn a false positive conclusion over the new therapy. Results are said to be "statistically significant" if the probability that the result is compatible with the null hypothesis is very small.

- We are already used to looking at the methods and results in order to evaluate whether they support the conclusion. Because there is always a leap of faith in applying the results of a study to your patients who, after all, were not in the study , perhaps a small improvement in the new therapy is not sufficient to cause you to alter your clinical approach.
- Statistical significance versus clinical significance. However, the blood pressure reduction is only 5 mmHg, which in clinical practice does not justify adopting the new drug.
- None Conflict of interest. When we catch a cold, we want to feel better, as quickly as possible.
- However, even a well-designed and properly conducted study with a statistical significant P-value does not always imply real clinical significance. In this case you might fail to detect an important difference between groups.

Significance does not tell us directly how big the difference was. Clinical significance, or clinical importance: Is the difference between new and old therapy found in the study large enough for you to alter your practice?

Because there is always a leap of faith in applying the results of a study to your patients who, after all, were not in the studyperhaps a small improvement in the new therapy is not sufficient to cause you to alter your clinical approach. Note that you would almost certainly not alter your approach if the study results were not statistically significant i.

- This seems more informative for the clinician. The number needed to treat.
- Perhaps it was a very small sample; perhaps it was a biased sample that is not equivalent to the people you are treating; perhaps the sample was large enough, but by chance or bad luck it contained people who gave wacky results. This is where there is an important, meaningful difference between the groups and the statistics also support this.
- On the other hand, the concept of clinical significance, also called clinical importance, can be summarized as a difference between two therapy results that is large enough to justify changing the standard of care. Results are said to be "statistically significant" if the probability that the result is compatible with the null hypothesis is very small.

But when is the difference between two therapies large enough for you to alter your practice? Statistics cannot fully answer this question.

It is one of clinical judgment, considering the magnitude of benefit of each treatment, the respective profiles of side effects of the two treatments, their relative costs, your comfort with prescribing a new therapy, the patient's preferences, and so on. But we can provide different ways of illustrating the benefit of treatments, in terms of the Number Needed to Treat. Yet another example of science offering only partial guidance to the art of medicine.

A partial way out of this uncertainty is to express study results using confidence intervals instead of significance levels. Confidence intervals show the likely range of results within which the true value is likely to lie.

- An adequate sample size reduces occurrences of type II errors;
- One thing that could be useful for establishing adequate clinical significance is to evaluate the confidence interval CI , which includes all values between the limits.

This seems more informative for the clinician. Conversely, the result may not be statistically significant because the study was so small or "under powered"but the difference is large and would seem potentially important from a clinical point of view.

You will then be wise to do another, perhaps larger, study.