In late 1999 the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) of the National Institutes of Health (NIH) announced the funding of the Spine Patient Outcomes Research Trial, known as the SPORT study.
In a well-publicized press release, the director of NIAMS, Stephan Katz, MD, stated, “Based on this [SPORT] trial we shall, for the first time, have scientific evidence regarding the relative effectiveness of surgical versus nonsurgical treatment of herniated lumbar disc, spinal stenosis, and degenerative spondylolisthesis.”
The $13.5 million dollar grant for the SPORT study closely coincided with the federal Agency for Healthcare Research and Quality’s publicly financed, evidence-based report entitled Treatment of Degenerative Lumbar Spinal Stenosis. After an exhaustive review and epidemiological analysis of published peer-reviewed literature, the report concluded, “Definitive evidenced-based statements about the treatment of spinal stenosis await the results of well-designed clinical trails.”
This report cited severe limitations in the quality of the design and execution of the studies published thus far, noting that only seven of 147 trials examining surgical treatment of spinal stenosis were randomized controlled trials. It further concluded, “Well-designed randomized clinical trials provide the highest quality of evidence for treatment effectiveness because each treatment arm is composed of comparable patient groups having the same characteristics. Nonrandomized trials may result in one treatment arm having patients who can appear to respond better or worse to treatment due to the characteristics of the patient group (i.e. selection bias or confounding).”
About SPORT
The SPORT study is a prospective, randomized, multicenter trial comparing surgery to nonoperative management for herniated lumbar disc, spinal stenosis, and degenerative spondylolisthesis. The study will enroll 1,450 patients at 11 orthopedic spine centers over a five-year period. Pre- and post-operative outcomes will be assessed using validated measurement instruments (for example, the SF-36 physical function subscale, and the Oswestry Disability Index). An “intent-to-treat”analysis will be utilized.
The stated power of this study, assuming a treatment effect magnitude of 10 points on the SF-36 questionnaire and a 20 percent “loss to follow-up,” is 85 percent. Analysis of the data to estimate the cost-effectiveness of surgical versus nonsurgical treatment also is planned, as is a concurrent prospective, observational cohort study of those patients who decline randomization. The principal investigator is James Weinstein, DO, of the Center for the Evaluative Clinical Sciences at Dartmouth Medical School.
Neurosurgery Identifies Concerns
Representatives from the American Association of Neurological Surgeons (AANS), Congress of Neurological Surgeons (CNS), and the AANS/CNS Section on Disorders of the Spine and Peripheral Nerves had the opportunity to review the design of the SPORT study. After thorough analysis, three main areas of concern were identified: hypothesis and data analysis; generalizability; and power.
Hypothesis and Data Analysis The SPORT study was designed to address the broad study hypothesis, Is surgery a more effective treatment for lumbar disc herniation, lumbar stenosis, and/or degenerative spondylolisthesis than nonoperative treatment? It was not designed to validly assess the more relevant question of which patients are most likely to benefit from a specific treatment, and therefore will not be able to do so. Nor will it identify factors that may influence outcome, irrespective of treatment. Essentially, such a broad study hypothesis assumes both a homogeneous patient population in each diagnostic group as well as a uniform response to treatment. In reality, there is significant heterogeneity with respect to clinical presentation (duration, type, distribution, severity of symptoms), natural hisstory, radiographic characteristics, psychosocial covariates (worker’s compensation, depression, litigation), and response to surgery for all of the conditions under study.
Indeed, class II evidence from prospective observational trials, further supported by empirical clinical experience and class III evidence, has established that the effects of a specific treatment for each condition under study are not uniform. Some patients benefit more than others. Some benefit not at all. This strongly suggests that other factors are associated with treatment outcomes, independent of the treatment. Numerous studies have shown that treatment outcomes are associated with specific defined characteristics and variables for each of the conditions under study.
In the case of herniated lumbar disc, for example, treatment outcomes may be associated with symptom location, distribution, severity, and duration, radiographic characteristics (bulge, protrusion, extrusion, sequestration), and psychosocial covariates (worker’s compensation, personal injury, depression). For lumbar stenosis treatment, outcomes may be associated with symptom severity and distribution, radiographic characteristics, medical co-morbidities such as diabetes, hip arthritis, and cardiovascular disease, as well as self-reported general health perception. For lumbar stenosis with degenerative spondylolisthesis, outcomes are additionally associated with whether or not fusion has been performed.
While the prospective, randomized study design will assure equal distribution of these factors across each treatment group, it will not specifically control for the independent effect of any variables on treatment outcome. Therefore, the true benefit of surgery may be underestimated or totally obscured. If, for example, surgery clearly benefits those patients with more severe symptoms while patients with milder symptoms benefit more from nonoperative care, based on the SPORT study design it is possible that these benefits will, in effect, cancel each other out. In essence, there will be a failure to reject the null hypothesis of no difference between surgery and nonoperative care for the conditions under study. Such a conclusion would not only be specious, but would disproportionately affect surgery and the patients with severe symptoms who are most likely to benefit from it.
Such a concern can only be addressed through stratified analysis within and between treatment groups to control for the effects of these variables. Unfortunately, since no subgroups are identified a priori in the SPORT study design on the basis of baseline characteristics or variables known to be associated with treatment outcomes, the SPORT study will neither identify nor quantify differences in treatment effects or benefits both within and between treatment groups based on these characteristics and variables. At best, these subgroups only will be identified and analyzed through secondary post hoc data analysis. Such after-the-fact analysis is fraught with methodological shortcomings and is often given little credence by methodologists and policy makers.
Ultimately, based on the broad nature of the study hypothesis and methods of data analysis, the SPORT study may not only underestimate or totally obscure the benefit of surgery in clearly identified subgroups, but it will fail to validly address the more important research question of which patients are most likely to achieve the greatest benefit from which specific treatment.
Generalizability With respect to generalizability there are two main questions. First, are the patients who agree to randomization different from those who do not in ways that are associated with treatment outcomes? The SPORT study will be able to control this to some degree through comparison with the observational trial cohort, but it is unclear what conclusions will be made if differences between the study groups are found.
This is of particular concern because of the widespread beliefs and established practice patterns related to the conditions under study. In the real world, surgical and nonoperative treatments are neither viewed interchangeably nor applied uniformly to the conditions under study. Patients with more severe symptoms are selected for — and ultimately choose — surgery, while patients with mild symptoms are usually managed with nonoperative care. In addition, irrespective of symptom severity the vast majority of patients are initially managed with nonoperative care. Most patients are offered and elect to undergo surgery only after a trial of nonoperative care fails to resolve their symptoms.
Based on these well-established perceptions and practice patterns, a concern is that a disproportionate number of patients with severe symptoms (that is, those most likely to benefit the most from surgery) will not be referred to the SPORT study centers; will not be offered or will not accept randomization but will opt for surgery if referred to the study centers; or will cross over to the surgical arm following randomization. All of these occurrences would serve to further underestimate the benefit of surgery in the randomized arm of the SPORT trial.
Secondly, do the orthopedic surgeons participating in the SPORT study utilize similar diagnostic and surgical selection criteria for lumbar disc herniation, spinal stenosis, and degenerative spondylolisthesis as neurosurgeons? And, in a similar vein, do they determine the surgical objective and successfully achieve the surgical objective in a manner and at a rate similar to that of neurosurgeons? These questions have not yet been adequately addressed.
Power The issue of power relates to SPORT’s use of an “intent to treat” analysis. Specifically, this is a compliance issue since up to 25 percent of patients randomized to nonoperative treatment will be non-compliant and will cross over to the surgical treatment arm. Any improvement in this group will be credited to nonoperative treatment. While such analysis does not affect the validity of SPORT’s findings, it may reduce the ability to detect a difference between treatments when in fact a difference truly exists. What is not clear from the study design is whether the anticipated 25 percent crossover rate was included in the calculation of the needed sample size to achieve the stated 85 percent probability of detecting a true treatment difference.
In summary, based on the heterogeneous nature of the conditions under study, the widespread beliefs and established practice patterns related to the treatments of these conditions, and some elements of the SPORT study design, the concern is that the benefits of surgical treatment will be underestimated at best, or totally obscured by the results of the SPORT study at worst. In the end, the concern is that the NIH will spend $13.5 million dollars on a highly publicized but underpowered and “biased toward the null” study that likely will fail to identify a difference between surgical and nonsurgical treatments when, in fact, a difference truly exists.
Further compounding the problem, public agencies and third party payers may interpret results of the SPORT study in such a way that there will be a disproportionate and unfair impact on the surgical options for these conditions, with the result that many patients may be denied the more appropriate and effective treatment.
Neurosurgery Responds With a New Study
The AANS Board of Directors, in conjunction with the executive committees of the CNS and the Spine Section, has responded to the SPORT study on several levels. First, neurosurgery’s concerns regarding the design and likely outcome of the SPORT study have been articulated both in a letter and in person to Dr. Weinstein. Secondly, neurosurgical representation is now in place on the SPORT study Data and Safety Monitoring Board.
Finally and most importantly, neurosurgery’s concerns with the SPORT study are being addressed through the development of a new study called the Stenosis Outcoome Study (SOS). Under the direction of the Spine Section, this prospective, randomized efficacy study is now being planned to validly assess the hypothesis, Is surgery more effective than nonoperative management for patients with moderately severe and severe lumbar spinal stenosis?
Strict exclusion criteria will assure that only the most ideal surgical candidates qualify for randomization. The study has been funded by contributions totaling $80,000 from the AANS, CNS, and the Spine Section and should be completed in two years. Mark Hadley, MD, who developed the study protocol, and Paul McCormick, MD, will be the principal co-investigators.
Paul McCormick, MD, MPH, is professor of clinical neurosurgery at the New York Neurological Institute and a member of the AANS Board of Directors.
For Further Information
|