Statistics: Why Meaningful Statistics Cannot Be Generated From a Private Practice
By Nicholas J. Gonzalez, MD; Linda L. Isaacs, MD
We often get asked by prospective patients, physicians, and others interested in our work the general question, “What are your statistics?” or at times, more specifically, “What is your 5-year success rate with breast cancer?” or “What is your 5-year success rate with pancreatic cancer?” Many, including highly trained scientists, think this is a simple question requiring that you only divide the number of patients who survive 5 years—the accepted standard for “cure”—by the total number treated. But there’s much more to it than that.
A few years ago, a well-known PhD epidemiologist contacted our office, wanting to do an evaluation of our “5-year survival” data for all patients we had treated with breast cancer, or, as he added, my “success rate with breast cancer.” I (Dr Gonzalez) said, having been approached many times before in this way, “Great, let’s do it.” He was pleased. Then I said to him, when you say breast cancer, what exactly do you mean? He went silent on the other end of the phone and I then continued, asking him which type, because there are many different histological types each with its own distinct natural history. For example, experts traditionally have considered ductal carcinoma in situ fairly indolent, though it can become invasive. Its unique natural history differs from that of lobular carcinoma, which in turn differs from invasive infiltrating ductal carcinoma, a more aggressive form of breast malignancy.
Not only are there different histological types each carrying with it a different prognosis, but to complicate the matter, pathologists divide each specific cell type into grades, another measure of aggressiveness based on the appearance of the cancer cells under the microscope. This classification scheme usually breaks down into 3 categories; well differentiated (least aggressive), moderately differentiated (moderately aggressive), and poorly differentiated (most aggressive), each in turn associated with a different prognosis. For each, different treatments might be suggested.
With infiltrating ductal carcinoma, the prognosis varies considerably depending on the size of the tumor, with tumors less than 2 cm having a better prognosis than tumors 2 to 5 cm, which have a better prognosis than tumors greater than 5 cm. Then there is the whole issue of lymph node involvement. The scientific literature supports the idea that the observation of no positive nodes carries a better prognosis than 1 to 3 nodes, which carries a better prognosis than greater than 3 nodes involved with cancer. Presence of 9 or more positive nodes is seen as a disaster in the making, even in the absence of overt distant metastatic disease. The precise location of the nodes in the axillary region adds another prognostic variable to the mix, with positive nodes lower in the axilla having a better prognosis than those higher up.
For all of the above, one must factor in hormone and human epidermal growth factor receptor 2 (HER2) status, because the prognosis for estrogen-receptor (ER) positive, progesterone-receptor (PR) positive, HER2-positive disease is quite different than that of ER-negative, PR-negative, HER2-negative cancers.
The survival for stage IV metastatic breast cancer (involvement of distant sites) remains dismal despite so-called advances in conventional therapy, but even among women diagnosed with stage IV disease, patients with cancer limited to bone can live for several years, as opposed to patients with brain and liver involvement who usually succumb fairly quickly, in the conventional world at least, often within months.
In any assessment of prognosis, one must consider prior therapy, which can affect the ultimate prognosis and outcome, because many of the aggressive therapies offered for breast cancer are quite toxic and mutagenic in and of themselves. We have seen patients die not directly as a result of their cancer, but because of the damage wrought by previous conventional treatments. In addition, these treatments often fail because they select for a resistant, more aggressive clone of cancer.
To complicate the issue still further, with a complex nutritional therapy such as ours, which patients must self-administer at home on a daily basis, compliance becomes a critical component in any evaluation of success. Determining compliance in a patient, particularly for a period of years, is no small undertaking, because each of our patients is prescribed multiple supplements that must be taken throughout the day, as well as a specified diet along with the various detoxification routines such as the coffee enemas.
By this point the PhD epidemiologist, realizing the work that would be involved, began to sound confused and somewhat exhausted, though I had done much of the talking. I then said to him to illustrate the complexities of his proposed project, a woman fully compliant with our regimen diagnosed with localized ductal carcinoma in situ who had undergone surgery before seeing me and lived 10 years proves nothing because most such patients with early stage ductal carcinoma in situ live 10 years after surgery without any other treatment—mine or someone else’s. However, a partially compliant stage IV breast cancer patient with metastases in the brain and liver who survives 3 years having failed chemotherapy, radiation, and hormonal blockade would be a near miracle (we actually have such patients who have survived more than 20 years).
The epidemiologist finally realized it would be a cumbersome and fairly meaningless exercise to try and prove my overall “success” rate with breast cancer by reviewing my files. Determining compliance alone could take months, if not years, of work because we had treated so many women with “breast cancer” during the time of our practice. He also began to understand my point that the so-called gold standard 5-year survival rate would be essentially meaningless to assess treatment effect. Eventually, he thanked me and went on his way to pursue easier tasks, though we remained friends.
One might argue that with a disease such as pancreatic cancer, the situation is somewhat simpler because of its well-documented deadly nature. But variability in prognosis and expected survival still exists depending on grade, stage, previous treatment received, and, important, performance status.
In addition, pancreatic cancer patients frequently develop complications such as biliary obstruction that can be life threatening or can affect their ability to comply with our treatment. For example, many of these patients require biliary stents, which frequently become obstructed or infected, a potentially deadly scenario. We have both lost patients whose pancreatic cancer seemed to be under control but who died 3 or 4 years after diagnosis because of stent infections that could not be controlled. In the context of a mean and median survival in the range of 3 to 6 months for patients with the advanced stages of pancreatic cancer, is such a patient who dies a treatment failure or success?
With so many variables at play among patients, even when considering the “success rate” of a single cancer like pancreatic, controlled clinical trials—set up precisely to reduce variables—would seem to be the way to go, if conducted in a fair and objective manner.
In 2010, I (Dr Gonzalez) was involved in another discussion in a far different context about measuring the “success rate” of pancreatic cancer patients treated in a private alternative medicine practice. I had several conversations about the issue with my friend Beth Clay, a seasoned Washington consultant specializing in health care issues, who previously worked at the National Institutes of Health (NIH) and served as a professional staff member of now-retired Congressman Dan Burton (R-Indiana). She decided, at my urging, to contact both MD Anderson Cancer Center in Houston and the Memorial Sloan Kettering Cancer Center in New York, specifically asking each of these internationally acclaimed cancer facilities if they could provide “statistics”—that is, their “success rate” with pancreatic cancer specifically. The MD Anderson spokesperson bluntly stated that no, MD Anderson did not release such statistics and believed the question irrelevant, but it “assumes” its patients do better at Anderson than the general population of pancreatic cancer patients treated elsewhere, even though the spokesperson could provide no substantiation at all for this claim.
The Sloan Kettering representative also reported that their institution had no statistics to release regarding their experience treating patients diagnosed with pancreatic cancer, again believing the question irrelevant, remarking their doctors at Sloan treated according to the best available information from clinical trials published in the scientific literature.
MD Anderson and Sloan Kettering are highly funded hospitals. For example, the endowment of the Memorial Sloan Kettering Cancer Center has been reported to be $3.5 billion, ranking it among the richest hospitals in the world, supported in great part by the largesse of wealthy Americans. The staff at each of these 2 institutions numbered in the thousands, including teams of epidemiologists and statisticians. Yet at the time of Ms Clay’s initial request, neither of these esteemed academic centers could provide an answer to the very simple question about their respective “success rate” with pancreatic cancer, the spokesperson in each case thinking the question was “irrelevant.”
The Cancer Treatment Centers of America (CTCA) collectively consist of a consortium of hospitals with key locations around the country, claiming it offers the “best” of conventional oncology care with a mixture of integrative approaches such as nutritional prescriptions and mind-body counseling. CTCA advertises heavily on television, and its Web site enthusiastically reports better results than would be expected for most cancers treated elsewhere, including pancreatic. For some years, CTCA has provided some information on “success rates” by cancer type, which it assess by comparing the survival of its patients with the Surveillance, Epidemiology, and End Results (SEER) data, the compendium of total US cancer statistics maintained by the National Cancer Institute (NCI).1,2
Over the years, CTCA has modified the format of the pancreatic cancer section on the Web site. In the latest incarnation, it provides a chart that directly compares CTCA survival rates with the SEER data for this cohort of patients. The 6-month survival of 62% at CTCA exceeds the 31% reported from SEER, the 1-year survival at CTCA of 31% compares with the SEER 14%, the 18-month survival of 17% at CTCA again exceeds the SEER 8%, and the 2-year CTCA survival of 10% again ranks somewhat higher than the SEER 5%. However, by 5 years, both the CTCA and SEER data converge at a meager 2% survival.
Although we appreciate CTCA’s attempt to provide “statistics,” the comparison to the SEER data has, in our opinion, very minimal value. The SEER numbers include all patients diagnosed with pancreatic cancer in the United States, including those who died immediately after diagnosis and those too sick to seek treatment at a distant center, such as CTCA. Patients seen at CTCA, or at any tertiary referral center, may well have better performance status than the average cancer patient, thereby creating a survival advantage having nothing to do with the treatment offered.3 In addition, it appears CTCA leaves out from consideration of the numbers patients who may have begun treatment elsewhere, before consulting there.
In 2012, I (Dr Gonzalez) asked Ms Clay to contact once again both MD Anderson and Memorial Sloan Kettering, asking for statistics specifically on pancreatic cancer. The spokesperson for MD Anderson responded:
MD Anderson does not have many published statistics on our website because many of our patients come here previously treated, so it would not accurately reflect MD Anderson. However, MD Anderson has an overall 5-year survival rate well above the national average. We see and treat more rare cancer diagnoses in a day that most hospitals see in 1 year.
The spokesperson didn’t answer Ms Clay’s specific question about pancreatic cancer, made a general statement about “5-year survival” backed by no documentation, and digressed to an irrelevant reference to “rare cancer diagnoses.” Pancreatic cancer, the fourth leading cancer killer in the United States, hardly falls into the category of “rare.”
Ms Clay learned that after her initial set of inquiries— beginning in March 2011—Sloan Kettering did begin providing survival statistics on its Web site for a number of cancer types using the SEER data as a comparison.4,5 But this is a recent development. For stage IV pancreatic cancer, the site describes a 30% survival at 1 year, approximately a 12% survival at 2 years, and a near 0% or 0% survival at 5 years (it is hard to read the chart precisely at 5 years).
In recent years, cancer centers such as CTCA, but also more academic institutions such as Memorial Sloan Kettering, have launched aggressive, apparently expensive, highly professional advertising campaigns on television and in the print media, often with moving testimonials from happy patients claiming great benefit from the treatment offered. In New York City, one can’t watch television for any length of time before an ad from CTCA or Sloan Kettering pops up on the screen.
From the time these promotionals became a regular part of television watching, we have noticed the paucity of hard statistics in these advertisements, replaced by very emotional appeals extolling the glories of the particular institution. Apparently, the explosion of such advertising by cancer centers in media markets throughout the country has attracted considerable attention within the medical community itself, generating some controversy. In the June 17, 2014, issue of Annals of Internal Medicine, an article entitled “What Are Cancer Centers Advertising to the Public? A Content Analysis”6 addressed just this issue. In their rather comprehensive article, apparently the first of its kind on the subject of cancer center advertising, the authors assessed the potential benefit—or possible harm—from the paid-for promotion by cancer centers in the television and print media aimed not at physicians, but the “consumer,” as the article refers at times to patients.
The authors evaluated some 409 advertisements placed by 102 US cancer centers. In their “Discussion” section of the article, they summarize their findings that these advertisements indeed relied primarily, at times exclusively, on emotional appeals, rather than scientific information or data. They write:
Advertisements commonly evoked hope for survival, promoted innovative treatment advances, and used language about fighting cancer while proving relatively limited information about benefits, risks, costs, or insurance coverage of advertised therapies. Patient testimonials focused on survival and rarely included disclaimers. …
These findings suggest that emotional appeals coupled with incomplete information are being widely used to promote services, even among the nation’s most prestigious cancer centers.6
It seems that even major cancer centers aggressively promote themselves without feeling the need to substantiate in detail the benefits claimed.
An editorial accompanying the article in Annals of Internal Medicine, written by Gregory A. Abel,7 MD, MPH, of Harvard’s Dana-Farber Cancer Center, provides an additional thoughtful perspective on the lack of valid data in cancer center advertisements:
What about the lack of data presented in cancer center advertisements (only 2%)? The authors argue that this is possibly detrimental to consumers because the alternative seems to be emotional appeals to fear or false hope. … As a result, unlike DTCM [direct-to-consumer marketing] for specific medications, if these advertisements were to present quantified data, they would likely manifest in the form of cancer center survival statistics. These data are notoriously easy to manipulate and would be difficult for consumers to evaluate, given the contribution of institutional case mix (referral bias).
Although “statistics” in the classic sense cannot reasonably or meaningfully be culled from a private practice, we believe it is still possible to generate useful information from such a situation. In 1981, while a medical student at Cornell University, I (Dr Gonzalez) began an investigation of the practice of the alternative cancer practitioner William Donald Kelley, DDS, under the direction of Robert A. Good, MD, PhD, at the time President of Sloan Kettering in New York. Although we both understood at the onset of the project the impossibility of obtaining definitive data from Kelley’s office records, Dr Good believed that if I could put together a series of appropriately diagnosed patients with obvious poor-prognosis or terminal disease who had done well for prolonged periods under his care, that would be a productive exercise, hopefully leading to expanded interest from the academic research world. After all, as Dr Good often said, advances in medicine often begin with a single case report of a patient responding in an unusual way to a new treatment. Specifically, he believed if I could uncover a single patient with properly diagnosed advanced pancreatic cancer who lived 5 years under Dr Kelley’s guidance, that would be an impressive finding, because no one else in medicine had such a case.
This review of Kelley’s records would eventually take 5 years, completed while I pursued my immunology fellowship under Dr Good. It was a daunting, difficult, time-consuming effort, because of the diversity among the patients Dr Kelley had treated in terms of their cancer types, previous conventional—and unconventional—therapies they had pursued, and of course compliance. Nonetheless, I found in Kelley’s extensive files many patients diagnosed with advanced, poor-prognosis, or terminal cancer, who had experienced regression of disease coupled with very unusual long-term survival while under Dr Kelley’s care, including a number diagnosed with metastatic pancreatic carcinoma. For the final report of my study, I wrote lengthy case reports of 50 such patients representing 26 types cancer. My book, One Man Alone, provides the results of that study in some detail, including the 50 patient histories with copies of relevant medical records.8 Though many of these patients had extraordinary success with Dr Kelley’s nutritional regimen, conventional physicians and researchers have, for the most part, either ignored the study or refused to take the data seriously.
In early 1993, NCI first approached me as part of its initial effort to evaluate “promising” unconventional cancer treatments. The senior staff at NCI, I learned, was very much aware of my Kelley study, which had been circulating in typescript form among certain government researchers. For a number of years, I had been providing copies to any interested scientist or physician, and apparently NCI had gotten hold of one such manuscript.
The NCI staff at the time, apparently, thought my approach to Kelley’s practice admirable, addressing, as it did, the enormous difficulties of determining “statistics” from a private-practice, uncontrolled situation with all the many variables in play. As NCI itself considered various methods to evaluate the practices of alternative practitioners, those assigned to the project realized it would be difficult to obtain meaningful statistical data in the standard sense for any cancer because of variables such as histology, stage, previous conventional therapy, etc—and, most important, compliance. As a result, the NCI scientists involved with the effort had developed what they called the “Best Case Series” approach, in which practitioners would present a group of their patients with appropriately diagnosed poor prognosis or terminal cancer who had experienced disease regression and/or long-term survival not usually seen for similar patients in conventional oncology. If a large number of impressive cases could be culled from a single physician’s experience, such findings might justify investment in controlled clinical trials. It is my understanding that NCI still finds the “Best Case” approach the most effective, in terms of evaluating patients treated outside of a formal controlled clinical study, in a private practice–type situation.
Dr Isaacs and I eventually obtained support from NCI to test our therapy in a controlled clinical trial in patients diagnosed with advanced pancreatic cancer. Unfortunately the study was so mismanaged by the academic supervisors assigned to the project that the data in our estimation has no meaning. My book, What Went Wrong, explains in some detail the tribulations and shortcomings of this effort.9 We learned from this experience that a clinical trial supervised at the highest level of academic medicine can go seriously awry, yielding, in our opinion, no information of value.
Recently, within the academic world there has been a resurgence of interest in case reports that document an unusual event or an unusual response to a new treatment. In the foreword to the book Clinical Case Reporting in Evidencebased Medicine, Dr René Mornex10 of the Claude Bernard University in France argues for the time-honored value in studying case histories of patients seen and treated outside of a formal clinical trial setting:
In addition to its pedagogical benefits, the presentation of clinical cases has heuristic implications. A case marks the beginning of a series of cases and therefore an epidemiological study. It is also the starting point of physiopathological hypotheses leading to confirmatory experimental studies. Numerous examples of this exist and communicable pathologies are a painful reality that have led to the opening of new nosological chapters stemming from a single case (eg, AIDS, mad cow disease). Many other examples can also be found in pharmacology (the discovery of sulfonylureas, for example), to the point where drug monitoring based on case studies has become a fully fledged discipline. In my area of specialty, endocrinology, it is well known that the fundamental role of the thyroid on all bodily functions was suggested to Kocher by the single observation of a patient who had recently undergone a thyroidectomy.
J. P. Vandenbroucke,11 writing in Annals of Internal Medicine, wrote a thoughtful defense of well-constructed case reports. In his article ,“In Defense of Case Reports and Case Series” he writes, “Case reports and series have a high sensitivity for detecting novelty and therefore remain one of the cornerstones of medical progress; they provide many new ideas in medicine.” So case reports, as distinct from “statistics,” do have value, potentially substantial value, as Dr Good said to me many years ago. Currently, I (Dr Gonzalez) am in the process of completing a set of more than 100 of our own patient case histories, written in some detail, to document the effect of our therapy in both cancer and nonmalignant illness such as chronic fatigue, Lyme disease, and multiple sclerosis. After our very troubling experience with the National Cancer Institute-National Institutes of Health, Dr Isaacs and I have come to believe this project is the best way, perhaps the only way, to get useful information out into the world about our regimen as applied today. The cancer cases, though describing patients not treated within the confines of an academic clinical study, should be convincing enough of the value of our treatment to anyone with an open mind. (Altern Ther Health Med. 2015;21(2):11-.)
1. Cancer Treatment Centers of America. Our cancer treatment statistics and results. http://www.cancercenter.com/ctca-results/. Accessed January 15, 2015.
2. Cancer Treatment Centers of America. CTCA survival comparison statistical methodology. http://www.cancercenter.com/ctca-results/survival-comparisonstatistical-methodology/. Accessed January 15, 2015.
3. Seve P, Sawyer M, Hanson J, Broussolle C, Dumontet C, Mackey JR. The influence of comorbidities, age, and performance status on the prognosis and treatment of patients with metastatic carcinomas of unknown primary site: a population-based study. Cancer. 2006;106(9):2058-2066.
4. Memorial Sloan Kettering Cancer Center. Our survival data. http:// www.mskcc.org/cancer-care/hospital-information/survival-data. Accessed January 15, 2015.
5. Memorial Sloan Kettering Cancer Center. Learn more about survival data. http:// www.mskcc.org/cancer-care/hospital-information/learn-more-about-survivaldata. Accessed January 15, 2015.
6. Vater LB, Donohue JM, Arnold R, White DB, Chu E, Schenker Y. What are cancer centers advertising to the public?: a content analysis. Ann Intern Med. 2014;160(12):813-820.
7. Abel GA. Public advertising by cancer centers: are more data needed? Ann Intern Med. 2014;160(12):870-871.
8. Gonzalez NJ. One Man Alone; An Investigation of Nutrition, Cancer, and William Donald Kelley. New York, NY: New Spring Press; 2010.
9. Gonzalez NJ. What Went Wrong: The Truth Behind the Clinical Trial of the Enzyme Treatment of Cancer. New York, NY: New Spring Press; 2012.
10. Jenicek M. Clinical Case Reporting in Evidence-based Medicine. 2nd ed. New York: Oxford University Press; 2001.
11. Vandenbroucke JP. In defense of case reports and case series. Ann Intern Med. 2001;134(4):330-334.
