2010)
Health Policy and Clinical Practice2
DC
device makers often fail to properly conduct safety studies and the US Food and
Drug Administration provides scant oversight. Jeanne Lenzer and Shannon Brownlee look
at some of the problems with post approval surveillance of novel devices
Administration’s neurological devices panel met to consider approval of a vagus
nerve stimulator (VNS). The manufacturer, Cyberonics, said it could prevent or
reduce seizures in patients with partial onset epilepsy who did not respond to
drug treatment. The device consists of a generator the size of a matchbox that
is implanted under the skin below the patient’s clavicle. Lead wires from the
generator are tunnelled up to the patient’s neck and wrapped around the left
vagus nerve at the carotid sheath, where it delivers electrical impulses to the
nerve lasting about 30 seconds every 3-5 minutes.
approval
definitive explanation during the FDA meeting of how the device stopped or
reduced seizures, but they had three studies, E03, E04, and E05, to show its
safety and efficacy.
studies, E03 and E05, involved 313 patients with treatment resistant partial
seizures randomised to high or low dose stimulation. The low stimulation arm
was intended to avoid the problem of an unblinded placebo arm because all
patients would be implanted and told they were receiving stimulation. The
studies did not include a medical treatment arm for comparison, leaving
unanswered the question of whether either treatment arm was superior to
existing care.
reported that 25% of patients in the high stimulation arms of the trials
achieved the primary end point: a 50% reduction in seizure frequency from baseline.
However, 20% of patients in the high stimulation arm had more seizures.1
about safety
cause of death among 17 patients of the 1000 implanted with the device
worldwide. The device had been approved in Europe and Australia before approval
in the US. The most common cause of death among young epileptics is sudden
unexpected death from epilepsy (SUDEP), a poorly understood complication that
is thought to be related to cardiac or respiratory arrest that sometimes occurs
shortly after a seizure. SUDEP is reported most often among younger patients
and those with poorly controlled seizures.2 3 The company told FDA panellists that
the deaths were from SUDEP or other causes unrelated to its product.1 However, one panellist, Steven
Piantadosi, professor of oncology and biostatistics and a clinical trial
methodologist at Johns Hopkins, expressed concerns, saying, “I’m still a little
worried about the death rates that we are seeing.”
data provided by Cyberonics, patients implanted with the device had a SUDEP
rate of 7.3/1000. Dr Piantadosi asked, “Should we be concerned by that?”
response to Dr Piantadosi’s questions, Ann Costello, an FDA reviewer, cited one
epidemiological study that showed an even higher SUDEP rate, 9.3/1000, among
treatment resistant patients about to have brain surgery to treat their severe
seizures.4 W Allen
Hauser, a member of Cyberonics’ scientific advisory board, also responded to Dr
Piantadosi’s concerns, saying, “I don’t think that the sudden death is an issue
specific to the device. It’s a specific issue in terms of people with bad
epilepsy.”
of approval
about the SUDEP rate, but the advisory panel nonetheless voted unanimously to
approve the vagus nerve stimulator for patients with treatment resistant
partial seizures. The FDA made the approval conditional: Cyberonics would have
to conduct a post-approval study to examine the safety of the device and it
would have to report promptly all serious adverse reactions to the agency.5
the 13 years since the device was approved in the US, more than 900 deaths have
been reported to the FDA, and it is still not clear what impact, if any, the
device has had on patient mortality. Although Cyberonics conducted
post-approval studies, none of the studies submitted to the FDA included
mortality data. The FDA did not specifically require Cyberonics to submit
mortality data as part of the follow up study, merely to “characterize
morbidity and mortality.” According to a spokesman for Cyberonics, the company
did collect mortality data from the Social Security Death Index, but they have
not published the results and they declined to show the data to the BMJ. The FDA told the BMJ that it has not
requested further studies concerning mortality.
with post-approval surveillance
monitor mortality data related to the VNS is but one example of the gap in
post-approval surveillance of medical devices. Most devices and drugs on the
market are supported by studies that are underpowered to detect rare but
potentially life threatening events that can kill tens of thousands of people
if the drug or device is widely used.6 The impracticality of conducting
large scale clinical trials before approval for every drug and device places a
burden on post-approval surveillance.
burden is especially important for devices because they are less likely than
drugs to be supported by clinical studies before use. Less than one third of
devices approved under FDA’s premarket approval process had been evaluated in a
randomised study, according to a review of 78 high risk cardiovascular devices
approved by the FDA from 2000 to 2007.7 Just 5% were supported by two or
more blinded, randomised studies. Most of the outcomes measured, according to
the FDA study, were surrogate markers.
monitor safety of devices, the FDA relies on reports of harms associated with
devices once they have been approved. According to the agency, the most
“comprehensive source of information about the safety and effectiveness” of
devices as they are used in everyday circumstances is its Manufacturer and User
Facility Device Experience (MAUDE) database.8 However, the data contained in the
database does not constitute a comprehensive record of the numbers of adverse
events or exposures, and thus can only pick up “signals” of possible safety
problems that could be used to trigger more definitive investigation (box). Of
course, these signals can be picked up only if someone is monitoring the data.
An FDA taskforce on device regulation concluded in August, however, that
“challenges” in “current data sources . . . make it difficult to . . .
effectively obtain complete information about the risks and benefits of
regulated products.”9
reports of adverse events associated with medical devices doubled from 2003 to
2007,8 the MAUDE
database remains an imperfect tool.6 10 Several factors can contribute to
under-reporting, including the voluntary nature of the reports; fear of
litigation by surgeons and others in a position to report the event; and
failure by patients and healthcare providers to connect new medical problems
with a device.11 Even
when device related adverse events do make it into FDA’s databases, 39% are
reported late.8 A review
by the BMJ of
deaths among vagus nerve stimulator patients listed in the FDA’s database shows
that some deaths weren’t reported until several years after the patient died.
the fact that manufacturers—not the FDA or any other independent body—can decide
whether the device is connected with a negative outcome. Manufacturers are not
required to report deaths or serious adverse events if they decide that the
events were unrelated to the device.12
FDA’s ability to detect potentially unsafe devices is further hampered by the
fact that many post-approval studies required as a condition of the device’s
approval are not conducted or conducted so poorly as to be meaningless. In
2005, the FDA evaluated the quality of post-approval studies. Susan Gardner,
director of the Office of Surveillance and Biometrics at the Centers for Device
and Radiological Health, which oversees approval and safety of devices, told an
FDA advisory panel that 45 of the 127 premarket approvals granted between 1998
and 2000 had orders requiring post-approval surveillance or studies. But more
than one fifth of the studies, she said, couldn’t be evaluated for quality
because there was no record at the FDA showing they had ever even been
conducted.13 FDA has
since automated its records but many companies continue to submit data deemed
“inadequate” by the agency, while other studies remain unreported.
signs
about the vagus nerve stimulator? Most MAUDE death reports, which are mainly
written by manufacturers, contain descriptions so brief that no definitive
cause of death can be inferred. Despite the paucity of information available
(narratives are often just a sentence or two in length), the device is often
exonerated. For example, one death associated with the VNS was reported to be
from SUDEP on the basis of a “visual autopsy.” Marcia Angell, a pathologist and
former editor in chief of the New
England Journal of Medicine, says that without an actual autopsy it
is impossible to determine whether it was SUDEP or some other factor, including
potentially the device itself, that caused the patient’s death, and even then
causality could remain uncertain.
narrative descriptions, however, are clear enough that further investigation of
the device would seem warranted. For example, on 4 December 2008, a patient
with a vagus nerve stimulator was admitted to hospital after an abrupt increase
in seizures and was found to have “severe asystolia” coincident with vagus
nerve stimulation. The device was disabled and the asystole ceased. The
database has more than a dozen reports in which the device’s activity seems to
be associated with asystole.
database also contains reports of what could conceivably be downstream effects
of asystole and bradycardia, such as increased “seizures,” new onset drop
attacks, and fainting—which in turn could be responsible for some instances in
which patients were reported to have died in a fall, by drowning, or in a road
crash.
case that ostensibly should have prompted further investigation by the FDA, was
that of 48 year old Dennis Fegan. According to medical records obtained by the BMJ with Mr Fegan’s
permission, he woke up in pain at 2 am on 2 July 2006, only to pass out. After
regaining consciousness, Mr Fegan, who had a 14 year history of partial complex
seizures, passed out several more times. He thought he was having an unusually bad
run of seizures. When an ambulance crew arrived, they witnessed Fegan having a
seizure and administered intravenous diazepam. Once he was at the hospital and
connected to a cardiac monitor, an apparent cause of his seizures emerged; at 3
minute intervals, his heart stopped for 30 seconds—synchronous with the firing
of his vagus nerve stimulator. As soon as the device was disabled, the asystole
ceased. His neurologist, a consulting cardiologist, and the treating emergency
physician concluded that the device was the likely cause of his asystole.
study bias, and failure to report
“is not intended to be used either to evaluate rates of adverse events or to
compare adverse event occurrence rates across devices.”14 Its purpose is to allow FDA
analysts and others to pick up “signals” of harm that can trigger further
investigation. Any investigation would need to evaluate the numerator (number
of adverse events) and the denominator (number of exposures), but this
information is routinely absent.
the case of the vagus nerve stimulator the denominator depends not only on the
number of patients implanted with the device but also on how many devices are
still functioning. Cyberonics reports that by 10 March 2010, they had received
registrations for 57 284 patients worldwide implanted with the device, yet it
acknowledges that it is impossible to know how many patients have had their
devices deactivated because it has no way to collect these data. The
combination of possible under-reporting of adverse events (the numerator) and
possible over-reporting of the number of active devices (the denominator) could
mean that the rate of deaths among device users is higher than is apparent. The
more than 900 deaths among relatively young people (most of those with implants
are 15 to 44 years old) 3 did not trigger a request for
further investigation.
what of the post-approval study, known as XE05, ordered by the FDA as a
condition of approval? A spokeswoman for the FDA cited XE05 as part of the
evidence that “supported the long term safety and effectiveness of the device.”
Cyberonics confirmed that only 50 patients were enrolled in the open label
study and deaths were not recorded. When asked how such a small study that
didn’t include mortality data could show the device’s “long term safety,”
Cyberonics replied that “the purpose of that study wasn’t to look at SUDEP or
mortality rates” but to evaluate other long term safety issues.
FDA gave the BMJ
references to five additional post-approval studies as evidence of the device’s
safety. Yet the five studies do not establish that the device wasn’t responsible
for deaths because none of them included mortality data. The largest study
consisted of 4743 patients in the company’s outcome registry. The other studies
evaluated subsets of registry patients. Cyberonics acknowledged that “mortality
was not an endpoint collected as part of the Epilepsy Patient Outcomes
Registry.”15
2005, the FDA approved the vagus nerve stimulator for the treatment of
depression, despite the unanimous recommendation against approval by its own
scientists. The FDA experts were concerned, in part, about the device’s safety.
Nicholas Stine, who was then a research associate at Public Citizen, a public
interest group, wrote to the agency, urging that the device should not be
approved for treating depression. He and his coauthor, Peter Lurie, now adviser
to FDA’s assistant commissioner for policy, said that “numerous reports” of
worsening depression, suicides, and sudden deaths during clinical trials of vagus
nerve stimulation “had not been adequately investigated,” and that the
manufacturer “had not demonstrated long term safety of the device.”16 However, Cyberonics told the BMJ: “The FDA concluded
that Cyberonics provided ‘reasonable assurance of safety and effectiveness’ of
VNS therapy for the treatment of depression based on valid scientific evidence.
There is no evidence linking VNS therapy to worsening depression, suicides, or
sudden deaths.”
company has also suggested that the stimulator might have a role in treating
obesity, stroke, traumatic brain injury, and other conditions, and has
previously taken out patents for these potential therapies.17
solutions
vagus nerve stimulator are emblematic of the FDA’s surveillance of all devices.
Many of the problems have relatively easy fixes. Although the MAUDE database is
flawed, it can still serve its purpose if qualitative signals are detected and
trigger better analysis and quantification of a potential safety issue.
Manufacturers could be required to regularly submit denominator data, which
could be facilitated by requiring companies to provide a website link that
patients or their surgeons could use to report whether the device is disabled
or removed. An independent review panel could be appointed to decide whether
adverse outcomes could be excluded from reporting. And, as the FDA has
suggested, mechanisms to limit widespread uptake of new devices could be put in
place so that fewer patients are harmed if it ultimately turns out that newly
approved devices are flawed.
JL and SB have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf
(available on request from JL and declare: no support from any organisation for
the submitted work; no financial relationships with any organisations that
might have an interest in the submitted work in the previous three years; no
other relationships or activities that could appear to have influenced the
submitted work.
review: Commissioned and peer reviewed.
Health Neurological Devices Panel. 10th Meeting of FDA’s Center for Devices and
Radiological Health Neurological Devices Panel. 27 June 1997. www.fda.gov/ohrms/dockets/ac/97/transcpt/3299t1.pdf.
Epilepsy, vagal nerve stimulation by the NCP system, all-cause mortality, and
sudden, unexpected, unexplained death. Epilepsia2000;41:549-53.
Mortality in epilepsy. Epilepsy Behav2007;10:363-76.
epilepsy and its relationship to sudden cardiac death. J
Clin Neurophysiol1991;8:216-22.