Measuring Performance

EMS was established in 1966 by the US Department of Transportation through the Highway Safety Act to reduce death and disability in the pre-hospital setting. EMS executes this objective through ex ante steps, such as training and community education, and ex post steps, such as patient stabilization and transport. With such an important mission, we all have an obligation to hold ourselves accountable, but how do we measure the effectiveness of our agency’s interventions?

Given the variability of EMS delivery and community needs, it has been difficult to develop a uniform body of standards. Evaluation is further complicated by a dual nature system where the majority of responses are for low acuity, time irrelevant conditions and a very small minority are for high acuity, time dependent conditions (e.g. cardiac arrest, uncontrolled hemorrhage and anaphylaxis). Many historical benchmarks, such as response times less than 9 minutes or need for ALS, are a legacy of a previous era (where only paramedics could defibrillate and lay rescuers did not know CPR) or reflect biased, non-evidence based agendas. Fortunately, a hierarchy of performance measures supported by a growing body of research is emerging. The measures are focused on EMS’s core mission (categorized by system design and structure, clinical care and outcomes, and response) and have been expanding to include EMS’s larger role in community health (human resources, finance/funding, quality management and community demographics).

The Hierarchy of EMS Performance

“Just Showing Up” – Transportation Measures

The minimum tools to evaluate an EMS agency.

  • Established emergency medical dispatch that can impact response mode and level
  • Timely arrival of care
  • Statutorily credentialed providers
  • Prescribed equipment
  • Delivery to appropriate, definitive care
  • Delay causing crash rate

“Doing a Good Job” – Clinical Indicators

It is difficult to separate EMS’s contribution to a patient’s clinical outcome from the rest of the healthcare system. The following is an approach developed by the 2007 Consortium of U.S. Metropolitan Municipalities’ EMS Medical Directors’ and is based on tracer conditions designed as a surrogate to measure clinical performance. Tracer conditions are clinical conditions where EMS has a high impact on mortality, morbidity, cost and frequency. The following evidence based treatment bundles have been shown to reduce harm. (Keep in mind that these interventions have been evaluated in bundles, it is not clear which individual interventions contribute to (or detract from) positive survival rates. Further, it is important to recognize that the effectiveness of the interventions do not depend on the credential level of the provider, but are contingent upon being performed correctly in appropriate circumstances. See R. Davis, “Inverse Lifesaving Function? More Paramedics Does Not Equal More Lives Saved,” USA Today, 2 March 2005, D-1.)

  • ST-Segment Elevation Myocardial Infarction (1 in 15 patients treated will avoid a stroke, a second MI or death)
    • Early Administration of Aspirin unless Contraindication
    • 12 Lead EKG Acquisition and Pre-Hospital Notification
    • Interval from EKG to PCI < 90 Minutes
  • Seizure (1 in 4 patients treated will avoid persistent seizure activity)
    • Blood Glucose Evaluation
    • Administration of Benzodiazepines for Status Epilepticus (seizure greater than 15 minutes)
  • Pulmonary Edema (1 in 6 patients treated will avoid the need to be intubated)
    • Nitroglycerin unless Contraindication
    • Non-Invasive Positive Pressure Ventilation
  • Trauma (1 in 11 patients treated will avoid death)
    • Transport to a Trauma Center for Patients Meeting Criteria
  • Cardiac Arrest (1 in 8 patients treated will avoid death)
    • Initiation of Chest Compression and Defibrillation < 5 Minutes from 911 Call
  • Intubation
    • Verification with ETCO2

“Excelling” – Whole-System Measures

Finally, it is important to account for all of the externalities that are produced by showing up and taking care of patients.

  • Rate of Adverse Events
  • Incidence of Occupational Injuries and Illnesses
  • Healthcare Cost per Capita
  • Satisfaction with Care Score
  • Pain Relief Rate
  • Annual Staff Turnover
  • Call Complaint Rate

It is possible to come up with many more proxies for quality, such as percentage of providers seeking advanced credentialing or number of community members taught CPR, however additional research is needed to provide the evidence to support their application even if common sense seems to dictate their persuasiveness.

All measures contain inherent disadvantages. Independent from the quality of care, outcomes will always vary due to patient characteristics, differences in measurement, and chance (random variation). Measuring the wrong things can produce perverse incentives such as increased costs, greater injuries, and less skilled providers. Ultimately, all measures should evaluate what the US National Highway Traffic Safety Administration has dubbed in its 1994 EMS Outcomes Project the “6D’s”: survival (death), impaired physiology (disease), disability, discomfort, satisfaction (dissatisfaction), and cost-effectiveness (destitution). Hopefully the above will serve as a starting point to begin our own self-evaluation.

Please see the following for further reference:

Evidence-Based Performance Measures

Measuring Quality in Emergency Medical Services

Emergency Medical Services Performance Measures

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s