As you go about making changes in your organization, a few questions will naturally arise:
By collecting data before, during, and after the change you implement, you can measure, evaluate, and compare your agency’s progress with respect to the goals you set out. The process of measuring change should speed the improvement process; you should begin with simple measures rather than spending time developing a complex measurement system.
Measuring the impact of change is an important aspect of successful organizational improvement. These steps are designed to help guide your agency in the timely and accurate measurement of change.
Change Teams establish clear measures and definitions prior to the start of a change project. The measures should clarify the project objectives and should be agreed upon by key stakeholders.
Think about what results you would see. For example:
Never start a change process without collecting baseline data. In doing so, you should clearly define a starting point for the change and work backwards to collect two to three months of baseline data before making any changes. This process anchors the change (pre-change) and enables your agency to measure the impact of the change over time (post-change). Baseline data should be collected using agreed upon measure definitions. More importantly, the collection of baseline data allows an agency to answer four questions and make adjustments as necessary:
Establish a clear improvement aim or target (e.g., reduce client no-shows from 65 percent to 25 percent). Such a target should:
Remember to be flexible when establishing an aim. If the change project uncovers new information that suggests changing the aim, change it.
If the aim is too ambitious, set a realistic aim that still challenges the agency to improve. If the aim is easily achieved, set a more ambitious aim that stretches the agency's capacity to improve.
The ability to establish consistent channels for collecting measurement data on a regular basis is a crucial part of the change process. Such channels may exist in existing data systems, but in other cases you may need to manually collect the data.
Often in the PDSA change process, it is easier to rely on manual collection for quick and rapid feedback on the success of the change. This means relying on small samples collected over short time periods to measure change progress.
For example, your agency might use reminder phone calls to reduce no-shows and your rapid measure might collect success or failure (e.g., simple tally marks) of the intervention for the next 10 clients with scheduled appointments. These results would then be compared to the baseline measure. Existing systems might be used for longer-term reporting (e.g., monthly) on the change progress. If your agency can affirmatively answer the questions in 2, you most likely have systems in place to consistently collect measurement data related to the change.
Over time your agency will collect both pre-change (baseline) and post-change data, and the data should be shared with the change team as well as others in your organization.
The most effective tool for sharing this information is charting your progress over time using simple line graphs created in Excel. These powerful visual aids should follow one simple axiom: one graph, one message.
Charts offer Change Team members, as well as the entire organization, several key pieces of information. Charts can be used to:
In developing charts, your agency should consider the following points:
In general, the charts should be used to compare your progress over time, not to compare programs or individuals within your organization. In some instances, it does make sense to compare performance by program or counselors (e.g., no-show or continuation rates), but using this comparative approach to change behavior requires careful consideration and implementation. Comparisons should not be used in a punitive way; they should only be used to improve performance
Measuring the impact of change does not stop here; in fact, it is only the beginning. The most important step in the process is to ask: What is the information telling me about change in my organization?
If change is successful, the information you have collected may tell you which intervention had the most success in meeting your aim. For example, one NIATx member sought to reduce the time from first contact to treatment (aim) by increasing professional staff availability. The change reduced the time from first call to first treatment from 18 days to 5 days, and in examining their data, the agency found that only physician and nurse practitioner availability played a role in the improvement.
Unsuccessful changes also afford your agency the opportunity to ask "Why?" Another NIATx member examined the characteristics of clients not continuing through the fourth treatment session and found that clients admitted to treatment on Fridays were more likely to drop out. The organization then stopped offering Friday admissions.