Am how to do it in practice

Am how to do it in practice

• 1.
  For the fullversions of these articles see bmj.com PRACTICE THE COMPETENT NOVICE Audit: how to do it in practice Andrea Benjamin BMJ Learning, BMA House, Junior doctors can find the process of doing effectiveness of audit and feedback was likely to be London WC1H 9JR greater when baseline adherence to recommended ABenjamin@bmjgroup.com an audit helpful in gaining an understanding practice was low and when feedback was delivered of the healthcare process—here’s how to more intensively. However, the evidence presented in BMJ 2008;336:1241-5 doi:10.1136/bmj.39527.628322.AD do one this review did not support mandatory use of audit and feedback as an intervention to change practice.5 In the United Kingdom, doctors in the first two years It seems ironic that the tool used to measure whether after graduation are asked to perform an audit. Audit we are doing the right thing in the right way—where measures practice against standards. Unlike research audit criteria should be derived from the best evidence (which asks the question, “what is the right thing to —is not itself supported by particularly strong evi- do?”), clinical audit asks, “are we doing the right thing dence. So why do we persist in audit? in the right way?”1 As audit is part of clinical governance, to assess Clinical audit forms part of clinical governance, whether patients are receiving the best quality of care it is which aims to ensure that patients receive the best essential that we measure practice to know when we need quality of care. Clinical governance is often defined as to change it. Audit provides us with the best available tool how NHS organisations are accountable for continu- to achieve this objective. An example of a success story is ally improving the quality of their services and the national audit of stroke in the United Kingdom safeguarding high standards of care.2 (excluding Scotland), which has improved quality of care Audit can include assessment of: for stroke patients across all three countries.6 The structure of care—for example, resources such as Audit is included in the foundation programme to the presence of a dedicated stroke unit allow you, as junior doctors, to gain an understanding of The process of care—for example, waiting times in how to obtain, maintain, and improve the services you clinics deliver now and in the future. Performing an audit may The outcome of care—for example, blood pressure also help you in your own learning and understanding of reduction in response to therapy. the healthcare process in a particular field. It may also Audit should also be transparent. It should not be allow you to contribute to constructing or refining a confrontational or judgmental—it is not an opportunity clinical protocol. to name, shame, and blame. Barriers to successful audit may have been a reason for the lack of effect of audit found in some of the trials Does audit work? included in the review mentioned above. So we all need There is conflicting evidence on whether audit works. Audit and feedback has not consistently been found to be effective.3 For every success story there is a project Stage 1: that has run into the ground without showing any Preparing for audit substantial contribution to quality of services. The most frequently cited barrier to successful clinical audit is the failure of organisations to provide sufficient protected Stage 5: Stage 2: Sustaining Selecting criteria time for healthcare teams to participate.4 improvements for audit review The most recent systematic review concluded that providing healthcare professionals with data about their performance in the form of audit and feedback Stage 4: Stage 3: This series aims to help junior doctors in their daily tasks and is may help improve their practice.5 However, in the Making Measuring level based on selected topics from the trials included in the review, the effects varied widely, improvements of performance UK core curriculum for foundation from an apparent negative to a very large positive years 1 and 2, the first two years after graduation from medical effect. When effective, the effects were mainly small to school. moderate. The review concluded that the relative The audit cycle BMJ | 31 MAY 2008 | VOLUME 336 1241
• 2.
  PRACTICE to try to overcome these barriers when we perform an audit. Summary of elements of effective clinical audit* Clinical audit should assess structure, process, or How can you conduct an audit so that it is a success outcomes of care rather than a failure? The audit should be part of a structured programme and If you are aware of the most frequent impediments to a should have a local lead successful audit, you can try to avoid them and make Audit should ideally be multidisciplinary your audit a success rather than a failure. Patients should ideally be part of the audit Good quality evidence on how to conduct an audit is Choose audit topics based on high risk, high volume, or lacking. However, the two elements that have been high cost problems or on national clinical audits, shown to be most effective in conducting an audit, national service frameworks, or NICE guidelines based on evidence from a review7 and an opinion Derive standards from good quality guidelines piece,8 are: an environment where audit is made a Use action plans to overcome the local barriers to priority by the trust board, so that it is encouraged and change, and identify those responsible for service supported; and the existence of a structured pro- improvement gramme for audit, where a trust has a central clinical Repeat the audit to find out whether improvements in audit office that coordinates audit activity and brings care have been implemented as a result of clinical audit together the results of audit for the trust as a whole. If Develop specific mechanisms and systems to monitor these elements are not present, the audit is less likely to and sustain service improvements once the audit cycle be a success. has been completed Most healthcare trusts should have a clinical audit *Based on A Practical Handbook for Clinical Audit16 lead. This person is responsible for creating and implementing a strategy for clinical audit, setting Have patients recommended topics? Patients’ priorities for audit, and implementing an audit programme. priorities can differ markedly from those of clin- Data for an audit are generally collected retrospec- icians. Practical approaches have been developed tively. However, prospective data collection can give a for involving patients in all stages of audit (including team immediate feedback on its current performance design), data collection, and implementing and act as positive reinforcement to improve or maintain change.11 12 Is good evidence available to inform standards— practice. Prospective audit usually requires good infor- mation technology resources.4 9 10 for example, systematic reviews or national clinical guidelines? The audit cycle and spiral Locate relevant information Clinical audit can be described as a cyclical or spiral Where can you find clinical guidelines? From the systematic process (figure), with the ultimate aim of National Institute for Health and Clinical Excellence improving care. The spiral suggests that as the process (www.nice.org.uk); National Library for Health continues, each cycle aspires to a higher level of quality. (www.library.nhs.uk); and Scottish Intercollegiate Guidelines Network (www.sign.ac.uk). Stage 1: Preparing for the audit Where can you find criteria for clinical audit? From Identify problem and local resources for audit clinical guidelines or local hospital guidelines. Selecting a topic for audit depends on the objectives of Where can you find information on service standards? the audit and is likely to involve measuring adherence to -From national service frameworks, which are long healthcare processes that have been shown to produce term strategies determined by various stakeholders— best outcomes for patients. Consider also incorporating such as health professionals, service users, and managers the views of all healthcare professionals involved in —to improve specific areas of care by setting measurable patient care, as in the national stroke audit.4 5 goals within set time frames (www.dh.gov.uk/en/Site The clinical team has an important role in prioritising map/DH_A-Z_AZSI). For example, the goal that no clinical topics. The following questions may help you patient should wait longer than one month from an select a topic: urgent referral by their GP with suspected cancer, to the Is the topic a priority for the organisation? For start of treatment.13 example, have problems been encountered in any of -From the National Centre for Health Outcomes the following areas? Development (http://nchod.uhce.ox.ac.uk) -High volume—such as requests for chest x rays in the -From the Health Commission Wales (Specialist accident and emergency department Services) (http://new.wales.gov.uk/topics/health/ -High risk to staff—such as needle stick injuries in an hcw/?lang=en). HIV unit Which organisations have information about clinical -High risk to patients—such as certain postoperative audit? The royal colleges and other professional complications bodies. The Clinical Effectiveness and Evaluation -High cost—such as trastuzumab for breast cancer. Unit of the Royal College of Physicians (www. 1242 BMJ | 31 MAY 2008 | VOLUME 336
• 3.
  PRACTICE Worked example: how to perform the perfect audit Stage 1: Preparing for audit Identify the problem and the local resources available for audit Dr Black is a foundation year 2 doctor on a respiratory firm. One month into his attachment, he has noticed that patients with chronic obstructive airways disease who met the criteria for starting non-invasive ventilation had not been given this treatment. He discusses this with his consultant, who encourages him to do an audit. Dr Black discovers that his hospital has a local audit lead and a clinical audit office. He contacts them and is told that they will be able to help him with the audit. In agreement with his local audit lead, he chooses the British Thoracic Society guidelines* on non-invasive ventilation in acute respiratory failure and finds these on the society’s website. Stage 2: Selecting criteria Determine what you are trying to measure and define gold standards Dr Black writes the following criterion statement, derived from the thoracic society’s guidelines: To measure the percentage of patients with an acute exacerbation of chronic obstructive pulmonary disease with a respiratory acidosis (pH 7.35), despite maximum medical treatment on controlled oxygen therapy, who received treatment with non-invasive ventilation. After discussion with the respiratory team, he chooses a standard of 90% as he thinks that in the given environment, as an optimum standard, this percentage is more realistic than an ideal standard of 100%. Stage 3: Measuring level of performance Collect data He decides to collect data from the medical notes of the last 50 patients admitted with an acute exacerbation of chronic obstructive pulmonary disease. He asks the audit office to obtain the notes for him. He creates the following audit proforma (then it takes him two afternoons to collect the data): 1 Does the patient have a respiratory acidosis (pH 7.35) despite maximum medical treatment on controlled oxygen therapy? Yes/No 2 Were they treated with non-invasive ventilation? Yes/No 3 If they were not treated with non-invasive ventilation, was a reason given? Yes/No Compare performance with criteria Dr Black analyses the data against the criterion statement. Forty of the 50 patients had a respiratory acidosis, despite maximum medical treatment on controlled oxygen therapy; 31 of these 40 patients (77.5%) were treated with non-invasive ventilation. This percentage did not meet the standard of 90%, so he looked at the reasons why non-invasive ventilation had not been started in nine of these patients. In four patients the respiratory acidosis was noted but it was documented to continue medical treatment. In two patients the respiratory acidosis was thought to be a metabolic acidosis. In two patients no non- invasive ventilation machine was available, and both patients required intubation. In one patient, by the time the decision to start non-invasive ventilation had been made by the medical registrar, the patient had required intubation. Stage 4: Making improvements Three months into his attachment, Dr Black presents the audit to the medical and emergency departments. The consensus is that the two main reasons that non-invasive ventilation was not used were lack of enough machines and lack of knowledge of medical staff on when to use it. The clinical directors of the medical and emergency departments agree on the following action plan to improve the use of non-invasive ventilation in these patients: 1 Educate all medical staff on the interpretation of arterial blood gases and use of non-invasive ventilation. This education was to be delivered by tutorials on blood gas interpretation for all junior doctors twice yearly and training in non-invasive ventilation. The respiratory consultants and specialist registrars were to give these in the first week for all new starters. 2 Display posters on when to use non-invasive ventilation in patients with chronic obstructive pulmonary disease in the emergency departments and medical wards. The posters were to be designed by the clinical director of the emergency department and produced by the medical illustration department. The plan was to display them within the next three months. 3 Develop a business plan to increase the number of non-invasive ventilation machines in thehospital. This was to be drafted by the clinical directors of the medical and emergency departments and presented to the chief executive within the next three months. Stage 5: Sustaining improvements Repeat audit After a year, although Dr Black is now in gastroenterology, he repeats the audit. The first two changes on the action plan have been implemented successfully, and the hospital now has three more non-invasive ventilation machines. He finds that 90% of patients are meeting the criteria. Owing to the improvements, Dr Black and the respiratory consultant devise a monitoring tool to ensure that the improvements are sustained. The tool consists of a short checklist on an A4 sheet of paper. This checklist is attached to the notes of all patients with chronic obstructive airways disease. It provides a simple way to ensure that when such patients meet the criteria for treatment with non-invasive ventilation, this treatment is used appropriately. The tool can be used for retrospective or prospective data collection. At the repeat audit, it may be appropriate to set the standard higher than 90%. * The society plans to update this guideline, with work to update it starting in 2008 BMJ | 31 MAY 2008 | VOLUME 336 1243
• 4.
  PRACTICE rcplondon.ac.uk/college/ceeu/index.htm) works to promote clinical standards through audits, guide- Tips for busy clinicians lines, and related activities to improve health care. Before you start an audit, find out whether your hospital has a local audit lead. If it does, contact him or her for Stage 2: Selecting audit review criteria advice and information You can use recommendations from clinical practice Choose a topic that reflects national clinical audits, guidelines to develop criteria and standards. This could national service frameworks, or NICE guidelines save you time and additional work. Use an audit proforma. This will make collecting the necessary data easier Determine what you are trying to measure Audit criteria are explicit statements defining an outcome to be measured. They should relate to important aspects of care and be derived from the collect data manually depending on the outcome being best available evidence. Having explicit selection measured. In either case, you will need to consider criteria will ensure that the data you collect are precise what data you need to collect, where you will find the and that you collect only essential information. data, and who will collect the data. Although clinical records are frequently used as the Define ideal standards source of data, they are often incomplete. Collecting For the criterion to be useful, you need to define the data from several sources—such as clinical records, standard (the level of care to be achieved for any blood results from patient administration systems, and particular criterion, which is usually expressed as a imaging from picture archiving and communications percentage).14 Ensure that the standard you choose is systems—can help to overcome this problem. realistic for your given environment (see the box with a Electronic information systems are useful not only worked example). for collecting data but also for improving access to Anderson, in his ABC of Audit, writes: “A minimum research evidence, prompting change through record standard describes the lowest acceptable standard of templates, and introducing revised systems of care. performance. Minimum standards are often used to distinguish between acceptable and unacceptable Compare performance with criteria practice. An ideal standard describes the care it This is the analysis stage. should be possible to give under ideal conditions, Compare the data collected with criteria and with no constraints. Such a standard by definition standards cannot usually be attained. An optimum standard Conclude how well the standards were met lies between the minimum and the ideal. Setting an If they were not met, identify reasons for this. optimum standard requires judgment, discussion In theory, if the standard was not met in 100% of the and consensus with other members of the team. Optimum standards represent the standard of care standard that was set, there is potential for improving care. most likely to be achieved under normal conditions Remember, the standard set may have been 90%. In of practice.”15 practice, if the results are close to 100% of the standard, Two examples of audit criteria (with a different you may decide that any further improvement will be standard defined for each criterion) are: difficult to achieve and that other standards, with results further away from 100%, are the priority targets for action. To measure what percentage of patients with septic However, this decision also depends on the topic—in shock were given anti-infective treatment. As septic some life threatening situations, it will be important to shock is a condition with a high mortality rate, it achieve 100% of the standard. would be appropriate to aim for a standard of 100%. To measure what percentage of patients in the Stage 4: Making improvements rheumatology outpatient clinic were seen within Implement change one hour. Here it may be acceptable for the standard Data collection has no chance of making any impact for the first audit to be 80%. unless you follow it up with the more difficult process of implementing changes. If you have completed Stage 3: Measuring levels of performance stages 1 to 3 early in your four month attachment, you Collect data will be able to present your results within this time Some hospitals have audit teams that may help with period. data collection. An audit proforma is useful for Once you have presented and discussed the audit collecting data and can be readily derived from results with the rest of the team and whoever else in the established guidelines and protocols. You need to hospital the audit is relevant to, you must agree on define the patients to be included and excluded in the recommendations for change. Use an action plan to audit, the audit review criteria, and the time period over record these recommendations, also indicating who which the criteria apply. has agreed to do what and by when. The data may be available in a computerised Disseminating educational materials, such as guide- information system, but it may also be appropriate to lines, has little effect unless accompanied by selected 1244 BMJ | 31 MAY 2008 | VOLUME 336
• 5.
  PRACTICE implementation methods, such as tutorials, reviews, or KEY POINTS reminders.4 Audit measures practice against performance Stage 5: Sustaining improvements The audit cycle involves five stages: preparing for audit; This stage is critical to the successful outcome of an selecting criteria; measuring performance level; making audit: it verifies whether the changes implemented improvements; sustaining improvements have had an effect and determines whether further Choose audit topics based on high risk, high volume, or high improvements are needed to achieve the standards cost problems, or on national clinical audits, national service identified in stage 2. frameworks, or guidelines from the National Institute for Health and Clinical Excellence Repeat the audit Derive standards from good quality guidelines Although as a foundation doctor you will not be able to Use action plans to overcome the local barriers to change complete the full audit cycle within your four month and identify those responsible for service improvement attachment, you will be able to collect and compare the Repeat the audit to find out whether improvements in care data within this time period. However, to complete the have been implemented after the first audit cycle, after an agreed period, the audit needs to be repeated. If you wish to complete the audit cycle yourself, you could come back and complete the cycle. If not, Finally, remember that audits are more likely to be you or your consultant must ensure that the cycle effective when baseline adherence to recommended is completed by someone else otherwise your time practice is low and feedback is delivered more spent on stages 1 to 3 of the audit cycle will have been intensively.5 wasted. You or whoever is going to complete the audit cycle This article is based on a previous BMJ Learning module by the same (possibly the next doctor on the rotation) should use the author (http://learning.bmj.com/learning/main.html). same strategies for doing the audit to ensure the original Contributors: AB is the sole contributor. audit is comparable. The repeat audit will hopefully Competing interests: None declared. Provenance and peer review: Commissioned; externally peer reviewed. show that changes have been implemented and improvements made. 1 Smith R. Audit and research. BMJ 1992;305:905-6. 2 Scally G, Donaldson LJ. Clinical governance and the drive for quality Develop tools to sustain improvements improvement in the new NHS in England. BMJ 1998;317:61-5. 3 Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L. If these improvements are sustained, some form of Changing provider behavior: an overview of systematic reviews of monitoring should replace a full audit. The team should interventions. Med Care 2001;39(suppl 2):II2-II45. develop structures and systems that integrate, monitor, 4 National Institute for Health and Clinical Excellence. Principles of best practice in clinical audit. London: NICE, 2002. and sustain the improvements implemented as part of 5 Jamtvedt G, Young JM, Kristoffersen DT, O’Brien MA, Oxman AD. Audit clinical audit. But if performance deteriorates, the full and feedback: effects on professional practice and health care audit should be reactivated. outcomes. Cochrane Database Syst Rev 2006;(2):CD000259. 6 National sentinel stroke audit 2006. Prepared on behalf of the Intercollegiate Stroke Working Party. London: Royal College of Physicians, 2007. What are the challenges? 7 Morrell C, Harvey G, Kitson A. Practitioner based quality improvement: The main challenge is how to make your audit a a review of the Royal College of Nursing’s dynamic standard setting system. Qual Health Care 1997;6:29-34. success. Firstly, you need support. Hopefully your trust 8 Fraser R, Baker R. The clinical audit programme in England: will have a clinical audit lead; if not, ensure that your achievements and challenges. Audit Trends 1997;5:131-6. consultant is on your side and enthusiastic, otherwise 9 Dixon N. Good practice in clinical audit. A summary of selected literature to support criteria for clinical audit. London: National Centre achieving success may be difficult. Tell your consultant for Clinical Audit, 1996. about a local audit presentation you attended where a 10 Kendall JM, McCabe SE. The use of audit to set up a thrombolysis repeat audit showed that patient care really had programme in the accident and emergency department. Emerg Med J 1996;13:49-53. improved after the audit or cite an audit success story, 11 Kelson M. Promoting patient involvement in clinical audit. Practical such as the national stroke audit. guidance on achieving effective involvement. London: College of Health and the Clinical Outcomes Group, 1998. Secondly, give yourself enough time—you should 12 Balogh R, Simpson A, Bond S. Involving clients in clinical audits of probably start thinking about choosing an audit topic mental health services. Int J Qual Health Care 1995;7:343-53. one month into your attachment. With access to a 13 Department of Health. The NHS cancer plan: a plan for investment, a plan for reform. computer and case notes, in a retrospective audit you 2000. www.dh.gov.uk/en/Publicationsandstatistics/Publications/ could collect the data within hours. PublicationsPolicyAndGuidance/DH_4009609. Thirdly, be realistic when setting the standards for 14 Irvine D, Irvine S. Making sense of audit. Oxford: Radcliffe Medical Press, 1991. your audit. Don’t choose a standard of 100% if you 15 Anderson DG. ABC of audit. Cleveland Vocational Training Scheme. know that your trust cannot possibly meet this ideal www.pdptoolkit.co.uk/Files/Guide%20to%20the%20PDP/content/ standard. Aim for an optimum rather than an ideal audit.htm. 16 NHS Clinical Governance Support Team. A practical handbook for standard—discuss with the team what the optimum clinical audit. 2005. www.cgsupport.nhs.uk/Resources/ standard should be. Clinical_Audit/7@Appendix_1.asp BMJ | 31 MAY 2008 | VOLUME 336 1245