The beast of exception reporting is rearing its head once again, this time in an article in the Health Service Journal (registration required) and in an editorial. What is being looked at here is raw practice data, similar to that produced routinely in Scotland without very much statistical analysis.

Helpfully there are some selected practice level details published by HSJ (5.6Mb Excel) and a summary at PCT level (PDF). In the articles this has been looked at in a journalistic way by finding the extremes and putting them in the headlines (and of course the blogging style is gross generalisation!). Simple things like the standard deviations are essential to give some idea of whether these extremes are the result of chance or other factors. For instance if we measured the height of all GPs we would be surprised if the tallest were ten times as tall as the average. However if we measured the number of suits owned it would be less surprising.

For a start I have looked at a box/whisker plot. In these the box contains the middle 50% of practices and the whiskers contain most of the rest with outliers plotted individually. We see from this that most practices are within quite small ranges.

I have written quite a lot about exception reporting. Analysis is difficult due to multitude of potential reasons for exceptions. We do not see any breakdown on the reason for exceptions in these statistics. QMAS collects the reasons to some extent, and this is visible at practice and PCT level. Although practices with high list growth are removed practices with high list turnover remain in the table. As new patients are automatically excepted this could have a significant effect on the data.

It is difficult to draw any conclusions. That would make the editorial a little dull though.

Many GPs will have made countless calls, sent innumerable letters, to try to goad their wayward patients to face up to their health risks. But the suspicion must remain that many patients have to all intents been dumped out of the NHS; the GP has given up on them, and too many PCTs are failing to bring these patients back.

I would suggest quite the opposite. These patients have given up on the GP and treatment. It is the place of the health service to inform and not to coerce. You can only try so hard. What is suggested is what has been described as a tyranny of health. The words goad and wayward suggest an extremely paternalistic view of the healthcare system. We can look back on the removal of patients from practice list for failure to comply with previous targets and are thankful that exception reporting has taken us away from there. We must not go back.

Updated 8th April

I have updated the boxplots with better ones (see the comment below). I should probably just leave the defaults on my stats package! There are quite a lot of points plotted but it is important to remember that there are around 8000 practices being plotted here. Even 1% of practices represents eighty of them.

Labels: exceptions

# posted by Gavin Jamie - QOF database @ 22:01 1 Comments Links to this post

In all of the general excitement(!) of the release of the 2006/7 QOF data it would be quite easy to miss the QOF exception bulletin produced by the Information Centre for England for the same year. Not perhaps the most gripping of documents but very useful none the less. It is rather dry with plenty of statistics but relatively little comment and no exploration of the reasons behind individual indicators. If you are not familiar with exception reporting in QOF it may be worth looking back at past exception articles.

I am not going to repeat any of the data there, rather to try to provide a little background to help understand what is going on. Page 11 (and to their credit the page numbered 11 is also the 11th page of the PDF - certainly not universal) shows a table of the top ten excepted indicators. There is also the bottom ten but I will concentrate, as I imagine most people will, on the highest figures.

Top of the list is CKD 3 (CKD and hypertension with BP less than 140/85) which has an exception rate of nearly 30%. The equivalent indicator for hypertension alone (BP5) does not even reach the top ten. What is going on here? Well firstly hypertension is very difficult to control in kidney disease so maximum tolerated can quite easily be reached. There is, however, a bigger and more technical issue. Following diagnosis of a condition a patient is automatically excepted for the next nine months if they don't meet the target. This was a new indicator this year and was not really a commonly made diagnosis before. With a simple assumption that practices started work on this QOF a year before (April 2006) then three quarters of the patients could have been excepted if they did not hit the target ( 9/12 ). Suddenly 30% seems fairly good. We can expect to see this drop next year.

Next is CHD 10 (beta blockers in CHD) which has always had a high exception reporting component. Rises a bit this year may be due to the advice that beta blockers are not much use after a year following a heart attack. They are also used much less first line for hypertension than previous due to new research. QOF is looking a bit dated here. Expect a rise again next year.

At third is AF 02 (ECG to diagnose atrial fibrillation) at 21%. Once again this indicator is for quite a short period - looking back over a year. Thus in this case 25% could be excepted automatically. Still fairly high though.

The timescale issue is also true of Asthma 8 (reversibility) at 20%, Stroke 11 (referred for investigation) at 18% and Dep 2 (depression scoring) at 17%. Again these only apply since first of April 2006.

MH 6 (comprehensive care plan) actually seems quite low at 17% due to the mental health register containing everyone who has ever had a psychosis or bipolar disorder - whether they still have the condition or not. MH 9 (annual review) is much the same at 15%.

Finally in the top ten is Epilepsy 8 (fit free for a year) at 17%. This reflects the difficulty in controlling some forms of epilepsy combined with a general lack of problem seen by some patients with occasional fits.

What is interesting is that only Epilepsy and beta blocker indicators have some clinical relevance in the exception reporting. All of the others (eight out of ten) say more about the business rules and the administrative nature of the indicator rather than patients or practices. So the take home message has to be don't place too much importance on exception reporting rates.

Labels: exceptions, published_data

# posted by Gavin Jamie - QOF database @ 22:14 0 Comments Links to this post

A few weeks ago the Centre for Health Economics at York University produced a report looking at some of the statistics in QOF. It looks in some detail at both disease prevalence and to some degree at exception reporting. They are particularly interested in the difference in behaviour between high scoring practice and lower scoring ones, although they also look at social and societal differences between practices.

They only looked at Scottish practices due to the rather better data that was available for them, which has got to be a pat on the back for ISD Scotland.

I won't go into detail about the mechanics of the analysis - you can read it yourself although I would warn you that some knowledge of statistics is needed. It is not a light read. health economics papers rarely are. Most of the really interesting findings are related to the differences between 2005 and 2006 in practices that did, and did not, get maximum points in a given area.

The results are interesting. In general terms those practices who hit the top indicator thresholds in the first year increased their prevalences in the second year relative to those practices which did not. Conversely those practices who did not reach the top thresholds tended to increase the amount of exception reporting they did.

Now there is probably nothing too surprising in that. It would be a rather worrying situation for an incentive scheme not to lead to changes in behaviour in the direction of the incentive. That is exactly what is happening here. Practices are tending to most work in the areas that lead to the greatest incentive. There are certainly issues with the underdiagnosis of chronic diseases and there are probably many people who could be exception reported and are not.

The report talks a lot about "gaming". It does not define this however and I struggle to find a good definition on the Internet. Perhaps the most benign definition would be, in this context "undertaking actions to increase revenue that would not improve patient care". Actually this would encompass all exception reporting. This is not a bad definition as they define altruism as precisely the converse (personally I think that is professionalism but lets not get bogged down in semantics)

The authors of the report do not look so kindly on gaming. They define it thus:

However, exception reporting also gives GPs the opportunity to exclude patients who should in fact be treated in order to achieve higher financial rewards. This is inappropriate use of exception reporting or "gaming".

You can see where we are going here, can't you? By page 15 they are just calling it cheating.

That is not to say that I disagree with their mathematical analysis. I actually think it is rather brilliant and represents an attempt to model QOF mathematically in a way that has not been seen before - in public at least.

However they fall over in the conclusions. They cannot see any reason for these variations except cheating and dishonesty. Now that is one possible explanation for their findings but it is not the only one by any means. They seem to have very little idea of how exceptions are actually used. They don't see practices a living organisations with priorities. If you incentivise them to look for more patients they will find them - there certainly seem to be plenty undiagnosed with diabetes and hypertension. If they are going to get extra cash for a more efficient exception reporting system then they are likely to do that. It could simply be an indication of priorities.

None of this needs dishonest exception reporting or fraudulent diagnosis, simply an understanding of where the statics come from. So are GPs cheating lying scoundrels? We some might be but there is no solid evidence of this on a large scale. It is reassuring (as a GP) to read their first conclusion.

The fact that practices could have treated substantially fewer patients (12.5%) without falling below the upper thresholds for indicators and thereby reducing practice revenue is compatible with altruistic motivation.

Not so bad after all!

Labels: analysis, exceptions

# posted by Gavin Jamie - QOF database @ 23:00 0 Comments Links to this post

Scotland and England have both now published some data on exception reporting. It is not co-incidence that these two countries published the data as both use the QMAS software. For the second year of running this collected data on exception reporting from practices. Indeed on the QMAS website practices and PCTs could see the breakdown of exceptions by reason. This could be compared with the national average.

Now this was a fairly positive development. Where people can see what others are doing they tend to fall into line. I have always been of the opinion that exceptions levels should be unexceptional. However in the published data England does not get down to practice level and Scotland does not break down the reasons for exceptions. There are a couple of reasons for this, one being that each patient can only be included in one exemption e.g. a new dissenting patient would only classify as one of these. This could be dealt with at the analysis stage an at least produce comparable results if it were not for a more significant problem. The English document states

The testing of patient exceptions on national QOF systems (such as QMAS) is primarily focused on ensuring that data values used for achievement calculations are accurate for payment purposes. Therefore any testing of the order of sequencing (ie the order whereby Different GP clinical information systems may follow different sequencing without this impacting on payment accuracy.

To translate into English this is simply to state that the method of deciding which exception applies was not actually tested on systems deployed to GP surgeries. Different computer systems may work this out differently. There is no way of checking as there is no set of business rules published for exception finding.

This hits plans for looking at individual practices quite hard. It become impossible to see whether the exceptions for an individual practice are entirely down to rapid practice turnover or mass patient dissent.

Analysis is still possible though. The Scottish data goes down to practice level and gives figures for exemptions and exceptions. Exemptions are simply those on the register to whom a particular criteria does not apply. An example would be a non smoker who would be exempt from smoking cessation advice. In theory the denominator of an indicator plus the exceptions plus the exemptions should add up to the register size. In practice it doesn't exactly due the the difference in the dates they are measured but it does get there roughly!

I expect we will see more advance analysis of the Scottish data in the coming weeks and months but it is certainly possible to identify practices at either end of the exception spectrum. Being out of ordinary does not automatically mean bad though.

In publishing the English data there is not a practice breakdown, but rather look at the individual indicators. Unsurprisingly there is more exception reporting achievement indicators than with monitoring one. There are, of course, more possible exceptions in these areas. Top of the list for exceptions is the use of beta blockers in CHD. Simply they are contraindicated in asthma, COPD and peripheral vascular disease - all more common in patient with CHD. Next at 18.8% was flu jabs in asthma, probably reflecting guidance from the chief medical officer that it was not indicated in large numbers of asthmatics. Epilepsy 4 at 16.8% reflects the fact that it is not always possible to completely control epilepsy no matter how many drugs you can persuade the patient to take. The rest of the top ten is more about flu jabs and getting to target.

In short there does not seem to be any evidence of systematic manipulation of exception reporting. More than that is difficult to say, other than the whole of the exception data is much less exciting than many people hoped, or possible feared!

Labels: exceptions

# posted by Gavin Jamie - QOF database @ 15:23 0 Comments Links to this post

Last week I looked at the reasons for exception reporting. In this entry I will go into some detail about how exception reporting actually works in practice. In particular how the business rules work out the exceptions and how practices decide what codes to enter.

When it comes to the business rules the exceptions fall into three main groups. Firstly there patients who are recently registered with the practice or recently diagnosed are automatically excepted. There is no need for practice intervention in this - the number of potential exceptions are simply dependant on the practice turnover and the number of new diagnoses. The length of the exemption is three months for most 'process' areas (e.g. blood pressure measurement) and nine months for 'outcome' measures (e.g. blood pressure below 150/90)

Secondly are the exceptions which apply to a whole domain. These are generally speaking due to reasons of patient dissent or unsuitability (e.g. hypertension in the terminally ill). Patient dissent is taken as being either actively expressed or a failure to respond to three invitations to review.

Thirdly exceptions may apply to a specific indicator. Patients may decline to have a flu jab or be allergic to a particular drug. Alternatively they may be on the maximum possible dose of treatment drugs and there simply is no further treatment.

To add to the complication each of the exceptions only count if the target is missed. If the patient subsequently makes the target in an area then the exception is ignored in that area. Thus a new patient will only be exempted from a target about having blood pressure measured until they actually have it measured or three months, whichever comes sooner.

In actually applying the codes there are further complications around whether the codes need to be repeatedly entered each year or not, but the above explanation should be enough to understand the basic process.

The latter two types of exception are controlled by the insertion of Read codes by the practice. Now it would be nice to think that the practice sat down in April and worked out who would be inappropriate to test or treat and entered the codes appropriately. They might invite all their patients to the surgery for review and code those who declined.

In reality, of course, it doesn't work like that. Most GPs don't particularly enjoy exception coding - it somehow feels like failure. Well it certainly doesn't in my surgery or any that I know of. Explicit dissent is recorded throughout the year until about January time then the figures are looked at closely. It is then that unsuitable patients are coded and the letters sent out. If the maximum threshold is crossed then we can all relax and stop exception coding.

So much for anecdote, but is there any sign that this is happening over a wide area? The answer is yes, at least in Brighton. A study there showed everyone getting much the same level of achievement in the areas that they looked at. There was a difference in that deprived areas had a much higher level of exception reporting. This could be interpreted as an increased level of exception reporting in reaction to targets being more difficult to reach. The alternative, and less politically correct interpretation, would be that patient in deprived areas are more resistant to treatment.

In the model presented here the two drivers to exception reporting are thus the practice list turnover and the practice's desire to seek out codes - the latter may be driven by likely achievement levels. There is also likely to be a direct population consent effect similar to that we see with immunisation uptake around the country.

Next time I will look a the currently published data and, using what we have explored so far, look at how they can be analysed. I will also look at what level of detail we can look at.

Labels: exceptions

# posted by Gavin Jamie - QOF database @ 20:53 0 Comments Links to this post

Exceptions have to be one of the most contentious issues in the QOF. Considered essential to many practices and built into the very fabric of the QOF. However it seems that few people other than GPs actually like them.

PCTs hate them as they have the vague and ultimately unprovable feeling that they may be being cheated. Statisticians hate them because it makes it very difficult to say what the real results are for the practice population. Certainly this latter argument misses the point somewhat. The point, of course being that they make the QOF rather saner than it would be otherwise.

This is not to say that there are not things that can be learnt through the exception reporting and it is those issues that I will explore over a series of articles. The actual nitty gritty of dealing with them I will leave to another day and for the moment concentrate on the question of "What are they for?

The current criteria are listed in the box. Their actual number seem to vary from source to source but this is more about layout than content. What is quite apparent is that they are designed to keep the QOF relevant. Some are about not penalising practices for patients informed decisions, something that had controversially not been included in the childhood vaccination targets. In a similar vein other exceptions are there to make sure that GPs are not encouraged to give treatments that are inappropriate or even harmfull. Finally some of the exceptions allow some time for the number to be produced after diagnosis or registration.

All of these codes provide a valuable services to prevent inappropriate care being incentives. There have been some calls for the abolition of exception codes though. There are some who would argue for the abolition of the codes, sometimes arguing that the fact that the points only score up to an achievement of 90% or less does the same job. The reality is that this latter mechanism is a blunt instrument, unresponsive to local circumstances. If anything it is these top thresholds that should be abolished with a continuation of the scoring up to 100%. It is a bizarre system that encourages clinicians to get to 90% and then stop.

It is however very clear that GPs are not stopping at the upper thresholds. Most of the achievement on this site is well over these thresholds. Exception reporting is essential in removing undue pressure on patients to conform to the medical model. It is probably the easiest target of cheap shots against QOF but the alternatives are likely contain more perverse incentives.

Having made the case for their existence, next time I will look at how they are implemented in practice by both practices and the business rules.

Labels: exceptions

# posted by Gavin Jamie - QOF database @ 21:40 0 Comments Links to this post

ISD Scotland has published some of the exception reporting data for Scotland. Once again they are ahead of the rest of the UK. It is, as they point out as frequently as they can, difficult data to use in any sensible way.

Over the next week or so I am planning a series of short articles about exception reporting and how it may be interpreted.

Labels: exceptions

# posted by Gavin Jamie - QOF database @ 12:34 0 Comments Links to this post