Risk Assessment: Fit for Purpose

Designing an effective model to assess the risk (and hence predict the future level of claims) of medical scheme members is, without doubt, a complex and arduous undertaking. There are an array of factors to consider. As a result, one can easily overlook the most seemingly basic question: “for what purpose?” The consideration of this question is of fundamental importance in designing any predictive model. Although the question may appear simple and obvious, determining an answer actually requires careful consideration of rather subtle aspects of the overall problem. Furthermore, the answer to such a question should play a direct part in the way in which the model is constructed, as well as the requirements of such a model. This paper takes a brief look at this basic question.

There are two main dimensions across which the above question can be looked at:

1. Whether the model requires accurate measurement of individual risk or group risk; and
2. The segment of the member population which the model primarily aims to target (for example, current low-cost or high-cost individuals).

These two dimensions will be explored in greater detail below.

‘The Individual’ versus ‘The Group’
In comparing models that are aimed at individual assessment of risk with those that are aimed at group assessment, it is useful to consider common applications of the risk assessment. The first of these is to identify (and in some cases, estimate the actual claims costs) of future high-cost individuals through predictive modelling. What exactly constitutes a ‘high-cost’ individual will vary from study to study. It may be based on a certain percentile of the member population (Shapiro, Childs & Getz, 2013; Zhao et al., 2003) or with respect to a pre-determined threshold amount (Dove, Duncan & Robb, 2003). The identification of future high-cost members is often performed as a means to target these individuals with managed care interventions and disease management programs (Fleishman & Cohen, 2010). In this context, it is clearly desirable for the model in question to have a sufficient level of predictive accuracy at the individual level, as the purpose is to identify those individuals that will incur high future costs.
Other common applications of risk assessment are for use in areas that require prediction of overall costs across certain groups of individuals. Measuring the relative risk differences among various groups can be used in pricing, as well as for risk adjustment . As Dove et al. (2003) point out, “the goal of risk adjustment is not to identify individual patients with high-cost conditions or to intervene in their care” but rather “to accurately predict the average annual expenditures for an individual patient to redistribute premiums to health plans”. In this context, predictive accuracy at an individual level becomes less of a necessity.

This may seem like a minor distinction to draw, but it warrants serious attention. Not least of all due to the difficulties of accurately predicting individual claims. Accurate risk assessment at an individual level is a far more onerous requirement, and it has been found that risk assessment is generally more efficient at explaining variations in costs over larger populations (American Academy of Actuaries, 2010). This is echoed by the American Medical Association (2009), who argue that attempts “to assign risk and severity scores to patient populations, as opposed to individual patients, are generally sound”. In essence, it is quite possible for a predictive model to be highly accurate at predicting overall expenditure of large groups of individuals, whilst failing to accurately identify specific individuals that will incur high costs in the future. Disregarding this subtle point could result in the misuse of models that aim to predict population expenditure, through an attempt to apply them at an individual level. An example of this is the Johns Hopkins ACG system which, although it assigns individual risk scores to each patient, is designed to be applied to entire populations or sub-populations of patients (Wiener & Abrams, 2011). As Fleishman & Cohen (2010) point out, “the utility of a predictive model depends on the specific context in which it will be used”. This context must be clearly established and understood before one attempts to design such a model. The purpose of the model will dictate who it is aimed at, and recognising this fact will create awareness of the level of predictive accuracy that is required.

“Risk assessment is the method payers use to evaluate the predicted overall health care claim dollars for each member relative to the average members in a given patient population” (American Academy of Actuaries, 2010). Predicting individual claims costs is simply one application of risk assessment. Another application is risk adjustment. All of these applications are of interest and so the discussion has not been limited to only the prediction of future claims costs. Thus, ‘risk assessment’ has been referred to here in its broader sense.
The American Academy of Actuaries (2010) define risk adjustment as “the process of adjusting payments to organisations based on differences in the risk characteristics of people enrolled in each plan”.

The target segment
The ‘segment’ of the population that a predictive model is aimed at is another issue that warrants attention. This is generally more relevant within the discussion of predictive models that aim to identify future high-cost individuals for managed care intervention. In this context, the ‘target segment’ refers to whether the model should target current low-cost members or current high-cost members. This is an interesting point to consider. Obviously it is possible for the model to aim to predict the future costs of all individuals, but it is still important to determine which segment’s risk characteristics are (and should be) most successfully measured. In other words, whether it is preferable to have a model that accurately identifies repeat high-cost members, or one that aims to identify current low-cost members that will transition to become high-cost in the future.
The former appears too often to be the focus, with majority of disease management programs targeting those beneficiaries that had high costs in the previous period (Dove et al., 2003). The problem with such an approach is that it implicitly assumes (often incorrectly), that the behaviour of this small group of individuals is replicated period after period. Dove et al. (2003) argue that this is not necessarily the case, as it ignores the well-known phenomenon of regression to the mean. They observed that very few patients remain consistently in the high-cost bracket. Furthermore, seeing as this high-cost group represent such a small proportion of the member population (only 1% of members in their study were classified as high cost), the overall reduction in costs that can be achieved by focusing attention on these few individuals is relativity low. “If the focus is on only the “repeaters” from the high-cost subgroups, relatively little total cost is identified for intervention and management” (Dove et al., 2003). On the other hand, they point out that low-cost members represented the majority of the population, and were found to consume a far larger percentage of overall costs in the subsequent period compared to high-cost members (due to the strong tendency for mean reversion).
Shapiro et al. (2013) also recognised the importance of mean reversion, arguing that “current high-cost beneficiaries should not all necessarily be the focus of care management”. Most of the existing research on identifying and predicting costs of high-cost members aimed to do so on the basis of a single year projection. Shapiro et al. (2013) broadened the existing research by extending this projection to the medium- and long-terms. In doing so, they found that predictions of beneficiaries who were previously low cost and transition to high cost were more accurate compared to the prediction of repeat high-cost members. It was also found that the predictive accuracy did not decrease with longer time horizons, whereas the predictions of repeat high-cost members became less accurate with increased time horizons. This, alongside the findings of Dove et al. (2003), seems to suggest that targeting predictions on current low-cost members has distinct advantages. That being said, as always, it depends on the specific purpose of the risk assessment in question. Another interesting point to consider is that instead of focusing on predicting repeat high-cost members, it may actually be more useful to try and predict those current high-cost members whose costs will greatly decline. This would allow schemes to redirect managed care resources away from these members, resulting in cost savings of a different form.
There are a wide variety of methodologies with which to measure the relative risk of individuals within a patient population, or that of different subpopulations. Importantly though, there are also a variety of different reasons to do so. It is precisely this (the underlying purpose), that should be kept in mind when deciding how best to design any predictive model.

References
American Academy of Actuaries. 2010. Risk Assessment and Risk Adjustment.
American Medical Association. 2009. An Introduction to Risk Assessment and Risk Adjustment Models.
Dove, H.G., Duncan, I. & Robb, A. 2003. A prediction model for targeting low-cost, high-risk members of managed care organizations. The American Journal of Managed Care. 9(5): 381-389.
Fleishman, J.A. & Cohen, J.W. 2010. Using Information on Clinical Conditions to Predict High‐Cost Patients. Health Services Research. 45(2): 532-552.
Shapiro, D., Childs, B. & Getz, C. 2013. Targeting high-cost beneficiaries in the medium-term with predictive modelling.
Weiner, J. & Abrams, C. 2011. The Johns Hopkins ACG system: technical reference guide, version 10.0. Baltimore: Johns Hopkins University Bloomberg School of Public Health.
Zhao, Y., Ash, A.S., Haughton, J. & McMillan, B. 2003. Identifying future high-cost cases through predictive modelling. Disease Management & Health Outcomes. 11(6): 389-397.

Julian Bloch