Risk Adjustment
This criterion is especially important for the purpose of public reporting, to assure that the results truly reflect the quality of care and not the case mix of the evaluated providers. The factors of influence are mainly patient characteristics such as the severity of illness and the comorbidity. Patient preferences can also be considered here. The assessment of this criterion requires both high methodological and high medical expertise on the part of the evaluators.
Definition
Significant factors of influence for the expression of the measured variable shall be identified, recorded and considered in the context of a risk adjustment. In the context of a risk adjustment only those factors of influence have to be considered which
- are not dependent on the care by the evaluated provider
- are not evenly distributed over all providers
- still exert relevant influence when viewed together with already considered risk factors
- truly distort the outcome of the quality goal if not considered (and therefore for example factors of influence which occur only very rarely and sporadically can be neglected if an influence on the results of the hospitals seems negligibly small in regard to the quality indicator).
In addition, the factors of influence considered should be measured reliably. Furthermore, the risk adjustment method (e.g. stratification, additive score) that is utilized should be chosen appropriately in order to enable an assessment of the quality goal that is as uninfluenced by impact factors as possible.
Core Statement
The following statements are assessed:
- “All known relevant factors of influence on the outcome of the quality indicators are considered”.
- “The risk factors are overall reliably measurable”.
- “The method of risk adjustment is appropriate to make an unbiased statement in regard to the quality goal”.
The core statement “the indicator is sufficiently risk-adjusted” is then assigned the least favorable vote from the three evaluated statements.
Information Base for the Assessment
For the final assessment of the core statement regarding a sufficient risk adjustment, separate assessments are made: for the completeness of the relevant risk factors, for the reliability of the risk factors utilized and for the method of risk adjustment. For each of these three assessments different information bases are compiled.
1. All known relevant factors of influence on the result of the quality indicators are considered.
Initially it is explicitly pointed out that factors of influence which
- depend on the quality of the provider
- are uniformly distributed across all providers
- viewed together with the risk factors under consideration have no relevant additional influence or do not distort the result for the quality goal
- shall not be considered for a risk adjustment or are not necessary for a risk adjustment.
For the assessment, initially a table with the factors of influence already considered is provided with the quality indicator. Potential factors of influence going beyond that are collected in a list of “so far not considered potential factors of influence”. This list is generated mainly from information about the measured variable in the scientific literature (for example international and national publications and guidelines). However, the information from the Structured Dialogue? (root cause analysis of conspicuous indicator results) may also be used, or the technical and clinical knowledge of the experts may be used to discuss potential factors of influence.
If potential factors of influence are present, which have not been included in the current risk adjustment method of the indicator, the relevant information (regarding an interrelationship between these factors of influence and the expression of the measured variable) including the literature search, are compiled. On the other hand, if data regarding the factors of influence and the measured variable already exist, a database analysis should be performed. The result of both, the evidence base and the analysis feed into an assessment. For categorical data, the p value of a chi square test for independence in a contingency table, can provide hints as to a possible interrelationship. Graphic depictions of the relative frequency of factors of influence within the individual hospitals can additionally strengthen or weaken the assumption of an equal distribution across the hospitals.
Besides the list of the factors of influence under consideration, and the list of the factors of influence not included in the current risk adjustment, it can be helpful to maintain a third list. This list includes potential factors of influence, which either depend on the quality of the providers or are uniformly distributed across all providers or when viewed together with the risk factors already included in the risk adjustment have no relevant additional influence. In addition, the list could include potential factors of influence that do not distort the outcome of the quality goal or for which an actual influence must be denied. This third list shows clearly that the factors listed were deliberately not considered, but it possibly also facilitates the handling of divergent opinions of individual evaluators.
2. “The risk factors used are overall measured reliably.”
An assessment takes place analogously to the criterion reliability, here focusing on factors of influence. For every individual risk factor the information base for the characteristic reliability is made available first as described in the chapter reliability. The corresponding individual assessments then serve as an information base for a comprehensive assessment of the reliability of all risk factors utilized for the quality indicator. For each risk factor, the relative risk (or alternatively the odds ratio) for the occurrence of an event for the measured variable is determined and the risk factors are divided into factors with high influence and factors with low influence. The description of the frequencies of individual assessments and the result of an assessment table serve as an information base for the comprehensive assessment, whereby factors with high influence are given a higher weight.
3. “The method of risk adjustment is appropriate to make an unbiased statement regarding the quality goal.”
Here the description of the quality goal in its exact wording and the risk adjustment procedure serves initially as an information base. Depending on the applied method of risk adjustment (and the quality goal) different information bases are provided in order to evaluate the suitability of the method itself and its appropriate application. For this purpose, the following methods are currently used at BQS: risk-standardized case constellation, stratification, additive scores, logistic regression. It is helpful to first compile the reasons why the respective method originally was chosen by the experts, as well as information about the number of cases in the basic population of the indicator necessary for an assessment of the quality goal. Beyond that the following information can be useful:
With risk adjustment not necessary: information as to whether factors of influence truly don’t exist, or whether a risk adjustment is justifiably not necessary.
With risk-standardized case constellation, stratification and additive score:
this provides information as to whether and why excluded cases, strata, or classes of scores are of no or only limited value for the assessment of the respective quality goal. In individual cases, a comparison of the ranking of the outcomes for the assessment that were utilized or not utilized can give hints that are beneficial for additional assessments of the quality goal. Additive scores that not only look at the score classes without risk factors or with all risk factors combined, may yield information about the outcome results in the presence of individual as well as combined risk factors. This can explain whether the additive weighing is reflected in the outcome results.
With logistic regression: during a new calculation of a logistic regression model the values of the Hosmer-Lemeshow test, the ROC curve, the relative comparison (of the model development), the values for the pseudo-R² and the Akaikes information criterion shall be made transparent. In a subsequent application without new calculations, information about the observed rate and the expected rate from the model will be made available, which should not lie too far apart. If during a subsequent application information is available about more recent models with a comparable population, or about comparisons of the regression coefficients of both models (and therefore possible significant differences), this could provide additional valuable information.
In all cases it is helpful if a team of biostatistical and medical experts develop and justify an assessment suggestion based on the available information.
Assessment Process
Separate assessments are undertaken: for the completeness of relevant risk factors, for the reliability of the risk factors utilized and for the method of risk adjustment. The core statement “the indicator is sufficiently risk-adjusted” is the least favorable result from the three assigned evaluated statements.
1. The statement regarding the completeness of risk factors to be evaluated, reads:
“All known relevant factors of influence on the outcome of the quality indicator are considered, which
- do not depend on the quality of the provider
- are not equally distributed across all providers
- viewed together with the considered risk factors still show relevant influence
- if not considered would distort the result for the quality goal”
Initially the information base is given and explained to all evaluators.
If the list of “not considered” potential factors of influence is filled and p values of the ?²- test from a data analysis of the “not considered” factors of influence can be used, an assessment recommendation is suggested as follows:
| Does not apply |
if the p-value of a not considered factor of influence that is < 5% or with a not considered factor of influence a medically relevant difference in the outcome for the measured variable is obviously recognizable by experts despite small case numbers |
| Rather does not apply |
if the p-value of a not considered factor of influence is between 5% and < 10% and the above condition is not fulfilled
|
| Rather applies |
if the p-value of a not considered factor of influence is between 10% and < 20% and both above conditions are not met
|
| Applies |
if the p-value for each not considered factor of influence is either >= 20% or a difference in the outcome is considered not medically relevant by experts
|
If data for the “not considered” potential factors of influence are not available, an assessment takes place on the basis of the scientific literature and the competency of the experts.
Subsequently the statement is assessed by all evaluators as described in Appendix 1.
Assessment Stages
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention
2. The statement to be evaluated regarding the reliability of the risk factors reads
“The risk factors used are overall reliably measurable.”
The evaluators are informed about the results of the reliability analysis of the individual risk factors. Afterwards, the available calculations regarding the relative risks are presented. An explanation is given as to how a classification into factors with high and low influence is determined. As a general rule, it turned out practical to classify risk factors with a relative risk of >= 2 as risk factors with a high influence and accordingly those with a relative risk < 2 as risk factors with a low influence. However, another classification, if necessary, can be reasonable, in particular if for example many risk factors are accumulated around the value “2”. The presentation of the frequencies of the individual assessment and the basis for the recommendation from the weighted assessment scheme can then for example be extracted from a table.
Table 3: Example of a simple weighted assessment suggestion. Risk factors with a relative risk >= 2 are weighted double in the example
| Factor of influence reliably measurable
|
Relative risk |
Weight
|
Applies
|
Rather applies |
Rather does not apply |
Does not apply
|
|
Risk factor 1
|
1,40
|
1
|
X
|
|
|
|
|
Risk factor 2
|
4,01
|
2
|
|
X X
|
|
|
|
Risk factor 3
|
1,84
|
1
|
|
|
|
X
|
|
Risk factor 4
|
5,45
|
2
|
XX
|
|
|
|
|
Risk factor 5
|
3,02
|
2
|
|
|
XX
|
|
|
Risk factor 6
|
1,59
|
1
|
X
|
|
|
|
|
Weighted number
|
|
|
4 / 9
44,4%
|
2 / 9
22,2%
|
2 / 9
22,2%
|
1 / 9
11,1%
|
A recommendation regarding the comprehensive assessment of the reliability of the risk factors used is suggested as follows from the weighted assessment scheme.
Does not apply
|
if >= 1/3 of the weighted number is in “does not apply” |
| Rather does not apply |
if the above condition is not fulfilled and >= 1/3 of the weighted number is in “does not apply” or “rather does not apply”
|
| Applies |
if > 2/3 of the weighted number is in “applies” and no risk factor with high influence is in “does not apply”
|
| Rather applies |
all other cases
|
In the example from Table 3 an assessment suggestion is “rather does not apply”.
Subsequently, the statement “the risk factors used are overall reliably measurable” is evaluated by all evaluators as described in Appendix 1.
Assessment Stages
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention
3. The statement to be evaluated regarding the method of risk adjustment reads:
“The method of risk adjustment is appropriate to make an unbiased statement regarding the quality goal”
After all evaluators have acknowledged and understood the information base, they assess the core statement. The process is described in detail in Appendix 1.
Assessment Stages
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention
Assessment of the Core Statement of the Criterion Risk Adjustment
“The indicator is sufficiently risk-adjusted”
The core statement is assigned the most unfavorable assessment from the three evaluated statements above.
Assessment Stages
1 = does not apply
2 = rather does not apply
3 = rather applies
4 = applies
Abstention
