Criteria and their Categorization

Overview
QUALIFY is an instrument for the structured assessment of quality indicators in the health care system. It tests, whether indicators are suitable for accurately demonstrating differences in quality of care. The instrument was designed in such a way that it can be used in all sectors of health care (ambulatory and hospital acute care, ambulatory and hospital rehabilitation). The necessity for the development resulted from the fact that a comparable instrument was not available.

The QUALIFY instrument is based on 20 criteria (Table 1), each of which is measuring a certain characteristic of the quality indicator being assessed.

Table 1: QUALIFY: Criteria and Categories

Individual criteria interact with each other: for example, clarity of definitions is an important prerequisite for a high reliability of a quality indicator. Vice versa a good reliability cannot be expected with unclear definitions.

The QUALIFY instrument can be used for different purposes:

• assessment of indicators already in use,

• assistance with new or further development of indicators.

• A practical application of QUALIFY was the identification of those indicators that are suitable for mandatory reporting to the public by German hospitals.

The approach and practical experiences can be found in the section practical application.

As a basis for a transparent assessment of quality indicators using QUALIFY, the following information is available for each individual criterion:

• A clear definition of the criterion
• A clear description of the assessment process (operationalization): this specifies all necessary single steps from the question to the assessment of the results

• A core statement: for each criterion there is a concrete statement which is evaluated (e.g. for the criterion risk adjustment: “all known relevant factors influencing the result of the quality indicators are considered.”)

• Assessment Stages: this pertains to defined assessment options related to the core statement (e.g. applies, does not apply, abstention)

• Information basis: besides the criteria, the evaluators receive structured materials prior to the assessment. These materials can be very different for the different criteria and different indicators.

• The criteria of the category relevance evaluate whether an indicator measures relevant quality aspects of a particular service. The relevance is directly dependent on the setting in which the quality indicators are applied (e.g. quality indicators for hip replacement procedures are highly relevant in Germany, but less relevant in regions with lower live expectancy).

• The indicators of the category scientific soundness examine direct methodological properties of a quality indicator. The assessment of these criteria can be justified methodologically.

• The indicators of the category feasibility examine the practical applicability in a concrete setting.

The categorization supports the understanding of the individual criterion and the relationship of the criteria to each other. In addition, a meaningful sequence can be derived from the categories during the assessment.

Assessment of Quality Indicators with QUALIFY: General Principles

For the assessment of quality indicators using QUALIFY one proceeds as follows:

1. Definition of the assessment’s objectives using QUALIFY. This step supports a goal oriented approach to the assessment.
2. Designation of the expert committee and the moderator (who participates in the discussion but does not vote). Patient representatives should absolutely be included in the assessment process.
3. Explanation of the methodology used in defining the information base of an assessment and for the actual assessment process.
4. Selection of criteria from the entire list which are relevant to the objectives. If individual criteria are not assessed, an explicit justification must be given.
5. Definition of the rules guiding the decision process, after which individual results will be synthesized, leading to an acceptance or rejection of individual quality indicators for that particular objective.
6. Assessment of quality indicators.

• The assessment follows a stepwise procedure. For practical reasons it makes sense to apply the criteria of relevance first. It is only worthwhile for relevant quality indicators to do a possibly extensive assessment of the scientific soundness of a quality indicator. Lastly, the practicability of the quality indicators can be evaluated, for example, the data availability. The sequence of the assessment can be varied as needed; the result of the assessment is not affected by it.
• It is advisable to evaluate the selected criteria successively (in order).
• Before using a criterion from the QUALIFY instrument, one must make sure that all evaluators have understood the definition, the core statement and the information base provided.
• The assessment takes place anonymously and without consulting with other members of the assessment team. This ensures equal representation of all members.

The special proceedings for each criterion are described in detail in the following chapters. Practical hints for the application of the instrument are also provided there.