Development of QUALIFY

Project Aim

The aim of the project was the development of an instrument for the assessment of quality indicators by means of criteria. With this instrument, quality indicators will be methodologically evaluated which have been developed for application in the health care system, which includes prevention, ambulatory and hospital acute care and ambulatory and hospital rehabilitation.

Development Methodology

A development team of methodologists was formed for the project:

From the BQS:

• A. Reiter
• B. Fischer
• J. Koetting
• H. Barlag
• K. Doebler

Scientific Support:

• M. Geraedts, University of Duesseldorf
• W. H. Jaeckel, University of Freiburg

Step 1: Literature Review

Proceeding:

1. Literature search in Medline and Embase and additionally on the internet (see search strategy)
2. Screening of abstracts: Assessment of the relevance of all articles found
3. Criteria, including possibly existing definitions and/or operationalizations were extracted from the original articles

Search strategy for literature databases:

Both databases were searched in September 2005 for articles which contained simultaneously a term of group 1, a term of group 2 and the main key word “health care quality”. Instead of “health care quality” other terms could also be used which are attributed to this term in the browser hierarchy (“term is exploded”).

The search was limited to articles in English, German and French. No restrictions were given for the date of publication. The search strategy was verified such that all articles already known to the participants were captured completely.

Search strategy websites:
The project participants compiled together a list of relevant institutions. The websites of these institutions including existing links were searched for criteria. The details are summarized in the Appendix 4.

Screening of abstracts:
For all identified citations from the literature search, the abstracts were printed out and read. All articles whose title, abstract text or keywords (EMTREE MEDICAL INDEX TERMS) had a hint for the subject of criteria were identified and ordered in full text.

Extracting from the original articles:
From all relevant articles, existing definitions and / or operationalizations were extracted.

Results:

• 1,128 articles found
• 270 articles categorized as potentially relevant
• 43 articles relevant
• 208 criteria, partly with definitions, extracted (including multiple mentions, e.g. 28 articles mentioned validity)

Step 2: Definition and Categorization
On the basis of 208 excerpts, terms were clustered and common elements as well as differences in the definitions were extracted from the different sources.  The attempt to develop a hierarchical taxonomy of criteria was abandoned due to the non-hierarchical interrelationship of the different criteria. 20 criteria were selected, definitions were developed and assigned to the categories relevance, scientific soundness and feasibility.

Result: 20 criteria with definitions harmonized to one another.

Step 3: Operationalization
For each criterion an operationalization was developed. Wherever possible, references were made to the relevant literature. Frequently, we had to develop our own operationalizations.

Result: Operationalization of 20 criteria.

Step 4: Assessment Procedure
For the assessment procedure, a review of consensus findings and assessment methods was conducted. On this basis, an assessment procedure was determined for each criterion. An important part of this was the previously developed definitions and operationalizations, as well as BQS specialty group concept.

Result: Assessment procedures and algorithms for the 20 criteria.

Step 5: Pretest and Refinement of the Instrument
On the basis of a selection of 6 quality indicators with different methodological characteristics, the instrument was tested by the project group. The following quality indicators were used for the pretest:

• Cardiac surgery: risk-adjusted mortality
• Breast surgery: X-rays of specimen
• Breast surgery: postoperative hematoma (not a BQS indicator)
• Obstetrics: maternal mortality
• Carotid artery surgery: indication for asymptomatic carotid artery stenosis
• Decubital ulcer (patient safety indicator according to definition of AHRQ, identified by means of administrative routine data, not a BQS indicator)

The selection considered indicators for the indication, processes and outcomes as well as sentinel event indicators and rate based indicators. Furthermore, indicators were selected which BQS from its practical work considered methodologically stable as well as at first glance weak-looking indicators (postoperative hematoma).

Proceeding for the Refinement of the Instrument:

1. Assessment of the indicators with the help of the instrument: For each criterion an independent assessment took place by the participants. After the assessment was completed the results were discussed, especially in the case of divergent assessments.
2. Need for revisions of definitions and operationalization steps on the basis of the results of the assessment: In this stage, only small deviations in the understanding of the criteria appeared, which could be resolved by clearer wording.
3. Adjustment of the instrument.

Result:
The QUALIFY instrument includes 20 criteria in the three categories relevance, scientific soundness, and feasibility. For each criterion a definition, an operationalization and an assessment procedure is available.