Appendix 2: Assessment of the Criteria

• Data collection effort
• Correctness of data can be verified
• Completeness of data can be verified
• Complete count of data sets can be verified

The assessment of quality indicators using these criteria takes place in maximally two rounds. The procedure is different from the procedure for the criteria in Appendix 1 in that the assessment stages and as a consequence also the analysis are changed.

1. For each criterion a different questionnaire is handed out with the following assessment scale:
   No = does not apply
   Yes = does apply
   Abstention
   
   All quality indicators to be assessed can be listed in a table on the questionnaire. For each quality indicator, only one answer is allowed. Abstentions are possible.
2. The results of the first assessment round are collected and analyzed.
3. If there is a complete count of votes and simultaneously a consensus is obtained from the assessments (for a criterion related to a quality indicator), the evaluation of this quality indicator for the respective criterion is completed.
   Complete count of votes: All assessments are valid and there is no abstention.
   Consensus: All assessments with yes (acceptance) or all evaluations with no (rejection).
   Example: Quality indicator wound infection, criterion data collection effort:
   - No abstention
   - 100% of votes: yes
4. In the absence of a complete count of votes (caused by abstentions, invalid assessments) or if there is disagreement in the first round of assessments, the aggregated results will be discussed in the evaluation committee with the goal of exchanging information. A consensus must not be forced.
   Example for disagreement: quality indicator wound infection, criterion data collection effort:
   - 80% of votes: yes
   - 20% of votes: no
5. The members vote again (second and final assessment round). The assessment is done as in the first round.
6. The assessments of the second assessment round are collected and analyzed:
   The assessment is valid if less than 1/3 of evaluators present did not participate in the assessment (abstention).
   With an invalid assessment in the second round, a further assessment is omitted.
7. Presentation of assessment results:
   In case of a valid assessment result: Reporting of the fraction of yes- and no-votes of all valid votes (abstentions are not included). If the fraction of yes-votes is above (>) 50%, this is considered an agreement.
   
   Example: Quality indicator wound infection, criterion data collection
   - 6 single assessments: yes, yes, yes, yes, no, no
   - Assessment result: yes (66% of valid votes)
   
   In case of an invalid assessment result: Quality indicator x: for the criterion y, a valid assessment is not obtained.