Implementation and outcome indicators

IMPLEMENTATION AND OUTCOME INDICATORS

This section describes the basic requirements to implement a post-introduction observation system and furnishes a series of outcome indicators for assessment of results. The aim is to guide and advise persons tasked with assessment on the different aspects to be borne in mind when it comes to implementing a post-introduction observation system, and provide them with a series of tools for analysing relevant aspects of post-introduction observation.

The implementation of a post-introduction observation system requires prior planning of the whole procedure, in close collaboration with a multidisciplinary team representative of all health professionals directly involved in the use of new technology. Before initiating a post-introduction observations a comprehensive assessment should be made of the information that must be collected, the data sources and resources available and the procedure for analysis of results.

Based on the results of the systematic review and the recommendations proposed by the working group, a post-introduction observation system for new technologies must comply with the following premises:

the outcome data to be collected should de part of the data that should be recorded in daily clinical practice because it is relevant for ensuring correct medical practice and patients' progress.
any increase in the number of diagnostic procedures and tests performed on the patient must be prevented.
the number of follow-up contacts should be in agreement with the routine check ups. It must be ensured that the number of follow-up visits is not increased save in specific cases in which there is no previous consensus among centres and a follow-up is agreed upon.
the variables to be collected must, as far as possible, be outcome variables that are either objective or capable of being rendered objective, with intermediate variables being avoided as far as possible. Validated scales are to be used for assessment of results (pain, quality of life, etc.).
the follow-up, albeit not too long, must nonetheless be sufficient for obtaining an adequate number of patients in each target study-subgroup and allowing for detection of short-/medium-term adverse effects. As a reference, the minimum recommended follow-up time is 1 year. In cases where the technology might be indicated in a small number of patients or subgroup analysis must be performed, this period could be extended until a minimum number of interventions (patients) was reached (guideline: 25-30).
a common database must be used for data-management and there should be a support from a specialised reference unit for the purpose of standardising analysis of results (for example, HTA agency).

The requirements deemed fundamental for implementing a post-introduction observation system are displayed in table 5.

ver imagen View full-size table

Table 5. Fundamental requirements for implementing a post-introduction observation system.
1.Scope of the study: It is necessary to delimit the health area, geographical area, province, region or country in which it is envisaged that the post-introduction observation procedure is to be implemented. It will also be necessary to define whether the post-introduction observation will solely consider centres that form part of the public health system or will also consider private centres. In addition, the scope of study must be specified, i.e., whether it is to include primary care, specialised care or all types of centres. 2. Participant centres: Contact must be made with the centres, departments and units that are involved in the use of the new technology, in order to request their collaboration and ask them to designate a contact person. 3. Study protocol: A study protocol must be drawn up in collaboration with a multidisciplinary group made up of health professionals involved in the planning, implementation, scheduling and use of the new technology, to agree on the characteristics of the post-introduction observation system. The proposed study protocol must be reviewed at each participant centre by the various specialists involved in the health care of these patients. The study protocol must address the following points: a. Definition of the goal and purpose of the study: The study objectives and purpose must be defined a priori. The objectives will be what determine the target population, the data to be collected and the outcome variables (outcome indicators). Hence, the relevance of these being carefully considered in collaboration with the multidisciplinary group. b. Target population: It is essential to establish and clearly define the clinical indications and specific characteristics of the patients (age range, co-morbidities, severity, etc.) who are to form part of the target study population. c. Study subgroups: It is essential to establish and define whether there are subgroups of patients that must be separately evaluated (clinical indications, age, symptoms, severity, co-morbidities, pregnancy, etc.). Similarly, in cases where there are similar though not identical procedures, procedures that call for a high degree of specialisation, or procedures that are used in combination with others, it is also necessary for the technique, manner and order of administration to be registered and for the study subgroups deemed of interest for analysis purposes (country, team, technique, etc.) to be established. d. Data-collection variables: The variables to be included in the study must be stipulated, the specific criteria that define such variables must themselves be defined, and the method of coding the variables must be determined. In the case of each technology, it is important for adverse effects that are going to be classified as mild/moderate and those that are going to be classified as severe to be defined. e. Systematic data-collection: The data sources and data-collection instruments (clinical history, specific questionnaires, etc.) must be chosen, and the procedure for collecting (person responsible, method of collection, measurement scales, etc.) and distributing the data (person responsible, channel and periodicity) established. f. Outcome indicators: The outcome indicators that are deemed relevant for each technology must be chosen by consensus. Depending on the technology in question, it may be relevant for all aspects (diffusion, accessibility, effectiveness, safety, resource use and financial impact) or only some aspects of post-introduction observation to be considered. Within these, it may in turn be deemed relevant for one or more outcome indicators to be analysed. g. Patient follow-up: Patient follow-up guidelines must be agreed upon and the duration of the study established. A follow-up time of 1 year is recommended but this may be increased if the intervention is indicated in few patients or if analysis by subgroup is required. h. Acceptable or desirable standards: The standards deemed acceptable or desirable for each outcome indicator targeted for assessment (diffusion, adequacy of use, safety, effectiveness, costs, etc.) must be defined a priori. The multidisciplinary group will be responsible for establishing and agreeing upon the values deemed acceptable or desirable. For the principal variables, the values to be used to determine the success or failure of the technology must be carefully established, taking into account all the pertinent information, i.e., results reported in the scientific literature, data published in diverse data sources and personal experiences. i. Data-analysis: The data-analysis procedure (unit responsible) must be established. j. Patient information: It is recommended that the written information to be furnished to the patients at the various centres be decided upon and rendered uniform. k. Data confidentiality: The procedure for maintaining the confidentiality of the data (masking of data, encryption, etc.) must be laid down. 4. Organisation: Once the planning stage has been finalised, it will be necessary to proceed to the implementation stage. This stage can be divided into 3 parts: a. Development of data-collection instruments (specific questionnaires). b. Drafting of detailed instructions covering the data-collection and dispatch procedure. These must include the definition and coding of all variables. c. Development of a common database. 5. Data-quality control: Provided that it is feasible, consideration should be given to an external review of 5% of entries in order to assess the quality of the data registered. This activity could be undertaken by the unit responsible for data analysis. 6. Plan for dissemination of results and alert system for unforeseen problems: It is recommended that, prior to the start of the study, agreement is reached on how to proceed in the event of detection of problems and how the results of the study are to be disseminated.

Table 5. Fundamental requirements for implementing a post-introduction observation system.

1.Scope of the study: It is necessary to delimit the health area, geographical area, province, region or country in which it is envisaged that the post-introduction observation procedure is to be implemented. It will also be necessary to define whether the post-introduction observation will solely consider centres that form part of the public health system or will also consider private centres. In addition, the scope of study must be specified, i.e., whether it is to include primary care, specialised care or all types of centres.
2. Participant centres: Contact must be made with the centres, departments and units that are involved in the use of the new technology, in order to request their collaboration and ask them to designate a contact person.
3. Study protocol: A study protocol must be drawn up in collaboration with a multidisciplinary group made up of health professionals involved in the planning, implementation, scheduling and use of the new technology, to agree on the characteristics of the post-introduction observation system. The proposed study protocol must be reviewed at each participant centre by the various specialists involved in the health care of these patients. The study protocol must address the following points:
- a. Definition of the goal and purpose of the study: The study objectives and purpose must be defined a priori. The objectives will be what determine the target population, the data to be collected and the outcome variables (outcome indicators). Hence, the relevance of these being carefully considered in collaboration with the multidisciplinary group.
- b. Target population: It is essential to establish and clearly define the clinical indications and specific characteristics of the patients (age range, co-morbidities, severity, etc.) who are to form part of the target study population.
- c. Study subgroups: It is essential to establish and define whether there are subgroups of patients that must be separately evaluated (clinical indications, age, symptoms, severity, co-morbidities, pregnancy, etc.). Similarly, in cases where there are similar though not identical procedures, procedures that call for a high degree of specialisation, or procedures that are used in combination with others, it is also necessary for the technique, manner and order of administration to be registered and for the study subgroups deemed of interest for analysis purposes (country, team, technique, etc.) to be established.
- d. Data-collection variables: The variables to be included in the study must be stipulated, the specific criteria that define such variables must themselves be defined, and the method of coding the variables must be determined. In the case of each technology, it is important for adverse effects that are going to be classified as mild/moderate and those that are going to be classified as severe to be defined.
- e. Systematic data-collection: The data sources and data-collection instruments (clinical history, specific questionnaires, etc.) must be chosen, and the procedure for collecting (person responsible, method of collection, measurement scales, etc.) and distributing the data (person responsible, channel and periodicity) established.
- f. Outcome indicators: The outcome indicators that are deemed relevant for each technology must be chosen by consensus. Depending on the technology in question, it may be relevant for all aspects (diffusion, accessibility, effectiveness, safety, resource use and financial impact) or only some aspects of post-introduction observation to be considered. Within these, it may in turn be deemed relevant for one or more outcome indicators to be analysed.
- g. Patient follow-up: Patient follow-up guidelines must be agreed upon and the duration of the study established. A follow-up time of 1 year is recommended but this may be increased if the intervention is indicated in few patients or if analysis by subgroup is required.
- h. Acceptable or desirable standards: The standards deemed acceptable or desirable for each outcome indicator targeted for assessment (diffusion, adequacy of use, safety, effectiveness, costs, etc.) must be defined a priori. The multidisciplinary group will be responsible for establishing and agreeing upon the values deemed acceptable or desirable. For the principal variables, the values to be used to determine the success or failure of the technology must be carefully established, taking into account all the pertinent information, i.e., results reported in the scientific literature, data published in diverse data sources and personal experiences.
- i. Data-analysis: The data-analysis procedure (unit responsible) must be established.
- j. Patient information: It is recommended that the written information to be furnished to the patients at the various centres be decided upon and rendered uniform.
- k. Data confidentiality: The procedure for maintaining the confidentiality of the data (masking of data, encryption, etc.) must be laid down.
4. Organisation: Once the planning stage has been finalised, it will be necessary to proceed to the implementation stage. This stage can be divided into 3 parts:
- a. Development of data-collection instruments (specific questionnaires).
- b. Drafting of detailed instructions covering the data-collection and dispatch procedure. These must include the definition and coding of all variables.
- c. Development of a common database.
5. Data-quality control: Provided that it is feasible, consideration should be given to an external review of 5% of entries in order to assess the quality of the data registered. This activity could be undertaken by the unit responsible for data analysis.
6. Plan for dissemination of results and alert system for unforeseen problems: It is recommended that, prior to the start of the study, agreement is reached on how to proceed in the event of detection of problems and how the results of the study are to be disseminated.

OUTCOME INDICATORS

This section of the guideline displays a list of key indicators for measuring the different aspects of post-introduction observation of new technologies. The purpose of these outcome indicators is to measure relevant aspects of the diffusion and use of new technologies once they are used in standard clinical practice, in order to furnish valid and reliable information on the quality of health services. In the provision of health services, "quality" measures to what extent the result obtained is the desired -optimal- according to the scientific knowledge and the context in which such services are offered. Limitations, in the sense of not achieving the desired effect, may stem from a flawed structure or incorrect process, due to over-, under- or misuse of medical technologies (25).

Table 6 shows the list of proposed indicators, grouped according to the respective aspects considered relevant for post-introduction observation of new technologies: diffusion, accessibility, adequacy of use, effectiveness, safety and economic impact. In the following tables, the specific characteristics of each indicator are described.

ver imagen View full-size table

Table 6. List of outcome indicators
Domain	Name of indicator
Difusion	Adoption of the new technology
Difusion	Coverage of the technology
Accessibility	Accessibility
Adequacy of use	Adequacy of patient selection criteria
Effectiveness	Effectiveness
Effectiveness	Effectiveness by subgroup
Safety	Severe adverse effects
	Mild adverse effects
	Adverse effects by subgroup
Economic impact	Adequacy of costs

Indicator	Adoption of the new technology
	In what proportion of centres has the new technology been adopted?
Justification	The approval/funding of a new technology does not assure that it will be efficaciously adopted within the health system. Frequently, the implementation of a new technology requires, not only financial investment, but also structural and organisational changes, availability of trained staff, and time. Ascertaining the centres that have adopted the new technology can be essential in order to detect problems of implementation.
Formula	Total number of centres that have adopted the new technology within a given period of time Total number of centres at which adoption of the new technology is deemed desirable within a given period of time X 100
Explanation of terms	Adoption of a new technology: implementation and use of the new technology in clinical practice in the centres considered within the target area. Centres at which the adoption of the new technology is deemed desirable: centres at which the adoption of the technology is deemed desirable/foreseeable must be defined a priori. Depending on the health system and/or type of technology, these could be all the centres within the target area, all centres with the relevant specialisation or solely reference centres or centres that have been authorised to use the new technology.
Scope of study	The scope of study (geographical area, province, region, country) must be defined in order to delimit the centres that are to be targeted by the study and are to form the numerators and denominators of the formula.
Guideline standard	80-100%.
Time frame	A mean follow-up of 1-year after approval/funding of the new technology is taken as reference. This period may be extended to 2-3 years in the case of technologies that may require major financial, organisational or structural changes.
Data-sources	Suppliers or manufactures of the new technology (technologies that require a specific apparatus, device or fungible goods for their use). Health-centre service catalogue (if existing and updated). Direct contact with each centre and/or department (innovative procedures or techniques based on the learning of skills). Contact with administrative centres tasked with approval of centres and allocation of resources for public health care.

Indicator	Coverage of the technology
	How many patients of those eligible to use the new technology, actually use it?
Justification	Ascertaining the relationship between patients eligible to use the new technology and those that use it in clinical practice is important when it comes to detecting problems of coverage, detecting whether there may be an over-use of the new technology with respect to what was envisaged, or detecting possible resistance to its use (under-use)
Formula	Total number of patients that have used the new technology within a given period of time Total number of patients eligible to use the new technology within a given period of time X 100
Explanation of terms	Patients that have used the new technology: consideration will only be given to subjects for whom results of applying the technology are available. Scheduled interventions that were not performed must be excluded. Patients eligible to use the new technology: patients in the target area who meet the selection criteria set a priori by the multidisciplinary group
Guideline standard	80-100%. This must be set a priori.
Scope of study	The scope of the study (hospital, health area, geographical area, province, region, country) must be defined in order to delimit the units and services to be targeted by the study.
Time frame	A mean follow-up of 1-year after approval/funding of the new technology is taken as reference, though this may be extended in the case of technologies indicated in a small number of patients. This must be set a priori.
Data sources	Patients that have used the new technology: clinical registries, medical records and/or MBDS registries Patients eligible to use the new technology: Disease frequency registries (number of cases, prevalence..) Studies published in the scientific literature Registries on the number of procedures using alternative techniques to those that this would replace.

Indicator	Accesibilidad
	What is the degree of use of the new technology in the various geographical areas?
Justification	When a new technology is introduced, an appropriate patient-reference circuit must be established to ensure that the degree of use is independent of this technology's availability in the health area. Calculating and comparing the use of a new technology in different geographical areas is fundamental for identifying problems of geographical accessibility
Formula	Number of patients residing in a predefined geographical area who have used the new technology within a given period of time Patients residing in a geographical area eligible for using the new technology within this period of time X 100
Explanation of terms	Subjects that have used the new technology: consideration will be given only to subjects for whom results of applying the technology are available. Scheduled interventions that were not performed must be excluded. Subjects susceptible to using the new technology: patients in the target area who meet the selection criteria set a priori by the multidisciplinary group.
Guideline standard	This must be set a priori
Scope of study	The scope of the study (health area, province, region, country) must be indicated, in order to delimit the health centres and areas to be targeted by the study.
Time frame	A mean follow-up of 1-year after approval/funding of the new technology is taken as reference, though this may be extended in the case of technologies indicated in a small number of patients. This must be set a priori.
Data sources	Patients that have used the new technology: clinical registries, medical records and/or MBDS registries Patients eligible to use the new technology: Disease frequency registries (number of cases, prevalence) Studies published in the scientific literature Registries on the number of procedures using previous alternative techniques for the same indication.

Indicator	Adecuación de criterios de selección de pacientes
	Are the patient-selection criteria used within the various centres, departments or units in line with what is deemed appropriate and/or desirable for this technology?
Justification	It is important to ascertain whether patients on whom the new technology is used meet the selection criteria. Inappropriate use could cause the loss of important resources without any proven benefit (financial, personal, time) or even impact negatively on the health of the population.
Formula de cálculo	Number of patients among whom the technology has been used, who meet the inclusion criteria Number of patients that have used the technology X 100
Descripción de términos	Inclusion criteria: consideration will be given to the clinical indications and to the different characteristics of the patients (age, degree of pain, ineligibility for other treatments) which are used to define patients to be treated/diagnosed with the new technology. The inclusion criteria will be established a priori, taking into account the authorised selection criteria for use of the technology within the health system (if these exist), scientific evidence and/or opinion of experts. Subjects that have used the new technology: consideration will only be given to subjects for whom results of applying the technology are available. Scheduled interventions that were not performed must be excluded.
Guideline standard	100%
Scope of study	The scope of the study (hospital, health area, geographical area, province, region, country) must be indicated, in order to delimit the departments and units that are going to be deemed the study target.
Time frame	A mean follow-up of 1-year after approval/funding of the new technology is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions (patients).This must be defined a priori.
Data sources	clinical registries, medical records and/or MBDS registries.

Indicator	Efectividad
	Is the new technology's degree of success in line with expectations?
Justification	The expectations foreseen in preliminary studies may not be fulfilled when the technology is applied in daily clinical practice (different level of experience, patient profiles, etc.). Should this be so, it may well be necessary for the use of such technology to be reconsidered.
Formula	Number of subjects among whom a beneficial effect has been observed Total number of subjects that have used the new technology X 100
Descripción de términos	Beneficial effect: effect deemed acceptable or desirable for this technology. According to the type of technology, the outcome variables to be used to define the new technology's success and the values deemed acceptable or desirable for this variable must be defined a priori. Total number of subjects that have used the new technology: deemed to be all subjects in the target population who have been treated or diagnosed with the new technology.
Outcome variables	They must be set a priori. As far as possible, these variables must be objective or capable of being rendered objective.
Guideline standard	90-100%. This must be set a priori..
Scope of study	The scope of the study (hospital, health area, geographical area, province, AR, country) must be indicated, in order to delimit the units and departments that are going to be considered.
Tiempo de seguimiento	SA mean follow-up of 1-year after approval/funding is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions (patients).
Data sources	Specific clinical registries Medical records Surveys/questionnaires

	Subgroup effectiveness
	Is the new technology's percentage of success in line with what was expected for certain subgroups of patients?
Justification	Preliminary studies often fail to envisage assessment of effectiveness in specific subgroups (children, the elderly, subjects with serious co-morbidities) and this may differ significantly from overall effectiveness. Similarly, where similar though not identical procedures or procedures that require a learning curve are involved, effectiveness may also differ significantly as between the various subgroups (team, degree of specialisation, etc.).
Formula	Number of patients in a study subgroup among whom a beneficial effect has been observed Total number of subjects in the subgroup that have used the new technology X 100
Descripción de términos	Study subgroup: subgroups susceptible to registering differences in terms of effectiveness. These must be established a priori. Beneficial effect: effect deemed acceptable or desirable for this technology. According to the type of technology, the outcome variables to be used to define the new technology's success and the values deemed acceptable or desirable for this variable must be defined a priori. Total number of subgroup subjects that have used the new technology: deemed to be all subjects in the target study subgroup who have been treated or diagnosed with the new technology.
Outcome variables	They must be set a priori. As far as possible, these must be objective or capable of being rendered objective.
Guideline standard	90-100%. This must be set a priori.
Scope of study	The scope of the study (hospital, health area, geographical area, province, region, country) must be indicated, in order to delimit the departments and units that are going to be considered.
Tiempo de seguimiento	A mean follow-up of 1-year after approval/funding is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions in each subgroup.
Data sources	Specific clinical registries Medical records Surveys/questionnaires

	Severe adverse effects
	What proportion of patients present with severe adverse effects?
Justification	In daily clinical practice, it frequently happens that severe adverse effects not envisaged in preliminary studies appear, or that these appear in a greater proportion of subjects. Quantifying these variations is important for ensuring patient safety.
Formula	Number of patients among whom a severe adverse effect has been observed Total number of patients on whom the new technology has been used X 100
Descripción de términos	Severe adverse effects: for practical purposes severe adverse effects are deemed to be any injury, damage or complication deriving from the use of the technology, including diagnostic errors (FP, FN) which can lead to death or major deterioration of patient's state of health. The type of adverse effects to be considered for study purposes must be established a priori. Total number of subjects that have used the new technology: deemed to be all subjects in the target population who have been treated or diagnosed with the new technology.
Guideline standardes	This must be defined a priori
Scope of study	The scope of the study (hospital, health area, geographical area, province, AR, country) must be indicated, in order to delimit the departments and units that are going to be considered objective of the study.
Tiempo de seguimiento	A mean follow-up of 1-year after approval/funding is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions (patients).This must be defined a priori.
Data sources	Specific clinical registries Medical records Surveys/questionnaires

	Mild to moderate adverse effects
	What proportion of patients present with moderate and/or mild adverse effects?
Justification	Analysis of adverse effects is the basis for identifying genuine opportunities for improvement. Adverse effects, albeit not severe, may affect patients' quality of life and, where frequent, can give rise to considerable resource use.
Formula	Number of subjects among whom moderate and/or mild adverse effects have been observed Total number of subjects on whom the new technology has been used X 100
Descripción de términos	Moderate and/or mild adverse effects: for practical purposes these are deemed to be any non-severe injury, damage or complication deriving from the use of the technology, as well as any diagnostic error (FP, FN) that has led to an alteration in the patients' state of health. The type of adverse effects to be considered for study purposes must be established a priori. Total number of subjects that have used the new technology: deemed to be all subjects in the target population who have been treated or diagnosed with the new technology.
Guideline standardes	These must be defined a priori
Scope of study	The scope of the study (hospital, health area, geographical area, province, region, country) must be indicated, in order to delimit the units and departments that are going to be considered objective of the study.
Tiempo de seguimiento	A mean follow-up of 1-year after approval/financing is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions (patients).This must be defined a priori.
Data sources	Specific clinical registries Medical records Surveys/questionnaires

	Adverse effects in subgroups
	Are the adverse effects in line with what was expected in especially susceptible subgroups?
Justification	Especially sensitive subgroups (children, the elderly, pregnant women, subjects with serious co-morbidities), subgroups with different degrees of specialisation or different ways of performing the intervention (apparatus, protocols, teams, etc.), could register previously unexpected adverse effects.
Formula	Number of subjects in a subgroup among whom an adverse effect has been observed Total number of subjects in the subgroup that have used the new technology X 100
Descripción de términos	Study subgroup: subgroups susceptible to present a higher proportion of adverse effects. These must be established a priori. Adverse effects: for practical purposes these are deemed to be any injury, damage or complication deriving from the use of the technology, as well as any diagnostic error (FP, FN) which has led to an alteration in the patient's state of health. The type of adverse effects to be considered for study purposes must be established a priori. Total number of subjects that have used the new technology: deemed to be all subjects in the target population who have been treated or diagnosed with the new technology.
Guideline standardes	These must be defined a priori.
Scope of study	The scope of the study (hospital, health area, geographical area, province, AR, country) must be indicated, in order to delimit the units and departments considered for study purposes.
Follow-up time	A mean follow-up of 1-year after approval/financing is taken as reference, though this may be extended in order to attain a minimum number of 25-30 interventions in each subgroup.
Data sources	Specific clinical registries Medical records Surveys/questionnaires

	Adequacy of costs
	Are the observed costs in line with what was expected?
Justification	For adequate planning, it is essential to ascertain whether the estimated costs are in line with the real costs of the technology.
Formula	Observed costs under specific chapters of major financial impact Estimated costs under specific chapters of major impact X 100
Description of terms	Costs: includes costs needed for implementing the new technology but also costs derived from its utilisation in clinical practice. Includes costs of apparatus, support material, fungible costs, maintenance costs, waste management, health service transport, etc.). The costs to be considered for study purposes must be established a priori. Costs must be measured in monetary units. Chapter of major impact: according to the type of technology, the cost chapters responsible for major resource use must be established a priori.
Guideline standardes aceptables	These must be defined a priori.
Scope of study	The health centres to be targeted by the study must be defined a priori.
Data sourcess	Specific registries Pharmaceutical companies, producers and distributers of the technology Ad hoc calculation

For new technologies that could have important implications in the consumption of resources it might be important to establish specific resource use indicators. These could include, among others, staff hours, length of hospital stay, operation time, etc.