Real World Research on Medical Devices – An Introduction to Quality Control Methods!
1. Data quality
According to the content of the General Principles, data quality can be evaluated in terms of representativeness, completeness, accuracy, authenticity, consistency and repeatability. For specific evaluation content, see Chapter 3 of the General Principles. Applicants must evaluate the quality of the data sources used in accordance with the above 6 Evaluate each aspect and present the evaluation results of each dimension in the form of a table.
2.Risk of bias
Bias may exist at all stages of real-world research design, implementation, analysis, and reporting. Applicants can describe in detail the measures used to control different risks of bias in the real-world research plan from three aspects: selection bias, information bias, and confounding bias. . For observational real-world studies, you can refer to the ROBINS-I evaluation tool for non-randomized interventional clinical studies to assess the risk of bias of the overall study. Here are just some of the types of bias found in real-world research:
(1) The research population lacks representativeness
In the design stage, it is also very important to set reasonable inclusion criteria. The inclusion criteria for the study consider whether the population included can represent the expected scope of application of the product. pRCT usually uses looser inclusion criteria, so it is less affected by the inclusion criteria. influence of selection bias. For prospective studies, consecutive enrollment is recommended to avoid patient selection. For certain devices that are easily affected by clinical institutions and physician levels, a multicenter design is recommended. For studies with controlled settings, especially case-control designs, measures need to be taken to avoid admission bias in the design, such as the experimental group and the control group being determined by sampling from the same population.
(2) Confounding bias
Confounding bias means that the degree of correlation (association) between exposure factors and intervention measures is distorted or interfered by other factors, so that the relationship between the research variables presented and the evaluation indicators or outcome variables is not true, but biased with the superposition of confounding effects. relation.
Randomization is a powerful means of controlling confounding, balancing both measurable and unmeasured confounding factors. Since the vast majority of real-world study designs (with the exception of pRCTs) do not use randomization, other methods such as restriction, paired, and stratified designs can be considered during the analysis stage to control for confounding. During the analysis stage, various adjustment statistical methods (such as stratified analysis, multivariable regression analysis, propensity score-based adjustment methods, etc.) can also be applied to control confounding.
(3) Deviation from intervention measures
In real-world studies, interventions may deviate midway through treatment due to various reasons, such as patients actively requesting to change treatment methods, doctors changing treatment strategies, etc., interventions with multiple treatments (such as hemodialysis) or interventions with long treatment times. (e.g., ventilators, extracorporeal membrane oxygenators), there is a greater likelihood of intervention deviation. When conducting real-world research, it is necessary to consider in advance the degree of risk of such bias in the device to be studied. If there is a non-negligible risk of bias in intervention measures, when selecting a real-world data source, it is necessary to consider whether the data source is detailed and accurate. Document the treatments used and any changes that occur during treatment.
In clinical practice, there may also be errors in the recording of intervention measures, such as errors in the manufacturer, model and specification of the device used, leading to information bias related to the intervention measures. When it is suspected that there is a possibility of recording errors, consider using the patient's imaging system The implant shape, marker point characteristics, price on the bill and other other information are verified.
(4) Measurement bias
In real-world research, accurate and precise measurements are important measures to reduce information bias. Imposing blinding can help overcome measurement bias caused by subjective factors of applicants or subjects. When blinding is difficult, objective hard endpoints (such as death, etc.) should be chosen as much as possible. During the implementation process, develop detailed operation manuals, train staff, standardize data collection procedures and monitor data collection activities, and use unified methods to collect, measure and interpret information; under applicable conditions, a third-party independent data monitoring committee can be set up Or unify standards and standardize the measurement results of indicators; when it is suspected that the data measurement is inaccurate, carry out data verification. In addition to the above commonly considered measures, corresponding measures need to be specified based on the specific types of measurement bias that may occur.
Measurement bias from subjects: Sufficient training is required so that subjects can correctly understand the questions and answer them accurately.
Measurement bias for the source of evaluators: This measurement bias can be reduced by using multiple evaluators for parallel measurements. Although in real-world studies, more often one person (i.e., the attending physician) completes the relevant measurement or evaluation activities, a certain In some cases (e.g. image-based measurements), measurements can be taken again by another evaluator afterwards.
Measurement bias in the source of evaluation tools: use measurement methods with proven reliability and validity, use precise instruments, etc.
(5) Recall bias
Try to avoid collecting information through the recall of the research population during the design stage, and try to record the data in documents as soon as it is generated. The nested case-control design can avoid the recall bias caused by the traditional case-control method of obtaining intervention measures, baseline data, etc. through recall.
In some cases, reviewing the patient's other health information may help confirm whether the patient's recollection is accurate. For example, if a patient recalls having pain or inflammation after receiving an intervention, the patient's health records, medication records, and electronic medical records on the corresponding dates can be reviewed to see if there is any relevant information for further support.
(6) Selection bias caused by loss to follow-up
It is necessary to set up adequate measures to prevent loss to follow-up as much as possible in the real-world research plan, including remedial measures that can be adopted after loss to follow-up, such as supplementing relevant data through additional follow-up methods (such as phone calls, home visits), and integrating with other data sources. (such as medical insurance data, death registration data, etc.) links, etc.;
In view of possible missing data when using retrospective data, the methods and principles for handling missing data need to be clarified in advance in the research plan. For missing data, the reasons for loss to follow-up need to be investigated as clearly as possible. If the loss to follow-up is not related to the intervention or outcome, it can be filled in according to the imputation methods and principles specified in the plan. A conservative approach can also be used for imputation, for example, the experimental group is imputed as invalid and the control group is imputed as valid.
(7)Reporting bias
Selectively presenting favorable results will cause selection reporting bias. The best way to avoid reporting bias is to pre-specify it in the protocol or statistical analysis plan. It is recommended that the protocol be pre-specified on public websites (such as China Clinical Trial Registration Center, ClinicalTrials.gov, etc.) register.
For real-world research using retrospective data, applicants must set up measures to ensure that researchers do not have access to outcome data before formal statistical analysis, to prevent researchers from conducting data mining in order to obtain expected statistical results before the start of the study. For example, when applying statistical analysis methods based on propensity scores, a two-stage design can be adopted. In the first stage, it is necessary to build an outcome data firewall, identify independent statisticians, identify confounding variables, and establish a propensity score estimation model. After a satisfactory balance of confounding variables is achieved in the first stage, a statistical analysis plan will be formulated in the second stage.
(8) Unmeasured confounding bias
If all confounders have been collected and modeled correctly, and the sample size is sufficient, estimation bias can be reduced or eliminated through appropriate analytical methods. However, in practice, it is difficult to obtain all confounding factor data, and some confounding factors are not or cannot be measured. The resulting bias is called unmeasured confounding bias. The effect size of unmeasured confounding is difficult to estimate, and sensitivity analysis can be attempted to assess its potential impact on conclusions.
3. Assess direction and magnitude of bias
Bias is directional, that is, the effect size of an intervention is underestimated or overestimated. Bias also varies in degree. Some relatively small biases may not affect the conclusion of the study. After completing the study, it is recommended to review and summarize any remaining biases during the study and assess the impact on the strength of the evidence.
How to assess bias varies depending on the specific study. For example, for selection bias caused by loss to follow-up, comparing the characteristics of the study population who were lost to follow-up with the characteristics of the study population who were not lost to follow-up, it may be due to the discovery that the intervention was ineffective due to loss to follow-up, thus determining the bias. existence and direction of bias. For measurement bias, some statistical indicators (such as intraclass correlation coefficient, coincidence rate, etc.) can be used to compare the measurement values of different people and different clinical institutions to help evaluate measurement bias.
