Using threshold analysis to assess the robustness of public health intervention recommendations from network meta-analyses: application to accident prevention in households with children under five

Background

In the appraisal of clinical interventions, complex evidence synthesis methods, such as network meta-analysis (NMA), are commonly used to investigate the effectiveness of multiple interventions in a single analysis. The results from a NMA can inform clinical guidelines directly or be used as inputs into a decision-analytic model assessing the cost-effectiveness of the interventions. However, there is hesitancy in using complex evidence synthesis methods when evaluating public health interventions. This is due to significant heterogeneity across studies investigating such interventions and concerns about their quality.

Threshold analysis has been developed to help assess and quantify the robustness of recommendations made based on results obtained from NMAs to potential limitations of the data. Developed in the context of clinical guidelines, the method may prove useful also in the context of public health interventions. In this paper, we illustrate the use of the method in public health, investigating the effectiveness of interventions aiming to increase the uptake of accident prevention behaviours in homes with children aged 0–5.

Methods

Two published random effects NMAs were replicated and carried out to assess the effectiveness of several interventions for increasing the uptake of accident prevention behaviours, focusing on the safe storage of other household products and stair gates outcomes. Threshold analysis was then applied to the NMAs to assess the robustness of the intervention recommendations made based on the results from the NMAs.

Results

The results of the NMAs indicated that complex intervention, including Education, Free/low-cost equipment, Fitting equipment and Home safety inspection, was the most effective intervention at promoting accident prevention behaviours for both outcomes. However, the threshold analyses highlighted that the intervention recommendation was robust for the stair gate outcome, but not robust for the safe storage of other household items outcome.

Conclusions

In our case study, threshold analysis allowed us to demonstrate that there was some discrepancy in the intervention recommendation for promoting accident prevention behaviours as the recommendation was robust for one outcome but not the other. Therefore, caution should be taken when considering such interventions in practice for the prevention of poisonings in homes with children aged 0–5. However, there can be some confidence in the use of this intervention in practice to promote the possession of stair gates to prevent falls in homes with children under 5. We have illustrated the potential benefit of threshold analysis in the context of public health and, therefore, encourage the use of the method in practice as a sensitivity analysis for NMA of public health interventions.