90

Can you spot weak evidence synthesis practices?. Design the quiz around scenario-based choices involving inclusion criteria, funnel plots, heterogeneity, fixed versus random effects, subgroup analysis, and sensitivity analysis. Keep answers explanatory and source-bound, emphasizing why a practice is risky rather than merely labeling it wrong.

Spot the misleading meta-analysis choice

This quiz uses short scenarios to test whether a meta-analysis is being handled in a way that could distort the pooled estimate. The focus is on six common weak practices plus publication-bias sensitivity analysis, with each answer explaining why the move is risky rather than just naming it as a mistake[1][2].

Scenario-based quiz

  1. 1) Inclusion criteria Scenario: A rapid review searches only one database, excludes gray literature, and limits studies to English, then presents the pooled estimate as if it covers the whole evidence base. Prompt: What is the main risk? Best answer: The risk is that narrow or improvised inclusion criteria can systematically miss relevant studies, so the pooled estimate may reflect what was easiest to find rather than the underlying evidence[3]. Why this matters: The practical guide warns that shortcut search choices can change how many studies are included and can bias the estimate and its interpretation[4].
  2. 2) Funnel plots and publication bias Scenario: A reviewer sees an asymmetric funnel plot and declares that publication bias has been proven. Prompt: What is the main risk? Best answer: The risk is over-interpreting funnel asymmetry as if it has only one explanation. Why this matters: The guide says funnel asymmetry can reflect bias or heterogeneity, not publication bias alone, and publication bias itself is a threat because missing or selectively published studies can distort quantitative synthesis[5][6].
  3. 3) Heterogeneity Scenario: Included studies differ in design, population, and outcome, but the paper emphasizes only the overall pooled effect and barely discusses between-study inconsistency. Prompt: What is the main risk? Best answer: The risk is pooling diverse studies as if they estimate one clean effect when heterogeneity may limit interpretability and generalizability[7]. Why this matters: The source notes that basic-research meta-analytic datasets are often diverse and inconsistent, so ignoring heterogeneity can make a summary effect look more stable or transportable than it really is[8].
  4. 4) Fixed versus random effects Scenario: Authors choose a fixed-effect model because it gives a narrower confidence interval, even though the studies differ meaningfully. Prompt: What is the main risk? Best answer: The risk is using a model that understates uncertainty and fails to represent real between-study variation[9]. Why this matters: The guide explains that random-effects models include between-study variance, whereas fixed-effect models do not, so the wrong choice can make the result look more precise than justified[10].
  5. 5) Subgroup analysis Scenario: After finding a weak overall effect, investigators run many subgroup analyses until one subgroup becomes statistically significant, then highlight that subgroup as the key finding. Prompt: What is the main risk? Best answer: The risk is data-driven subgroup fishing that inflates false-positive risk[11]. Why this matters: The guide warns that multiple comparisons can make subgroup differences look more meaningful than they really are, especially when power is low and many splits are tried[12].
  6. 6) Sensitivity analysis Scenario: One study strongly influences the pooled result, so the team removes it and reports the new pooled estimate without explaining why that study was excluded. Prompt: What is the main risk? Best answer: The risk is using sensitivity analysis as a hidden result-editing tool instead of a transparency tool[13]. Why this matters: The guide says influential studies should not be blindly discarded; sensitivity analysis is meant to show how dependent conclusions are on analytic choices, and unjustified removal can reshape the pooled result while hiding fragility[14].
  7. 7) Publication-bias sensitivity analysis Scenario: A meta-analysis has only a small number of studies, and the authors say publication bias cannot be assessed at all, so they ignore it. Prompt: What is the main risk? Best answer: The risk is treating publication-bias assessment as impossible rather than using cautious sensitivity approaches[15]. Why this matters: The publication-bias paper describes sensitivity-analysis approaches, including a priori weight functions, as a way to examine how missing or selectively published studies could change the synthesis, even in small meta-analytic datasets[16].

Key takeaway

Across these scenarios, the common problem is not just that a step is technically wrong, but that it can make the pooled estimate look more complete, more precise, or more certain than the evidence really supports. Good evidence synthesis is therefore about transparency, checking assumptions, and stress-testing conclusions rather than simply producing one summary number[17][18].


Related Content From The Pandipedia