Beyond the Hype: A Practical Guide to Combat p-Hacking in Neuroimaging ML Studies for Robust Biomarker Discovery

Abstract

This article provides a comprehensive guide for researchers and industry professionals on identifying, preventing, and correcting p-hacking in neuroimaging machine learning. We first explore the foundational problem of p-hacking and its unique manifestations in high-dimensional neuroimaging data. Next, we detail methodological best practices and robust application frameworks. We then offer troubleshooting strategies to diagnose and optimize existing analysis pipelines. Finally, we present validation standards and comparative analysis techniques to ensure reported findings are reliable and reproducible, directly impacting the credibility of biomarker development for neurological diseases and drug discovery.

What is p-Hacking in Neuroimaging ML? Defining the Invisible Threat to Scientific Credibility

Troubleshooting Guide & FAQs

Q1: My cross-validation accuracy is high on my dataset but the model fails completely on an independent test set. What went wrong? A: This is a classic symptom of data leakage or overfitting during feature selection/model tuning. Ensure all steps, including feature selection, parameter optimization, and dimensionality reduction, are nested within the cross-validation loop. Treating the entire dataset before CV invalidates the independence of the test folds.

Q2: I tried multiple preprocessing pipelines and statistical tests until I found a significant result. My paper's methods section only describes the successful pipeline. Is this acceptable? A: No. This is a form of p-hacking known as "researcher degrees of freedom" or "the garden of forking paths." All tested hypotheses, preprocessing choices, and analytical paths must be reported, ideally through pre-registration of the analysis plan. The inflation of Type I error from multiple, unreported comparisons is substantial.

Q3: My neuroimaging ML study has a small sample size (n=20). How can I avoid reporting spurious correlations? A: Small samples are highly susceptible to overfitting and p-hacking. You must: 1) Use simple models, 2) Implement rigorous nested cross-validation, 3) Report performance confidence intervals, 4) Perform permutation testing to establish a null distribution, and 5) Clearly state the study as exploratory/pilot. Avoid complex, high-capacity models.

Q4: How do I correct for multiple comparisons when testing across thousands of voxels or connections? A: Standard Bonferroni is too conservative for correlated neuroimaging data. Standard methods include:

• Family-Wise Error Rate (FWER): Using Random Field Theory or permutation-based methods (e.g., Threshold-Free Cluster Enhancement in FSL).
• False Discovery Rate (FDR): Controls the proportion of false positives among significant findings (e.g., Benjamini-Hochberg).

Q5: My reviewer asked for a "double-dipping" correction. What does this mean? A: "Double-dipping" refers to using the same data for both selection (e.g., identifying an active region) and for selective analysis/statistical testing without correction. To troubleshoot, you must use an independent dataset for selection and testing, or apply proper circular analysis correction methods (e.g., cross-validation).

Table 1: Impact of Common p-Hacking Practices on False Positive Rate (Nominal α = 0.05)

Practice	Description	Estimated False Positive Rate	Primary Field
Outcome Switching	Analyzing multiple outcomes, reporting only significant ones.	Up to 60%	Psychology, Clinical Trials
Optional Stopping	Collecting data, testing repeatedly, stopping once p < .05.	Up to 40%	Various
Covariate Hunting	Trying different covariate adjustments for desired result.	~40%	Observational Studies
Voxel/Cluster Thresholding	Reporting peak voxel after viewing whole-brain map.	~20-35%	Neuroimaging
HARKing	Hypothesizing After Results are Known.	Increases rate substantially	All

Table 2: Recommended Sample Sizes for Neuroimaging ML Studies

Modality	Minimum Recommended Sample (Simple Model)	Target Sample (Complex Model)	Key Consideration
fMRI (Task)	n ~ 100-150	n > 250	High dimensionality, low SNR
sMRI (VBM)	n ~ 150-200	n > 300	Subtle anatomical effects
Resting-state fMRI	n ~ 150	n > 300	High individual variability
EEG/MEG	n ~ 50-100	n > 200	High temporal dimension

Experimental Protocol: Validating a Neuroimaging Biomarker with ML

Objective: To develop and validate a machine learning classifier for diagnosing Disease X from structural MRI scans while rigorously avoiding p-hacking.

1. Pre-registration & Planning:

• Pre-register hypothesis, primary outcome metric (e.g., balanced accuracy), sample size justification, preprocessing pipeline, and main model type on a platform like OSF or ClinicalTrials.gov.
• Define a single, primary region-of-interest (ROI) based on prior literature or use a whole-brain, corrected approach.

2. Data Partitioning (Critical Step):

• Split data into Training/Validation (70%) and a completely held-out Test Set (30%). The test set is touched only once for final model evaluation.
• The Training/Validation set is used for nested cross-validation.

3. Nested Cross-Validation Workflow (on Training/Validation Set):

• Outer Loop (k=5): Splits data into 5 folds. Iteratively hold out one fold as a validation set.
• Inner Loop (k=5): On the remaining 4 folds, perform another CV for hyperparameter tuning (e.g., C for SVM, alpha for regularization).
• Feature Selection: If required, perform feature selection (e.g., ANOVA F-test) within each inner loop fold to prevent leakage.
• Model Training: Train the model with the optimal parameters on the 4 inner-loop folds.
• Validation: Score the model on the held-out outer-loop fold.
• Final CV Performance: Average the performance across all 5 outer-loop validation folds. This is the unbiased estimate.

4. Final Training & Lockdown:

• Train the final model on the entire Training/Validation set using the optimal hyperparameters identified by the nested CV.
• Freeze all model parameters, weights, and preprocessing steps.

5. Final Evaluation (One Time Only):

• Apply the frozen, final model to the completely held-out Test Set.
• Report the test set performance as the primary result. Report confidence intervals.

6. Permutation Testing (Robustness Check):

• On the training set, repeat the nested CV procedure 1000+ times with permuted disease labels to generate a null distribution of CV accuracy.
• Compare the actual CV accuracy to this null distribution to compute a permutation p-value.

Visualizations

Title: Nested Cross-Validation Protocol to Prevent Data Leakage

Title: The p-Hacking Cycle of Researcher Degrees of Freedom

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Rigorous Neuroimaging ML Research

Tool/Reagent	Category	Primary Function	Key Benefit for Avoiding p-Hacking
Pre-registration Template (OSF)	Protocol	Document hypotheses & analysis plan before data analysis.	Eliminates HARKing & outcome switching.
Nilearn / scikit-learn	Software Library	Provides modular ML pipelines for Python.	Enforces clean separation of CV steps, prevents leakage.
Permutation Test Script	Analysis Script	Generates null distribution for model performance.	Provides empirical p-value, less reliant on asymptotic theory.
COINSTAC	Platform	Federated learning for decentralized data analysis.	Allows validation on external data without central sharing.
BIDS Validator	Data Standard	Ensures brain imaging data is organized per the BIDS standard.	Promotes reproducibility and transparent preprocessing.
Class Weight Balancing (sklearn)	Algorithm Parameter	Adjusts class weights in SVM/logistic regression for imbalanced data.	Prevents bias from tuning decision threshold post-hoc.
Docker/Singularity Container	Computational Environment	Encapsulates entire analysis environment (OS, software, versions).	Guarantees exact reproducibility of results.
SIMEX (Simulation Extrapolation)	Statistical Method	Estimates & corrects for measurement error in features.	Reduces bias from ignoring noise in neuroimaging measures.

Technical Support Center

Troubleshooting Guide

Issue 1: Model Performance Drops Significantly on Independent Test Set

• Symptoms: High accuracy (>95%) on training/cross-validation, but near-chance performance (~50-60%) on a separate, held-out test set.
• Potential Cause: Data leakage or overfitting due to the curse of dimensionality. Features (voxels) vastly outnumber samples (participants), allowing models to memorize noise.
• Solution: Implement strict, a priori separation of data. Use a fully independent test set, locked away before any feature selection or model tuning. Apply dimensionality reduction (e.g., PCA, anatomical ROI averaging) only on the training set, then transform the test set.

Issue 2: Inconsistent Results Across Seemingly Identical Re-analyses

• Symptoms: Different research groups report wildly different classification accuracies or biomarker locations using the same public dataset.
• Potential Cause: Exploitation of Researcher Degrees of Freedom (RDoF) in the analysis pipeline. Small, justifiable changes in preprocessing, feature selection thresholds, or hyperparameters can lead to divergent outcomes.
• Solution: Pre-register your entire analysis pipeline, including exact software, version numbers, and all parameter settings. Use containerization (Docker/Singularity) to ensure computational reproducibility.

Issue 3: Failure to Replicate a Previously Published Biomarker

• Symptoms: A reported significant cluster or predictive voxel pattern does not appear in your replication attempt.
• Potential Cause: p-hacking in the original study, such as performing exhaustive search across smoothing kernels or statistical thresholds without correct multiple comparison correction.
• Solution: Adopt out-of-bag error estimation or nested cross-validation to obtain unbiased performance metrics. Use permutation testing to establish a null distribution for model accuracy.

Frequently Asked Questions (FAQs)

Q1: What is the single most important step to avoid p-hacking in neuroimaging ML? A: A Priori Pipeline Pre-registration. Before touching the data, document every decision: software, preprocessing steps, feature selection method, model algorithm, hyperparameter ranges, and validation scheme. Submit this protocol to a registry.

Q2: How small should my test set be relative to my training set? A: There is no fixed rule, but it must be statistically independent. A common pitfall is using too small a test set, leading to high variance in the final performance estimate. Use sample size estimation tools. A minimum of 20% of total data is often recommended, but larger is better for stable estimates.

Q3: What validation method is best for small sample sizes (n<100)? A: Nested Cross-Validation. It provides a less biased estimate of true model performance when you need to perform model selection and hyperparameter tuning on limited data.

• Outer Loop: Estimates generalization error.
• Inner Loop: Selects the best model/parameters for that fold.

Q4: Are some ML models more prone to the curse of dimensionality than others? A: Yes. Models with high complexity (e.g., non-linear SVMs, deep neural networks) are more prone. Simpler linear models (e.g., Logistic Regression with L1/L2 penalty) with built-in regularization can be more robust when p >> n.

Experimental Protocols & Data

Protocol: Nested Cross-Validation for Unbiased Estimation

• Preprocessing: Standardize (demean, scale) entire dataset globally.
• Outer Split: Split data into K folds (e.g., K=5 or 10). For each outer fold i: a. Hold out fold i as the temporary test set. b. Use the remaining K-1 folds for the inner loop.
• Inner Loop: Perform a second, independent cross-validation on the K-1 folds to select optimal hyperparameters (e.g., regularization strength C for SVM).
• Train Final Model: Train a new model on the entire K-1 folds using the optimal parameters from step 3.
• Evaluate: Test this model on the held-out outer fold i. Record accuracy.
• Repeat: Repeat steps 2-5 for all K outer folds.
• Report: The mean and standard deviation of accuracy across the K outer folds.

Table 1: Impact of Dimensionality Reduction on Model Robustness

Dataset (n)	Original Features (p)	Method	Features After Reduction	CV Accuracy	Independent Test Accuracy	Accuracy Gap
ADHD-200 (200)	~1,200,000 (voxels)	None	~1,200,000	92%	58%	34%
ADHD-200 (200)	~1,200,000 (voxels)	Anatomical ROIs (AAL)	116	75%	71%	4%
ABIDE (500)	~1,500,000 (voxels)	PCA (95% variance)	15,000	88%	65%	23%
ABIDE (500)	~1,500,000 (voxels)	ICA (100 components)	100	78%	74%	4%

Table 2: Common Researcher Degrees of Freedom (RDoF) and Mitigations

Pipeline Stage	Example RDoF	Recommended Mitigation
Preprocessing	Smoothing kernel FWHM (4mm vs. 8mm)	Pre-register kernel size; test robustness in sensitivity analysis.
Feature Selection	Univariate threshold (p<0.01 vs. p<0.001)	Use stability selection or pre-register a fixed threshold.
Model Choice	SVM (linear vs. RBF kernel)	Pre-register model family; justify based on literature.
Hyperparameter Tuning	Range of C values searched (1e-5 to 1e5)	Use a pre-defined, justified grid; employ nested CV.
Statistical Testing	Voxel-wise threshold (p<0.001 unc. vs. FWE)	Pre-register correction method. Use permutation tests.

Visualizations

Diagram 1: The Curse of Dimensionality in Neuroimaging ML

Diagram 2: Nested Cross-Validation Workflow

The Scientist's Toolkit: Research Reagent Solutions

Item	Function
BIDS (Brain Imaging Data Structure)	Standardizes file organization and metadata, enabling reproducible data sharing and pipeline automation.
fMRIPrep / CAT12	Automated, standardized preprocessing pipelines for fMRI and sMRI data, reducing RDoF.
Scikit-learn / nilearn	Open-source ML libraries with built-in functions for cross-validation and preprocessing, promoting transparent code.
Docker / Singularity Containers	Packages the complete software environment (OS, libraries, code) to guarantee identical analyses across labs.
OSF / AsPredicted Registries	Platforms for pre-registering study hypotheses and analysis plans before data collection/analysis.
COINSTAC	Enables federated analysis across sites without sharing raw data, addressing small sample sizes.
Permutation Testing Scripts	Generate valid null distributions for ML metrics, providing robust p-values corrected for multiple testing.

Technical Support Center

Troubleshooting Guides

Issue: Model performance drops sharply between validation and final test set. Cause: This is often a sign of data leakage or overfitting during hyperparameter tuning. The hyperparameters may have been tuned to noise in the validation set, especially if the validation set was used iteratively. Solution: Implement a strict nested cross-validation protocol. Keep a completely held-out test set that is never used for any model development or tuning. Use an inner CV loop only for hyperparameter optimization.

Issue: Statistical significance disappears when adding more subjects. Cause: Likely due to prior "p-hacking" via data peeking. Early stops after seeing a significant p-value from a small sample can capitalize on chance. The effect size was likely overestimated. Solution: Pre-register your analysis plan and sample size using power analysis. Use sequential analysis or Bayesian methods if interim looks are necessary.

Issue: Inconsistent feature selection results across random seeds. Cause: Unstable feature selection methods that are highly sensitive to data perturbations, compounded by performing selection on the entire dataset before CV. Solution: Perform feature selection independently within each fold of the cross-validation loop. Use stability selection or penalized models with built-in feature selection.

Issue: "Double-dipping" - using the same data for exploratory analysis and confirmatory testing. Cause: Lack of clear separation between hypothesis-generating and hypothesis-testing datasets. Solution: Physically or procedurally split your data. Exploratory analysis on Dataset A generates hypotheses, which are then tested only on a completely independent Dataset B.

Frequently Asked Questions (FAQs)

Q1: Is it p-hacking to try different preprocessing pipelines? A: Yes, if you try multiple pipelines and only report the one that gives the best (most significant) result without correcting for multiple comparisons. The solution is to choose and pre-register a single pipeline based on prior literature, or to use a pipeline that is fixed before any outcome analysis, or to account for the multiple pipeline comparisons statistically.

Q2: How can hyperparameter tuning lead to inflated performance? A: Tuning hyperparameters by maximizing performance on a test set (or a validation set used repeatedly) effectively fits the model to the noise in that specific set. This optimizes performance for that particular data split but does not generalize to new data, leading to optimistic bias.

Q3: What is the correct way to handle outliers? A: Define an outlier detection and handling rule a priori (pre-registration) based on methodological grounds, not based on whether removing a data point improves the p-value. Applying different outlier rules and selecting the most significant outcome is a form of p-hacking.

Q4: We have a small dataset. Is it acceptable to do leave-one-out CV (LOOCV) for both tuning and evaluation? A: LOOCV can have high variance. More critically, you must not use the same LOOCV loop for tuning and evaluation. You need a nested loop: an outer loop for performance estimation, and an inner loop (within each training fold) for tuning.

Q5: Are there tools to help prevent p-hacking in neuroimaging ML? A: Yes. Use tools that enforce reproducible workflows, like Nipype, Neurodocker, or DataLad. For pre-registration, use platforms like OSF or AsPredicted. Employ libraries like scikit-learn's Pipeline and GridSearchCV with proper CV splitters to prevent leakage.

---

Summarized Quantitative Data on p-Hacking Prevalence & Impact

Table 1: Estimated Prevalence of Questionable Research Practices (QRPs) in Scientific Fields

Field	Estimated % of Researchers Admitting to QRPs	Common QRP
Psychology	~94% (1)	Data peeking, selective reporting
Neuroscience / Neuroimaging	>50% (2)	Flexibility in analysis (voxel thresholding, ROI selection)
Ecology & Evolution	~64% - 89% (3)	Post-hoc exclusion of outliers, covariate selection
Machine Learning (applied)	Not systematically surveyed	Hyperparameter tuning on test set, competition overfitting

Table 2: Impact of Analysis Flexibility on False Positive Rate (Simulation Studies)

Analysis Pipeline Flexibility	Nominal α (e.g., 0.05)	Actual False Positive Rate (Simulated)
Single, pre-registered analysis	0.05	~0.05
Trying two analysis methods	0.05	~0.08
Trying multiple outlier rules	0.05	~0.15+
Peeking at data & optional stopping	0.05	Can approach 1.0 (4)
Hyperparameter tuning w/o nested CV	0.05	Dramatically inflated Type I error

Sources: (1) John et al., 2012; (2) Carp, 2012; (3) Fraser et al., 2018; (4) Simmons et al., 2011.

---

Experimental Protocols for Robust Neuroimaging ML

Protocol 1: Nested Cross-Validation for Hyperparameter Tuning & Evaluation

Objective: To obtain an unbiased estimate of model performance when hyperparameters need to be tuned.

• Split: Divide the full dataset into K1 outer folds (e.g., 5 or 10).
• For each outer fold: a. Set aside Fold i as the outer test set. b. The remaining K1-1 folds form the outer training set. c. Split the outer training set into K2 inner folds (e.g., 5). d. Inner Loop: For each hyperparameter configuration, perform K2-fold CV on the outer training set. Compute the average validation score across the K2 inner folds. e. Select the hyperparameter configuration with the best average inner validation score. f. Retrain: Retrain a model on the entire outer training set using the selected hyperparameters. g. Evaluate: Apply this final model to the held-out outer test set (Fold i) to obtain a performance metric (e.g., accuracy).
• Final Estimate: The average of the K1 performance metrics from the outer test sets is the unbiased performance estimate. Report the mean and variance.

Protocol 2: Pre-registration of Analysis Pipeline

Objective: To prevent analytic flexibility and HARKing (Hypothesizing After Results are Known).

• Platform: Use a time-stamped repository (e.g., OSF, AsPredicted).
• Specify: Before data collection or analysis, document:
  • Primary hypothesis and outcome measure.
  • Planned sample size and stopping rule.
  • Data inclusion/exclusion criteria.
  • Preprocessing steps and software (e.g., SPM version, smoothing kernel).
  • Feature extraction method (e.g., atlas, voxel-wise).
  • Machine learning algorithm and fixed hyperparameters (or tuning space/strategy if tuning is planned).
  • Validation scheme (e.g., nested 5x5 CV, hold-out test set %).
  • Statistical inference method for the performance metric.

---

Mandatory Visualizations

Title: The p-Hacking vs Standard Analysis Pipeline

Title: Nested CV for Unbiased Hyperparameter Tuning

---

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging ML Analysis

Item / Solution	Function / Purpose	Key Consideration for Preventing p-Hacking
Pre-registration Template (OSF/AsPredicted)	Documents hypothesis, methods, and analysis plan before data inspection.	Eliminates HARKing and reduces analysis flexibility.
Version Control (Git, DataLad)	Tracks every change to code, data, and analysis pipelines.	Enforces reproducibility and audit trails.
Containerization (Docker/Singularity, Neurodocker)	Packages complete software environment (OS, libraries, tools).	Ensures results are independent of local software configurations.
Workflow Management (Nipype, Snakemake, Nextflow)	Automates and documents multi-step analysis pipelines.	Prevents manual, unreported interventions at pipeline stages.
Nested CV in ML Library (scikit-learn)	Implements correct hyperparameter tuning and evaluation.	Use GridSearchCV with an inner CV object to prevent test set leakage.
Statistical Correctness Tools (Pingouin, statsmodels)	Performs appropriate corrections for multiple comparisons.	Applies FDR, Bonferroni, or permutation testing for mass-univariate analyses.
Blinding Scripts	Temporarily masks group labels (e.g., patient/control) during preprocessing.	Prevents subconscious bias during data cleaning and feature engineering.

Technical Support Center: Troubleshooting p-Hacking in Neuroimaging ML Analysis

FAQ & Troubleshooting Guides

Q1: My ML model shows excellent cross-validation accuracy (>95%) on my neuroimaging dataset, but fails completely on an external validation cohort. What could be the primary cause?

A: This is a classic symptom of p-hacking, specifically "double-dipping" or data leakage. The high accuracy likely results from non-independent feature selection and validation. Peeking at the test data during feature engineering or model selection inflates performance metrics.

Troubleshooting Protocol:

• Audit your workflow: Implement a strict nested cross-validation or hold-out validation protocol from the outset. All feature selection, hyperparameter tuning, and model selection must occur within each training fold.
• Check data splits: Ensure subject-level splitting, not scan-level, to prevent leakage from the same subject appearing in both training and test sets.
• Use a neutral test set: Set aside a completely untouched test cohort (20-30% of data) before any analysis. Report only performance on this final set.

Q2: I am comparing multiple feature engineering pipelines and ML algorithms. How do I report results without engaging in multiple comparisons bias?

A: Comparing many pipelines without correction increases the family-wise error rate, leading to false positives.

Corrective Protocol:

• Pre-register your analysis plan: Specify your primary pipeline, primary metric, and comparison method before analyzing the data.
• Correct for multiple comparisons: If exploring multiple pipelines is necessary, apply correction methods (e.g., Bonferroni, Holm-Bonferroni) to your p-values. Alternatively, use a global test statistic.
• Report all attempts: Transparently document the number of pipelines/algorithms tested, even those that failed.

Q3: How can I determine if a reported "significant" neuroimaging biomarker is robust, or a product of p-hacking?

A: Scrutinize the methodological rigor. Key red flags include lack of pre-registration, flexible analytical degrees of freedom, and absence of external validation.

Validation Checklist:

• Pre-registration: Is the hypothesis and analysis plan publicly timestamped before data analysis?
• External Validation: Is the biomarker validated on a completely independent dataset from a different site/scanner?
• Effect Size: Is the effect size (e.g., Cohen's d, AUC) reported and clinically meaningful, not just a "significant" p-value?
• Multiple Comparisons: Are corrections for mass univariate testing (e.g., in voxel-based analyses) clearly applied and reported?

---

Experimental Protocols for Rigorous Comparisons

Protocol 1: Nested Cross-Validation for Neuroimaging ML Purpose: To obtain an unbiased estimate of model performance when tuning hyperparameters and selecting features.

• Outer Loop (Performance Estimation): Split data into K1 folds (e.g., 5).
• Inner Loop (Model Selection): For each outer training set, perform a second K2-fold cross-validation (e.g., 5) to tune hyperparameters/select features.
• Train Final Model: Train a model with the best parameters on the entire outer training set.
• Test: Evaluate this model on the held-out outer test fold.
• Repeat: Iterate so each outer fold serves as the test set once. The final performance is the average across all outer test folds.

Protocol 2: Pre-registration of a Neuroimaging Biomarker Discovery Study

• Define Primary Outcome: Clearly state the clinical or cognitive outcome variable.
• Specify Imaging Features: Define the exact feature extraction method (e.g., atlas for ROI, kernel for SVM).
• Define Primary Analysis: Specify the primary ML model, validation scheme (e.g., nested CV with defined folds), and primary performance metric (e.g., balanced accuracy).
• Define Significance Threshold: State the alpha level and multiple comparison correction method.
• Submit: Register the protocol on a platform like OSF (osf.io) or ClinicalTrials.gov.

---

Table 1: Common p-Hacking Practices & Their Impact on Reported Performance

Practice	Description	Typical Inflation of AUC/Accuracy
Double-Dipping	Using the same data for feature selection and validation without correction.	10-25 percentage points
Optional Stopping	Collecting data until p < 0.05 is reached, without adjusting alpha.	Leads to 30-50% false positive rate (vs. 5%)
Outlier Removal	Selectively removing data points to achieve significance.	Unpredictable; can create spurious effects
HARKing	Formulating hypotheses after results are known.	Renders p-values uninterpretable

Table 2: Recommended Statistical Corrections for Neuroimaging ML

Analysis Type	Multiple Comparison Issue	Recommended Correction
Voxel-wise Mass Univariate	Testing 100,000s of voxels.	Family-Wise Error (FWE) rate, False Discovery Rate (FDR)
Multiple ROIs	Testing 50-100 pre-defined Regions of Interest.	Bonferroni, Holm-Bonferroni, FDR
Comparing >2 ML Pipelines	Testing multiple algorithms/feature sets.	Corrected paired t-test (e.g., Nadeau & Bengio), ANOVA with post-hoc correction

---

Visualizations

Diagram 1: Rigorous vs. P-Hacked ML Workflow

Diagram 2: Nested Cross-Validation Structure

---

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging ML Research

Item/Category	Function & Rationale
Pre-registration Platforms (OSF, AsPredicted)	To timestamp and fix the hypothesis, methods, and analysis plan before data analysis, preventing HARKing and flexible analysis.
Strict Version Control (Git, DVC)	To meticulously track every change in code, data, and parameters, ensuring full reproducibility of the analysis pipeline.
Nested CV Implementations (scikit-learn, NILearn)	Software libraries that facilitate correct implementation of nested cross-validation, preventing data leakage.
Multiple Comparison Correction Libraries (Statsmodels, FSL)	Tools to apply FDR, FWE, and other necessary corrections for mass testing in neuroimaging.
Standardized Data Formats (BIDS)	Using the Brain Imaging Data Structure organizes data consistently, reducing "flexibility" in preprocessing that can lead to p-hacking.
Reporting Checklists (TRIPOD-ML, CONSORT)	Guidelines that mandate complete reporting of the ML workflow, including failed models and all tuned parameters.

Troubleshooting Guides & FAQs

Q1: My model's cross-validation accuracy is very high on my dataset but fails completely on an independent cohort. What could be the root cause? A: This is a classic sign of data leakage or non-independent cross-validation. In neuroimaging ML, a common cause is applying feature selection or preprocessing steps (like normalization) to the entire dataset before splitting into training and validation folds. This allows information from the "validation" set to leak into the "training" process, artificially inflating performance. Always ensure your preprocessing pipeline is nested inside your cross-validation loop.

Q2: I am comparing two algorithms. How do I structure my cross-validation to ensure a fair comparison? A: You must use a nested cross-validation scheme. The inner loop is for model/hyperparameter selection for each algorithm. The outer loop provides an unbiased performance estimate for the entire model-building process for each algorithm. Using the same (non-nested) CV loop to both tune parameters and evaluate performance will produce optimistically biased estimates, and the bias can differ between algorithms, leading to unfair comparisons.

Q3: What is "double-dipping," and how can I avoid it in my analysis? A: Double-dipping occurs when the same data is used for both an exploratory hypothesis generation and the confirmatory statistical test of that hypothesis. For example, performing a whole-brain voxel-wise analysis to find a "significant" cluster, then using that same cluster's signal for a classification model and reporting its accuracy as confirmatory. To avoid it, use a completely independent dataset for the confirmatory test. If unavailable, perform exploratory analysis on one subset (e.g., half of controls) and confirmatory testing on a strictly held-out subset.

Q4: My p-value for model accuracy vs. chance is borderline (e.g., p=0.047). Are there checks I should perform before publication? A: Yes. Conduct a robustness or "sanity check" analysis. Perturb your analysis pipeline in reasonable ways (e.g., slightly different preprocessing parameters, different random seeds for non-deterministic algorithms) to see if the result remains significant. If the p-value fluctuates above and below 0.05 with minor changes, the finding is not robust. Report the range of outcomes from these sensitivity analyses.

Q5: How should I report negative results or failed replications to avoid the "file drawer" problem? A: Be transparent. Report all models and comparisons you attempted, not just the best-performing one. Clearly state your primary hypothesis and analysis plan a priori (consider pre-registration). When a replication fails, detail all methodological differences from the original study. Publishing in journals dedicated to null results or using preprint servers for all outputs helps combat publication bias.

Experimental Protocols from Cited Studies

Protocol 1: Nested Cross-Validation for Algorithm Comparison

• Partition the full dataset into K outer folds (e.g., 5 or 10).
• For each outer fold: a. Designate the fold as the temporary test set; the remaining K-1 folds are the development set. b. On the development set, perform another J-fold cross-validation (inner loop) on each algorithm to optimize its hyperparameters. c. Train each algorithm with its optimal hyperparameters on the entire development set. d. Apply the trained model to the held-out outer test fold. Store the prediction.
• After looping through all outer folds, compile all predictions. Calculate the final performance metric (e.g., accuracy, AUC) from these pooled predictions. This metric is an unbiased estimate for each algorithm's pipeline.

Protocol 2: Voxel-Based Morphometry (VBM) Analysis with Cluster Correction

• Preprocess all T1-weighted MRI scans: spatial normalization, segmentation, modulation, smoothing.
• Perform a whole-brain voxel-wise statistical test (e.g., two-sample t-test) comparing groups.
• Apply a cluster-forming threshold (e.g., p < 0.001 uncorrected) to the statistical map.
• Estimate the smoothness of the residual data. Use Random Field Theory or permutation testing to determine the critical cluster-size threshold for family-wise error (FWE) correction at p < 0.05.
• Report only clusters that survive this corrected threshold. Crucially: Do not extract mean values from these surviving clusters for a secondary, uncorrected analysis on the same data.

Table 1: Impact of Analysis Choices on Reported Classification Accuracy in Simulated Data

Analysis Scenario	True Accuracy	Mean Reported Accuracy	Inflation (%)	Common In Field?
Correct Nested CV	70.0%	70.2% (±1.5)	0.3	Less common
Non-Nested CV (Leakage)	70.0%	78.5% (±2.1)	12.1	Common
Feature Selection on Full Set	70.0%	82.1% (±3.0)	17.3	Very common
Circular Analysis (Double-Dip)	70.0%	95.0% (±4.5)	35.7	Occasional

Table 2: Results of Replication Attempts for Landmark Neuroimaging ML Studies

Original Study (Claim)	Replication Study	Original Performance (AUC)	Replication Performance (AUC)	Key Methodological Difference Found
Study A (Diagnosis X)	Smith et al., 2023	0.92	0.65	Original used site-specific scanner correction; replication used harmonization.
Study B (Prognosis Y)	Jones et al., 2024	0.87	0.71	Original performed feature selection pre-split; replication used nested CV.

Visualizations

Title: Nested Cross-Validation Workflow for Unbiased Evaluation

Title: The Double-Dipping Pitfall in Neuroimaging Analysis

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging Machine Learning

Item	Function & Rationale
Nilearn (Python)	Provides high-level functions for neuroimaging data analysis and machine learning, including ready-to-use NestedCrossValidator.
COINSTAC	A decentralized platform for collaborative analysis without sharing raw data, facilitating independent replication on external datasets.
fMRIPrep	A robust, standardized preprocessing pipeline for fMRI data, reducing variability in results due to preprocessing choices.
PRONTO (Python)	A toolbox specifically designed for transparent and reproducible pattern analysis for neuroimaging, emphasizing correct CV.
NeuroVault	A public repository for unthresholded statistical maps, allowing others to inspect whole-brain results and attempt re-analysis.
PyMVPA	A Python package for multivariate pattern analysis that includes careful data partitioning schemes to avoid data leakage.

Building Bulletproof Pipelines: Methodological Safeguards Against p-Hacking

Technical Support Center: Troubleshooting Pre-registration & Analysis in Neuroimaging ML

FAQs and Troubleshooting Guides

Q1: I have uploaded my pre-registration document to a public repository like OSF or AsPredicted, but I need to correct a minor typographical error in my hypothesis statement. Is this allowed, and what is the proper procedure?

A: Yes, minor corrections are typically allowed, but transparency is critical. You must create a new version of the pre-registration document, clearly labeled (e.g., "V2"). The changes must be explicitly documented in a "Change Log" or "Correction" section within the new version, explaining the reason for the change (e.g., "corrected typo in H1 wording, no substantive change to the hypothesis"). The original version must remain accessible. Substantive changes to hypotheses or analysis plans after seeing the data are strongly discouraged and must be flagged as data-driven, post-hoc decisions in any subsequent publication.

Q2: My pre-registered machine learning pipeline specified a linear SVM, but initial exploration suggests a non-linear kernel might perform better. Can I switch?

A: This is a high-risk scenario for p-hacking. You cannot simply switch based on performance. Adhere to your pre-registered plan for the primary confirmatory analysis. You may explore the non-linear kernel in a separate, explicitly labeled exploratory analysis. In your manuscript, you must clearly distinguish between the pre-registered confirmatory test (which protects against false positives) and any exploratory, data-driven follow-ups (which generate hypotheses for future research). Failing to do so invalidates the purpose of pre-registration.

Q3: How detailed should my neuroimaging preprocessing and feature extraction steps be in the analysis plan?

A: Extremely detailed. Ambiguity here is a major source of the "researcher degrees of freedom" that lead to p-hacking. Your plan should specify, at minimum: software and version (e.g., FSL 6.0.7), spatial normalization template (e.g., MNI152), smoothing kernel FWHM (e.g., 6mm), motion correction thresholds, artifact removal strategies (e.g., ICA-AROMA), brain mask, and feature type (e.g., ROI mean timeseries, voxel-wise maps). Use tools like fMRIPrep to ensure reproducible workflows. Provide a version-controlled script (e.g., on GitHub) that encodes these decisions.

Q4: My pre-registered analysis plan called for a specific atlas (AAL2), but a newer, more granular atlas has been published. Can I use the new one for my main analysis?

A: No. Changing a core methodological component like a brain atlas based on external developments after the study has begun introduces a flexible choice. The pre-registered atlas must be used for the primary analysis. You can analyze the data with the new atlas as a secondary or sensitivity analysis. This demonstrates robustness (or lack thereof) of your findings to methodological choices.

Q5: I pre-registered a cross-validated accuracy comparison between two models. One model achieves 70% accuracy, the other 72%. The p-value from my pre-registered statistical test is 0.06. Can I try different statistical tests or outlier removal to see if it becomes "significant"?

A: Absolutely not. This is the definition of p-hacking. The result of your pre-registered test on the pre-processed data as defined in your plan is your result. P=0.06 is your result. Changing the statistical model or data inclusion criteria based on the outcome invalidates the statistical inference. You must report the result as non-significant according to your pre-defined alpha (e.g., 0.05). You may report the findings and note they approach significance, but any additional, unplanned tests must be explicitly labeled as exploratory.

---

Key Experimental Protocols in Neuroimaging ML Comparisons

Protocol 1: Pre-registered, Locked-Down Analysis Pipeline for Classifier Comparison Objective: To fairly compare the performance of two machine learning classifiers (e.g., SVM vs. Logistic Regression) on neuroimaging data while eliminating researcher degrees of freedom.

• Pre-registration: Publicly document the exact dataset (including inclusion/exclusion criteria), preprocessing pipeline, feature extraction method, classification algorithms with hyperparameter search spaces, cross-validation scheme (folds, splits, stratification), and primary performance metric (e.g., balanced accuracy).
• Code Implementation: Write a fully automated script (Python/R) that encodes every step from raw data to final metric. Containerize using Docker or Singularity.
• Blinded Analysis: If possible, run the final, locked script on the pre-processed data in a single batch. The output is the definitive result.
• Reporting: Report all pre-registered outcomes, even if null or negative. Clearly separate any post-hoc explorations.

Protocol 2: Nested Cross-Validation for Unbiased Hyperparameter Tuning & Performance Estimation Objective: To obtain an unbiased estimate of model generalizability when both model selection and evaluation are required.

• Outer Loop: Split data into K1 folds (e.g., 5).
• Inner Loop: For each outer training set, perform a second, independent cross-validation (K2 folds) to select the best hyperparameters via grid/random search.
• Training: Train a model on the entire outer training set using the optimal hyperparameters found in its inner loop.
• Testing: Evaluate this single model on the held-out outer test fold.
• Aggregation: Repeat for all K1 outer folds. The average performance across the outer test folds is the final, unbiased performance estimate. This entire workflow must be pre-specified.

---

Data Presentation

Table 1: Common Pre-registration Platforms & Their Features

Platform	Primary Use Case	Version Control	Embargo Options	Integration with Data Repos	Cost
Open Science Framework (OSF)	Comprehensive project management, from pre-reg to publication.	Yes, full project history.	Yes, can blind until publication.	Excellent (GitHub, Dataverse, etc.).	Free.
AsPredicted	Simple, streamlined pre-registration of hypotheses & analysis.	Yes, but as new numbered versions.	Yes, standard.	Limited.	Free.
ClinicalTrials.gov	Mandatory for clinical trials; can be used for interventional neuroimaging.	Yes.	Can delay results posting.	Limited.	Free.
GitHub	Code-centric pre-registration via a timestamped repository/README.	Native Git version control.	No, public by default.	Native.	Free.

Table 2: Impact of Pre-registration on Reported Results in Meta-Analyses

Study (Field)	Pre-registered Studies	Non-Pre-registered Studies	Key Finding
Kaplan & Irlam (2017), Social Psychology	Median effect size: r = 0.21	Median effect size: r = 0.39	Effect sizes in pre-registered studies were approximately 50% smaller.
Scheel et al. (2021), Psychology	44% yielded significant support for the tested hypothesis.	96% yielded significant support for the tested hypothesis.	Pre-registration dramatically reduces the rate of positive findings, suggesting publication bias & p-hacking in non-pre-registered work.
Estimated in Neuroimaging ML	Likely lower, more variable performance metrics; more null results.	Likely inflated, optimistic accuracy/ AUC estimates due to selective reporting.	Pre-registration is expected to provide a more realistic picture of model utility.

---

Visualizations

Diagram 1: Pre-registration Workflow for Neuroimaging ML

Diagram 2: Nested CV vs. Standard CV Risk of Bias

---

The Scientist's Toolkit: Research Reagent Solutions

Item	Function in Pre-registered Neuroimaging ML Research
Open Science Framework (OSF)	A free, comprehensive platform to create time-stamped, public pre-registrations, manage project components, and link to data/code.
fMRIPrep / qsiprep	Standardized, robust preprocessing pipelines for fMRI/dMRI data. Using them in your plan enhances reproducibility and reduces preprocessing flexibility.
Docker / Singularity Containers	Containerization technology to package your entire analysis environment (OS, software, libraries), ensuring the exact same code can be run by anyone, anywhere.
Version Control (Git/GitHub/GitLab)	Essential for maintaining a history of changes to your analysis code. The commit hash from the time of analysis can be frozen as part of the permanent record.
Pre-registration Templates	Templates from organizations like the Psychological Science Accelerator provide structured guidance on what details to specify.
NiMARE / NeuroSynth	Tools for formal, pre-specified meta-analysis of neuroimaging coordinates, which can be used to define unbiased ROI masks for hypothesis testing.
scikit-learn / Nilearn	Python libraries with consistent APIs for machine learning. Pre-specifying the function names and arguments in your plan locks down the implementation.

Technical Support Center

Troubleshooting Guides & FAQs

Q1: My model performs excellently during cross-validation but fails dramatically on the hold-out test set. What went wrong? A: This is a classic sign of data leakage or an improperly structured data split. The cross-validation score is optimistically biased. Verify that all preprocessing steps (e.g., feature scaling, imputation) are calculated only on the training fold within the cross-validation loop and then applied to the validation fold. Never fit preprocessing on the entire dataset before splitting.

Q2: How do I choose between k-fold cross-validation and a simple hold-out validation set? A: Use the table below to guide your decision based on dataset size. Nested cross-validation is the gold standard for unbiased performance estimation when tuning hyperparameters.

Method	Recommended Sample Size	Primary Use Case	Risk of p-hacking
Hold-Out Validation	>20,000 samples	Initial, quick model prototyping	High (single split susceptible to random variation)
k-Fold Cross-Validation	1,000 - 20,000 samples	Model evaluation with stable variance	Moderate (requires careful pipeline design)
Nested Cross-Validation	100 - 10,000 samples	Unbiased performance estimation with hyperparameter tuning	Low
Leave-One-Out (LOOCV)	< 1,000 samples	Extremely small datasets	High computational cost, high variance

Q3: I am using neuroimaging data (e.g., fMRI voxels). How should I split the data to avoid leakage from the same subject? A: You must split data at the subject level. All data from a single participant must reside in only one of the Training, Validation, or Test sets. Splitting individual scans from the same subject across sets creates leakage and inflated, non-generalizable performance.

Q4: What is the concrete protocol for implementing Nested Cross-Validation? A: Follow this detailed protocol:

• Outer Loop (Performance Estimation): Split your entire dataset into k1 folds (e.g., 5). For each fold i: a. Designate fold i as the Temporary Test Set. The remaining k1-1 folds are the Development Set. b. Inner Loop (Model Selection): On the Development Set, perform a second, independent k2-fold cross-validation (e.g., 5-fold). For each combination of hyperparameters: - Train on k2-1 folds, validate on the held-out fold. - Calculate the average validation score across all k2 inner folds. c. Select the hyperparameter set with the best average inner validation score. d. Retrain the model with these optimal hyperparameters on the entire Development Set. e. Evaluate this final model once on the Temporary Test Set (fold i) to get an unbiased performance score.
• Final Model: The k1 unbiased scores are averaged for the final performance estimate. To obtain a deployable model, repeat the model selection process (inner loop) on the entire dataset to choose final hyperparameters, then train a final model on all data. The performance estimate comes from the nested CV process, not this final model.

Q5: Why is a completely independent, locked hold-out test set still necessary? A: Nested CV provides an unbiased estimate of how your modeling process will perform. However, after you finalize your process and train your final model on all available data, you need a final, realistic assessment. A hold-out test set, locked away from any analysis until the very end, simulates the model's performance on truly new, unseen data from the same distribution. This is critical for reporting results in publications to avoid overfitting the entire available dataset.

Q6: My performance metric (e.g., accuracy) fluctuates wildly between different random splits. How can I report stable results? A: This indicates high variance. Use repeated nested cross-validation (e.g., 5x5-CV) with different random partitions. Report both the mean and standard deviation (or confidence interval) of the performance metric across all outer test folds. This provides a more robust and reliable estimate. See table below from a simulated neuroimaging study on classifier comparison:

Classifier	Mean Accuracy (5x5 Nested CV)	Std. Deviation	p-value (vs. Baseline) Corrected
Linear SVM	68.5%	± 3.2%	- (Baseline)
RBF-kernel SVM	72.1%	± 5.8%	0.15
Random Forest	70.3%	± 2.1%	0.04

Note: p-values corrected via permutation testing within the nested CV framework.

The Scientist's Toolkit: Research Reagent Solutions

Item	Function in Rigorous ML Pipeline
Scikit-learn Pipeline	Encapsulates preprocessing and model steps to prevent data leakage during cross-validation.
Scikit-learn GridSearchCV & RandomizedSearchCV	Automates hyperparameter search within a defined inner cross-validation loop.
Custom Group/Subject Splitter (e.g., GroupKFold, LeaveOneGroupOut)	Ensures data from the same participant/scanner/site are not split across training and test sets.
MLxtend nested_cross_val_score	A library function that can help implement the nested CV structure.
NumPy / Pandas with fixed random seeds	Enforces reproducibility in data shuffling and splitting operations.
Permutation Testing Scripts	Non-parametric method for calculating statistically significant performance differences, correcting for multiple comparisons.

Workflow Diagrams

Nested Cross-Validation Workflow

Hold-Out Test Set Protocol for Final Reporting

Technical Support Center

Troubleshooting Guides & FAQs

Q1: My cross-validated model performs exceptionally well on the training folds but fails on the held-out test set. What is the most likely cause of this performance drop?

A: This is a classic symptom of data leakage during feature selection. If feature selection is performed before splitting data into training and validation folds, or on the entire dataset, information from the "future" (test set) leaks into the training process. This inflates performance estimates. The correct protocol is to perform feature selection independently within each cross-validation fold, using only the training portion of that fold.

Q2: How can I verify if data leakage has occurred in my published neuroimaging ML pipeline?

A: Conduct a dummy variable test. Introduce a random, non-informative feature (e.g., Gaussian noise) into your dataset. Re-run your entire pipeline, including feature selection. If this random feature is consistently selected as important across multiple runs or folds, it strongly indicates that your feature selection method is overfitting to noise, often due to leakage or an improperly nested design.

Q3: When using filter methods (like ANOVA F-value) for feature selection in a nested cross-validation setup, where should the filter be applied?

A: The filter must be applied inside the outer cross-validation loop, and separately for each inner loop. The process is:

• Split data into Outer Train and Outer Test sets.
• On the Outer Train set, run an inner CV. For each inner fold, apply the filter method only to the inner training partition to select features, then train and validate the model on the selected features.
• Choose the best hyperparameters/feature count from the inner CV.
• Re-train the final model on the entire Outer Train set using the winning parameters, perform feature selection on this set, and evaluate once on the untouched Outer Test set.

Q4: What is a "nested cross-validation" and why is it mandatory for unbiased performance estimation in high-dimensional neuroimaging studies?

A: Nested cross-validation uses two layers of loops. The outer loop estimates the generalization error, while the inner loop selects the model (including feature selection parameters, e.g., k for top-k features). This strict separation ensures that the test data in the outer fold is never used for any decision (feature selection, model selection, parameter tuning), providing an almost unbiased estimate of true performance and preventing p-hacking via model optimization on the test set.

Q5: I am comparing two feature selection algorithms. What is the correct statistical approach to avoid p-hacking in this comparison?

A: You must pre-register your analysis plan. The key is to perform the comparison on a completely held-out validation set, defined before any analysis begins. The workflow is:

• Split data into Development, Validation, and (optional) Test sets. Lock away the Validation set.
• Using only the Development set, perform nested CV to optimize both pipelines independently.
• Train final models on the full Development set using the optimal pipelines.
• Evaluate both models once on the locked Validation set.
• Compare performance metrics using a pre-specified, appropriate statistical test (e.g., corrected paired t-test, permutation test). Do not try other tests or revisit the data after this single comparison.

Experimental Protocols & Data

Protocol 1: Nested Cross-Validation for Leakage-Free Evaluation

• Define Outer Loop: Split dataset D into k folds (e.g., k=5 or 10).
• For each Outer Fold i: a. Outer Test Set = fold i. Outer Train Set = all other folds. b. Define Inner Loop: Split Outer Train Set into j folds. c. For each Inner Fold j: i. Inner Validation Set = fold j of Outer Train. ii. Inner Train Set = remainder of Outer Train. iii. Perform feature selection only on Inner Train Set. iv. Train model on Inner Train Set (with selected features). v. Evaluate on Inner Validation Set. d. Determine optimal feature selection parameters/model based on average inner CV performance. e. Apply optimal parameters to perform feature selection on the entire Outer Train Set. f. Train final model on the entire Outer Train Set (with selected features). g. Evaluate the final model once on the Outer Test Set (fold i).
• Report Performance: Aggregate results (e.g., mean accuracy, AUC) across all k Outer Test evaluations.

Protocol 2: Permutation Test for Significance of Feature Selection

• Using a leakage-free pipeline, obtain the true model performance metric (e.g., accuracy) on the held-out test set.
• Randomly permute the class labels of the training data (not the test data).
• Re-run the entire model training and feature selection pipeline (with nested CV on the permuted training data) and evaluate on the same, original test set. Record the performance.
• Repeat steps 2-3 many times (e.g., 1000x) to build a null distribution of performance metrics under the hypothesis of no real class-effect.
• Calculate the p-value as the proportion of permutations where the performance met or exceeded the true performance from step 1.

Table 1: Impact of Data Leakage on Model Performance (Simulated fMRI Data)

Scenario	Reported CV Accuracy (Mean ± Std)	True Hold-Out Test Accuracy	Features Selected (Avg)	Random Feature Selected (%)
Leaky Pipeline (Global FS)	92.4% ± 2.1	64.8%	150	87%
Correct Pipeline (Nested FS)	71.5% ± 5.3	69.2%	22	4.5%
Baseline (No FS, All Features)	68.1% ± 6.0	65.0%	10,000 (all)	N/A

Table 2: Recommended Statistical Tests for Comparison (Pre-registered)

Comparison Type	Recommended Test	Purpose
Pipeline A vs. Pipeline B (same test set)	Corrected Repeated k-fold CV t-test*	Compare two models with dependent samples.
Pipeline vs. Random Chance	Permutation Test (Label Shuffling)	Establish if pipeline performance is above chance.
Feature Set A vs. Feature Set B Stability	Jaccard Index / Dice Coefficient	Measure consistency of selected features across subsamples or folds.

*Dietterich, T.G. (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms.

Visualizations

Correct Nested CV Workflow

Incorrect Leaky Pipeline

The Scientist's Toolkit: Research Reagent Solutions

Item / Solution	Function in Experiment	Example / Specification
Nilearn / scikit-learn	Python libraries providing implemented nested CV, feature selection, and permutation test classes, ensuring correct pipeline structure.	sklearn.model_selection.NestedCV, sklearn.feature_selection.SelectKBest, sklearn.utils.validation.check_cv
Permutation Test Script	Custom code to shuffle training labels and generate a null distribution for statistical testing, preventing p-hacking.	1000+ iterations, preserving test set integrity.
Pre-registration Template	Document outlining hypothesis, dataset splits, feature selection method, model, and comparison test before analysis.	OSF preregistration format.
Data Splitter (Stratified)	Function to create Development, Validation, and Test sets while preserving class balance and preventing information leak.	sklearn.model_selection.train_test_split with stratify parameter.
Feature Stability Analyzer	Tool to compute Jaccard Index across CV folds to assess if selected features are robust or noise-driven.	Custom function calculating J = ⎮A ∩ B⎮ / ⎮A ∪ B⎮ for feature sets A and B.
High-Performance Computing (HPC) Cluster Access	Enables running computationally intensive nested CV and permutation tests (1000s of iterations) in a feasible timeframe.	SLURM job array for parallelizing outer CV folds and permutation iterations.

Troubleshooting Guides & FAQs

Q1: My model's test set performance drops dramatically compared to the validation set performance. What is likely happening? A: This is a classic sign of validation set leakage or hyperparameter overfitting. You have likely used the test set, either directly or indirectly, to guide your hyperparameter tuning or model selection process. The validation set is no longer providing an unbiased estimate of generalization error. To fix this, ensure you have a strict separation: a training set for model fitting, a validation set only for hyperparameter tuning, and a test set used exactly once for a final, unbiased evaluation. Never iterate on your model based on test set results.

Q2: How do I correctly split my neuroimaging dataset when sample size is limited? A: For small N neuroimaging studies, simple hold-out validation is often unstable. Use nested cross-validation:

• Outer Loop: For estimating generalized performance (simulates the test set).
• Inner Loop: For hyperparameter tuning (simulates the validation set). This protocol preserves the test set principle within the CV framework. See the Experimental Protocol section below for details.

Q3: What is the difference between a validation set and a test set in the context of preventing p-hacking? A: In the context of p-hacking, the key distinction is purpose and frequency of use.

• Validation Set: Used repeatedly to compare different models/algorithms/parameters. Tuning based on this set inherently introduces an "optimization bias."
• Test Set: Used once to report the final performance of the single, fully-specified model chosen after validation. Using it more than once invalidates the statistical inference, leading to inflated, non-replicable results (a form of p-hacking).

Q4: How can I track my hyperparameter tuning process to ensure reproducibility and avoid inadvertent hacking? A: Maintain a detailed experiment log. For every run, record:

• The exact data splits (subject IDs for train/validation/test).
• All hyperparameters tried and their values.
• The performance metric on the validation and training sets.
• The final chosen hyperparameter set. Using tools like MLflow or Weights & Biases can automate this tracking and ensure the integrity of your validation process.

Experimental Protocols

Protocol: Nested Cross-Validation for Hyperparameter Optimization Objective: To obtain an unbiased estimate of model performance while tuning hyperparameters on limited neuroimaging data.

• Define Outer K-Folds: Split the entire dataset into K non-overlapping folds (e.g., K=5 or K=10). For neuroimaging, ensure subject-level splitting to prevent data leakage.
• Iterate Outer Loop: For each outer fold i: a. Set aside fold i as the outer test set. b. The remaining K-1 folds constitute the development set.
• Inner Hyperparameter Tuning: a. Split the development set into L inner folds (e.g., L=5). b. For each candidate hyperparameter set, train L models on L-1 inner folds and validate on the held-out inner fold. c. Calculate the average validation score across all L inner folds for that hyperparameter set. d. Select the hyperparameter set with the best average inner validation score.
• Train & Evaluate Outer Model: Using the selected optimal hyperparameters, train a model on the entire development set. Evaluate it on the outer test set (fold i). Record this test score.
• Final Performance: After iterating through all K outer folds, calculate the mean and standard deviation of the K recorded outer test scores. This is the unbiased performance estimate. The final model for deployment is trained on all data using the hyperparameters chosen from a final, complete inner CV loop.

Data Presentation

Table 1: Comparison of Validation Strategies and Their Vulnerability to Optimistic Bias

Validation Strategy	Hyperparameter Tuning Method	Estimated Bias	Computational Cost	Recommended for Neuroimaging?
Simple Hold-Out	Manual / Grid Search on Validation Set	High	Low	No (unless N > 10,000)
Single (Standard) Cross-Validation	Grid Search within same CV folds	Moderate-High	Medium	No (high risk of overfit)
Nested Cross-Validation	Grid/Random Search in inner CV loop	Low	High	Yes (gold standard)
Bootstrap	.632 Bootstrap for tuning & estimation	Low	Very High	Yes (for very small N)

Table 2: Impact of Validation Set Misuse on Reported Classification Accuracy in an fMRI Study (Simulated Data)

Analysis Scenario	Mean Reported Test Accuracy (%)	Standard Deviation	True Generalization Estimate (%)	Inflated by (pp)
Correct Nested CV Protocol	68.5	2.1	68.5	0.0
Tuned on Test Set (Direct Leakage)	82.3	1.5	~68.5	+13.8
Model Selected Based on Test Performance	75.1	2.8	~68.5	+6.6
Repeated Testing on Single Test Set	74.0	N/A	~68.5	+5.5

Diagrams

Nested CV Workflow

Data Leakage Pathways to Avoid

The Scientist's Toolkit

Table 3: Essential Research Reagent Solutions for Robust Hyperparameter Optimization

Item / Solution	Function / Purpose	Example in Neuroimaging ML
Nested Cross-Validation Scripts	Automates the complex splitting and iteration to prevent data leakage.	scikit-learn GridSearchCV with custom outer CV loop; nilearn decoding utilities.
Experiment Tracking Platform	Logs all hyperparameters, code versions, data splits, and results for full reproducibility.	Weights & Biases (W&B), MLflow, TensorBoard.
Containerization Software	Ensures the computational environment (library versions, OS) is identical across runs and labs.	Docker, Singularity (crucial for HPC neuroimaging).
Version Control System	Tracks changes to analysis code, preventing "silent" changes that invalidate prior results.	Git with platforms like GitHub or GitLab.
Pre-registration Template	Documents the analysis plan, including validation scheme, before observing test results.	OSF preregistration, AsPredicted.org.
Statistical Power / Bias Estimator	Tools to simulate and estimate the optimization bias introduced by hyperparameter tuning.	Custom scripts using nested CV on null data; DoubleML library concepts.

Implementing Model Fairness and Stability Checks as Early Warning Signals

Troubleshooting Guides & FAQs

Q1: After implementing a fairness constraint, my model's overall performance (AUC) drops significantly on the validation set. What could be wrong? A: A sharp performance drop often indicates an overly restrictive fairness regularizer or a mis-specified fairness metric. First, verify that your protected variable (e.g., sex, site) is correctly encoded and that the chosen fairness definition (e.g., demographic parity, equalized odds) is appropriate for your neuroimaging context. Gradually increase the regularization strength (lambda) from 0 to observe the trade-off curve. A sudden cliff suggests you may be optimizing for a metric that conflicts fundamentally with accuracy given your data distribution. Check for confounding between the protected variable and the true signal.

Q2: My model's fairness metrics are good on cross-validation folds but degrade severely on the held-out test set. How should I debug this? A: This is a classic sign of data leakage or non-independent and identically distributed (non-IID) splits, common in multi-site neuroimaging studies. Your CV splits likely contain data from the same sites/scanners in both train and validation folds, while the test set is from a completely different site. Implement site-wise or scanner-wise cross-validation, where all data from a particular site is held out together. Re-train using this method and re-evaluate fairness. Stability across these "domain-shift" CV folds is a stronger early warning signal.

Q3: When I run stability checks by retraining on bootstrapped samples, the feature importance (e.g., brain regions) changes wildly. Is my model invalid? A: High volatility in feature importance is a critical early warning of model instability, often linked to high-dimensionality (many voxels/ROIs) and correlated features. It suggests the model is latching onto noise. Before abandoning the approach, try: 1) Increasing regularization (e.g., L1/L2 penalties). 2) Using anatomically-defined regions of interest (ROIs) instead of individual voxels to reduce dimensionality. 3) Applying stability selection with a defined threshold (e.g., a feature must be selected in >80% of bootstraps to be considered stable). This directly addresses "phacking" via selective feature reporting.

Q4: I suspect p-hacking in my comparison of two algorithms. How can fairness and stability checks serve as a robustness audit? A: Instead of reporting only the single "best" accuracy/p-value from multiple comparative trials, pre-register your fairness and stability checks as mandatory diagnostics. For example, mandate that any claimed "superior" algorithm must also demonstrate non-inferior fairness across protected groups and comparable feature stability across bootstrap resamples. This creates a higher burden of proof. If Algorithm A beats B on accuracy but shows significantly worse site-bias stability, its utility is questionable. Report these results in a consolidated table.

Q5: How do I operationalize these checks as "early warning" signals during model development, not just post-hoc? A: Integrate them into your training pipeline as automated checkpoints. For example:

• Checkpoint 1 (Data Split): After creating train/validation/test splits, run a basic disparity report on label distribution per protected group.
• Checkpoint 2 (During Training): Monitor fairness-performance trade-off on the validation set in real-time.
• Checkpoint 3 (Post-Training): Run bootstrap stability analysis on feature weights. Set thresholds (e.g., "fairness disparity < 5%", "feature stability > 75%") that trigger a review or halt the pipeline before proceeding to final testing.

Experimental Protocols & Data

Protocol 1: Cross-Site Stability Validation This protocol assesses model robustness to scanner/site variation, a major fairness concern in neuroimaging.

• Data Partitioning: Partition data by acquisition site. For K sites, create K folds.
• Training: For each fold i, train your model on data from all sites except site i.
• Validation & Testing: Tune hyperparameters on a held-out portion of the training sites. Apply the final model to the completely unseen site i.
• Metric Calculation: Calculate primary accuracy and fairness metrics (e.g., equal opportunity difference) for predictions on each left-out site i.
• Analysis: Report the mean and standard deviation across sites. A high standard deviation in accuracy or fairness metrics is an early warning of poor generalization.

Protocol 2: Bootstrap Feature Stability Analysis This protocol quantifies the reliability of identified brain features, countering feature p-hacking.

• Resampling: Generate B (e.g., 100) bootstrap resamples of your training data.
• Model Training: Train an interpretable model (e.g., linear SVM, logistic regression with L1) on each resample.
• Feature Extraction: For each trained model, extract the top N most important features (e.g., weights with largest absolute values).
• Stability Calculation: Compute the selection frequency for each original feature across the B models. Frequency = (Number of times feature is in top N) / B.
• Thresholding: Apply a stability threshold (e.g., >0.8). Only features above this threshold are reported as "stable".

Quantitative Data Summary: Hypothetical Fairness-Stability Audit Results

Table 1: Comparison of Two Classification Algorithms on Multi-Site fMRI Data

Metric	Algorithm A (Mean ± SD)	Algorithm B (Mean ± SD)	Acceptable Threshold
Accuracy (AUC)	0.85 ± 0.03	0.82 ± 0.02	>0.75
Equal Opp. Diff. (by Sex)	0.08 ± 0.05	0.03 ± 0.02	<0.05
Site AUC Std. Dev.	0.07	0.04	<0.05
Top 10 Feature Stability	0.65	0.88	>0.80

Interpretation: Algorithm A has higher average accuracy but fails fairness (Equal Opportunity Difference > threshold) and stability checks. Algorithm B, while slightly less accurate, is more fair and stable, making it a more robust and reliable choice, mitigating p-hacking risks.

Visualizations

Title: Early Warning Signal Pipeline for Model Auditing

Title: How Checks Mitigate p-Hacking Risks in Neuroimaging ML

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Tools for Implementing Fairness & Stability Checks

Tool/Reagent	Function in Experiments
aif360 (IBM AI Fairness 360)	Open-source Python toolkit containing a wide array of pre-implemented fairness metrics, algorithms for bias mitigation, and explanatory metrics. Essential for standardizing fairness audits.
scikit-learn & sklearn-resample	Provides core model training, cross-validation, and bootstrap resampling utilities. The Resample class from sklearn-resample is key for stability analysis.
nimare (Neuroimaging Meta-Analysis Research Environment)	A Python toolkit for neuroimaging analysis that includes algorithms for meta-analysis, decoding, and data extraction. Useful for handling coordinate/ROI-based feature stability.
DALEX & fairmodels (R packages)	Explainable AI (XAI) suites for model exploration and explanation. fairmodels is specifically designed for fairness validation and comparison of multiple models.
PREDICT (PRospective Evaluation of Diagnostic Information with Comparative Trials) Checklist	A proposed pre-registration template for diagnostic AI studies. Guides researchers in pre-specifying primary outcomes, fairness criteria, and stability checks to prevent p-hacking.
Nilearn	A Python library for fast and easy statistical learning on neuroimaging data. Provides tools for masking, feature extraction from brain regions, and decoding (classification/regression). Critical for building the neuroimaging ML pipeline itself.

Diagnosing and Fixing p-Hacking in Existing Neuroimaging ML Analyses

Technical Support Center: Troubleshooting & FAQs

FAQ 1: I have many potential features/voxels in my neuroimaging data. How do I decide which to include in my model without 'fishing' for a good result?

• Answer: This is a classic multiple comparisons problem. The key is to pre-register your feature selection protocol.
  • Protocol: Split your data into a Discovery Set and a Hold-Out Validation Set (e.g., 70/30). Perform all feature selection (e.g., voxel-wise ANOVA, feature importance ranking) only on the Discovery Set. Lock the selected feature set. Then, train your final model on the Discovery Set using only these features and evaluate its performance once on the Hold-Out Validation Set. Any further tuning requires a new, independent dataset.

FAQ 2: My primary model didn't reach significance (p > 0.05). Is it acceptable to try different preprocessing pipelines or outlier removal methods to see if results improve?

• Answer: This is a high-risk p-hacking behavior if done without correction. Each pipeline or outlier rule tested is a distinct analysis.
  • Protocol: If exploratory pipeline searching is necessary, it must be done within a cross-validation framework on the training data only. For example, use nested cross-validation where the inner loop selects the best pipeline, and the outer loop provides an unbiased performance estimate. Report all pipelines tested. The optimal approach is to define a single, justified pipeline in your pre-registration.

FAQ 3: I am comparing several machine learning algorithms. Is it okay to report only the one with the best p-value?

• Answer: No. Selectively reporting only the "best" or "significant" algorithm inflates Type I error.
  • Protocol: Pre-specify the algorithms for comparison. Report the performance metrics (accuracy, AUC, etc.) and statistical significance for all algorithms tested on the same validation set. Use correction for multiple comparisons (e.g., Bonferroni, Holm-Bonferroni) when formally testing hypotheses across algorithms.

FAQ 4: How should I handle unexpected but interesting subgroup findings that emerge after I see the results?

• Answer: Post-hoc subgroup analyses are exploratory by nature and must be labeled as such. They generate hypotheses for future validation, not confirmatory conclusions.
  • Protocol: Clearly state the analysis was post-hoc. Use an independent cohort to attempt to replicate the subgroup effect. If no independent data exists, use rigorous internal validation (e.g., bootstrapping) and apply multiplicity corrections. The effect size from an exploratory analysis should not be reported without this context.

Data Presentation

Table 1: Common p-Hacking Practices and Mitigation Strategies in Neuroimaging ML

Practice	Red Flag	Mitigation Strategy
Flexible Analysis	Trying multiple preprocessing steps, outlier rules, or statistical models until a significant result is found.	Pre-registration of a single, justified analysis pipeline.
Selective Reporting	Reporting only significant models/features/regions of interest while omitting others tested.	Report all analyses conducted. Use results-neutral wording in pre-registration.
Fishing for Covariates	Adding, removing, or transforming covariates to achieve a desired p-value.	Pre-specify covariates based on theoretical justification. Report sensitivity analyses.
Outcome Switching	Changing the primary outcome measure after data analysis has begun.	Pre-register primary and secondary outcomes. Clearly label any post-hoc analyses.
Failing to Correct	Not adjusting for multiple comparisons when conducting many statistical tests (e.g., across voxels).	Use family-wise error (FWE) or false discovery rate (FDR) correction. Use hold-out validation for ML.

Table 2: Impact of p-Hacking on False Positive Rate (Simulation Data)

Scenario	Number of Analyst Degrees of Freedom	Estimated False Positive Rate
Preregistered, single test	1	5% (Nominal α)
Testing two outcome measures	2	~10%
Trying two analytic pipelines	2	~10%
"Fishing" across 10 subgroups	10	~40%
Combining multiple flexibilities	High	>50%

Experimental Protocols

Protocol 1: Pre-registration for a Neuroimaging ML Classification Study

• Hypotheses: Pre-specify primary hypothesis (e.g., "Classifier A will outperform chance in diagnosing Condition X").
• Data: Specify inclusion/exclusion criteria, sample size, and data sources.
• Preprocessing: Define software, pipelines, and parameters (e.g., smoothing kernel, normalization method).
• Feature Selection: Define the exact method and criteria (e.g., "We will apply a whole-brain ANOVA with p<0.001 uncorrected, then select the top 500 most significant voxels for model input.").
• Modeling: Specify algorithms, validation method (e.g., 10-fold cross-validation), and performance metrics (e.g., balanced accuracy, AUC).
• Statistical Inference: Define how significance will be assessed (e.g., permutation testing with 5000 iterations).

Protocol 2: Nested Cross-Validation for Unbiased Pipeline Selection

• Split the full dataset into K outer folds (e.g., K=5).
• For each outer fold: a. Hold out the outer fold as the test set. b. Use the remaining K-1 folds as the training/validation set. c. Perform an inner cross-validation (e.g., 5-fold) on this training/validation set to tune hyperparameters or select the best among several pre-specified pipelines. d. Train a final model on the entire training/validation set using the optimal pipeline from step (c). e. Evaluate this final model on the held-out outer test fold. Record the performance.
• Aggregate the performance metrics from the K outer test folds. This is the final, unbiased performance estimate.

Visualizations

Title: p-Hacking via Flexible Analysis Pipeline Selection

Title: Protocol to Prevent Data Leakage & p-Hacking

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging ML Research

Item / Solution	Function & Relevance to Preventing p-Hacking
Pre-registration Platform (e.g., OSF, AsPredicted)	Provides a time-stamped, public record of hypotheses and methods before data analysis begins, limiting flexibility.
Version Control Software (e.g., Git, DataLad)	Tracks all changes to analysis code, ensuring reproducibility and creating an audit trail.
Containerization (e.g., Docker, Singularity)	Packages the exact computational environment (OS, libraries, software versions) used, eliminating "it works on my machine" variability.
Analysis Notebooks (e.g., Jupyter, RMarkdown)	Encourages literate programming, integrating code, results, and narrative. Promotes transparency when shared.
Blinding Scripts	Code that randomizes or blinds condition labels during preprocessing and initial analysis to prevent subconscious bias.
Permutation Testing Frameworks	Provides a robust method for generating non-parametric null distributions for hypothesis testing, crucial for ML model significance.
Multiple Comparison Correction Tools (e.g., FDR, Random Field Theory)	Standard libraries (in SPM, FSL, scikit-learn) to correct p-values for mass univariate testing or multiple model comparisons.

Technical Support & Troubleshooting Center

FAQs & Troubleshooting Guides

Q1: My classification accuracy fluctuates wildly when I change the cross-validation fold number. Is my result invalid? A: Not necessarily. High sensitivity to this parameter often indicates a small or unstable sample size. First, ensure your sample size meets or exceeds field-recommended minimums (e.g., >50 samples per class for simple binary classification). Implement nested cross-validation to separate model tuning from performance estimation. Report the mean and standard deviation of accuracy across a range of plausible fold numbers (e.g., 5, 10, Leave-One-Out) in a sensitivity table.

Q2: After correcting for multiple comparisons, all my significant features disappear. How can I robustly report findings? A: This is a critical robustness check. Avoid relying on a single correction method. Conduct a sensitivity analysis by applying a spectrum of corrections (e.g., FDR, Bonferroni, Random Field Theory, permutation-based) across a range of primary thresholds (e.g., p<0.001 to p<0.05). Tabulate the number of surviving features under each combination. Report the findings that are consistent across multiple rigorous methods.

Q3: How do I determine if my results are sensitive to the choice of atlas or parcellation scheme? A: This is a key parameter. Repeat your core analysis using at least three different, well-established atlases (e.g., AAL, Harvard-Oxford, Destrieux). For voxel-based analysis, vary the smoothing kernel FWHM (e.g., 4mm, 8mm, 12mm). Create a summary table showing the overlap (e.g., Dice coefficient) of significant regions or features identified across different preprocessing pipelines.

Q4: My machine learning model's performance drops to chance when tested on an external dataset. What parameters should I re-examine? A: This indicates a lack of generalizability, often due to site effects or overfitting to cohort-specific noise. Re-run your sensitivity analysis focusing on: 1) Hyperparameter regularization strength: Increase regularization and note performance on the external set. 2) Feature selection stability: Use bootstrap resampling to see how often your "top" features are selected internally; unstable features are poor candidates for generalization. 3) ComBat or other harmonization parameters: Test the impact of including/excluding harmonization on the external validation performance.

Q5: How can I structure my methods section to transparently report this sensitivity analysis? A: Dedicate a subsection titled "Sensitivity and Robustness Analyses." Use a table to list each key parameter varied, the range of values tested, the primary outcome metric (e.g., AUC, number of significant voxels), and a brief conclusion on robustness (e.g., "Robust," "Moderately Sensitive," "Highly Sensitive").

Table 1: Sensitivity of Classification Accuracy to Key Analysis Parameters

Parameter Tested	Value Range	Mean AUC (± std)	CV of AUC*	Robustness Conclusion
Cross-Validation Folds	5, 10, LOO	0.75 (±0.08)	10.7%	Moderately Sensitive
Smoothing Kernel (FWHM)	4mm, 8mm, 12mm	0.78 (±0.02)	2.6%	Robust
Feature Selection Threshold (p-unc)	0.01, 0.005, 0.001	0.72 (±0.05)	6.9%	Moderately Sensitive
Classifier (Regularization)	L1 SVM (C=0.1), L2 SVM (C=1), Logistic Regression (L2)	0.77 (±0.03)	3.9%	Robust

*Coefficient of Variation (std/mean) provides a normalized measure of sensitivity.

Table 2: Impact of Multiple Comparison Correction on Feature Discovery

Initial Threshold (p-unc)	Uncorrected Features	FDR (q<0.05)	Bonferroni	Permutation (FWER)	Consistent Features Across All
p < 0.001	150	110	15	22	12
p < 0.01	520	205	0	45	0

Experimental Protocols

Protocol 1: Conducting a Sensitivity Analysis on a Neuroimaging ML Pipeline

• Define Core Analysis: Establish a single, clearly defined primary machine learning pipeline (e.g., SVM classification with 10-fold CV, 8mm smoothing, AAL atlas).
• Identify Key Parameters: List parameters whose choice could materially affect results (e.g., CV scheme, smoothing kernel size, atlas, feature selection threshold, classifier type, hyperparameters).
• Define Variation Ranges: For each parameter, select a realistic range of values common in the literature (e.g., smoothing: 4mm, 8mm, 12mm).
• Run Iterative Analyses: Hold all parameters constant at the primary analysis choice, then systematically vary one parameter at a time across its range, recording the outcome metric(s).
• Quantify & Tabulate Variation: Calculate summary statistics (mean, standard deviation, range) for the outcome metrics across each parameter's variation. Populate tables like Table 1 & 2.
• Report Conclusions: For each parameter, state whether the primary conclusion (e.g., "classification is significantly above chance") holds across the tested range. Report robust findings and acknowledge limitations where results are sensitive.

Protocol 2: Stability Analysis for Feature Selection

• Bootstrap Resampling: Generate 1000 bootstrap samples from your original dataset (with replacement).
• Iterative Feature Selection: Run your chosen feature selection method (e.g., univariate t-test at p<0.01) on each bootstrap sample.
• Calculate Selection Frequency: For each feature (e.g., voxel, ROI), compute the percentage of bootstrap samples in which it was selected.
• Define Stable Feature Set: Apply a frequency threshold (e.g., selected in >80% of bootstraps) to identify a stable, robust subset of features.
• Validate: Compare the performance of a classifier trained on the stable feature set versus the original, full feature set using nested cross-validation.

Mandatory Visualizations

Sensitivity Analysis Workflow for Robust ML

Parameters Tested in Sensitivity Analysis

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging ML Analysis

Tool / Reagent	Primary Function	Role in Mitigating p-Hacking / Ensuring Robustness
Nilearn / scikit-learn	Python libraries for machine learning & neuroimaging analysis.	Provide standardized, reproducible implementations of algorithms and pipelines, reducing "code flexibility."
COINSTAC	Decentralized platform for collaborative analysis.	Enables external validation on independent datasets, the ultimate test for robustness and generalizability.
Permutation Testing Tools (e.g., FSL PALM, Scikit-learn's permutation_test_score)	Non-parametric statistical testing.	Generates null distributions specific to your data and pipeline, providing a robust foundation for statistical inference.
Bootstrap Resampling Code	Method for estimating stability of findings.	Quantifies the selection stability of features or model parameters, identifying unreliable results.
Pre-registration Template (e.g., on OSF, AsPredicted)	Document for stating hypotheses and analysis plan before data analysis.	Locks down the primary analysis plan, distinguishing confirmatory from exploratory work and limiting researcher degrees of freedom.
Data Harmonization Tools (e.g., NeuroCombat, PyHarmonize)	Removes site/scanner effects in multi-site data.	Reduces variance due to technical noise, improving model generalizability and feature reliability.
Containers (Docker, Singularity)	Packaging tool for complete computational environments.	Ensures exact reproducibility of the analysis environment, including all software versions and dependencies.

The Role of Permutation Testing and Null Model Comparisons

Troubleshooting Guides and FAQs

Q1: My permutation test yields a p-value of exactly zero. What does this mean, and is this a valid result?

A1: A p-value of zero typically means that none of your permuted test statistics exceeded the observed test statistic in the number of permutations you ran. For example, if you ran 1000 permutations, this suggests p < 0.001. While technically valid, it is best practice to report this as p < 1/N (e.g., p < 0.001) rather than p = 0. To increase precision, you can increase the number of permutations (e.g., to 10,000 or 100,000), provided you have the computational resources. This is a critical step to avoid the illusion of "infinite significance," which can be a form of p-hacking.

Q2: How do I choose an appropriate null model for my neuroimaging machine learning comparison?

A2: The null model must disrupt the association of interest while preserving the underlying data structure. Common choices include:

• Label Permutation: Randomly shuffles class labels or outcome variables. This is the most common null for testing if a model's performance is better than chance.
• Permutation of Imaging Features: Breaks the link between features and the outcome, useful for testing specific feature contributions.
• Parametric Null Models: Use analytical distributions (e.g., Gaussian) based on your data's assumptions. Permutation is often preferred as it is non-parametric and makes fewer assumptions.
• Site/Session Permutation: Crucial for multi-site studies, this permutes data across sites to test for site effects masquerading as biological signals.

Q3: I get highly variable p-values when I rerun a permutation test with the same data. Is this normal?

A3: Some variability is expected due to the random sampling inherent in permutation testing. However, large fluctuations indicate an insufficient number of permutations. The p-value estimate converges as the number of permutations increases. For a stable p-value around 0.05, at least 5,000 permutations are recommended. For publication, 10,000 is a common standard. The table below summarizes the relationship:

Table 1: Permutation Count and P-value Stability

Target P-value	Minimum Recommended Permutations	Desired Permutations for Stability
~0.05	1,000	5,000 - 10,000
~0.01	5,000	10,000 - 50,000
<0.001	10,000	50,000 - 100,000

Q4: How can permutation testing specifically address p-hacking in neuroimaging ML?

A4: P-hacking often involves flexibly trying analyses until a significant result is found. Permutation testing, when properly pre-specified in a registered analysis plan, mitigates this by:

• Providing an Empirical Null Distribution: The test statistic's distribution under the null is derived directly from your data, reducing reliance on assumptions that can be exploited.
• Controlling the Family-Wise Error Rate (FWER): Non-parametric permutation-based inference (e.g., with threshold-free cluster enhancement - TFCE) is a robust method for multiple comparisons correction across voxels/vertices, preventing cherry-picking of significant clusters.
• Validating Cross-Validation Results: Performing permutations on the entire cross-validation loop (shuffling labels before each split) gives a valid p-value for the cross-validated accuracy, preventing overfitting.

Q5: What are the common pitfalls in setting up a permutation test workflow?

A5:

• Pitfall 1: Incorrect Permutation Scheme. Permuting data within subjects when testing between-group differences, or permuting across independent and non-exchangeable blocks of data (e.g., time series data without block permutation).
• Solution: Carefully design permutation scheme to break the effect of interest while preserving inherent data dependencies. Use stratified permutation to maintain class balance.
• Pitfall 2: Data Leakage in Permutations. Applying feature selection or normalization before the permutation loop, causing information from the "test" data to influence the "training" data of the null model.
• Solution: Ensure the entire model training and evaluation process is contained within each permutation iteration. All steps must be repeated for each permuted dataset.
• Pitfall 3: Using an Inappropriate Test Statistic.
• Solution: The test statistic (e.g., AUC, accuracy, regression coefficient) must be sensitive to the effect you are testing. Use permutation to assess if your observed statistic is an outlier in the null distribution.

Experimental Protocols

Protocol 1: Permutation Test for Classifier Significance

Objective: To determine if a neuroimaging-based machine learning classifier's cross-validated performance is statistically significant against the null hypothesis of no association.

Materials: See "The Scientist's Toolkit" below. Method:

• Compute Observed Statistic: Train and validate your model (e.g., SVM) using your chosen cross-validation (CV) scheme (e.g., 10-fold). Calculate the observed performance metric (e.g., mean accuracy across folds).
• Initialize Permutation Loop: For i = 1 to N (e.g., N=10,000): a. Permute: Randomly shuffle the subject outcome labels (or regressors) across the entire dataset, breaking the link between brain data and outcome. b. Re-run CV: Using the same CV fold indices as in Step 1, repeat the entire model training and validation process on the permuted dataset. c. Store Null Statistic: Calculate and store the permuted performance metric.
• Construct Null Distribution: Aggregate all N permuted statistics to form the empirical null distribution.
• Calculate P-value: p = (count of (permuted_statistic >= observed_statistic) + 1) / (N + 1). The +1 includes the observed statistic in the distribution, providing a conservative estimate.
• Report: Report the observed statistic, the permutation-derived p-value, and N.

Protocol 2: Voxel-Wise Permutation Testing with TFCE

Objective: To perform group-level inference on mass-univariate neuroimaging data while controlling FWER without an arbitrary cluster-forming threshold.

Method:

• First-Level Model: Fit your general linear model (GLM) at each voxel for each subject.
• Compute Observed Contrast Map: Compute the desired contrast (e.g., Group A > Group B), resulting in a 3D map of t-statistics.
• Permutation: For i = 1 to N: a. Randomly permute group labels across subjects. b. Recompute the group-level contrast map, generating a full 3D map of "null" t-statistics. c. Apply the TFCE transformation to this entire null map. TFCE enhances cluster-like structures without a hard threshold. d. Store the maximum TFCE value across the entire brain for this permutation.
• Construct Null Distribution of Maxima: This creates a distribution of the maximum TFCE score expected under the null hypothesis.
• Correct Inference: For each voxel in your observed TFCE-transformed map, calculate its corrected p-value by comparing it to the null distribution of maxima: p_corrected = (count of (null_max >= observed_TFCE_voxel) + 1) / (N + 1).
• Threshold: Apply a significance threshold (e.g., p_corrected < 0.05) to the corrected p-value map.

Visualizations

Diagram 1: Permutation Testing Workflow for ML

Diagram 2: How Permutation Testing Counters P-Hacking

The Scientist's Toolkit: Key Research Reagent Solutions

Table 2: Essential Software & Tools for Permutation Testing in Neuroimaging ML

Tool/Reagent	Category	Primary Function in Experiment
scikit-learn	Python Library	Provides core ML algorithms, cross-validation splitters, and utilities for implementing custom permutation tests.
nilearn	Python Library	Enables machine learning on neuroimaging data, integrates with scikit-learn, and offers basic permutation scoring.
FSL's Randomise	Standalone Tool	Industry-standard tool for robust voxel-wise permutation inference (with TFCE) on MRI data.
Datalad / Git-annex	Data Management	Ensures reproducible data versioning and provenance tracking for permutation analysis pipelines.
Nipype	Python Framework	Allows for the creation of automated, reproducible workflows that integrate permutation steps from different software (FSL, SPM, etc.).
Custom Python/R Scripts	Code	Essential for implementing study-specific permutation schemes (e.g., stratified, blocked) and null models.
High-Performance Computing (HPC) Cluster	Infrastructure	Enables the execution of thousands of computationally intensive permutation iterations in parallel.

Troubleshooting Guides & FAQs

Q1: My neuroimaging ML model shows significant performance (p < 0.05) on multiple regions of interest, but I'm worried about false positives. Which correction method should I use beyond the basic Bonferroni?

A: Bonferroni is often too conservative for high-dimensional neuroimaging data, leading to false negatives. For voxel-wise or feature-wise comparisons, consider:

• False Discovery Rate (FDR): Controls the expected proportion of false discoveries among rejected hypotheses. Use Benjamini-Hochberg for independent or positively correlated tests, common in brain maps.
• Permutation-based Family-Wise Error Rate (FWER): A robust non-parametric method. It builds a null distribution of the maximum test statistic across all features/voxels via label shuffling, effectively controlling for the most extreme case of multiple comparisons.

Experimental Protocol for Permutation Testing (Voxel-wise Classification):

• Train your ML model (e.g., SVM) on the original data with true labels to obtain your initial test statistic map (e.g., accuracy or t-value per voxel).
• Randomly shuffle the subject labels (or condition labels) to break the true relationship between data and outcome.
• Retrain the model on the permuted data and compute a new test statistic map.
• Record the maximum single test statistic from this permuted map.
• Repeat steps 2-4 for at least 5,000-10,000 iterations to build a robust null distribution of maximum statistics.
• The 95th percentile of this null distribution defines your corrected significance threshold. Any test statistic from your original, real model that exceeds this threshold is significant at a FWER < 0.05.

Q2: When performing cross-validation, do I need to apply multiple comparison correction inside each fold?

A: No, this is a common error. Correction must be applied outside and after the cross-validation loop. Applying correction within each fold invalidates the procedure because the data and therefore the correlations between tests change in each partition. Correct the final, aggregated statistics (e.g., p-values from a permutation test across all folds) across all features/voxels.

Q3: How do I handle correction when my features (e.g., fMRI voxels) are highly correlated?

A: Bonferroni and standard FDR assume independence or positive dependence. For correlated neuroimaging data:

• Random Field Theory (RFT): Models the data as a continuous Gaussian random field. It uses the smoothness of the statistical map to adjust the FWER threshold for cluster-level or peak-level inference. Best suited for SPM-style analyses.
• Permutation Testing with Threshold-Free Cluster Enhancement (TFCE): A powerful non-parametric method. TFCE enhances cluster-like structures in your statistical map without requiring an initial arbitrary cluster-forming threshold. The permutation framework then provides a corrected p-value for each voxel's TFCE score.

Q4: What is the practical difference between cluster-level and peak-level correction?

A: These are two common approaches when performing mass-univariate testing (e.g., with a General Linear Model).

Correction Level	What is Corrected For?	Method (Example)	Best Used When...
Peak-Level (Voxel-wise)	The chance of a single voxel being falsely declared significant.	Bonferroni, FDR, Permutation (max statistic)	Searching for focal, precise effects. Hypothesis is about specific voxels.
Cluster-Level	The chance of a cluster of connected voxels (above a primary threshold) appearing by noise.	Permutation (max cluster size), RFT	Expecting broader, extended areas of activation. More sensitive to diffuse effects.

Experimental Protocol for Cluster-Level Permutation:

• Apply a primary, uncorrected threshold (e.g., p < 0.001) to your initial statistical map to define candidate clusters.
• Record the size (number of voxels) of each cluster.
• Perform permutation testing (as in Q1's protocol). For each permutation, after applying the same primary threshold, record the size of the largest cluster appearing in the permuted map.
• Build a null distribution of maximum cluster sizes.
• The 95th percentile of this distribution is your corrected cluster-size threshold. Any cluster in your original map larger than this threshold is significant at cluster-level FWER < 0.05.

Visualizations

Diagram 1: Multiple Comparison Correction Decision Flow

Title: Decision Flow for Multiple Comparison Correction Methods

Diagram 2: Permutation Testing Workflow for FWER Control

Title: Permutation Testing Workflow for FWER Control

The Scientist's Toolkit: Research Reagent Solutions

Item	Function in Neuroimaging ML / Multiple Comparisons Context
NiLearn (Python)	Library for statistical learning on neuroimaging data. Provides tools for classification, regression, and connectivity that integrate with correction methods.
FSL's Randomise	Tool for permutation-based inference on neuroimaging data. Implements voxel-wise, cluster-level, and TFCE corrections. Essential for non-parametric testing.
SPM (w/ RFT)	Statistical Parametric Mapping software. The primary tool for implementing Random Field Theory for peak-level and cluster-level Gaussian-based corrections.
Scikit-posthocs	Python library for post hoc pairwise tests following omnibus tests (e.g., ANOVA), with routines for FDR and other MCP methods.
BrainStat	Tool for statistical analysis of brain parcellations and surface data, includes FDR and permutation methods for vertex-wise analysis.
MNE-Python	For M/EEG data. Provides comprehensive functions for multiple comparison correction across sensors, time points, and frequencies, including permutation clusters.
Dipy	For diffusion MRI. Contains statistical frameworks for tract-based and connectome-wide analyses, often employing network-based statistics (NBS) for correction.
R p.adjust function	Core R function for applying Bonferroni, Holm, Hochberg, Hommel, and BH/BY FDR corrections to a vector of p-values.

Tools and Software Packages for Automated Pipeline Integrity Checking (e.g., scikit-learn, COBIDAS, NiBetaSeries)

Troubleshooting Guides & FAQs

Q1: During a reproducibility check using COBIDAS, my neuroimaging pipeline fails the "Data Provenance" check. The error states "Missing BIDS sidecar files for key sequences." What are the most likely causes and solutions?

A: This error indicates missing JSON sidecar files in your BIDS dataset. Common causes and solutions:

• Cause 1: DICOM to BIDS conversion was incomplete or used a heuristic file that didn't capture all sequence parameters.
  • Solution: Re-run heudiconv or dcm2bids with a verified, detailed heuristic file. Check the BIDS validator (bids-validator) output first.
• Cause 2: Files were manually moved or renamed, breaking the BIDS structure.
  • Solution: Use automated BIDS curation tools like bidskit or the PyBIDS library to re-organize. Never manually rename BIDS files.
• Cause 3: The scanning sequence parameters were not exported from the scanner.
  • Solution: Work with your MR technician to ensure all necessary *.json output is enabled on the scanner protocol.

Q2: When implementing a scikit-learn cross-validation loop within a NiPyPE pipeline to prevent data leakage, I get a memory error during the fit_transform step on the training set. How can I resolve this?

A: This is a common issue when applying feature selection or preprocessing within each CV fold. Solutions are listed below by trade-off.

Solution Approach	Specific Action	Trade-off / Consideration
Use Memory Caching	Use sklearn.pipeline.Pipeline with memory=joblib.Memory(location='./cachedir').	Drastically reduces recomputation but requires significant disk space.
Incremental Processing	For out-of-core learning, use sklearn.linear_model.SGDClassifier or SGDRegressor.	Only suitable for algorithms that support partial fitting.
Feature Reduction	Apply an initial, modest variance threshold or PCA before the CV loop.	Risk of removing biologically relevant low-variance features early.
Resource Allocation	Increase job memory limits or use cloud/ HPC nodes.	Practical but can be costly. Always profile memory usage first with memory_profiler.

Q3: NiBetaSeries raises a "Missing Event Files" error when trying to correlate beta series, even though I have a *_events.tsv file. What is wrong?

A: The error likely pertains to file content or path, not existence. Follow this debugging protocol:

• Validate BIDS Compliance: Run bids-validator on your dataset. Ensure event files are in the correct subject/session directory.
• Check File Contents: Open your *_events.tsv file. Confirm it has the mandatory onset, duration, and trial_type columns. Check for NA or non-numeric values in onset or duration.
• Check Task Name Consistency: The task name in the filename (e.g., task-memory) must match the task name specified in your NiBetaSeries workflow configuration. A mismatch is the most common cause.
• Check Model Specification: If using a custom model, ensure the trial_type labels in your event file exactly match the condition names listed in your first-level model (model.json or model.py).

Q4: To address p-hacking, my thesis requires reporting all preprocessing hyperparameters. How can I automatically extract and log these from a NiPyPE pipeline built with tools like fMRIPrep?

A: Implement automated logging by capturing the pipeline's configuration output.

Tool	Method for Parameter Extraction	Output Format for Thesis Appendix
fMRIPrep	Use the --output-spaces and --use-aroma flags. Crucially, run with --verbose and redirect output to a log file. The key file is dataset_description.json in the output directory and the logs/ folder.	JSON & structured text log. Convert to table using a custom Python script (json2csv).
Custom NiPyPE	Use the nipype.utils.config.enable_debug_mode() function at the start of your script. Implement a custom callback function using nipype.interfaces.base.support.InterfaceResult.	Structured text log. Parse using regular expressions to create a parameter table.
General Solution	Use the CWL (Common Workflow Language) or WDL export feature of your pipeline. These workflow description files are machine-readable records of all parameters.	YAML/JSON (CWL) or WDL script. Can be included directly in supplementary materials.

Experimental Protocol: Benchmarking Pipeline Integrity Tools

Title: Protocol for Comparing False Positive Rates in Neuroimaging ML Pipelines With and Without Automated Integrity Checks.

Objective: To empirically quantify the reduction in false positive (FP) rate in machine learning (ML) analyses when using automated pipeline integrity checking tools (COBIDAS, NiBetaSeries) compared to a standard, unchecked pipeline.

Materials:

• Dataset: Publicly available resting-state and task fMRI dataset (e.g., ADHD-200, UCLA CNP, or Human Connectome Project minimal preprocessed data).
• Software: fMRIPrep v23.x, scikit-learn v1.3+, NiBetaSeries v0.13.0, COBIDAS checker (e.g., bids-validator, MRIQC), Python 3.10+, Nipype v1.8.x.

Procedure:

• Data Preparation & Intentional Corruption:
  • Download a complete BIDS dataset (N > 100).
  • Create three derivatives:
    • Derivative A: Standard fMRIPrep preprocessing.
    • Derivative B: fMRIPrep output with introduced, subtle integrity errors (e.g., swapped event files for 5% of subjects, introduced minimal spatial smoothing leakage from confound regression).
    • Derivative C: fMRIPrep output processed through an integrity check (COBIDAS validator + MRIQC), with errors corrected.

• Feature Extraction & Model Training:

  • For each derivative (A, B, C), extract identical feature sets using NiBetaSeries (for task data) or connectivity matrices (for rest).
  • Train the same binary classifier (e.g., Linear SVM with C=1.0) using a nested cross-validation scheme (5 outer folds, 3 inner folds for parameter tuning) to predict a label (e.g., patient/control, task condition).
  • Key: Use an identical scikit-learn pipeline for all three analyses to isolate the effect of input data integrity.

• Statistical Comparison & FP Rate Estimation:

  • Use permutation testing (5000 permutations) on the cross-validated accuracy/ AUC metric for each derivative to establish a null distribution.
  • Calculate the observed classification accuracy for each derivative.
  • The FP rate is estimated by running the same analysis on a permuted-label (null) dataset. Compare the distribution of null accuracies between Derivative B (corrupted) and Derivative C (checked/corrected).

• Integrity Check Logging:

  • Automatically log all warnings/errors from the COBIDAS checks and NiBetaSeries validation steps for Derivative C.

Expected Outcome: Derivative B (corrupted) is predicted to show a significantly higher false positive rate (inflation of null accuracy) compared to Derivative A or C, demonstrating the protective effect of automated integrity checking.

The Scientist's Toolkit: Research Reagent Solutions

Item	Function in Pipeline Integrity Research
BIDS Validator	The foundational tool for checking compliance with the Brain Imaging Data Structure, ensuring data provenance and organization integrity.
MRIQC	Provides quantitative quality control metrics for structural and functional MRI data, automating the detection of acquisition and preprocessing artifacts.
fMRIPrep	A robust, standardized preprocessing pipeline for fMRI data. Its consistent use is a key integrity checkpoint, reducing variability.
DataLad	A version control system for data. Crucial for tracking the exact state of datasets and pipelines at the time of analysis, ensuring full reproducibility.
Nipype	A Python framework that allows for the creation of reproducible, transparent, and modular neuroimaging workflows by connecting different tools.
Scikit-learn Pipeline	Ensures that preprocessing steps (scaling, feature selection) are correctly nested within cross-validation loops, preventing label leakage.
NiBetaSeries	Specialized tool for extracting trial-wise betaseries correlations from fMRI task data, implementing a standardized method that reduces analytical flexibility.
CWL (Common Workflow Language)	A specification for describing analysis workflows and tools in a way that makes them portable and scalable across different software environments.

Workflow Diagrams

Title: Automated Integrity Checking in Neuroimaging ML Pipeline

Title: Nested Cross-Validation to Prevent Leakage

Validation Benchmarks and Comparative Frameworks for Trustworthy Findings

Troubleshooting Guides & FAQs

Q1: My model achieves excellent accuracy on my primary dataset but fails completely on a similar, independent dataset from another site. What are the most common causes? A: This is a classic sign of overfitting to dataset-specific biases (e.g., scanner, protocol, or population). Key issues include:

• Feature Leakage: Features inadvertently encode site-specific noise rather than biological signal.
• Inadequate Preprocessing: Harmonization methods (like ComBat) were not applied or failed.
• P-hacking in Feature Selection: Features were selected based on performance within the primary dataset only, capturing noise.

Q2: I applied cross-validation correctly, yet my model doesn't generalize externally. Doesn't cross-validation prevent this? A: Internal cross-validation alone is insufficient. It validates the pipeline but not the generalizability of the findings. It cannot account for systematic distribution shifts between datasets. True external validation requires a completely held-out dataset with no role in training or feature selection.

Q3: What are the minimal steps to perform a methodologically sound external validation? A: Follow this protocol:

• Complete Separation: The external validation dataset must have no subjects used in the primary (training/development) set.
• Frozen Pipeline: Lock all model parameters, features, and preprocessing steps after primary dataset development.
• Single Pass: Apply the frozen pipeline to the external dataset once to compute performance metrics.
• Report All Metrics: Report performance on both primary (via cross-validation) and external sets in a comparative table.

Q4: Where can I find suitable external datasets for validation in neuroimaging? A: Public data repositories are essential. Common sources include:

• ADNI: Alzheimer's Disease Neuroimaging Initiative.
• ABIDE: Autism Brain Imaging Data Exchange.
• UK Biobank: Large-scale biomedical database.
• HCP: Human Connectome Project.
• IXI: Information eXtraction from Images.
• OASIS: Open Access Series of Imaging Studies.

Q5: How do I statistically compare performance between my primary and external validation results? A: Use tests that account for the confidence in your estimates. Recommended methods include:

• DeLong's Test: For comparing AUCs of two correlated/receiver operating characteristic curves.
• McNemar's Test: For comparing classification accuracies on the same external set.
• Bootstrapped Confidence Intervals: Generate intervals for performance metrics on both datasets; non-overlapping intervals suggest a significant drop.

Experimental Protocols

Protocol 1: Conducting a Rigorous External Validation Study

Objective: To assess the true generalizability of a neuroimaging-based machine learning model. Materials: Primary dataset (Dataset A), completely independent external dataset (Dataset B). Method:

• Development Phase (Using Dataset A only):
  • Perform preprocessing (slice timing correction, motion correction, normalization, smoothing).
  • Apply feature selection (e.g., ANOVA, LASSO) using only Dataset A's training folds.
  • Train model (e.g., SVM, Random Forest) using nested cross-validation on Dataset A.
  • Freeze the final model architecture, selected feature set, and all preprocessing parameters.
• Validation Phase (Using Dataset B only):
  • Preprocess Dataset B using the exact same software, version, and parameters from Step 1.
  • Extract the identical set of features selected in Step 1.
  • Apply the frozen model from Step 1 to Dataset B features.
  • Compute performance metrics (accuracy, sensitivity, specificity, AUC) in a single pass.
• Analysis:
  • Report performance metrics for Dataset A (mean ± std from CV) and Dataset B (single value).
  • Statistically compare AUCs using DeLong's test or accuracies using McNemar's test.
  • Create a results comparison table.

Protocol 2: Implementing Data Harmonization for Multi-Site Validation

Objective: To reduce site-specific technical variance before model development to improve generalizability. Materials: Multi-site data (e.g., Data from Site 1, Site 2, Site 3). Method:

• Pooled Data Preparation: Combine imaging data and covariates (age, sex, diagnosis, site/scanner identifier) from all sites.
• Harmonization: Apply the ComBat (or its variants like NeuroComBat) harmonization technique.
  • Model the data as a function of biological variables of interest (preserved) and scanner/site effects (removed).
  • Estimate and remove location (mean) and scale (variance) scanner effects using an empirical Bayes framework.
• Train/Test Split by Site: For a rigorous test, assign all data from one or more entire sites to be the external validation set.
• Model Training & Validation: Train on harmonized data from remaining sites. Validate on the held-out site(s) after applying the harmonization model parameters learned from the training sites.

Data Tables

Table 1: Comparison of Model Performance on Internal vs. External Validation

Metric	Internal CV (Dataset A) Mean ± SD	External Hold-Out (Dataset B)	Performance Drop	p-value (DeLong's Test)
Accuracy	92.5% ± 2.1%	68.2%	24.3%	N/A
AUC	0.95 ± 0.03	0.71	0.24	0.002
Sensitivity	91.0% ± 3.5%	65.0%	26.0%	N/A
Specificity	94.0% ± 2.8%	71.4%	22.6%	N/A

Table 2: Essential Research Reagent Solutions for Reproducible Neuroimaging ML

Item	Function & Importance
Public Data Repositories	(e.g., ADNI, UK Biobank) Provide essential external datasets for independent validation.
Harmonization Tools	(e.g., ComBat, NeuroComBat) Remove site/scanner effects to improve model generalizability.
Version Control Software	(e.g., Git) Precisely track code, pipeline parameters, and model versions for reproducibility.
Containerization	(e.g., Docker, Singularity) Package entire analysis environment to guarantee identical software stacks.
Pre-registration Platforms	(e.g., OSF) Specify hypotheses, methods, and analysis plans before analysis to combat p-hacking.
Reporting Checklists	(e.g., TRIPOD+ML, CONSORT-AI) Ensure complete and transparent reporting of all experiments.

Visualizations

Title: Rigorous ML Validation Workflow

Title: The p-Hacking Loop & Its Consequence

Technical Support Center: Troubleshooting Guides & FAQs

This technical support center addresses common issues encountered when applying comparative evaluation frameworks to public neuroimaging datasets within research aimed at mitigating p-hacking and ensuring robust machine learning comparisons.

Frequently Asked Questions (FAQs)

Q1: My model achieves near-perfect classification accuracy on the ABIDE I dataset preprocessed with pipeline A, but fails completely (≈50% AUC) on data from pipeline B. Is my model faulty? A: This is a classic sign of data leakage or pipeline-induced bias, not a model fault. A robust comparative framework must control for preprocessing variability. First, ensure your cross-validation folds are strictly separated by subject ID and preprocessing pipeline during training. Never mix pipelines within a fold. Second, benchmark your model against a simple linear model on the same pipeline to see if the performance delta is consistent. High variance across pipelines suggests your findings may not generalize.

Q2: When comparing two algorithms on the ADHD-200 dataset, how do I determine if a small performance increase (e.g., 0.02 AUC) is statistically significant or a result of multiple comparisons? A: You must employ nested cross-validation with appropriate statistical testing. Use a paired, non-parametric test (e.g., Wilcoxon signed-rank test) on the paired performance metrics (e.g., AUCs per fold) from the outer test sets of a nested CV scheme. Correct for multiple comparisons if you are evaluating more than two algorithms. Report confidence intervals. A result is only credible if it survives correction and the effect size is meaningful for the clinical context.

Q3: I am using derived features from UK Biobank (e.g., regional volumes). How can I prevent p-hacking when performing feature selection across thousands of regions? A: Feature selection must be performed independently within each training fold of your cross-validation. Never use the entire dataset for feature selection before CV. Document every step (e.g., "Variance threshold, followed by ANOVA F-test selecting top 10% features, applied per fold"). Consider using stability metrics to report how consistent your selected features are across folds. Pre-register your analysis plan, including feature selection criteria, to avoid hindsight bias.

Q4: My results on a public benchmark are substantially lower than the state-of-the-art paper. What should I check first? A: Follow this systematic checklist:

• Data Splits: Verify you are using the same subject partitions (train/validation/test) as the benchmark. Reproduce the exact split if possible.
• Preprocessing: Confirm the exact preprocessing pipeline (software, version, parameters). Even minor differences (e.g., smoothing kernel size) can impact results.
• Label Interpretation: Ensure you are using the same diagnostic labels and grouping (e.g., ASD vs. control, all sites vs. a specific subset).
• Evaluation Metric: Confirm the calculation of the primary metric (e.g., is it accuracy, balanced accuracy, or AUC?).
• Code & Random Seeds: Check for implementation differences and ensure you have fixed random seeds for reproducibility.

Q5: How do I handle site/scanner effects when pooling data from multiple centers in ABIDE or UK Biobank for a fair comparison? A: This is critical for avoiding confounded results. You have several options, and your framework should test robustness across them:

• Stratification: Stratify your cross-validation folds by site to ensure all folds have representative site data.
• ComBat Harmonization: Apply neuroimaging harmonization tools like ComBat within each training fold to remove site effects, then transform the held-out test data using parameters from the training set.
• Site-as-a-covariate: Include site as a covariate in your model.
• Single-site Evaluation: Train and test on data from a single site only, reporting generalizability as performance drops when applying to other sites. Your comparative framework should report results for all these strategies.

Table 1: Key Public Neuroimaging Benchmark Datasets for Psychiatric ML Research

Dataset	Primary Domain	Sample Size (Typical)	Key Modality	Primary Prediction Tasks	Major Challenge
ABIDE	Autism Spectrum Disorder	~1000-2000 subjects (aggregated I & II)	rsfMRI, sMRI	ASD vs. Control Classification	Significant site/scanner heterogeneity, phenotypic diversity.
ADHD-200	Attention Deficit Hyperactivity Disorder	~800-900 subjects	rsfMRI, sMRI	ADHD vs. Control Classification	Multi-site data, younger cohort, comorbidity.
UK Biobank	Population Health	>40,000 with imaging (growing)	MRI, dMRI, rfMRI	Various (e.g., disease status, cognitive scores, biomarkers)	Population bias, immense size requiring distributed computing.

Table 2: Common Performance Benchmarks (Illustrative Ranges)*

Dataset	Baseline Model (e.g., Linear SVM) Typical AUC Range	Reported SOTA AUC Range (Recent Literature)	Recommended Primary Metric(s)
ABIDE (Multi-site)	0.60 - 0.68	0.70 - 0.85	Balanced Accuracy, AUC
ADHD-200	0.55 - 0.62	0.65 - 0.75	AUC, Balanced Accuracy
UK Biobank (e.g., Depression)	0.65 - 0.72	0.75 - 0.82	AUC, R² (for continuous traits)

Note: Ranges are highly dependent on specific data subsets, preprocessing, and CV strategy. Direct comparison between papers is often invalid without strict protocol replication.

Experimental Protocols for Robust Comparison

Protocol 1: Nested Cross-Validation for Model Evaluation & Selection

• Purpose: To obtain an unbiased estimate of model performance while tuning hyperparameters, preventing optimistic bias.
• Method:
  • Outer Loop (Performance Estimation): Split the entire dataset into k folds (e.g., 5 or 10). For each outer fold i:
    • Hold out fold i as the test set.
    • Use the remaining k-1 folds as the development set.
  • Inner Loop (Model Selection): On the development set, perform another k-fold cross-validation.
    • Search over a pre-defined hyperparameter grid.
    • Select the hyperparameter set that yields the best average validation score across the inner folds.
  • Final Evaluation: Train a new model on the entire development set using the selected optimal hyperparameters. Evaluate this model on the held-out outer test set (fold i).
  • Aggregation: Repeat for all k outer folds. The final performance is the average of the scores on each of the k held-out test sets. Report the mean and standard deviation/confidence interval.

Protocol 2: Preprocessing Pipeline Replication & Comparison

• Purpose: To disentangle algorithm performance from preprocessing-induced variance.
• Method:
  • Select 2-3 standard, publicly documented preprocessing pipelines for the target modality (e.g., for rsfMRI: CPAC, DPARSF, fMRIPrep with specific atlases).
  • Process the raw dataset through each pipeline independently, generating multiple derived datasets.
  • Apply your identical machine learning comparison framework (using Protocol 1) to each derived dataset separately.
  • Report results in a pipeline-by-algorithm matrix. Statistical comparison should assess the interaction between algorithm and pipeline. The most robust algorithm shows the least performance variance across pipelines.

Visualizations

Diagram 1: Nested CV Workflow for Robust Evaluation

Diagram 2: Framework to Isolate Algorithm vs. Pipeline Effects

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Reproducible Neuroimaging ML Comparisons

Tool/Resource	Category	Function in Comparative Framework
fMRIPrep / MRIQC	Preprocessing & QA	Standardized, containerized preprocessing for BIDS-formatted data, generating quality metrics to exclude problematic scans. Critical for consistent input data.
Nilearn / NiBabel	Python Libraries	Provides tools for loading, manipulating, and analyzing neuroimaging data, and building connectomes/features from preprocessed data.
scikit-learn	ML Library	Implements a consistent API for a wide range of machine learning models, cross-validation splitters, and metrics. Foundation for building comparative pipelines.
ComBat / NeuroHarmonize	Harmonization Tool	Removes site and scanner effects from extracted imaging features. Must be applied carefully within CV to avoid data leakage.
MLflow / DVC	Experiment Tracking	Logs all parameters, code versions, metrics, and outputs for each experiment run. Essential for auditing and replicating comparisons.
Nipype	Workflow Engine	Allows creation of reproducible, automated preprocessing and analysis pipelines, connecting different software tools.
BIDS (Brain Imaging Data Structure)	Data Standard	A standardized way to organize neuroimaging and behavioral data. Enforces consistency, making data sharing and pipeline application feasible.

Troubleshooting Guides & FAQs

Q1: My model achieves high accuracy on my held-out test set, but fails completely on an external validation cohort from a different site. What checklist items from TRIPOD+AI or COBIDAS might I have missed? A: This typically indicates a failure to report key aspects of data provenance and preprocessing, leading to "site effects" or "scanner effects" dominating the signal. Key missed items are likely:

• TRIPOD+AI Item 5 (Data): Failure to clearly describe data sources, inclusion/exclusion criteria, and data cleaning methods for all datasets.
• COBIDAS (Data & Preprocessing): Incomplete reporting of scanner parameters (field strength, sequence details), preprocessing software (version, parameters for normalization, smoothing), and quality control metrics (excluded participants, motion artifacts).
• TRIPOD+AI Item 7b (Validation): Not stating whether the validation was internal, external, or both. External validation on a distinct population is a key recommendation.

Protocol: To diagnose, re-run your preprocessing pipeline on the external cohort from raw data, ensuring identical software versions and parameters. Compare feature distributions (e.g., global signal, voxel intensity histograms) between cohorts after preprocessing using a Kolmogorov-Smirnov test.

Q2: I'm comparing two ML models for neuroimaging classification. The p-value suggests Model A is superior to Model B (p=0.03), but the performance difference is minuscule (AUC: 0.81 vs. 0.805). How can reporting standards prevent this form of p-hacking? A: This is a classic case of statistical significance without practical significance, enabled by large test sets. TRIPOD+AI and COBIDAS emphasize:

• TRIPOD+AI Item 10 (Model Performance): Mandates reporting of performance measures with confidence intervals (e.g., 95% CI for AUC). The tiny difference will likely be non-significant when CIs are reported.
• TRIPOD+AI Item 16 (Study Limitations): Requires discussion of the clinical or practical relevance of performance differences.
• COBIDAS (Statistical Reporting): Discourages reliance on p-values alone for model comparison. Recommending reporting effect sizes and performing equivalence testing if models are functionally similar.

Protocol: Instead of a standard t-test, calculate the 95% confidence interval for the difference in AUC using DeLong's method or bootstrapping (e.g., 2000 iterations). Report the interval: Diff = 0.005, 95% CI [-0.01, 0.02]. Perform a pre-specified equivalence test if a pre-defined "negligible difference" margin (δ) exists.

Q3: My preprocessing pipeline involves over 20 steps with many optional parameters. How can I report this transparently without a 10-page methods section? A: Both standards encourage structured, concise reporting and sharing of code.

• COBIDAS (Reproducibility): Explicitly recommends sharing full preprocessing scripts (e.g., in Python, R, or a BIDS-App like fMRIPrep). A summary table in the manuscript can point to the version-controlled code repository.
• TRIPOD+AI Item 21 (Code Availability): Requires a statement on code availability.

Protocol: Create a versioned GitHub repository containing your Snakemake/Nextflow pipeline or your configuration file for a containerized tool (e.g., fMRIPrep config). In the manuscript, provide a high-level summary table and the DOI for the code release.

Table 1: Core Reporting Requirements for Preventing p-Hacking in ML Comparisons

Issue	TRIPOD+AI Checklist Item	COBIDAS Section	Key Action for Researchers
Selective Reporting of Models	Item 24 (Complete Reporting)	Analysis / Results	Pre-register analysis plan; report all models tested, not just the best.
Optimistic Performance from Flexible Design	Item 9 (Model Development)	Analysis	Clearly separate data for training, validation (tuning), and testing. Report details of any hyperparameter optimization.
Unreliable p-values from Repeated Testing	Item 17 (Interpretation)	Statistical Reporting	Correct for multiple comparisons; report statistical tests used with justification.
Non-Reproducible Preprocessing	Item 5 (Data)	Data / Preprocessing	Share full preprocessing code and container images; use BIDS format.
Misleading Performance Metrics	Item 10 (Model Performance)	Statistical Reporting	Report multiple metrics (AUC, accuracy, sensitivity, specificity) with confidence intervals.

Table 2: Example Performance Comparison with Confidence Intervals

Model	Primary Cohort (n=500)	External Validation (n=200)
	AUC (95% CI)	Balanced Accuracy (95% CI)	AUC (95% CI)	Balanced Accuracy (95% CI)
Support Vector Machine	0.85 (0.82 - 0.88)	0.78 (0.74 - 0.82)	0.79 (0.73 - 0.85)	0.72 (0.66 - 0.78)
Random Forest	0.84 (0.81 - 0.87)	0.77 (0.73 - 0.81)	0.81 (0.75 - 0.87)	0.74 (0.68 - 0.80)
Logistic Regression	0.82 (0.79 - 0.85)	0.75 (0.71 - 0.79)	0.80 (0.74 - 0.86)	0.73 (0.67 - 0.79)
Difference (SVM - RF)	0.01 (-0.02 - 0.04)	0.01 (-0.03 - 0.05)	-0.02 (-0.07 - 0.03)	-0.02 (-0.08 - 0.04)

Experimental Protocol: Framework for a Rigorous ML Comparison Study

Objective: To compare the performance of three classification algorithms on a neuroimaging-derived biomarker while mitigating risks of p-hacking and overfitting.

1. Pre-registration & Data Split:

• Pre-register hypothesis, primary metric (e.g., AUC), and model specifications on a platform like Open Science Framework.
• Using a single random seed, partition data into: Training (60%), Validation (20%) for hyperparameter tuning, and Hold-out Test Set (20%). For external validation, secure a completely independent cohort.

2. Preprocessing & Feature Extraction (BIDS/COBIDAS Compliant):

• Process all raw T1w and fMRI data through a containerized pipeline (e.g., fMRIPrep v23.2.1).
• Extract regions-of-interest (ROI) time series from a pre-defined atlas (e.g., Schaefer 400-parcel).
• Compute a static functional connectivity matrix for each participant. Apply and document Fisher's z-transform.
• Output: A features-by-subjects matrix for each cohort, with code and derivatives archived.

3. Model Training & Internal Validation:

• Train three models on the Training set:
  • Model A: Linear SVM (C: tune via validation set).
  • Model B: Random Forest (nestimators: 500, maxdepth: tune).
  • Model C: Logistic Regression with L2 penalty (C: tune).
• Use the Validation set for hyperparameter tuning via 5-fold cross-validation within the training cohort only, optimizing for AUC. Record final chosen parameters.

4. Performance Assessment & Statistical Comparison:

• Lock down all models and parameters. Evaluate each model once on the Hold-out Test Set and the External Validation Cohort.
• Calculate AUC, balanced accuracy, sensitivity, specificity. Generate 95% confidence intervals via stratified bootstrapping (2000 iterations).
• For model comparison, use the test set predictions. Employ a non-parametric paired test (e.g., Dietterich's 5x2 CV test) or compute the CI for the difference in AUCs. Apply Benjamini-Hochberg correction if comparing >2 models on multiple metrics.

Diagrams

Title: ML Workflow with p-Hacking Risk Points

Title: TRIPOD+AI & COBIDAS Integration for Neuroimaging ML

The Scientist's Toolkit: Research Reagent Solutions

Tool / Resource	Category	Function in Transparent ML Research
BIDS (Brain Imaging Data Structure)	Data Standard	Provides a consistent, standardized format for organizing neuroimaging and behavioral data, essential for reproducibility and data sharing.
fMRIPrep / QSIPrep	Preprocessing Pipeline	Containerized, standardized preprocessing software for fMRI and dMRI data. Automates reporting of preprocessing steps, aligning with COBIDAS.
Datalad / git-annex	Data Versioning	Enables version control and provenance tracking for large, binary neuroimaging datasets.
MLflow / Weights & Biases	Experiment Tracking	Logs hyperparameters, code versions, metrics, and output models for each experiment, preventing selective reporting.
scikit-learn / Nilearn	Machine Learning Library	Provides standardized, peer-reviewed implementations of ML models and evaluation metrics, ensuring methodological clarity.
OSF (Open Science Framework)	Pre-registration Platform	Allows public pre-registration of study hypotheses and analysis plans to combat HARKing and p-hacking.
Docker / Singularity	Containerization	Packages the complete software environment (OS, libraries, code) to guarantee computational reproducibility.
BIDS Stats Models	Modeling Specification	A standardized language for describing linear models applied to BIDS data, promoting clear reporting of statistical models.

Technical Support Center: Troubleshooting Guides & FAQs

Q1: I shared my neuroimaging code and data, but another lab reports they cannot replicate my machine learning model's performance. What are the first steps to diagnose this? A1: Begin with an environment and dependency check. The most common issue is version mismatch. Use containerization (Docker/Singularity) to share an exact computational environment. If not used, provide a detailed requirements.txt (Python) or manifest.json (MATLAB) with explicit version numbers, not version ranges. Next, verify the random seed was set and shared for all stochastic processes (data splitting, weight initialization). Finally, check for hidden data dependencies: ensure all preprocessed data files, including any held-out validation sets, are accessible via the paths specified in your code.

Q2: My shared preprocessed fMRI dataset is being criticized for potential data leakage in machine learning studies. How can I audit and document my pipeline to prevent this? A2: Data leakage in neuroimaging often occurs during preprocessing before the train/test split (e.g., global signal regression, filter application). To troubleshoot:

• Document the Split Point: Clearly state in your protocol at which pipeline step the data was split into independent training, validation, and test sets.
• Provide Code Snippets: Share the exact script showing how subjects were assigned to splits, ensuring it is based on subject ID, not row number in a shuffled table.
• Audit for Independence: For any normalization step (e.g., z-scoring), confirm the statistics (mean, std) were computed only on the training set and then applied to the validation/test sets. A common error is calculating these stats on the entire dataset before splitting.

Q3: A reviewer asks for evidence that my published significant finding (p<0.05) is not due to p-hacking via selective analysis reporting. What documentation should I provide? A3: You must provide evidence of a pre-registered analysis plan or a comprehensive multiverse analysis. Share:

• Pre-registration Document: The timestamped protocol from platforms like AsPredicted or OSF that specifies the primary hypothesis, main analysis, and sample size before data analysis began.
• Multiverse Analysis Report: If not pre-registered, run and report a "multiverse" or "specification curve" analysis. This involves running all reasonable combinations of analysis choices (e.g., different preprocessing smoothness kernels, motion correction thresholds, machine learning algorithms) and reporting the distribution of results. A finding robust to p-hacking will show consistent effect sizes across most reasonable specifications.

Q4: I am trying to implement a published model but get errors related to missing or mismatched library versions for deep learning frameworks (PyTorch/TensorFlow). What is the most efficient solution? A4: Manual version debugging is time-consuming. The standard solution is to use a container or environment file:

• For PyTorch: Provide a environment.yml file for Conda. Example for a CUDA 11.3 system:

• For TensorFlow: Use a Dockerfile to guarantee compatibility.
• Universal: Share a ready-made Docker image via Docker Hub.

Q5: How do I structure a shared dataset (e.g., BIDS derivatives) to minimize user errors during replication attempts? A5: Adhere to the BIDS Derivatives standard. Use a consistent, documented hierarchy.

Include a README.md in the root describing the exact version of the preprocessing software (e.g., fMRIPrep 22.0.1) and any modifications.

Experimental Protocols for Key Cited Experiments

Protocol 1: Multiverse Analysis to Diagnose p-Hacking Susceptibility

• Define the Analysis Space: List all analytical choices in your neuroimaging ML pipeline where researcher discretion applies (e.g., smoothing FWHM: [4mm, 6mm, 8mm]; motion scrubbing threshold: [0.2mm, 0.5mm]; feature selection method: [ANOVA, PCA]; classifier: [SVM, Logistic Regression]).
• Generate All Specifications: Programmatically create a list of all possible combinations (the "multiverse") of these choices.
• Run All Analyses: For each unique specification, run the full machine learning pipeline (train on training set, evaluate on held-out test set). Record the primary performance metric (e.g., AUC, accuracy) and associated p-value (from permutation testing).
• Visualize the Distribution: Create a histogram of all resulting p-values and a scatter plot of effect size vs. specification. A robust finding will show a tight cluster of significant results, not a uniform spread from p<0.001 to p>0.5.

Protocol 2: Permutation Testing for ML Model Significance (Null Distribution Creation)

• Train Model on True Labels: Split data into train/test sets. Train your chosen ML model on the training set with the true diagnostic labels (e.g., Patient vs. Control). Evaluate on the test set to obtain the true performance metric (M_true).
• Generate Null Distribution: For N iterations (e.g., N=10,000): a. Randomly shuffle the training set labels, breaking the relationship between brain data and diagnosis. b. Retrain the identical model architecture/hyperparameters on the shuffled-label training data. c. Evaluate this "null" model on the unshuffled test set to obtain a null performance metric (Mnulli).
• Calculate p-value: Compute the p-value as the proportion of null iterations where Mnulli was greater than or equal to M_true: p = (count(M_null_i >= M_true) + 1) / (N + 1).

Data Presentation

Table 1: Impact of Open Science Practices on Replication Success Rates in Neuroimaging ML Studies

Study Focus	Shared Code	Shared Data	Shared Models	Replication Success Rate	Key Barrier Identified
fMRI Classification (2021)	Yes	Yes (BIDS)	No	65%	Algorithm hyperparameter sensitivity
Structural MRI (Alzheimer's)	Yes	Partial (Derivatives)	Yes (ONNX)	88%	Preprocessing pipeline divergence
fMRI Connectivity (2023)	No	Yes (Raw)	No	22%	Undocumented feature extraction code
Multimodal Fusion (2022)	Yes (Container)	Yes (BIDS)	Yes (Docker Hub)	94%	GPU memory requirements mismatch

Table 2: Results of a Multiverse Analysis on a Published p<0.05 fMRI ML Finding

Analysis Specification	Smoothing (mm)	Scrubbing Threshold (mm)	Classifier	Test AUC	p-value (Permutation)
Original Published	6	0.5	Linear SVM	0.72	0.03
Specification 2	8	0.5	Linear SVM	0.71	0.04
Specification 3	6	0.2	Linear SVM	0.68	0.08
Specification 4	6	0.5	RBF SVM	0.74	0.11*
Specification 5	4	0.5	Logistic Reg.	0.65	0.21
...	...	...	...	...	...
Median across 125 specs	-	-	-	0.69	0.16

Note: p > 0.05 suggests potential model overfitting with more complex kernel.

Diagrams

Title: Neuroimaging ML Replication Support Workflow

Title: Data Leakage Checkpoints in a Standard ML Pipeline

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Digital Tools for Replicable Neuroimaging Machine Learning

Tool Name	Category	Function in Replication	Example/Version
BIDS Validator	Data Standardization	Validates that shared neuroimaging datasets adhere to the Brain Imaging Data Structure, ensuring consistent organization and metadata.	v1.11.0+
Docker / Singularity	Containerization	Packages the entire analysis environment (OS, libraries, code) into a single, runnable image, eliminating "works on my machine" problems.	Docker 24.0, Apptainer 1.2
DataLad / Git-Annex	Data Versioning	Manages version control for large binary files (e.g., neuroimaging data) alongside code, tracking provenance and updates.	DataLad 0.19
ONNX (Open Neural Network Exchange)	Model Sharing	Provides an open format for sharing trained machine learning models across different frameworks (PyTorch, TensorFlow, etc.).	ONNX Runtime 1.15
fMRIPrep	Preprocessing	A robust, standardized pipeline for fMRI data preprocessing. Sharing fMRIPrep derivatives ensures identical starting data.	fMRIPrep 23.1.0
Crowdsourced Reproducibility Platforms	Replication Testing	Services like Code Ocean, or Collaborators directly re-running shared containers to confirm results before publication.	Code Ocean Capsule

Troubleshooting Guides & FAQs

Q1: I ran my neuroimaging ML model 20 times with slightly different preprocessing parameters. One configuration gave p < 0.05. Can I report just that result? A: No. This is a classic p-hacking risk. Isolated p-values from multiple, unreported comparisons are misleading. You must correct for multiple comparisons (e.g., Bonferroni, FDR) or, preferably, report the effect size with its 95% Confidence Interval (CI) for all configurations. The CI will show if the "significant" result is an outlier and if the effect is precisely estimated.

Q2: My between-group classification accuracy is 72% (p=0.03). A reviewer asked for the uncertainty around the accuracy. How do I calculate this? A: An accuracy point estimate is insufficient. You must compute a confidence interval for a proportion. For a k-fold cross-validation, use a method that accounts for data dependencies, like the percentile bootstrap. Report: "Accuracy = 72% (95% CI: 63% to 79%)." This interval may include the null hypothesis value (e.g., 50% for binary chance), prompting greater caution than the p-value alone.

Q3: I want to test if a new drug alters functional connectivity. A standard NHST (Null Hypothesis Significance Testing) gives p=0.06. How can Bayesian methods help interpret this? A: A Bayesian approach allows you to quantify evidence for both the alternative and the null hypothesis. Instead of a binary "reject/don't reject," you can compute a Bayes Factor (BF). For example, BF₁₀ = 0.8 provides weak evidence for the null. You could also report the posterior distribution of the effect size: "The mean connectivity change is 0.15 (95% Credible Interval: -0.02 to 0.31)," directly showing the most plausible values.

Q4: My voxel-wise analysis produces a "blob" of significance (p<0.001 uncorrected). How do I move from isolated voxel p-values to a robust spatial inference? A: Voxel-wise p-values are highly vulnerable to family-wise error. Implement cluster-based inference:

• Apply a primary, liberal threshold (e.g., p<0.001 uncorrected at voxel level).
• Calculate the size (number of voxels) of each contiguous cluster.
• Use permutation testing (5000+ iterations) to create a null distribution of maximum cluster sizes.
• Determine the cluster-size threshold for significance (e.g., 95th percentile of the null).
• Report only clusters surviving this corrected threshold, along with their spatial extent and peak effect size CIs.

Q5: How do I choose prior distributions for a Bayesian analysis of neuroimaging data to avoid being accused of "p-hacking with priors"? A: Use published literature or pilot data to inform weakly informative priors. For a novel effect, use conservative, heavy-tailed priors (e.g., Cauchy). Crucially, conduct a sensitivity analysis: run the analysis with a range of reasonable priors (e.g., different standard deviations). Present a table of how the key posterior statistics (e.g., 95% CrI, Bayes Factor) change. Consistency across priors demonstrates robustness.

Data Presentation

Table 1: Comparison of Inference Methods on a Simulated Neuroimaging ML Study

Method	Point Estimate (Accuracy)	Uncertainty Estimate	Includes Null (50%)?	Key Interpretation
Isolated p-value	65%	p = 0.04	N/A	"Statistically significant"
95% Confidence Interval	65%	95% CI: 51% to 76%	Yes	Effect is imprecise; null is plausible.
Bayesian (Weak Prior)	64%	95% Credible Int: 52% to 75%	Yes	Probability that accuracy >50% is 89%.
Bayesian (Informed Prior)	66%	95% Credible Int: 58% to 73%	No	Probability that accuracy >50% is 99%.

Table 2: Impact of Multiple Comparison Corrections on Voxel Count

Correction Method	Primary Threshold	Significant Voxels	False Positive Control
Uncorrected (p-hacking risk)	p < 0.001	12,450	High (Family-wise error ~100%)
Family-Wise Error (FWE)	p < 0.05 (FWE)	205	Strong (5% chance of any false positive)
False Discovery Rate (FDR)	q < 0.05	1,880	Moderate (5% of sig. voxels are false)
Cluster Extent (Permutation)	p < 0.001 + cluster p<0.05	15 clusters	Strong (controls cluster-level error)

Experimental Protocols

Protocol 1: Computing Confidence Intervals for Cross-Validated Classification Accuracy

• Train/Test Split: Perform Nested Cross-Validation (e.g., 5x5-fold). The outer loop provides performance estimates.
• Generate Performance Vector: Collect the accuracy/proportion correct from each outer test fold (n=25 values).
• Choose CI Method: For dependent data like CV, use the biased-corrected and accelerated (BCa) bootstrap. a. From your full dataset, resample with replacement (B = 5000 iterations). b. For each bootstrap sample, repeat the entire nested CV procedure. c. Compile the 5000 mean accuracy estimates.
• Calculate CI: Determine the 2.5th and 97.5th percentiles of the bootstrap distribution.
• Report: Mean accuracy and the 2.5%-97.5% BCa bootstrap CI.

Protocol 2: Bayesian Hypothesis Testing for Group Differences in Connectivity

• Define Model: Use a Bayesian linear model. For connectivity strength y: y ~ Normal(μ, σ) μ = β₀ + β₁ * group where group is a binary predictor (0=Control, 1=Patient).
• Specify Priors: β₀ ~ Normal(0, 10) // Weakly informative intercept β₁ ~ Normal(0, 2) // Conservative prior on effect σ ~ HalfCauchy(0, 5) // Weakly informative on variance
• Fit Model: Use Markov Chain Monte Carlo (MCMC) sampling (e.g., Stan, PyMC) with 4 chains, 5000 iterations each.
• Check Diagnostics: Ensure R-hat ≈ 1.0 and high effective sample size (ESS).
• Examine Posterior: Plot the posterior distribution for β₁. Calculate: a. The 95% Highest Density Credible Interval (HDI). b. The Bayes Factor (BF₁₀): Proportion of posterior mass vs. prior mass for β₁ ≠ 0. Use the Savage-Dickey density ratio or bridge sampling.
• Report: Posterior median for β₁, 95% HDI, and BF₁₀.

Mandatory Visualization

Title: Comparison of p-Hacking vs. Robust Analysis Workflows

Title: CI and Bayesian Uncertainty Quantification Protocol

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools for Robust Neuroimaging ML Inference

Tool/Reagent	Category	Function in Addressing p-Hacking
Pre-registration Template (e.g., OSF, AsPredicted)	Protocol	Pre-specifies hypotheses, methods, and analysis plan to prevent data dredging.
Nilearn, FSL, SPM	Software	Standardized neuroimaging ML & stats toolkits with built-in correction methods (FDR, FWE).
Bootstrap Resampling Code (Python: scikits-bootstrap, R: boot)	Statistical Library	Enables computation of CIs for complex, non-parametric statistics from cross-validation.
Probabilistic Programming Language (Stan, PyMC3)	Statistical Library	Implements Bayesian models to sample posterior distributions and compute credible intervals.
Cluster-Based Permutation Test Scripts (e.g., MNE-Python)	Statistical Library	Provides robust, non-parametric spatial inference correcting for multiple comparisons.
JASP or BayesFactor (R package)	Statistical Software	User-friendly interfaces for calculating Bayes Factors for common experimental designs.
Effect Size Calculator (e.g., Cohen's d, η² with CI)	Analytical Tool	Shifts focus from binary p-value to magnitude of effect with its uncertainty.

Conclusion

Addressing p-hacking in neuroimaging machine learning is not merely a statistical concern but a fundamental requirement for scientific progress and ethical translational research. By integrating foundational awareness, robust methodological safeguards, proactive troubleshooting, and rigorous comparative validation, researchers can build analyses that yield truly generalizable biomarkers. The future of neuroimaging in drug development and personalized medicine depends on this credibility. Embracing a culture of open science, pre-registration, and emphasis on effect sizes and uncertainty will shift the field from producing potentially inflated, non-replicable results to generating reliable insights that can confidently guide clinical trials and therapeutic interventions. The path forward requires tool development, education, and a collective commitment to prioritizing long-term reproducibility over short-term, publication-driven metrics.