Machine-learning phishing detectors achieve strong results when trained and tested on random splits of a single corpus, but collapse when the source or the time period changes: the PhishBench benchmarking study (Zeng et al., 2020) measured 21–25% accuracy for email classifiers transferred between the Nazario corpus and its Bluefin corpus of institutionally missed phishing emails, and subsequent work has repeatedly re-demonstrated this fragility for email, URL, and website detection. The open problem is to develop and rigorously demonstrate phishing detection approaches whose performance generalizes across diverse data sources and over temporal shifts — detectors trained on particular corpora and time periods that maintain detection performance, under leakage-free evaluation, on corpora from different sources and later periods, without relying on source-identifying or temporal dataset artifacts — and to characterize how much of the measured cross-source and temporal generalization gap can be closed and why.

Background

Phishing is a form of cybercrime in which an attacker impersonates a trustworthy entity — typically via email, a URL, or a cloned website — to trick victims into disclosing credentials or executing malicious actions; it is among the most common initial entry points for data breaches. Most academic defenses are supervised machine-learning classifiers: features are engineered from email headers and bodies (e.g., the public Nazario phishing corpus and SpamAssassin/Enron legitimate-mail archives), from URL strings (e.g., feeds such as PhishTank, OpenPhish, and the Anti-Phishing Working Group), or from website content, and a classifier is trained to separate phishing from legitimate instances. A recurring weakness of this literature, documented in the systematization of Das et al. (SoK, IEEE Communications Surveys & Tutorials 2020), is evaluation practice: models are typically validated with random train/test splits inside a single dataset, which cannot reveal whether the model has learned general phishing behavior or merely the idiosyncrasies of one corpus.

The benchmarking framework PhishBench (Zeng et al., 2020) made this gap concrete. The paper contributed diverse datasets for phishing email, URL, and website detection — including the Email Benchmark Dataset (Nazario phishing emails, including its 2015–2017 additions, plus legitimate mail from Wikileaks archives, Enron, and SpamAssassin) and Bluefin, roughly 300 phishing emails collected over 2013–2018 that evaded the authors' institutional spam filters and were therefore never caught in deployment — together with a framework for benchmarking features and classifiers under metrics suited to imbalanced data such as F1-score and MCC (the Matthews correlation coefficient, a balanced measure that remains informative when one class is much rarer than the other). Beyond benchmarking, the authors ran a cross-dataset experiment: a boosting classifier trained on Nazario (with Email Benchmark legitimate mail) achieved only 21% accuracy on Bluefin, and one trained on Bluefin achieved only 25% on Nazario, while a SpamAssassin-trained model reached 78% on Nazario — evidence that models latch onto source-specific patterns and that the more recent, filtered-email attacks in Bluefin behave differently. The authors also stress that no security dataset for phishing can ever be considered complete, because attackers continuously evolve new tactics, and that the field lacks agreed standards for dataset quality and diversity.

This is an instance of distribution shift (the training and deployment data follow different probability distributions): cross-source shift arises because corpora differ in collection mechanism, available header fields, language, and preprocessing; temporal shift, often called concept drift in the online-learning literature, arises because phishing tactics, infrastructure, and vocabulary change over time. Related mitigations studied in machine learning include domain adaptation (aligning feature distributions between a labeled source and an unlabeled target domain) and continual learning (updating a model incrementally as new data arrives while limiting forgetting of earlier knowledge). The generalization problem is made harder by corpus-specific artifacts — tokens, date patterns, and formatting quirks that let a model identify which dataset an email came from rather than whether it is phishing — and by prevalence mismatch, since benchmark datasets are often balanced while real inbox traffic contains only a tiny fraction of phishing. Later work has confirmed each of these failure modes quantitatively: phishing-URL features were shown to be dataset-dependent under explainable-AI analysis (Mia et al., 2024); detectors trained on 2018 data lost over 20% accuracy by 2020 (Ejaz et al., 2023); and leave-one-corpus-out email experiments showed that models learn highly predictive source markers and that cross-corpus transfer fails operationally (Bhuiyan & Bhuiyan, 2026). Proposed mitigations — unsupervised domain adaptation, continual learning, ensemble transfer learning, multimodal models built on BERT (Bidirectional Encoder Representations from Transformers, a pretrained language-model architecture) — yield partial, setting-specific improvements but none establishes generalizable phishing detection.

Problem Statement

Achieve generalizable phishing detection across diverse data sources and temporal shifts, as posed by the cross-dataset experiments of Zeng et al. (2020) and repeatedly re-demonstrated since: develop phishing detection approaches — for emails, URLs, and/or websites — such that a detector trained on one or more phishing corpora from particular collection sources and time periods maintains its detection performance, under leakage-free evaluation protocols, on corpora from different sources and later time periods, rather than exploiting source-identifying or temporal artifacts of its training data. Demonstrating such generalization requires evaluation under both cross-source transfer (train on one source or leave-one-source-out, test on unseen sources) and temporally ordered splits (train on earlier data, test on later data), reporting of the generalization gap relative to in-distribution baselines using imbalance-appropriate metrics (e.g., F1, MCC, precision/recall under realistic prevalence), and evidence, such as artifact or feature-attribution analysis, that the detector is not merely recognizing the source corpus. The task additionally requires a rigorous empirical characterization that quantifies how much of the measured generalization gap (including the 21–25% cross-source accuracy observed by Zeng et al.) is attributable to identifiable causes — corpus artifacts, prevalence mismatch, temporal concept drift — and how much of it each mitigation closes.

References

  1. Victor Zeng, Shahryar Baki, Ayman El Aassal, Rakesh Verma, Luis Felipe Teixeira De Moraes, Avisha Das. Diverse Datasets and a Customizable Benchmarking Framework for Phishing. Proceedings of the Sixth International Workshop on Security and Privacy Analytics (WOSPA '20, co-located with CODASPY '20), pp. 35–41, 2020. DOI: 10.1145/3375708.3380313. https://doi.org/10.1145/3375708.3380313
  2. Avisha Das, Shahryar Baki, Ayman El Aassal, Rakesh Verma, Arthur Dunbar. SoK: A Comprehensive Reexamination of Phishing Research From the Security Perspective. IEEE Communications Surveys & Tutorials, 2020. DOI: 10.1109/COMST.2019.2957750. https://doi.org/10.1109/COMST.2019.2957750
  3. Asif Ejaz, Adnan Noor Mian, Sanaullah Manzoor. Life-long phishing attack detection using continual learning. Scientific Reports, 2023. DOI: 10.1038/s41598-023-37552-9. https://doi.org/10.1038/s41598-023-37552-9
  4. Fariza Rashid, Ben Doyle, Soyeon Caren Han, Suranga Seneviratne. Phishing URL detection generalisation using Unsupervised Domain Adaptation. Computer Networks, 2024. DOI: 10.1016/j.comnet.2024.110398. https://doi.org/10.1016/j.comnet.2024.110398
  5. Maraz Mia, Darius Derakhshan, Mir Mehedi Ahsan Pritom. Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI. Proceedings of the 11th International Conference on Networking, Systems, and Security (ICNSS 2024), 2024. DOI: 10.1145/3704522.3704532. https://doi.org/10.1145/3704522.3704532
  6. Gabriel Loiseau, Valentin Lefils, Maxime Meyer, Damien Riquet. WikiPhish: A Diverse Wikipedia-Based Dataset for Phishing Website Detection: Data/Toolset Paper. Proceedings of the Fourteenth ACM Conference on Data and Application Security and Privacy (CODASPY '24), pp. 361–366, 2024. DOI: 10.1145/3626232.3653283. https://doi.org/10.1145/3626232.3653283
  7. Yi Wei, Masaya Nakayama, Yuji Sekiya. Enhancing Generalization in Phishing URL Detection via a Fine-Tuned BERT-Based Multimodal Approach. IEEE Access, 2025. DOI: 10.1109/ACCESS.2025.3591843. https://doi.org/10.1109/ACCESS.2025.3591843
  8. Istiaque Bhuiyan, Tanvir Bhuiyan. The Fragility of Phishing Detection Models: Evidence from Cross-Corpus Transfer, Prevalence Shift, Artifact Learning, and Evasion Risk. Big Data and Cognitive Computing, 10(7):211, 2026. DOI: 10.3390/bdcc10070211. https://doi.org/10.3390/bdcc10070211
  9. Saviour Henry, Bliss Utibe-Abasi Stephen, Philip Asuquo. Ensemble transfer learning for cross-dataset phishing detection. Franklin Open, 2026. DOI: 10.1016/j.fraope.2026.100579. https://doi.org/10.1016/j.fraope.2026.100579