Machine-learning phishing email detectors routinely report near-ceiling benchmark scores, yet recent evidence shows that these scores can collapse under realistic deployment: models trained on one email corpus transfer unstably to other corpora, fail asymmetrically when the phishing prevalence (base rate) shifts between training and deployment, and partly rely on corpus-identity artifacts rather than genuine phishing signal. The open problem is to develop and validate learning methods and training regimes that make email phishing detection deployment-robust in a jointly verifiable sense: stable transfer to email corpora unseen during training, calibrated and prevalence-robust ranking behavior including under operational alert-budget constraints, and demonstrated independence from source-corpus artifacts. These criteria were established empirically by Bhuiyan & Bhuiyan (2026); the cross-dataset experiments of the PhishBench benchmarking framework independently corroborate the cross-source generalization failure mode. As of mid-2026 no published method jointly satisfies the three conditions for email corpora; adjacent methodological progress exists only for URL-based phishing detection, and the 2025 E-PhishGen assessment confirms that cross-source generalization of email detectors remains unresolved.

Background

Phishing emails — messages impersonating trusted parties to induce credential disclosure, malicious clicks, or fraudulent payments — are most commonly countered by statistical text classifiers, and a large literature reports near-perfect accuracy for such detectors. Those benchmark numbers, however, are usually obtained by randomly splitting a single pooled corpus into training and test sets, which shares the corpus's distribution, formatting conventions, and class balance between training and evaluation.

The standard baselines in this literature are linear classifiers over bag-of-words features: term frequency–inverse document frequency (TF-IDF) weighting converts an email into a vector of word scores, where common words are down-weighted, and logistic regression or a linear support vector classifier (SVC) then learns a separating hyperplane between phishing and legitimate mail. The relevant deployment realities are threefold. First, cross-corpus transfer: a model trained on one email corpus (e.g., CEAS_08, Enron, Ling, Nazario, Nigerian Fraud, or SpamAssassin, the six public corpora used by Bhuiyan & Bhuiyan in 2026) may be applied to mail collected elsewhere, and its F1 score — the harmonic mean of precision (the fraction of flagged emails that are truly phishing) and recall (the fraction of phishing emails actually flagged) — varies substantially across source–target pairs. Second, prevalence shift: the phishing base rate (the fraction of phishing emails in the stream) at deployment generally differs from the training prevalence, and detection quality then depends on the score threshold; under operational alert budgets — when analysts can investigate only the top few percent of risk-scored emails — a high-prevalence-trained model's recall can collapse, while a low-prevalence-trained model becomes overly conservative (high precision, poor recall) in high-threat environments. Calibration here means that the detector's output probabilities actually match observed error frequencies under the shifted base rate; miscalibrated scores make fixed thresholds misleading. Third, artifact (shortcut) learning: because each corpus has its own collection process, time period, headers, and formatting, corpus identity is highly learnable from email text (Bhuiyan & Bhuiyan report dataset-identification accuracies of 0.97–0.98), and top-feature inspection shows detectors leaning on date tokens, corpus markers, and URL/domain terms rather than general phishing indicators — spurious correlations that inflate benchmark scores while weakening transfer.

The evidence base for these failure modes is recent but convergent. The PhishBench framework and datasets (Zeng et al., 2020) were built precisely because prior evaluations used limited datasets and metrics, and its cross-dataset experiments found that models trained on one source perform poorly on another, framing generalizable detection across sources and temporal shifts as open. Bhuiyan & Bhuiyan (2026) systematized the three failure modes across the six public corpora, showing that leave-one-corpus-out pooled training (training on all corpora but the test one) improves but does not eliminate instability (F1 roughly 0.76–0.90 across unseen corpora), that prevalence-shift failure is asymmetric and threshold-dependent, and that artifact-removal ablation and targeted masking of high-weight features perturb detection. Independent confirmation that cross-source generalization is unresolved comes from E-PhishGen (Pajola et al., 2025), which re-implemented machine-learning and large-language-model detectors, observed near-perfect performance only when trained and tested on the same dataset, and showed far weaker performance on new LLM-generated phishing corpora, concluding that phishing email detection is still an open problem. Adjacent methodological progress exists for URL-based (not email) phishing: unsupervised domain adaptation, which aligns training and target data distributions without labeled target examples (Rashid et al., 2024), ensemble transfer learning that harmonizes disjoint feature spaces (Henry et al., 2026), and multimodal approaches that fine-tune BERT, a pre-trained transformer language model (Wei et al., 2025), while Mia et al. (2024) showed with explainable-AI analysis — methods that attribute a model's decisions to its input features — that URL features themselves are not trustworthy across datasets. None of these addresses email corpora, prevalence-shift calibration, or corpus-artifact elimination jointly, and the trustworthiness-calibration evaluation framework of Ganiuly & Smaiyl (2025) for LLM-based email detectors measures calibration and cross-dataset stability without providing a method that achieves them. Bhuiyan & Bhuiyan (2026, Section 5.6) list as future research directions whether calibration methods such as Platt scaling, isotonic regression, temperature scaling, or prevalence-adjusted thresholding reduce deployment failure, whether transformer-based and large-language-model-based detectors behave better under the same failure-mode framework, hybrid defenses that combine pooled multi-corpus training, artifact-reduced preprocessing, prevalence-aware threshold selection, and calibrated probability outputs with ensemble models, and standard cross-dataset evaluation protocols.

Problem Statement

Develop and validate learning methods and training regimes for phishing email detection that are deployment-robust in the following jointly verifiable sense, established as open by Bhuiyan & Bhuiyan (2026, Sections 5.4–5.7), with the cross-source generalization failure mode independently corroborated by PhishBench (Zeng et al., 2020):

  1. Cross-corpus generalization. Trained on one or more public email corpora (from CEAS_08, Enron, Ling, Nazario, Nigerian Fraud, SpamAssassin), the detector must maintain stable precision, recall, and F1 (the harmonic mean of precision and recall) on held-out corpora unseen during training — not merely on random pooled train–test splits — including under leave-one-corpus-out evaluation. Following the source's design, held-out targets are the binary-testable corpora (CEAS_08, Enron, Ling, SpamAssassin), each evaluated after training on the remaining corpora; the phishing-only corpora (Nazario, Nigerian Fraud) serve as phishing-positive training sources — with sensitivity analysis over their inclusion — and are not held-out binary test targets.

  2. Prevalence-robust calibrated behavior. When the phishing base rate at deployment differs from the training prevalence, the detector's scores must remain calibrated (output probabilities that match observed error frequencies), or its thresholds must be demonstrably adjustable to the deployment prevalence, and it must retain usable recall under operational alert-budget constraints (e.g., when only the top 1%, 5%, or 10% risk-scored emails can be investigated) — avoiding both documented asymmetric failure modes (recall collapse of high-prevalence-trained models under alert budgets, and conservative under-detection — high precision, low recall — by low-prevalence-trained models in high-threat environments).

  3. Artifact independence. The detector's decisions must be demonstrably not driven by source-corpus identity artifacts (corpus markers, date tokens, header and formatting conventions), assessed with the source's artifact diagnostics: dataset-identification probes (classifiers trained to recover the source corpus from the detector's features; high probe accuracy indicates residual corpus-identity signal) on its feature space, top-feature inspection, artifact-removal ablation, and targeted feature-masking sensitivity analysis. The source supplies no numeric acceptance thresholds for these diagnostics — no maximum probe accuracy, no masking-sensitivity bound — and cautions that high dataset-identification accuracy alone does not prove that artifacts cause poor generalization and that masking results are diagnostic evidence of feature reliance; establishing quantitative artifact-independence criteria is therefore itself part of the open evaluation-protocol task below.

The problem comprises two parts. The first is to develop and validate a method or training regime that meets the three conditions above; the second is to develop a standard evaluation protocol for cross-dataset phishing detection — pooled baseline testing, single-corpus transfer, leave-one-corpus-out testing, prevalence-shift simulation, alert-budget evaluation, artifact analysis, and adversarial robustness testing — under which such claims are tested. The source paper contributes its own deployment-aware, adversary-aware evaluation framework but leaves the standardized protocol as future work. The source states no numeric acceptance boundaries for predicates such as 'stable' or 'usable recall' — it describes leave-one-corpus-out F1 of 0.8201–0.8994 as 'sustained' yet insufficient because pooled training 'does not remove all deployment risks' — so proposing and justifying such criteria is part of the open problem rather than a supplied boundary.

References

  1. Istiaque Bhuiyan and Tanvir Bhuiyan, 'The Fragility of Phishing Detection Models: Evidence from Cross-Corpus Transfer, Prevalence Shift, Artifact Learning, and Evasion Risk', Big Data and Cognitive Computing 10(7):211, 2026. DOI 10.3390/bdcc10070211, https://doi.org/10.3390/bdcc10070211
  2. Victor Zeng, Shahryar Baki, Ayman El Aassal, Rakesh Verma, Luis Felipe Teixeira De Moraes, and Avisha Das, 'Diverse Datasets and a Customizable Benchmarking Framework for Phishing', Proceedings of the Sixth International Workshop on Security and Privacy Analytics (IWSPA), pp. 35–41, 2020. DOI 10.1145/3375708.3380313, https://doi.org/10.1145/3375708.3380313
  3. Luca Pajola, Eugenio Caripoti, Stefan Banzer, Simeone Pizzi, Mauro Conti, and Giovanni Apruzzese, 'E-PhishGEN: Unlocking Novel Research in Phishing Email Detection', Proceedings of the 18th ACM Workshop on Artificial Intelligence and Security (AISec), pp. 64–76, 2025. DOI 10.1145/3733799.3762967, https://doi.org/10.1145/3733799.3762967 (arXiv 2509.01791, https://arxiv.org/abs/2509.01791)
  4. Fariza Rashid, Ben Doyle, Soyeon Caren Han, and Suranga Seneviratne, 'Phishing URL detection generalisation using Unsupervised Domain Adaptation', Computer Networks 245:110398, 2024. DOI 10.1016/j.comnet.2024.110398, https://doi.org/10.1016/j.comnet.2024.110398
  5. Saviour Henry, Bliss Utibe-Abasi Stephen, and Philip Asuquo, 'Ensemble transfer learning for cross-dataset phishing detection', Franklin Open 15:100579, 2026. DOI 10.1016/j.fraope.2026.100579, https://doi.org/10.1016/j.fraope.2026.100579
  6. Yi Wei, Masaya Nakayama, and Yuji Sekiya, 'Enhancing Generalization in Phishing URL Detection via a Fine-Tuned BERT-Based Multimodal Approach', IEEE Access 13:131197–131216, 2025. DOI 10.1109/access.2025.3591843, https://doi.org/10.1109/access.2025.3591843
  7. Daniyal Ganiuly and Assel Smaiyl, 'Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models', arXiv 2511.04728, 2025, https://arxiv.org/abs/2511.04728
  8. Maraz Mia, Darius Derakhshan, and Mir Mehedi Ahsan Pritom, 'Can Features for Phishing URL Detection Be Trusted Across Diverse Datasets? A Case Study with Explainable AI', Proceedings of the 11th International Conference on Networking, Systems, and Security (NSysS), pp. 137–145, 2024. DOI 10.1145/3704522.3704532, https://doi.org/10.1145/3704522.3704532