Heterogeneous defect prediction (HDP) transfers a defect-prediction model from a source project to a target project whose software metrics differ, by statistically matching metrics with similar value distributions. The method rests on an unverified assumption: that a distribution-similar matched pair preserves the direction (tendency) of the relationship between metric values and defect-proneness. When the tendency is reversed between source and target, matched features act as inverted predictors and transfer performance can fall below chance. Nam, Fu, Kim, Menzies, and Tan (IEEE TSE 2018) left as future work the design of an automatic filter that detects and discards such 'noisy' metric matches without access to target labels; separately, in their threats-to-validity discussion, they left as future work identifying a proper prediction-probability threshold that would allow within-project and heterogeneous models to be fairly compared in precision and recall. No existing method certifies or filters tendency-reversed matches using no target labels: later work either replaces interpretable metric matching with latent-space alignment or refines purely distribution-based matching, which cannot determine tendency direction from unlabeled data. The problem remains open.

Background

Software defect prediction (SDP) trains machine-learning classifiers on module-level measurements of a code base (software metrics, e.g. lines of code or static-analysis counts) to flag which modules are likely defective. When a project lacks labeled history, cross-project defect prediction (CPDP) reuses labeled data from another project; most CPDP methods require the source and target projects to record exactly the same metrics. Heterogeneous defect prediction (HDP) removes that requirement: Nam, Fu, Kim, Menzies, and Tan (IEEE TSE 44(9):874-896, 2018) select the most informative source metrics (by Gain Ratio, an information-gain-based feature-selection criterion), measure the similarity of each source-target metric pair with the two-sample Kolmogorov-Smirnov (KS) test, a nonparametric test of whether two samples come from the same distribution, discard pairs below a cutoff (p-value 0.05), and solve a maximum weighted bipartite matching (an assignment problem pairing source metrics to target metrics maximizing total similarity) to obtain a matched metric set on which a logistic-regression classifier is trained on source data and applied to the target. Evaluated on 34 projects from five dataset groups (AEEEM, ReLink, MORPH, NASA, SOFTLAB) covering 962 source-target combinations, HDP reached a median AUC of 0.711 versus 0.732 for within-project prediction (AUC, the area under the receiver operating characteristic curve, measures ranking quality independent of any decision threshold).

HDP's transfer is only as good as its matches. Distribution similarity says nothing about the sign of the relation between a metric and the defect label: in the source project larger values of a matched metric may mark defect-prone modules, while in the target project the paired metric's larger values may mark clean modules. In such 'loss cases' the transferred classifier systematically inverts its ranking, and the authors show AUC can drop below the 0.5 random baseline. Because the whole premise of CPDP is that the target project has no (or almost no) labeled data - in the source's words, labels of target instances cannot be known in advance - a repair filter cannot test tendency direction on the target directly, since tendency is a property relating metrics to unavailable labels. Two further considerations shape the problem. First, the original evaluation used only the threshold-free AUC measure; reporting deployable precision and recall requires a decision threshold on predicted probabilities, and identifying a proper prediction-probability threshold that would allow within-project and heterogeneous models to be fairly compared in precision and recall was posed by the source as a separate future-work item. Second, later replication studies sharpened the stakes: Chen, Mu, Liu, Cui, and Ni (Information and Software Technology 130:106441, 2021) showed that five state-of-the-art HDP methods do not significantly outperform simple unsupervised baselines, and that the original matching procedure fails to find any matched metrics in 339 of 962 combinations; Jahanshahi, Cevik, and Basar (2020/2021) additionally documented severe sensitivity of feasibility to parameter choice. Predecessors and successors built alternatives - CCA-based transfer (Jing, Wu, Dong, Qi, and Xu, ESEC/FSE 2015), cost-sensitive transfer kernel canonical correlation analysis (Li et al., Automated Software Engineering 2017), where CCA (canonical correlation analysis) finds linear combinations of two feature sets that are maximally correlated, and kernelized variants extend this to nonlinear relations - but none addresses the question of when a matched pair is actively misleading.

Problem Statement

Given a source project with labeled modules and a target project whose metric set differs from the source's and whose modules are entirely unlabeled, devise and validate an automatic method that determines which statistically matched source-target metric pairs carry opposite defect-proneness tendencies - that is, pairs where larger values of the source metric mark defect-prone modules while larger values of the matched target metric mark clean modules, or vice versa, what the source calls a 'noisy metric matching' - and discards or repairs those misleading matches, so that the Loss results the source attributes to noisy metric matching - prediction combinations in which the transferred HDP model performs significantly worse than baselines such as within-project prediction - are minimized, without collapsing the matching: the retained matched set must still support transfer performance at least on par with the original HDP matching (as originally posed by Nam, Fu, Kim, Menzies, and Tan, IEEE TSE 2018). Both parts of the method are mandatory - an explicit label-free criterion that decides which matched pairs carry opposite tendencies, and the discard-or-repair treatment it applies to those pairs - and its validation must be comparative, conducted on heterogeneous source-target project pairs by comparing the filtered matching with the unfiltered HDP matching. Consistent with the source's formulation of the problem, the only information constraint is that labels of target instances cannot be known in advance: the method may not use any target defect labels at matching, filtering, or model-building time. The source separately lists, as an independent future-work item in its threats-to-validity discussion, the identification of a proper prediction-probability threshold that would allow within-project and heterogeneous models to be fairly compared in precision and recall; that item is noted here as background and is not part of the target question.

References

  1. Jaechang Nam, Wei Fu, Sunghun Kim, Tim Menzies, Lin Tan. Heterogeneous Defect Prediction. IEEE Transactions on Software Engineering 44(9):874-896, 2018. DOI 10.1109/TSE.2017.2720603. https://doi.org/10.1109/TSE.2017.2720603 (accepted author manuscript: https://timm.github.io/old/pdf/17hdp.pdf)
  2. Jaechang Nam, Sunghun Kim. Heterogeneous defect prediction. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), pp. 508-519. DOI 10.1145/2786805.2786814. https://doi.org/10.1145/2786805.2786814
  3. Xiaoyuan Jing, Fei Wu, Xiwei Dong, Fumin Qi, Baowen Xu. Heterogeneous cross-company defect prediction by unified metric representation and CCA-based transfer learning. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), pp. 496-507. DOI 10.1145/2786805.2786813. https://doi.org/10.1145/2786805.2786813
  4. Zhiqiang Li, Xiao-Yuan Jing, Fei Wu, Xiaoke Zhu, Baowen Xu, Shi Ying. Cost-sensitive transfer kernel canonical correlation analysis for heterogeneous defect prediction. Automated Software Engineering 25(2):201-245, 2018. DOI 10.1007/s10515-017-0220-7. https://doi.org/10.1007/s10515-017-0220-7
  5. Zhiqiang Li, Xiao-Yuan Jing, Xiaoke Zhu, Hongyu Zhang. Heterogeneous Defect Prediction Through Multiple Kernel Learning and Ensemble Learning. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 91-102. DOI 10.1109/ICSME.2017.19. https://doi.org/10.1109/ICSME.2017.19
  6. Zhiqiang Li, Xiao-Yuan Jing, Xiaoke Zhu, Hongyu Zhang, Baowen Xu, Shi Ying. On the Multiple Sources and Privacy Preservation Issues for Heterogeneous Defect Prediction. IEEE Transactions on Software Engineering 45(4):391-411, 2019. DOI 10.1109/TSE.2017.2780222. https://doi.org/10.1109/TSE.2017.2780222
  7. Zhiqiang Li, Xiao-Yuan Jing, Xiaoke Zhu, Hongyu Zhang, Baowen Xu, Shi Ying. Heterogeneous defect prediction with two-stage ensemble learning. Automated Software Engineering 26(3):599-651, 2019. DOI 10.1007/s10515-019-00259-1. https://doi.org/10.1007/s10515-019-00259-1
  8. Zhou Xu, Peipei Yuan, Tao Zhang, Yutian Tang, Shuai Li, Zhen Xia. HDA: Cross-Project Defect Prediction via Heterogeneous Domain Adaptation With Dictionary Learning. IEEE Access 6:57597-57613, 2018. DOI 10.1109/ACCESS.2018.2873755. https://doi.org/10.1109/ACCESS.2018.2873755
  9. Xu Zhou, Sizhe Ye, Tao Zhang, Zhen Xia, Shuai Pang, Yong Wang, Yutian Tang. MVSE: Effort-Aware Heterogeneous Defect Prediction via Multiple-View Spectral Embedding. In 2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS), pp. 10-17. DOI 10.1109/QRS.2019.00015. https://doi.org/10.1109/QRS.2019.00015
  10. Xiang Chen, Yanzhou Mu, Ke Liu, Zhanqi Cui, Chao Ni. Revisiting heterogeneous defect prediction methods: How far are we? Information and Software Technology 130:106441, 2021. DOI 10.1016/j.infsof.2020.106441. https://doi.org/10.1016/j.infsof.2020.106441 (preprint: arXiv 1908.06560, https://arxiv.org/abs/1908.06560)
  11. Hadi Jahanshahi, Mucahit Cevik, Ayse Basar. Moving from Cross-Project Defect Prediction to Heterogeneous Defect Prediction: A Partial Replication Study. CASCON 2020; arXiv 2103.03490, 2021. DOI 10.48550/arXiv.2103.03490. https://arxiv.org/abs/2103.03490
  12. Haowen Chen, Xiao-Yuan Jing, Zhiqiang Li, Di Wu, Yi Peng, Zhiguo Huang. An Empirical Study on Heterogeneous Defect Prediction Approaches. IEEE Transactions on Software Engineering 47(12):2803-2822, 2021. DOI 10.1109/TSE.2020.2968520. https://doi.org/10.1109/TSE.2020.2968520
  13. Haowen Chen, Xiao-Yuan Jing, Baowen Xu. Heterogeneous Defect Prediction through Joint Metric Selection and Matching. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), pp. 367-377. DOI 10.1109/QRS54544.2021.00048. https://doi.org/10.1109/QRS54544.2021.00048
  14. Haowen Chen, Xiao-Yuan Jing, Yuming Zhou, Bing Li, Baowen Xu. Aligned metric representation based balanced multiset ensemble learning for heterogeneous defect prediction. Information and Software Technology 147:106892, 2022. DOI 10.1016/j.infsof.2022.106892. https://doi.org/10.1016/j.infsof.2022.106892
  15. Pravas Ranjan Bal, Sandeep Kumar. A Data Transfer and Relevant Metrics Matching Based Approach for Heterogeneous Defect Prediction. IEEE Transactions on Software Engineering 49(3):1232-1245, 2023. DOI 10.1109/TSE.2022.3173678. https://doi.org/10.1109/TSE.2022.3173678
  16. Yuwei Zhang, Dahai Jin, Ying Xing, Yunzhan Gong. Automated defect identification via path analysis-based features with transfer learning. Journal of Systems and Software 166:110585, 2020. DOI 10.1016/j.jss.2020.110585. https://doi.org/10.1016/j.jss.2020.110585
  17. Ayse Tosun, Ayse Bener. Reducing false alarms in software defect prediction by decision threshold optimization. In 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 477-480, 2009. DOI 10.1109/ESEM.2009.5316006. https://doi.org/10.1109/ESEM.2009.5316006
  18. Shensi Tong, Qing He, Yuting Chen, Ye Yang, Beijun Shen. Heterogeneous Cross-Company Effort Estimation through Transfer Learning. In 23rd Asia-Pacific Software Engineering Conference (APSEC), pp. 169-176, 2016. DOI 10.1109/APSEC.2016.033. https://doi.org/10.1109/APSEC.2016.033
  19. Xiaoning Shen, Jiaqi Lu, Shuxian Li, Liyan Song. Feature mapping based on heterogeneous cross-company effort estimation. Software Quality Journal 32(4):1717-1761, 2024. DOI 10.1007/s11219-024-09697-x. https://doi.org/10.1007/s11219-024-09697-x
  20. Yujie Ren, Bin Liu, Shihai Wang. Joint Instance and Feature Adaptation for Heterogeneous Defect Prediction. IEEE Transactions on Reliability 73(1):741-756, 2024. DOI 10.1109/TR.2023.3305356. https://doi.org/10.1109/TR.2023.3305356
  21. Radowanul Haque, Aftab Ali, Sally I. McClean, Ian Cleland, Joost Noppen. Heterogeneous Cross-Project Defect Prediction Using Encoder Networks and Transfer Learning. IEEE Access 12:409-419, 2024. DOI 10.1109/ACCESS.2023.3343329. https://doi.org/10.1109/ACCESS.2023.3343329
  22. Xing Zong, Guiyu Li, Shang Zheng, Haitao Zou, Hualong Yu, Shang Gao. Heterogeneous Cross-Project Defect Prediction via Optimal Transport. IEEE Access 11:12015-12030, 2023. DOI 10.1109/ACCESS.2023.3241924. https://doi.org/10.1109/ACCESS.2023.3241924
  23. Meetesh Nevendra, Pradeep Singh. Meta network attention-based feature matching for heterogeneous defect prediction. Automated Software Engineering 32(1), 2024. DOI 10.1007/s10515-024-00480-7. https://doi.org/10.1007/s10515-024-00480-7
  24. Xu Yu, Jiaqi Yan, Qinqin Gao, Qinglong Peng, Bin Yu, Junwei Du, Ying Xing, Dunwei Gong. Feature Disentanglement-Based Heterogeneous Defect Prediction. ACM Transactions on Software Engineering and Methodology 35(4), 2026. DOI 10.1145/3742474. https://doi.org/10.1145/3742474
  25. Xiao-Yuan Jing, Haowen Chen, Baowen Xu. Heterogeneous Defect Prediction. In Intelligent Software Defect Prediction (Springer), pp. 65-138, 2023. DOI 10.1007/978-981-99-2842-2_5. https://doi.org/10.1007/978-981-99-2842-2_5
  26. Osayand P. Omondiagbe, Sherlock A. Licorish, Stephen G. MacDonell. Negative Transfer in Cross Project Defect Prediction: Effect of Domain Divergence. In 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 1-4, 2022. DOI 10.1109/SEAA56994.2022.00010. https://doi.org/10.1109/SEAA56994.2022.00010