The Impact of Feature Selection on Machine Learning Model Performance

Feature selection can reduce input dimensionality and improve interpretability, but its effect on predictive performance depends on the data and learning algorithm. This study experimentally compares three classifiers Logistic Regression, Support Vector Machine (SVM) with a radial basis function kernel, and Random Forest under three feature configurations using the Wisconsin Breast Cancer Diagnostic dataset. A stratified five-fold cross-validation design was used, with scaling and feature selection fitted only within each training fold to reduce data leakage. The conditions included all 30 predictors, the 10 highest-ranked predictors by ANOVA F-score, and the 10 highest-ranked predictors by mutual information. Mean accuracy, precision, recall, F1-score, and ROC-AUC were recorded, treating malignant cases as the positive class. SVM with all features achieved the highest mean accuracy (97.72%) and F1-score (0.9688), while Logistic Regression with all features achieved the highest ROC-AUC (0.9953). Reducing the input to 10 features lowered mean accuracy for all three classifiers, although the magnitude of the change varied by model. The findings indicate that feature selection should be evaluated jointly with the classifier and the intended performance objective rather than assumed to improve prediction automatically. Because the experiment uses one benchmark dataset, the results are methodological evidence rather than clinical validation.