Introduction
Feature selection is a crucial step in building machine learning models, as irrelevant or redundant features can hinder model performance. In this blog post, we will explore two essential feature selection methods and apply them to a real-world dataset: eliminating low variance features and recursive feature elimination using cross-validation.
Eliminating Low Variance Features:
One of the initial steps in feature selection involves identifying and removing features with low variance. These features are less informative as they contain a high fraction of the same values. In our example, we use the VarianceThreshold function from scikit-learn to eliminate features with variance below a specified threshold (in this case, 80%).
# Define the variance threshold and fit it to the feature array
p = 0.8
th = p * (1-p)
sel = fs.VarianceThreshold(threshold=th)
Features_reduced = sel.fit_transform(Features)
Selecting k Best Features Using Cross-Validation:
After eliminating low variance features, the next step is to determine the importance of the remaining features. We use recursive feature elimination with cross-validation (RFECV), a robust method that recursively removes features and evaluates model performance through cross-validation. AUC (area under the curve) is used as the model selection metric due to imbalanced labels.
# Define the logistic regression model
logistic_mod = linear_model.LogisticRegression(C = 10, class_weight = {0:0.45, 1:0.55})
# Apply RFECV to determine which features to retain
selector = fs.RFECV(estimator=logistic_mod, cv=feature_folds, scoring='roc_auc')
selector = selector.fit(Features_reduced, Labels)
The resulting support shows which features are selected or eliminated. In our case, 16 features are retained.
selector.support_
Further analysis reveals the relative ranking of features.
selector.ranking_
The number of features is reduced to 16, indicating the removal of two unimportant features.Nested Cross-Validation for Model Optimization:
With the selected features, nested cross-validation is employed to optimize the logistic regression model’s hyperparameters. The inside folds are used for grid search to find the best hyperparameter values.
# Perform the grid search over the parameters
clf = ms.GridSearchCV(estimator=logistic_mod, param_grid=param_grid, cv=inside, scoring='accuracy', return_train_score=True)
The optimal hyperparameter values are obtained, and the outer loop of cross-validation is performed to assess overall model performance.
cv_estimate = ms.cross_val_score(clf, Features, Labels, cv=outside)
The mean performance metric provides an estimate of the model’s generalization performance.
Model Testing and Evaluation:
The final step involves testing the model on an independent dataset. The logistic regression model is trained using the optimal hyperparameter values and the reduced feature subset.
logistic_mod = linear_model.LogisticRegression(C=100, class_weight={0:0.6, 1:0.4})
logistic_mod.fit(X_train, y_train)Model scores and probabilities are calculated for the test set, and performance metrics are displayed, including confusion matrix, accuracy, AUC, precision, recall, and F1 score.
print_metrics(y_test, probabilities, 0.3)
plot_auc(y_test, probabilities)
Conclusion:
Feature selection is a critical aspect of building machine learning models, and effective techniques can significantly improve model performance. By eliminating low variance features and using recursive feature elimination with cross-validation, we can identify and retain the most informative features. The nested cross-validation process helps optimize model hyperparameters, ensuring robust and generalized performance. Testing the model on an independent dataset provides a realistic evaluation of its capabilities. Remember that thoughtful feature selection is essential for creating accurate and efficient machine learning models.
Leave a Reply