Bug prediction using decision trees, random forests and classifiers.

Overview

In this project, we explored the implementation and evaluation of classification models, including decision trees, bagging, and random forests. We implemented these algorithms using Python, and subsequently analyzed the Eclipse bug dataset for software bug prediction.

The project consisted of two parts: programming and data analysis. In the programming segment, key functions were implemented to construct and utilize classification trees and ensemble methods. During analysis, we apply these functions to a real-world dataset to extract insights and evaluate model performance.

Part 1: Programming

For the first phase, we implemented modular functions to construct classification trees and build ensemble models, in this case bagging and random forests. We designed the functions in such a way that offered flexibility, allowing different parameters for different use cases.

Key Functionalities

• Tree Construction: The tree_grow function creates a decision tree using the Gini index to evaluate split quality.
• Prediction: The tree_pred function predicts class labels for new data based on the constructed tree.
• Bagging: The tree_grow_b function builds an ensemble of trees on bootstrap samples, while tree_pred_b aggregates predictions using majority voting.
• Random Forests: The implementation incorporates feature subsetting, selecting a random subset of predictors at each split to enhance model diversity.

Part 2: Data Analysis

We used the Eclipse bug dataset to test the implemented models. The dataset included software metrics and bug information at the package level, with the goal of predicting whether a package has post-release bugs. Training data was drawn from Eclipse release 2.0, while release 3.0 served as the test set.

Analytical Approach

• A single classification tree was trained using parameters chosen to balance performance and interpretability.
• Bagging was applied to create an ensemble of trees, improving robustness against overfitting.
• A random forest model was built, leveraging feature subsetting to enhance generalization.

Key metrics such as accuracy, precision, and recall were used to evaluate model performance. Statistical testing was conducted to assess the significance of observed differences between model results.

Results and discussion

The Random Forest model outperformed the Single Classification Tree and Bagging models in terms of overall performance. This was in line with expectations, as Random Forest is an ensemble method that aggregates the results of multiple decision trees, each trained on a random subset of the data. This reduces variance and overfitting, leading to improved generalization and a stronger ability to balance precision and recall. While Bagging shows higher precision, its lower recall than Random Forest suggests it may not capture as many defect cases. To properly test if these differences in precision and recall are significant, an additional ANOVA and Tukey test would need to be done for these metrics.

In contrast, the Single Classification Tree model struggled to maintain a balance between precision and recall, missing a significant number of defect cases.

As seen in the figure below, we created a visualisation that shows the first splits in the single tree model. In the first leaf node NSM max, we see that software that has a low number of sub-modules, indicating the code is less complex, is reasonably classified as 'no bugs'. In the second leaf VG max, the majority of cases are classified as 'bugs'. This classification is also reasonable in line with the notion that complex software is more prone to error. The third leaf NBD sum classifies software with a moderate sum of block depth as containing bugs. This seems intuitive as well, considering that nesting code brings higher complexity. Lastly, the fourth leaf pre classifies the software as containing bugs when the number of pre-release bugs is relatively high (but ≤ 11.5). Overall, given the attributes involved, the classification rules seem reasonable and align with general expectations of software complexity.

Figure: First three splits in the single decision tree model