.. _library_c45_classifier:

c45_classifier

This library implements the C4.5 decision tree learning algorithm. C4.5 is an extension of the ID3 algorithm that uses information gain ratio instead of information gain for attribute selection, which avoids bias towards attributes with many values (see below for implementation details).

The library implements the classifier_protocol defined in the classification_protocols library. It provides predicates for learning a decision tree from a dataset, optionally prune it, using it to make predictions, and exporting it as a list of predicate clauses or to a file.

Datasets are represented as objects implementing the dataset_protocol protocol from the classification_protocols library. See test_files directory for examples.

API documentation

Open the `../../apis/library_index.html#c45_classifier <../../apis/library_index.html#c45_classifier>`__ link in a web browser.

Loading

To load all entities in this library, load the loader.lgt file:

::

| ?- logtalk_load(c45_classifier(loader)).

Testing

To test this library predicates, load the tester.lgt file:

::

| ?- logtalk_load(c45_classifier(tester)).

Implemented features

• Information gain ratio for attribute selection (avoids bias towards attributes with many values)
• Handling of discrete (categorical) attributes with multi-way splits
• Handling of continuous (numeric) attributes with binary threshold splits (selects the threshold with the highest gain ratio from midpoints between consecutive sorted values)
• Handling of missing attribute values (represented using anonymous variables): examples with missing values for an attribute are distributed to all branches during tree construction; gain ratio computation uses only examples with known values for the attribute being evaluated; prediction with missing values uses majority voting by exploring all possible branches and selecting the most common class
• Tree pruning using pessimistic error pruning (PEP): estimates error rates using the upper confidence bound of the binomial distribution (Wilson score interval) with a configurable confidence factor (default 0.25) and minimum instances per leaf (default 2); replaces subtrees with leaf nodes when doing so would not increase the estimated error; helps reduce overfitting and improve generalization
• Export of learned decision trees as predicate clauses
• Pretty-printing of learned decision trees

Learned tree representation

For discrete attributes, the learned decision tree is represented as leaf(Class) for leaf nodes and tree(Attribute, Subtrees) for internal nodes with discrete attributes, where Subtrees is a list of Value-Subtree pairs.

For continuous (numeric) attributes, the tree uses binary threshold splits represented as tree(Attribute, threshold(Threshold), LeftSubtree, RightSubtree) where LeftSubtree corresponds to values =< Threshold and RightSubtree to values > Threshold.

Limitations

• No incremental learning (the tree must be rebuilt from scratch when new examples are added)

References

• Quinlan, J.R. (1986). Induction of Decision Trees. Machine Learning, 1(1), 81-106. https://doi.org/10.1007/BF00116251
• Quinlan, J.R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers. https://doi.org/10.1016/C2009-0-27846-9
• Mitchell, T.M. (1997). Machine Learning. McGraw-Hill. Chapter 3: Decision Tree Learning.

Usage

To learn a decision tree from a dataset:

::

| ?- c45_classifier::learn(play_tennis, Tree).

To prune a learned tree with the default parameters (confidence factor 0.25, minimum instances per leaf 2):

::

| ?- c45_classifier::learn(breast_cancer, Tree), c45_classifier::prune(breast_cancer, Tree, PrunedTree).

To prune a learned tree with both custom confidence factor and minimum instances per leaf:

::

| ?- c45_classifier::learn(breast_cancer, Tree), c45_classifier::prune(breast_cancer, Tree, 0.1, 3, PrunedTree).

To export the tree as a list of predicate clauses:

::

| ?- c45_classifier::learn(play_tennis, Tree), c45_classifier::tree_to_clauses(play_tennis, Tree, classify, Clauses).

To export the tree to a file:

::

| ?- c45_classifier::learn(play_tennis, Tree), c45_classifier::tree_to_file(play_tennis, Tree, classify, 'tree.pl').

To print the tree to the current output:

::

| ?- c45_classifier::learn(play_tennis, Tree), c45_classifier::print_tree(Tree).

To predict the class for a new instance (as a list of attribute-value pairs):

::

| ?- c45_classifier::learn(play_tennis, Tree), c45_classifier::predict(Tree, [outlook-sunny, temperature-hot, humidity-high, wind-weak], Class).