Multilabel Rule Learning
Final Report Abstract
Inductive rule learning is one of the most traditional research areas in machine learning. Rule learning algorithms are typically used when not only the accuracy of predictions is of interest, but also the interpretability and transparency of the model, which ideally yields new insights into the domain of the dataset. In this project, we tackled the problem of learning interpretable rule models for the task of multi-label classification, which differs from conventional multi-class classification in that not only a single label but an entire subset of labels has to predicted. This problem is challenging, because dependencies between the labels have to be taken into account in order to obtain performant models. We formalized this problem in a formal way and clearly stated the associated challenges, such as whether individual rules should only predict single labels or multiple labels, whether they should make only positive or also negative predictions, or whether the rule body can contain predictions to other labels, which may result in circular reasoning structures. During the course of the project, we studied important subproblems, such as how multi-label losses for individual rules can be minimized, how it can be decided whether a rule should predict a label or abstain, how multiple predictions of individual rules can be aggregated into overall multi-label predictions, what strategies can be used to learn rule sets or decision lists, and how to deal with mixed dependency rule sets, which include both input features as well as label predictions of other rules in the rule bodies. These studies culminated in the development of BOOMER, a highly performant multi-label rule learning algorithm based on a gradient boosting methodology, which has previously been used in single-label rule learning. The learner is freely available and has already been adopted by other research groups.
Publications
- Learning interpretable rules for multi-label classification. In Jair Escalante, H. et al. (eds.), Explainable and Interpretable Models in Computer Vision and Machine Learning, The Springer Series on Challenges in Machine Learning, pp. 81–113. Springer-Verlag, 2018. ISBN 978-3-319-98131-4
Loza Mencía, E., Furnkranz, J., Hullermeier, E., and Rapp, M.
(See online at https://doi.org/10.1007/978-3-319-98131-4_4) - On the trade-off between consistency and coverage in multi-label rule learning heuristics. In Kralj Novak, P. et al.(eds.), Discovery Science, pp. 96–111, Cham, October 2019. Springer International Publishing. ISBN 978-3-030-33778-0
Rapp, M., Loza Mencía, E., and Fürnkranz, J.
(See online at https://doi.org/10.1007/978-3-030-33778-0_9) - Conformal rule-based multi-label classification. In Schmid, U., Klugl, F., and Wolter, D. (eds.), KI 2020: Advances in Artificial Intelligence. Springer, Cham, September 2020. ISBN 978-3-030-58284-5
Hüllermeier, E., Fürnkranz, J., and Loza Mencía, E.
(See online at https://doi.org/10.1007/978-3-030-58285-2_25) - Learning gradient boosted multi-label classification rules. In Proceedings of the European Conference of Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), pp. 124–140. Springer, 2020
Rapp, M., Loza Mencía, E., Fürnkranz, J., Nguyen, V.-L., and Hüllermeier, E.
(See online at https://doi.org/10.1007/978-3-030-67664-3_8) - On aggregation in ensembles of multilabel classifiers. In Appice, A. et al. (eds.), Discovery Science, pp. 533–547, Cham, October 2020. Springer International Publishing. ISBN 978- 3-030-61527-7
Nguyen, V.-L., Hüllermeier, E., Rapp, M., Loza Mencía, E., and Fürnkranz, J.
(See online at https://doi.org/10.1007/978-3-030-61527-7_35) - Reliable multilabel classification: Prediction with partial abstention. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):5264–5271, Apr. 2020
Nguyen, V.-L. and Hüllermeier, E.
(See online at https://doi.org/10.1609/aaai.v34i04.5972) - Rule-based multi-label classification: Challenges and opportunities. In Gutierrez-Basulto, V. et al. (eds.), Rules and Reasoning, pp. 3–19. Springer International Publishing, August 2020. ISBN 978-3-030-57977-7
Hüllermeier, E., Fürnkranz, J., Loza Mencía, E., Nguyen, V.-L., and Rapp, M.
(See online at https://doi.org/10.1007/978-3-030-57977-7_1) - Gradient-based label binning in multi-label classification. In Proceedings of the European Conference of Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), Part III, pp. 462–477, 2021
Rapp, M., Loza Mencía, E., Fürnkranz, J., and Hüllermeier, E.
(See online at https://doi.org/10.1007/978-3-030-86523-8_28) - Multilabel classification with partial abstention: Bayes-optimal prediction under label independence. Journal of Artificial Intelligence Research, 72:613–665, 2021
Nguyen, V.-L. and Hüllermeier, E.
(See online at https://doi.org/10.1613/jair.1.12610)
