Result Metric
Result metric (aitoolbox.experiment.core_metrics
) is an abstraction built around the calculation of the single
performance metric. It helps keep the code base more reusable and better structured, especially when used as part of
the encapsulating Result Package.
AIToolbox comes out of the box with implemented several commonly used performance evaluation metrics implemented as result metrics. These can be found in:
Use of Result Metrics inside Result Packages
As it is described in the Implementing New Result Packages section, result metrics come in handy when developing
the result packages which are wrapping together multiple metrics needed to evaluate a certain ML task. To support this
chaining together of multiple performance metrics, the result metric abstraction offers a convenient metric
concatenation and result package dictionary creation via the +
operator. To create the dictionary holding all
the performance metric results the user can simply write: metric_1 + metric_2 + metric_3 + ...
. This makes the use
of the +
operator very convenient because the produced results dictionary format exactly matches that which is
required when developing an encapsulating result package.
Example of result metric concatenation:
from aitoolbox.experiment.core_metrics.classification import \
AccuracyMetric, ROCAUCMetric, PrecisionRecallCurveAUCMetric
accuracy_result = AccuracyMetric(y_true, y_predicted)
roc_auc_result = ROCAUCMetric(y_true, y_predicted)
pr_auc_result = PrecisionRecallCurveAUCMetric(y_true, y_predicted)
results_dict = accuracy_result + roc_auc_result + pr_auc_result
# results_dict will hold:
# {'Accuracy': 0.95, 'ROC_AUC': 0.88, 'PrecisionRecall_AUC': 0.67}
Implementing New Result Metrics
When the needed result metric is not available in the AIToolbox, the users can easily implement their own new metrics. The approach is very similar to that of the new result package development.
In order to implement
a new result metric, the user has to create a new metric class which inherits from the base abstract result metric
aitoolbox.experiment.core_metrics.abstract_metric.AbstractBaseMetric
and implements the abstract method
calculate_metric()
.
As part of the calculate_metric()
the user has to implement the logic for the performance metric calculation and
return the metric result from the method. Predicted values and ground truth values normally needed for the performance
metric calculations are available inside the metric as object attributes and can thus be accessed as: self.y_true
and self.y_predicted
throughout the metric class, calculate_metric()
included.
Example Result Metric implementation:
from sklearn.metrics import accuracy_score
from aitoolbox.experiment.core_metrics.abstract_metric import AbstractBaseMetric
class ExampleAccuracyMetric(AbstractBaseMetric):
def __init__(self, y_true, y_predicted, positive_class_thresh=0.5):
# All additional attributes should be defined before the AbstractBaseMetric.__init__
self.positive_class_thresh = positive_class_thresh
AbstractBaseMetric.__init__(self, y_true, y_predicted, metric_name='Accuracy')
def calculate_metric(self):
if self.positive_class_thresh is not None:
self.y_predicted = self.y_predicted >= self.positive_class_thresh
return accuracy_score(self.y_true, self.y_predicted)