Python API

import fairmedfm as fm

fm.auc_gap(y_true, y_score, sensitive_features=group)                      # one metric, one number
report = fm.evaluate(y_true, y_score, sensitive_features)                  # binary classification
report = fm.evaluate_segmentation(sensitive_features, dice=dice)           # segmentation from Dice scores
report = fm.evaluate_segmentation(sensitive_features, pred_masks=p, true_masks=t)   # ... or from masks

Both return a FairnessReport with pandas tables (summary, by_group) and JSON export. See Evaluate your model for examples and Fairness metrics for the definitions.

Single metrics

Each function returns one float and accepts the same inputs and options as evaluate or evaluate_segmentation. With several sensitive attributes, their combinations are compared.

auc_gap

auc_gap(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Largest minus smallest group ROC AUC (0 is fair).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

worst_group_auc

worst_group_auc(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Lowest group ROC AUC (higher is better).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

accuracy_gap

accuracy_gap(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Largest minus smallest group accuracy at the threshold with the best overall F1 (0 is fair).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

bce_gap

bce_gap(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Largest minus smallest group binary cross-entropy (0 is fair).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

ece_gap

ece_gap(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Largest minus smallest group expected calibration error (0 is fair).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

equal_opportunity_difference

equal_opportunity_difference(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

Largest minus smallest group true positive rate at the best-F1 threshold (eo; 0 is fair).

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

equalized_odds_score

equalized_odds_score(
    y_true,
    y_score,
    *,
    sensitive_features,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
)

1 - (TPR gap + TNR gap) / 2 at the best-F1 threshold (eod; 1 is fair).

A score, not a difference: unlike fairlearn's equalized_odds_difference (0 is fair), higher is fairer.

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample (0/1, booleans, one-hot rows, or any labels with pos_label).	required
y_score	Any	positive-class probabilities, logits, or a (samples, classes) matrix such as predict_proba or softmax output.	required
sensitive_features	Any	group of each sample, e.g. sex or site: an array, Series, list, or a DataFrame or dict of several attributes (their combinations are compared).	required
pos_label	Any	the positive class, when labels are not 0/1 or booleans (one-vs-rest with more classes).	None
score_type	str	"auto", "probability" or "logit": how to read y_score.	'auto'
score_column	Optional[int]	column of the positive class in a (samples, classes) y_score.	None
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]} (cut points) or {"age": 4} (quantiles).	None

See :func:fairmedfm.evaluate for details.

dice_gap

dice_gap(
    dice=None, *, sensitive_features, bins=None, **masks
)

Largest minus smallest group mean Dice (delta_dice; 0 is fair).

Parameters:

Name	Type	Description	Default
dice	Any	Dice score per sample. Alternatively pass pred_masks and true_masks (arrays, tensors or file paths) and Dice is computed per sample.	None
sensitive_features	Any	group of each sample, as in the classification functions.	required
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]}.	None
**masks	Any	pred_masks and true_masks instead of dice, and optionally label, mask_threshold and empty_score, as in :func:fairmedfm.evaluate_segmentation.	{}

worst_group_dice

worst_group_dice(
    dice=None, *, sensitive_features, bins=None, **masks
)

Lowest group mean Dice (min_dice; higher is better).

Parameters:

Name	Type	Description	Default
dice	Any	Dice score per sample. Alternatively pass pred_masks and true_masks (arrays, tensors or file paths) and Dice is computed per sample.	None
sensitive_features	Any	group of each sample, as in the classification functions.	required
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]}.	None
**masks	Any	pred_masks and true_masks instead of dice, and optionally label, mask_threshold and empty_score, as in :func:fairmedfm.evaluate_segmentation.	{}

dice_std

dice_std(
    dice=None, *, sensitive_features, bins=None, **masks
)

Standard deviation of the group mean Dice scores (std_dice; 0 is fair).

Parameters:

Name	Type	Description	Default
dice	Any	Dice score per sample. Alternatively pass pred_masks and true_masks (arrays, tensors or file paths) and Dice is computed per sample.	None
sensitive_features	Any	group of each sample, as in the classification functions.	required
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]}.	None
**masks	Any	pred_masks and true_masks instead of dice, and optionally label, mask_threshold and empty_score, as in :func:fairmedfm.evaluate_segmentation.	{}

dice_skewness

dice_skewness(
    dice=None, *, sensitive_features, bins=None, **masks
)

(1 - worst group Dice) / (1 - best group Dice) (skewness_dice; 1 is fair, NaN if a group is perfect).

Parameters:

Name	Type	Description	Default
dice	Any	Dice score per sample. Alternatively pass pred_masks and true_masks (arrays, tensors or file paths) and Dice is computed per sample.	None
sensitive_features	Any	group of each sample, as in the classification functions.	required
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]}.	None
**masks	Any	pred_masks and true_masks instead of dice, and optionally label, mask_threshold and empty_score, as in :func:fairmedfm.evaluate_segmentation.	{}

equity_scaled_dice

equity_scaled_dice(
    dice=None, *, sensitive_features, bins=None, **masks
)

Mean Dice divided by 1 + the standard deviation of group means (es_dice; higher is better).

Parameters:

Name	Type	Description	Default
dice	Any	Dice score per sample. Alternatively pass pred_masks and true_masks (arrays, tensors or file paths) and Dice is computed per sample.	None
sensitive_features	Any	group of each sample, as in the classification functions.	required
bins	Optional[Bins]	groups for continuous attributes, e.g. {"age": [40, 60]}.	None
**masks	Any	pred_masks and true_masks instead of dice, and optionally label, mask_threshold and empty_score, as in :func:fairmedfm.evaluate_segmentation.	{}

Full reports

evaluate

evaluate(
    y_true,
    y_score,
    sensitive_features,
    *,
    pos_label=None,
    score_type="auto",
    score_column=None,
    bins=None,
    intersectional=False,
)

Fairness of a binary classifier across the groups of one or more sensitive attributes.

Parameters:

Name	Type	Description	Default
y_true	Any	ground truth per sample: 0/1, booleans, one-hot rows, or any labels together with pos_label.	required
y_score	Any	model output per sample: positive-class probabilities, logits, or a (samples, classes) array of probabilities or logits. Lists, NumPy arrays, pandas objects and PyTorch tensors all work.	required
sensitive_features	Any	a pandas DataFrame (one column per attribute), a dict of name to values, a Series, or one array. Values can be strings or numbers; missing values exclude the sample for that attribute.	required
pos_label	Any	the positive class in y_true. Needed when labels are not 0/1 or booleans; with more than two classes, the evaluation is one-vs-rest for this class.	None
score_type	str	"auto" treats scores in [0, 1] as probabilities and anything else as logits (sigmoid for one column, softmax for several); "probability" or "logit" force one interpretation.	'auto'
score_column	Optional[int]	column of the positive class when y_score has several columns. By default, columns are taken to be in sorted class order, as in scikit-learn's predict_proba: the column of pos_label among the sorted labels, or the second of two columns for 0/1 labels.	None
bins	Optional[Bins]	how to group continuous attributes, e.g. {"age": [40, 60]} (cut points: <40, 40-60, >=60) or {"age": 4} (quartiles). Numeric attributes with more than 20 distinct values must be binned.	None
intersectional	bool	also evaluate the combination of all attributes (e.g. "F & >=60").	False

Returns:

Name	Type	Description
A	FairnessReport	class:FairnessReport. report.summary has, per attribute, overall-auc, overall-acc,
	FairnessReport	overall-bce, overall-ece, worst-auc, auc-gap, acc-gap, bce-gap, ece-gap, eo
	FairnessReport	(TPR gap) and eod (1 - (TPR gap + TNR gap) / 2, where 1 is fair), as in the FairMedFM paper.

evaluate_segmentation

evaluate_segmentation(
    sensitive_features,
    *,
    dice=None,
    pred_masks=None,
    true_masks=None,
    label=None,
    mask_threshold=0.5,
    empty_score=1.0,
    bins=None,
    intersectional=False,
)

Fairness of a segmentation model across the groups of one or more sensitive attributes.

Give either per-sample Dice scores (dice) or predicted and ground-truth masks (pred_masks and true_masks), from which Dice is computed per sample.

Parameters:

Name	Type	Description	Default
sensitive_features	Any	as in :func:evaluate.	required
dice	Any	Dice score in [0, 1] per sample (image or volume).	None
pred_masks	Any	one predicted mask per sample: a list of arrays or tensors, a stacked array, or file paths (.npy, .npz, .png, .jpg, .tif, .nii, .nii.gz; images and NIfTI need pip install "fairmedfm[io]"). Probability masks in [0, 1] are thresholded at mask_threshold.	None
true_masks	Any	one ground-truth mask per sample, in the same forms.	None
label	Any	the class to evaluate in multi-class masks; by default any non-zero value is foreground.	None
mask_threshold	float	threshold for probability masks.	0.5
empty_score	float	Dice when both masks of a sample are empty (default 1.0, a correct empty prediction). The benchmark trainer's torchmetrics Dice gives 0 in this case; pass empty_score=0 to match it.	1.0
bins	Optional[Bins]	as in :func:evaluate.	None
intersectional	bool	as in :func:evaluate.	False

Returns:

Name	Type	Description
A	FairnessReport	class:FairnessReport. report.summary has, per attribute, mean_dice, min_dice,
	FairnessReport	max_dice, delta_dice, skewness_dice, std_dice and es_dice.

FairnessReport

Result of :func:evaluate or :func:evaluate_segmentation.

Attributes:

Name	Type	Description
task		"cls" (classification) or "seg" (segmentation).
summary		one row per sensitive attribute with the fairness metrics (see the metric reference).
by_group		one row per (attribute, group) with the group's sample count and metrics. Groups that could not be evaluated have a skipped reason and are left out of the attribute's gaps.
overall		metrics on all samples.
warnings		notes about how the input was interpreted or which groups were skipped.
metadata		how the input was read, e.g. the score transform or the positive label.

to_dict

to_dict()

JSON-compatible dictionary; NaN becomes None.

to_json

to_json(path=None, indent=2)

JSON text of :meth:to_dict; also written to path if given.

Building blocks

binary_classification_report

binary_classification_report(
    prob, label, threshold=0.5, suffix=""
)

AUC, accuracy, BCE, ECE, TPR, TNR and confusion counts at one decision threshold.

expected_calibration_error

expected_calibration_error(
    prob,
    label,
    num_bins=10,
    metric_variant="abs",
    quantile_bins=False,
)

Calibration error: the sample-weighted mean, over bins of prob, of |mean prob - fraction positive|.

Bins are num_bins equal-width intervals over the range of prob (pandas.cut), or equal-count bins with quantile_bins=True. metric_variant="squared" averages squared differences and "rmse" takes the square root of that. See Guo et al. (2017), https://arxiv.org/abs/1706.04599, and Nixon et al. (2019), https://arxiv.org/abs/1904.01685.

binary_cross_entropy

binary_cross_entropy(prob, label)

Mean binary cross-entropy, with log terms clamped at -100 as in torch.nn.BCELoss.

find_threshold

find_threshold(prob, label)

Decision threshold with the highest F1 score on these samples (the lowest such threshold on ties).

Choosing the operating point by F1 follows the chest X-ray fairness evaluations of Seyyed-Kalantari et al. (Nature Medicine, 2021) and Zhang et al. (CHIL, 2022).

Earlier interfaces

classification_fairness and segmentation_fairness from version 0.1 are deprecated: they emit a FutureWarning and will be removed in 1.0. Use evaluate, evaluate_segmentation or the single metrics instead. They take one sensitive attribute, require 0/1 labels and positive-class probabilities, and raise an error for a group with only one label.

classification_fairness

classification_fairness(prob, label, group)

Fairness of a binary classifier across the groups of one sensitive attribute.

Parameters:

Name	Type	Description	Default
prob	ArrayLike	positive-class probability for each sample.	required
label	ArrayLike	ground-truth label (0 or 1) for each sample.	required
group	Sequence[Any]	sensitive attribute value for each sample, e.g. "F"/"M" or age bins. Two or more groups; every group must contain both labels.	required

Deprecated since 0.4.1 and to be removed in 1.0: use :func:fairmedfm.evaluate, which accepts more input forms and reports groups it cannot evaluate instead of failing, or a single metric such as :func:fairmedfm.auc_gap.

Returns:

Type	Description
Dict[str, Any]	summary: the FairMedFM fairness metrics (overall-auc, worst-auc, auc-gap, acc-gap,
Dict[str, Any]	bce-gap, ece-gap, eod, eo and the overall accuracy, BCE and ECE). Gaps are the largest
Dict[str, Any]	minus the smallest group value. eod is 1 - (TPR gap + TNR gap) / 2 (higher is fairer) and
Dict[str, Any]	eo is the TPR gap; both use the threshold with the best overall F1, as does accuracy.
Dict[str, Any]	overall: metrics on all samples at threshold 0.5 and at the best-F1 threshold (@best_f1).
Dict[str, Any]	groups: the same metrics for each group, keyed by group value as a string, with sample counts.

segmentation_fairness

segmentation_fairness(dice, group)

Fairness of a segmentation model from per-sample Dice scores across the groups of one attribute.

Parameters:

Name	Type	Description	Default
dice	ArrayLike	Dice similarity coefficient in [0, 1] for each sample (image or volume).	required
group	Sequence[Any]	sensitive attribute value for each sample; two or more groups.	required

Deprecated since 0.4.1 and to be removed in 1.0: use :func:fairmedfm.evaluate_segmentation or a single metric such as :func:fairmedfm.dice_gap.

Returns:

Type	Description
Dict[str, Any]	summary: mean_dice (over all samples), min_dice/max_dice (worst and best group mean),
Dict[str, Any]	delta_dice (max minus min), skewness_dice ((1 - min) / (1 - max); None when the best
Dict[str, Any]	group has a mean Dice of exactly 1), std_dice (standard deviation of the group means) and
Dict[str, Any]	es_dice (equity-scaled Dice, mean_dice / (1 + std_dice)).
Dict[str, Any]	groups: mean Dice and sample count per group, keyed by group value as a string.

evaluate_binary, organize_results and evaluate_seg keep the interfaces of the original benchmark code and are used by its trainers. evaluate_seg returns inf for skewness_dice when the best group's mean Dice is 1 and lets NaN Dice scores propagate, as the original did.

evaluate_binary

evaluate_binary(pred, Y, A)

Overall metrics and per-group metric lists (groups in sorted order).

organize_results

organize_results(overall_metrics, subgroup_metrics)

Fairness summary from the output of :func:evaluate_binary.

evaluate_seg

evaluate_seg(dsc_list, sensitive_list)

Segmentation fairness summary as logged by the trainers; see :func:segmentation_fairness.

Unlike :func:segmentation_fairness, skewness_dice is inf when the best group has a mean Dice of 1, and NaN Dice scores propagate into the means instead of raising.