Quickstart Guide¶
Get up and running with TrustBench in 5 minutes using either the Command Line Interface (CLI) or the Python API.
1. Quickstart via CLI¶
Step 1: Initialize Configuration¶
Generate a starter trustbench.yaml configuration in your project root:
Step 2: Validate Configuration¶
Check that your configuration paths and parameters are valid:
Step 3: Run Evaluation¶
Execute evaluation and generate structured artifacts:
This will run inference, compute classification, calibration, and uncertainty metrics, and output interactive HTML and machine-readable JSON reports into artifacts/.
2. Quickstart via Python API¶
Basic Classification Evaluation¶
import numpy as np
from sklearn.linear_model import LogisticRegression
from trustbench import TrustBench, InMemoryDatasetAdapter
# 1. Train or load your model
X_train = np.random.randn(100, 4)
y_train = np.random.randint(0, 2, size=100)
clf = LogisticRegression().fit(X_train, y_train)
# 2. Wrap evaluation dataset
X_test = np.random.randn(50, 4)
y_test = np.random.randint(0, 2, size=50)
dataset = InMemoryDatasetAdapter(inputs=X_test, labels=y_test)
# 3. Evaluate with TrustBench
bench = TrustBench(model=clf, dataset=dataset)
result = bench.evaluate()
print(f"Accuracy: {result.metrics['classification']['accuracy']:.2f}")
print(f"Brier Score: {result.metrics['calibration']['brier_score']:.4f}")
Basic Regression Evaluation¶
import numpy as np
from sklearn.linear_model import Ridge
from trustbench import evaluate_regression, InMemoryDatasetAdapter
X = np.linspace(0, 10, 100).reshape(-1, 1)
y = 2.5 * X.squeeze() + np.random.normal(0, 0.5, size=100)
reg = Ridge().fit(X, y)
dataset = InMemoryDatasetAdapter(inputs=X, labels=y)
result = evaluate_regression(reg, dataset)
print(f"MAE: {result.metrics['mae']:.3f}")
print(f"R²: {result.metrics['r2']:.3f}")
[!NOTE] Prediction Intervals & Uncertainty (PICP / MPIW): Standard point regressors like
Ridgeproduce scalar predictions without variance estimates, so interval metrics (picp_95,mpiw_95) are omitted. When evaluating probabilistic models (such as Gaussian Processes, deep ensembles, or custom adapters exposingpredict_uncertainty(inputs)), TrustBench automatically extracts predictive standard deviations \(\sigma\) and computes 95% coverage (picp_95) and interval width (mpiw_95).