Model Adapters Reference¶
Model adapters decouple the evaluation core from specific machine learning frameworks. TrustBench supports PyTorch, Scikit-learn, ONNX Runtime, Keras, JAX, and custom adapters via runtime protocols.
The ModelAdapter Protocol¶
Any model adapter must implement the structural protocol defined in trustbench.core.protocols.ModelAdapter:
class ModelAdapter(Protocol):
@property
def num_classes(self) -> int:
"""Total number of output classes."""
...
@property
def supports_predict_proba(self) -> bool:
"""Whether predict_proba is implemented."""
...
@property
def supports_predict_logits(self) -> bool:
"""Whether predict_logits is implemented."""
...
def predict(self, inputs: Any) -> np.ndarray:
"""Return predicted class indices (N,)."""
...
def predict_proba(self, inputs: Any) -> np.ndarray:
"""Return class probabilities (N, C)."""
...
def predict_logits(self, inputs: Any) -> np.ndarray:
"""Return raw unnormalized logits (N, C)."""
...
Built-In Adapters¶
1. PyTorch (PyTorchModelAdapter)¶
Wraps any torch.nn.Module.
- Automatically moves inputs to device (cpu, cuda).
- Employs non-blocking asynchronous CUDA streaming (tensor.to(device, non_blocking=True)).
- Executes under torch.inference_mode().
from trustbench.models.pytorch import PyTorchModelAdapter
adapter = PyTorchModelAdapter(model, num_classes=10, device="cuda")
2. Scikit-learn (SklearnModelAdapter)¶
Wraps any estimator supporting predict() and fit().
- Automatically discovers classes from estimator.classes_.
- Handles binary, multi-class, and continuous regression estimators.
- Converts decision_function to probabilities when predict_proba is absent.
from trustbench.models.sklearn import SklearnModelAdapter
adapter = SklearnModelAdapter(logistic_regression_model)
3. ONNX Runtime (ONNXModelAdapter)¶
Wraps an ONNX model file or pre-created onnxruntime.InferenceSession.
- Works on CPU or CUDA execution providers.
- Automatically discovers session input/output bindings.
from trustbench.models.onnx import ONNXModelAdapter
adapter = ONNXModelAdapter("model.onnx", num_classes=10)
4. Keras / TensorFlow (KerasModelAdapter) (Experimental)¶
Wraps tf.keras.Model or keras.Model instances.
[!NOTE] The Keras adapter is currently experimental. Ensure TensorFlow or Keras 3 is installed via
pip install "trustbench[tensorflow]".
from trustbench.models.keras import KerasModelAdapter
adapter = KerasModelAdapter(keras_model, num_classes=10)
5. JAX / Flax (JAXModelAdapter) (Experimental)¶
Wraps functional models via apply_fn(params, inputs).
[!NOTE] The JAX adapter is currently experimental. Ensure JAX and Flax are installed via
pip install "trustbench[jax]".
from trustbench.models.jax import JAXModelAdapter
adapter = JAXModelAdapter(apply_fn=model.apply, params=params, num_classes=10)
Creating a Custom Adapter¶
You can implement a custom adapter by implementing the ModelAdapter protocol or subclassing trustbench.models.base.ModelAdapter:
import numpy as np
from trustbench import ModelAdapter
class ConstantModelAdapter(ModelAdapter):
def __init__(self, constant_class: int = 0, num_classes: int = 2):
self._class = constant_class
self._num_classes = num_classes
@property
def num_classes(self) -> int:
return self._num_classes
def predict(self, inputs):
return np.full(len(inputs), self._class, dtype=int)