validmind.vm_models

Models entrypoint

class VMInput(abc.ABC):

Base class for ValidMind Input types

def with_options(self, **kwargs) -> VMInput:

Allows for setting options on the input object that are passed by the user when using the input to run a test or set of tests

To allow options, just override this method in the subclass (see VMDataset) and ensure that it returns a new instance of the input with the specified options set.

Arguments:
  • **kwargs: Arbitrary keyword arguments that will be passed to the input object
Returns:

VMInput: A new instance of the input with the specified options set

class VMDataset(validmind.vm_models.VMInput):

Base class for VM datasets

Child classes should be used to support new dataset types (tensor, polars etc) by converting the user's dataset into a numpy array collecting metadata like column names and then call this (parent) class __init__ method.

This way we can support multiple dataset types but under the hood we only need to work with numpy arrays and pandas dataframes in this class.

Attributes:
  • raw_dataset (np.ndarray): The raw dataset as a NumPy array.
  • input_id (str): Identifier for the dataset.
  • index (np.ndarray): The raw dataset index as a NumPy array.
  • columns (Set[str]): The column names of the dataset.
  • target_column (str): The target column name of the dataset.
  • feature_columns (List[str]): The feature column names of the dataset.
  • feature_columns_numeric (List[str]): The numeric feature column names of the dataset.
  • feature_columns_categorical (List[str]): The categorical feature column names of the dataset.
  • text_column (str): The text column name of the dataset for NLP tasks.
  • target_class_labels (Dict): The class labels for the target columns.
  • df (pd.DataFrame): The dataset as a pandas DataFrame.
  • extra_columns (Dict): Extra columns to include in the dataset.
VMDataset( raw_dataset: numpy.ndarray, input_id: str = None, model: VMModel = None, index: numpy.ndarray = None, index_name: str = None, date_time_index: bool = False, columns: list = None, target_column: str = None, feature_columns: list = None, text_column: str = None, extra_columns: dict = None, target_class_labels: dict = None)

Initializes a VMDataset instance.

Arguments:
  • raw_dataset (np.ndarray): The raw dataset as a NumPy array.
  • input_id (str): Identifier for the dataset.
  • model (VMModel): Model associated with the dataset.
  • index (np.ndarray): The raw dataset index as a NumPy array.
  • index_name (str): The raw dataset index name as a NumPy array.
  • date_time_index (bool): Whether the index is a datetime index.
  • columns (List[str], optional): The column names of the dataset. Defaults to None.
  • target_column (str, optional): The target column name of the dataset. Defaults to None.
  • feature_columns (str, optional): The feature column names of the dataset. Defaults to None.
  • text_column (str, optional): The text column name of the dataset for nlp tasks. Defaults to None.
  • target_class_labels (Dict, optional): The class labels for the target columns. Defaults to None.
def with_options(self, **kwargs) -> VMDataset:

Support options provided when passing an input to run_test or run_test_suite

Example:

# to only use a certain subset of columns in the dataset:
run_test(
    "validmind.SomeTestID",
    inputs={
        "dataset": {
            "input_id": "my_dataset_id",
            "columns": ["col1", "col2"],
        }
    }
)

# behind the scenes, this retrieves the dataset object (VMDataset) from the registry
# and then calls the `with_options()` method and passes `{"columns": ...}`
Arguments:
  • **kwargs: Options:
    • columns: Filter columns in the dataset
Returns:

VMDataset: A new instance of the dataset with only the specified columns

def assign_predictions( self, model: VMModel, prediction_column: str = None, prediction_values: list = None, probability_column: str = None, probability_values: list = None, prediction_probabilities: list = None, **kwargs):

Assign predictions and probabilities to the dataset.

Arguments:
  • model (VMModel): The model used to generate the predictions.
  • prediction_column (str, optional): The name of the column containing the predictions. Defaults to None.
  • prediction_values (list, optional): The values of the predictions. Defaults to None.
  • probability_column (str, optional): The name of the column containing the probabilities. Defaults to None.
  • probability_values (list, optional): The values of the probabilities. Defaults to None.
  • prediction_probabilities (list, optional): DEPRECATED: The values of the probabilities. Defaults to None.
  • kwargs: Additional keyword arguments that will get passed through to the model's predict method.
def prediction_column( self, model: VMModel, column_name: str = None) -> str:

Get or set the prediction column for a model.

def probability_column( self, model: VMModel, column_name: str = None) -> str:

Get or set the probability column for a model.

def add_extra_column(self, column_name, column_values=None):

Adds an extra column to the dataset without modifying the dataset features and target columns.

Arguments:
  • column_name (str): The name of the extra column.
  • column_values (np.ndarray, optional): The values of the extra column.
df: pandas.core.frame.DataFrame

Returns the dataset as a pandas DataFrame.

Returns:

pd.DataFrame: The dataset as a pandas DataFrame.

x: numpy.ndarray

Returns the input features (X) of the dataset.

Returns:

np.ndarray: The input features.

y: numpy.ndarray

Returns the target variables (y) of the dataset.

Returns:

np.ndarray: The target variables.

def y_pred(self, model) -> numpy.ndarray:

Returns the predictions for a given model.

Attempts to stack complex prediction types (e.g., embeddings) into a single, multi-dimensional array.

Arguments:
  • model (VMModel): The model whose predictions are sought.
Returns:

np.ndarray: The predictions for the model

def y_prob(self, model) -> numpy.ndarray:

Returns the probabilities for a given model.

Arguments:
  • model (str): The ID of the model whose predictions are sought.
Returns:

np.ndarray: The probability variables.

def x_df(self):

Returns a dataframe containing only the feature columns

def y_df(self) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the target column

def y_pred_df(self, model) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the predictions for a given model

def y_prob_df(self, model) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the probabilities for a given model

def target_classes(self):

Returns the target class labels or unique values of the target column.

class VMModel(validmind.vm_models.VMInput):

An base class that wraps a trained model instance and its associated data.

Attributes:
  • model (object, optional): The trained model instance. Defaults to None.
  • input_id (str, optional): The input ID for the model. Defaults to None.
  • attributes (ModelAttributes, optional): The attributes of the model. Defaults to None.
  • name (str, optional): The name of the model. Defaults to the class name.
def serialize(self):

Serializes the model to a dictionary so it can be sent to the API

def predict_proba(self, *args, **kwargs):

Predict probabilties - must be implemented by subclass if needed

@abstractmethod
def predict(self, *args, **kwargs):

Predict method for the model. This is a wrapper around the model's

Inherited Members
VMInput
with_options
@dataclass
class Figure:

Figure objects track the schema supported by the ValidMind API

Figure( key: str, figure: Union[matplotlib.figure.Figure, plotly.graph_objs._figure.Figure, plotly.graph_objs._figurewidget.FigureWidget, bytes], ref_id: str, _type: str = 'plot')
def to_widget(self):

Returns the ipywidget compatible representation of the figure. Ideally we would render images as-is, but Plotly FigureWidgets don't work well on Google Colab when they are combined with ipywidgets.

def serialize(self):

Serializes the Figure to a dictionary so it can be sent to the API

def serialize_files(self):

Creates a requests-compatible files object to be sent to the API

@dataclass
class ModelAttributes:

Model attributes definition

ModelAttributes( architecture: str = None, framework: str = None, framework_version: str = None, language: str = None, task: validmind.vm_models.model.ModelTask = None)
@classmethod
def from_dict(cls, data):

Creates a ModelAttributes instance from a dictionary

R_MODEL_TYPES = ['LogisticRegression', 'LinearRegression', 'XGBClassifier', 'XGBRegressor']
@dataclass
class ResultTable:

A dataclass that holds the table summary of result

ResultTable( data: Union[List[Any], pandas.core.frame.DataFrame], title: Optional[str] = None)
def serialize(self):
@dataclass
class TestResult(validmind.vm_models.result.result.Result):

Test result

TestResult( result_id: str = None, name: str = 'Test Result', ref_id: str = None, title: Optional[str] = None, doc: Optional[str] = None, description: Union[str, validmind.ai.utils.DescriptionFuture, NoneType] = None, metric: Union[int, float, NoneType] = None, tables: Optional[List[ResultTable]] = None, raw_data: Optional[validmind.RawData] = None, figures: Optional[List[Figure]] = None, passed: Optional[bool] = None, params: Optional[Dict[str, Any]] = None, inputs: Optional[Dict[str, Union[List[VMInput], VMInput]]] = None, metadata: Optional[Dict[str, Any]] = None, _was_description_generated: bool = False, _unsafe: bool = False)
test_name: str

Get the test name, using custom title if available.

def add_table( self, table: Union[ResultTable, pandas.core.frame.DataFrame, List[Dict[str, Any]]], title: Optional[str] = None):

Add a new table to the result

Arguments:
  • table (Union[ResultTable, pd.DataFrame, List[Dict[str, Any]]]): The table to add
  • title (Optional[str]): The title of the table (can optionally be provided for pd.DataFrame and List[Dict[str, Any]] tables)
def remove_table(self, index: int):

Remove a table from the result by index

Arguments:
  • index (int): The index of the table to remove (default is 0)
def add_figure( self, figure: Union[matplotlib.figure.Figure, plotly.graph_objs._figure.Figure, plotly.graph_objs._figurewidget.FigureWidget, bytes, Figure]):

Add a new figure to the result

Arguments:
  • figure (Union[matplotlib.figure.Figure, go.Figure, go.FigureWidget, bytes, Figure]): The figure to add (can be either a VM Figure object, a raw figure object from the supported libraries, or a png image as raw bytes)
def remove_figure(self, index: int = 0):

Remove a figure from the result by index

Arguments:
  • index (int): The index of the figure to remove (default is 0)
def to_widget(self):

Create an ipywdiget representation of the result... Must be overridden by subclasses

def serialize(self):

Serialize the result for the API

async def log_async( self, section_id: str = None, position: int = None, unsafe: bool = False):
def log( self, section_id: str = None, position: int = None, unsafe: bool = False):

Log the result to ValidMind

Arguments:
  • section_id (str): The section ID within the model document to insert the test result
  • position (int): The position (index) within the section to insert the test result
  • unsafe (bool): If True, log the result even if it contains sensitive data i.e. raw data from input datasets
Inherited Members
validmind.vm_models.result.result.Result
show
@dataclass
class TestSuite:

Base class for test suites. Test suites are used to define a grouping of tests that can be run as a suite against datasets and models. Test Suites can be defined by inheriting from this base class and defining the list of tests as a class variable.

Tests can be a flat list of strings or may be nested into sections by using a dict

TestSuite( sections: List[validmind.vm_models.test_suite.test_suite.TestSuiteSection] = None)
def get_tests(self) -> List[str]:

Get all test suite test objects from all sections

def num_tests(self) -> int:

Returns the total number of tests in the test suite

def get_default_config(self) -> dict:

Returns the default configuration for the test suite

Each test in a test suite can accept parameters and those parameters can have default values. Both the parameters and their defaults are set in the test class and a config object can be passed to the test suite's run method to override the defaults. This function returns a dictionary containing the parameters and their default values for every test to allow users to view and set values

Returns:

dict: A dictionary of test names and their default parameters

class TestSuiteRunner:

Runs a test suite

TestSuiteRunner( suite: TestSuite, config: dict = None, inputs: dict = None)
async def log_results(self):

Logs the results of the test suite to ValidMind

This method will be called after the test suite has been run and all results have been collected. This method will log the results to ValidMind.

def summarize(self, show_link: bool = True):
def run(self, send: bool = True, fail_fast: bool = False):

Runs the test suite, renders the summary and sends the results to ValidMind

Arguments:
  • send (bool, optional): Whether to send the results to ValidMind. Defaults to True.
  • fail_fast (bool, optional): Whether to stop running tests after the first failure. Defaults to False.