# Copyright © 2023-2026 ValidMind Inc. All rights reserved.
# Refer to the LICENSE file in the root of this repository for details.
# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial

- id: conceptual_soundness
  title: Conceptual Soundness
  index_only: true
  sections:
    - id: model_overview
      title: Model Overview
      guidelines:
        - Provide a clear and concise description of the model's main concept,
          explaining the underlying financial theory or economic rationale.
        - Justify the choice of the model, algorithm, or approach, relating it
          to the financial institution's needs, objectives, and any relevant
          industry standards.
        - Discuss any alternative models or approaches considered during the
          model development process, and explain why they were not selected.
        - Describe any key assumptions made in the model and their potential
          implications on the model's results and performance.
        - Explain how the model's structure and design contribute to its
          robustness, stability, and reliability in a financial context.
      parent_section: conceptual_soundness
      contents:
        - content_id: model_overview
          content_type: text
    - id: intended_use_business_use_case
      title: Intended Use and Business Use Case
      index_only: true
      condensed: true
      parent_section: conceptual_soundness
      sections:
        - id: intended_use
          title: Intended Use
          parent_section: intended_use_business_use_case
          guidelines:
            - Clearly describe the specific business use case(s) for which the
              model is designed, including the intended users and the financial
              institution's objectives.
            - Explain how the model fits into the overall business strategy and
              decision-making processes of the financial institution.
        - id: regulatory_requirements
          title: Regulatory Requirements
          parent_section: intended_use_business_use_case
          guidelines:
            - Detail any specific risks, regulatory requirements, or other
              considerations associated with the model's intended use, and how
              they have been addressed in the model development process.
        - id: model_limitations
          title: Model Limitations
          parent_section: intended_use_business_use_case
          guidelines:
            - Discuss the model's intended scope, including any limitations,
              boundaries, or exclusions in its applicability.
            - Describe any potential external factors, such as economic or
              market conditions, that could impact the model's performance, and
              how they have been considered in the model development process.
    - id: model_selection
      title: Model Selection
      guidelines:
        - Provide a detailed description of the selected model, including its
          algorithm, mathematical foundations, and key features that make it
          suitable for the intended use case.
        - Explain the rationale behind choosing the specific model, and how it
          addresses the financial institution's objectives, regulatory
          requirements, and risk management needs.
        - Compare the selected model with alternative models or approaches that
          were considered during the model development process, highlighting
          their strengths and weaknesses, and explaining why the chosen model is
          the most appropriate.
        - Describe any model customizations or adaptations made to better align
          it with the financial institution's needs, and discuss the potential
          impact of these changes on the model's performance.
        - Explain any potential limitations or drawbacks of the selected model
          in the context of the intended use case, and how they have been
          mitigated or addressed during the model development process.
      parent_section: conceptual_soundness
- id: data_preparation
  title: Data Preparation
  index_only: true
  sections:
    - id: data_description
      title: Data description
      guidelines:
        - Provide a comprehensive overview of the data sources used in the
          model, including internal and external sources, and specify the data's
          time period, frequency, and granularity.
        - Describe the main variables and features used in the model, including
          both input and output variables, and explain their relevance to the
          model's purpose and intended use case.
        - Detail any data transformations, preprocessing, or feature engineering
          performed on the raw data to prepare it for model input, and explain
          the rationale for these transformations.
        - Address any data quality concerns, such as missing values, outliers,
          or inconsistencies, and describe the methods used to handle these
          issues.
        - Discuss any potential biases, dependencies, or limitations in the data
          that could impact the model's performance, and explain how they have
          been considered or mitigated during the data preparation process.
      contents:
        - content_id: text_data_quality
          content_type: text
          options:
            default_text: |
              # Text Data Quality Metrics and Tests
        - content_type: test
          content_id: validmind.data_validation.Duplicates
        - content_type: test
          content_id: validmind.data_validation.nlp.StopWords
        - content_type: test
          content_id: validmind.data_validation.nlp.Punctuations
        - content_type: test
          content_id: validmind.data_validation.nlp.CommonWords
        - content_type: test
          content_id: validmind.data_validation.nlp.LanguageDetection
        - content_type: test
          content_id: validmind.data_validation.nlp.Toxicity
        - content_type: test
          content_id: validmind.data_validation.nlp.PolarityAndSubjectivity
        - content_type: test
          content_id: validmind.data_validation.nlp.Sentiment
      parent_section: data_preparation
    - id: feature_selection
      title: Feature Selection and Engineering
      guidelines:
        - Describe the process used to select the most relevant features for the
          model, including any feature selection techniques or criteria applied,
          such as correlation analysis, mutual information, or forward/backward
          selection.
        - Explain the rationale behind including or excluding specific features,
          and discuss their importance and contribution to the model's
          performance and intended use case.
        - Detail any feature engineering techniques applied to create new
          features or transform existing ones, such as dimensionality reduction,
          aggregation, or interaction terms, and explain their relevance and
          purpose in the context of the model.
        - Describe any data normalization or scaling techniques used to
          standardize the input features, and explain the rationale for their
          application in the model.
        - Discuss potential multicollinearity, redundancy, or other issues among
          the selected features, and describe the methods used to address these
          concerns during the feature selection and engineering process.
      parent_section: data_preparation
      contents:
        - content_id: feature_selection
          content_type: text
- id: model_development
  title: Model Development
  index_only: true
  sections:
    - id: model_training
      title: Model Training
      guidelines:
        - Describe the model training process, including the algorithm used, any
          hyperparameters or settings, and the optimization techniques employed
          to minimize the loss function or maximize the objective function.
        - Detail the model validation and selection process, including the use
          of cross-validation, holdout samples, or other techniques to assess
          the model's performance and prevent overfitting.
        - Provide a summary of the training results, including performance
          metrics such as accuracy, precision, recall, F1 score, or other
          relevant measures, depending on the model's intended use case.
        - Discuss any challenges, issues, or trade-offs encountered during the
          model training process, such as overfitting, underfitting, or class
          imbalance, and explain how they were addressed or mitigated.
        - Describe any tuning or optimization steps performed to improve the
          model's performance, such as hyperparameter tuning, feature selection,
          or other adjustments, and explain the rationale for these changes.
      parent_section: model_development
      contents:
        - content_id: model_training
          content_type: text
    - id: prompt_validation
      title: Prompt Validation
      guidelines:
        - Provide documentation of the prompt used for the model. Describe the
          type of prompting used (e.g. few-shot, zero-shot, chain-of-thought
          etc.). Explain how the prompt was constructed and how it was
          validated.
      parent_section: model_development
      contents:
        - content_type: test
          content_id: validmind.prompt_validation.Bias
          options:
            title: Bias
        - content_type: test
          content_id: validmind.prompt_validation.Clarity
          options:
            title: Clarity
        - content_type: test
          content_id: validmind.prompt_validation.Conciseness
          options:
            title: Conciseness
        - content_type: test
          content_id: validmind.prompt_validation.Delimitation
          options:
            title: Delimitation
        - content_type: test
          content_id: validmind.prompt_validation.NegativeInstruction
          options:
            title: Negative Instruction
        - content_type: test
          content_id: validmind.prompt_validation.Specificity
          options:
            title: Specificity
    - id: model_evaluation
      title: Model Evaluation
      parent_section: model_development
      guidelines:
        - Describe the process used to evaluate the model's performance on a
          test or validation dataset that was not used during training, to
          assess its generalizability and robustness.
        - Present the key performance metrics for the model evaluation, such as
          accuracy, precision, recall, F1 score, AUC-ROC, mean squared error, or
          other relevant measures, depending on the model's intended use case.
        - Provide graphical representations of the model's performance, such as
          confusion matrices, ROC curves, or residual plots, to help visualize
          its effectiveness and identify any areas for improvement.
        - Discuss the model's performance in the context of its intended use
          case, and compare it to any benchmarks, industry standards, or
          alternative models, as appropriate.
        - Identify any limitations, weaknesses, or areas for improvement in the
          model's performance, and discuss potential strategies for addressing
          these concerns in future iterations or updates.
      sections:
        - id: embedding_model
          title: Embedding Model
          parent_section: model_evaluation
          guidelines:
            - Embedding model guidelines
          contents:
            - content_type: test
              content_id: validmind.model_validation.embeddings.StabilityAnalysisRandomNoise
            - content_type: test
              content_id: validmind.model_validation.embeddings.StabilityAnalysisSynonyms
            - content_type: test
              content_id: validmind.model_validation.embeddings.StabilityAnalysisTranslation
            - content_type: test
              content_id: validmind.model_validation.embeddings.CosineSimilarityHeatmap
            - content_type: test
              content_id: validmind.model_validation.embeddings.CosineSimilarityDistribution
            - content_type: test
              content_id: validmind.model_validation.embeddings.PCAComponentsPairwisePlots
        - id: retrieval_model
          title: Retrieval model
          parent_section: model_evaluation
          guidelines:
            - Retrieval model guideline
          contents:
            - content_type: test
              content_id: validmind.model_validation.ragas.ContextRecall
            - content_type: test
              content_id: validmind.model_validation.ragas.ContextEntityRecall
            - content_type: test
              content_id: validmind.model_validation.ragas.ContextPrecision
            - content_type: test
              content_id: validmind.model_validation.ragas.ContextPrecisionWithoutReference
        - id: rag_model
          title: RAG model
          parent_section: model_evaluation
          guidelines:
            - RAG model guideline
          contents:
            - content_type: test
              content_id: validmind.model_validation.ragas.SemanticSimilarity
            - content_type: test
              content_id: validmind.model_validation.ragas.Faithfulness
            - content_type: test
              content_id: validmind.model_validation.ragas.ResponseRelevancy
            - content_type: test
              content_id: validmind.model_validation.ragas.AnswerCorrectness
            - content_type: test
              content_id: validmind.model_validation.TokenDisparity
            - content_type: test
              content_id: validmind.model_validation.BleuScore
            - content_type: test
              content_id: validmind.model_validation.RougeScore
            - content_type: test
              content_id: validmind.model_validation.MeteorScore
            - content_type: test
              content_id: validmind.model_validation.ragas.AspectCritic
            - content_type: test
              content_id: validmind.model_validation.ToxicityScore
            - content_type: test
              content_id: validmind.model_validation.RegardScore
- id: monitoring_governance
  title: Monitoring and Governance
  index_only: true
  sections:
    - id: monitoring_plan
      title: Monitoring Plan
      guidelines:
        - Describe the plan for ongoing monitoring of the model's performance,
          including the frequency of evaluations, the performance metrics to be
          assessed, and any thresholds or triggers for action.
        - Explain the process for identifying and addressing any changes in the
          model's performance or the underlying data that may require model
          updates, recalibration, or retraining.
        - Detail the procedures for model validation and backtesting, to ensure
          the model remains accurate, reliable, and compliant with regulatory
          requirements and industry standards.
        - Discuss the governance framework in place to oversee the model's use,
          including the roles and responsibilities of various stakeholders, such
          as model developers, validators, and risk managers.
        - Describe the model's documentation and version control procedures, to
          ensure that changes, updates, and improvements are properly tracked
          and recorded.
      contents:
        - content_id: monitoring_plan
          content_type: text
          options:
            default_text: >
              To ensure the ongoing effectiveness of the model, it will be
              monitored on a regular basis. The monitoring plan includes the
              following steps:


              1. **Performance Metrics Monitoring:** The key performance metrics
              such as Accuracy, Precision, Recall, F1 Score, and ROC-AUC will be

              tracked on a regular basis. These metrics will be computed for
              both the training, validation and test datasets to identify any
              signs of

              overfitting or underperformance.

              2. **Data Drift Monitoring:** Over time, the distribution of the
              input data may change, a phenomenon known as data drift. This
              could

              impact the model's performance. We will monitor the distributions
              of the input features and the target variable to detect any
              significant

              changes that may require retraining of the model.

              3. **Outcome Monitoring:** The actual outcomes (whether a customer
              exited the bank or not) will be compared with the model's
              predictions

              to assess the model's performance in a real-world setting.

              4. **Feature Importance Monitoring:** The importance of different
              features for the model's predictions will be tracked. If there are

              significant shifts in feature importance, this could indicate
              changes in the underlying patterns in the data.

              5. **Periodic Model Retraining:** Depending on the findings from
              the above monitoring activities, the model may need to be
              retrained

              periodically. This will be done using the most recent data to
              ensure that the model stays up-to-date with the latest patterns
              and trends.


              In terms of governance, a clear process will be put in place for
              managing these monitoring activities. This will include clear
              roles

              and responsibilities, documentation of the monitoring results, and
              a process for deciding when and how to take action based on the

              monitoring results (for example, when to retrain the model). Any
              major decisions about the model (such as changes to the model or
              its

              features) will be made in a transparent and accountable manner,
              with appropriate documentation and sign-off.
      parent_section: monitoring_governance
    - id: monitoring_implementation
      title: Monitoring Implementation
      guidelines:
        - Describe the tools, systems, or platforms used to implement the
          monitoring plan, including any relevant software, data pipelines, or
          reporting tools.
        - Detail the process for collecting and storing the data needed for
          ongoing monitoring, including any data preprocessing, cleaning, or
          transformation steps required.
        - Explain the procedures for analyzing the model's performance metrics
          and generating monitoring reports, including any statistical tests or
          visualizations used to assess the model's performance and stability.
        - Discuss the escalation process and communication channels for
          reporting any significant deviations in the model's performance, as
          well as the decision-making process for determining appropriate
          actions, such as model updates or recalibration.
        - Describe any training or educational programs in place to ensure that
          relevant stakeholders, such as model developers, validators, and risk
          managers, are equipped to understand, interpret, and act on the
          monitoring results.
      parent_section: monitoring_governance
      contents:
        - content_id: monitoring_implementation
          content_type: text
          options:
            default_text: >
              Implementing the monitoring plan requires a systematic approach
              with clear steps and procedures. Here are the steps for
              implementing

              the monitoring plan:


              1. **Establish Baseline Metrics:** At the outset, we'll establish
              baseline performance metrics for the model. These will serve as

              reference points for future comparison.

              2. **Automate Metric Calculation:** Performance metrics such as
              Accuracy, Precision, Recall, F1 Score, and ROC-AUC will be
              automatically

              calculated and recorded for both the training and validation
              datasets after each run of the model.

              3. **Set Up Data Drift Monitors:** We'll set up automated
              monitoring of the distributions of the input features and the
              target

              variable. Any significant changes in these distributions will
              trigger alerts.

              4. **Implement Outcome Monitoring:** We'll compare the model's
              predictions with the actual outcomes on a regular basis.
              Discrepancies

              will be investigated to understand the root cause.

              5. **Track Feature Importance:** We'll keep a record of feature
              importance as indicated by the model. Any significant shifts in

              feature importance over time will be thoroughly reviewed.

              6. **Establish Retraining Protocols:** Based on the findings from
              the above monitoring activities, we'll establish clear protocols

              for when and how the model should be retrained. This may include
              criteria for triggering a retraining, procedures for carrying out

              the retraining, and protocols for testing and validating the
              retrained model before it is put back into service.

              7. **Document and Review:** All monitoring activities and their
              results will be documented in a transparent and accessible manner.

              These documents will be reviewed regularly by a designated team or
              individual to ensure that any issues are promptly identified and
              addressed.


              By implementing this monitoring plan, we can ensure that the model
              continues to perform well and that any issues are quickly

              identified and addressed. The ultimate aim is to ensure that the
              model continues to provide accurate and reliable predictions that

              can support the bank's decision-making processes.
    - id: governance_plan
      title: Governance Plan
      guidelines:
        - Describe the overall governance framework and processes established to
          ensure proper oversight and management of the model, including the
          roles and responsibilities of key stakeholders such as model
          developers, validators, and risk managers.
        - Detail the policies and procedures for model risk management,
          including model risk identification, assessment, and mitigation
          strategies.
        - Explain the model approval process, including any internal or external
          reviews, audits, or regulatory assessments that must be completed
          before the model is put into production.
        - Discuss the procedures for ongoing model maintenance, updates, and
          improvements, including the documentation and version control
          processes to track and record changes to the model.
        - Describe the contingency plans in place to manage potential model
          failures or performance issues, such as fallback models, alternative
          data sources, or manual processes, and explain the criteria for
          activating these contingency measures.
      parent_section: monitoring_governance
      contents:
        - content_id: governance_plan
          content_type: text
          options:
            default_text: >
              Effective governance of the model is crucial to ensure its
              reliability, security, and compliance with regulatory
              requirements.

              Here is the plan for model governance:


              1. **Roles and Responsibilities:** Clear roles and
              responsibilities will be assigned for model development,
              validation, deployment,

              monitoring, and retraining. This will include a model owner, who
              will have overall responsibility for the model, as well as others

              responsible for specific tasks.

              2. **Model Documentation:** Comprehensive documentation will be
              maintained for the model, including details of its development,

              validation, and performance, as well as any changes made to the
              model or its inputs over time. This documentation will be updated

              regularly and will be accessible to all relevant stakeholders.

              3. **Change Control:** Any changes to the model or its inputs will
              be subject to a strict change control process, including

              documentation of the proposed change, review and approval by a
              designated authority, testing and validation of the change, and a

              post-implementation review.

              4. **Security and Access Control:** Measures will be put in place
              to ensure the security of the model and its data, including

              access controls, data encryption, and regular security audits.

              5. **Regulatory Compliance:** The model will be designed and
              operated in compliance with all relevant regulatory requirements,

              and its compliance will be regularly reviewed and confirmed.

              6. **Auditability:** The model and its operations will be
              auditable, with clear and accessible records that can be reviewed
              by

              internal or external auditors.

              7. **Training and Awareness:** All individuals involved in the
              development, operation, and oversight of the model will receive

              appropriate training and will be kept aware of their
              responsibilities, the model's performance, and any relevant
              developments or issues.
