Custom Evaluation for Generative AI – Comprehensive Guide

⚠️ Important Note (Must Read)

• You must create an object store secret with a user defined name (for eg: default) to store output artifacts from orchestration runs. This is mandatory.
• For input artifacts, you may create additional object store secrets with different names if needed.
• If a user defined name (for eg: default) is not configured, orchestration runs will fail due to missing output target setup.

In this evaluation workflow, prompts can be provided in two different ways.
Before proceeding, understand the available approaches and choose the one that fits your requirement.

🔹 Option 1 – Prompt Template + Model (Prompt Registry)

- The prompt is stored in the Prompt Registry

- The model is referenced directly in the evaluation configuration

- Prompts are reusable and version-controlled

- Best suited for standardized or production-grade workflows

📌 When to use this?

If you want reusable, versioned prompts that can be managed independently.

👉 If you would like to see this approach in action, refer to the [Evaluation Quickstart tutorial](LINK TO ADD), where we demonstrate the Prompt Registry method.

🔹 Option 2 – Orchestration Registry (Inline Prompt)

- The prompt is defined directly inside the orchestration configuration

- No separate prompt registry entry is required

- Ideal for ad-hoc, experimental, or one-time evaluations

📌 When to use this?

If the prompt is specific to this evaluation and does not need reuse or versioning.

In this tutorial, we will use the Orchestration Registry (Inline Prompt) approach.

Create Orchestration Registry Configuration

Metrics determine how your model outputs are evaluated during an evaluation run. They define the scoring logic that SAP AI Core uses to compare models, measure quality, and validate improvements over time.

In SAP AI Core, metrics are configured during the Create Evaluation Configuration step:

"metrics": "Content Filter on Input,Pointwise Instruction Following,Content Filter on Output"

You can specify one or multiple metrics (comma-separated).

Types of Metrics

SAP AI Core supports two major types:

1. System-defined Metrics (Ready to use)

2. Custom Metrics (User-defined)

1. System-defined Metrics

These are built-in metrics provided by SAP AI Core. No additional setup required.

They are grouped into two categories:

Computed Metrics

These use reference data, schema validation, or deterministic logic.

👉 Use computed metrics when:

- You have ground truth/reference answers

- You need deterministic validation

- You want schema validation

model-as-a-judge metrics

These use a judge LLM to evaluate responses qualitatively.

| Name | Description | Reference required |——————————————————————————————————————–
| Pointwise Instruction Following | assess the model’s ability to follow instructions provided in the user prompt | No |
| Pointwise Correctness | assess the model’s ability to provide a correct response based on the user prompt | Yes |
| Pointwise Answer Relevance | assess the model’s response is related to user prompt | No |
| Pointwise Conciseness | assess the model’s response is a short and concise answer to user prompt | No |
|

Entries marked with an asterisk () are experimental metrics.

👉 Use model-as-a-judge metrics when:

- You need qualitative evaluation

- No exact ground truth exists

- You want human-like evaluation logic

Custom Metrics (User-defined metrics)

When system metrics are insufficient, you can define your own metric.

Custom metrics can be used to evaluate the LLM outputs according to the unique needs of a use case. A user-defined llm-as-a-judge metric uses a judge LLM along with a rubric to compute a metric rating. The output of a llm-as-a-judge metric can be numeric or text.

The system defines a structure for the judge prompts and users provide the metric definition in the pre-defined format. Relevant instructions, such as output instructions, are automatically added to ensure the desired output from the LLM.

Custom Metric Definition Structure

{
  "scenario": "genai-evaluations",
  "metricName": "my_custom_metric",
  "version": "0.0.1",
  "type": "structured",
  "model_configuration": {
    "model_name": "string",
    "model_version": "string"
  },
  "prompt_configuration": {
    "evaluation_task": "Describe the goal of this evaluation.",
    "criteria": "Explain how evaluation is performed.",
    "rating_rubric": [
      {
        "rating": 1,
        "rule": "Poor quality response"
      },
      {
        "rating": 5,
        "rule": "Excellent response"
      }
    ],
    "include_properties": ["prompt", "reference"],
    "examples": [
      {
        "prompt": "Sample prompt",
        "response": "Sample response",
        "reference": "Expected answer",
        "rating": 5,
        "explanation": "Why this rating was given"
      }
    ]
  }
}

NOTE: “scenario” and “metricName” and “version” is a required parameter for the custom metric in evaluation configuration.

NOTE: The user must provide at least one prompt, system or user prompt, or both prompts can be provided.

Model Availability Notice

⚠️ If gpt-4.1 (2025-04-14) is not available in your region:

- LLM-as-a-Judge metrics cannot be executed

- Evaluation service depends on this specific model version

Metrics determine how your model outputs are evaluated during an evaluation run. They define the scoring logic that SAP AI Core uses to compare models, measure quality, and validate improvements over time.

Metrics must be supplied before creating an Evaluation Configuration.

Note

To evaluate and compare multiple models in a single execution, you must create a distinct orchestration registry ID for each model you wish to test. Assign a different foundation model to each registry ID, and then pass this list of registry IDs into your evaluation configuration. This ensures the system generates separate, comparable runs for each model simultaneously.

After creating the Evaluation Configuration, the next step is to execute it.

Execution triggers the evaluation workflow, which:

- Reads the test dataset

- Generates submissions to the orchestration service

- Collects model outputs

- Computes all selected metrics

- Produces aggregate and raw evaluation results

The process is identical for SAP AI Launchpad, Python, and Bruno, with only the invocation method differing.

Once the evaluation execution is complete, SAP AI Core generates both aggregated metrics and detailed instance-level results.
These results help compare model performance, understand quality metrics, and debug issues.

Over time, your workspace may accumulate old configurations, executions, and metrics.
SAP AI Core allows you to safely delete these resources once they are no longer needed.

This section explains how to delete:

- Evaluation Executions

- Evaluation Configurations

⚠️ Important:

Deletions are permanent and cannot be undone.