Optimize Prompts for Tool Calling Using a Custom LLM-as-a-Judge Metric in SAP AI Core

Before writing any code, it helps to hold the whole flow in your head. Every step below re-uses a single authenticated client object and builds on variables created earlier, so the notebook cells (and the steps here) are meant to run in order, top to bottom.

The end-to-end pipeline is:

 ┌─────────────────┐     ┌──────────────────┐     ┌────────────────────┐
 │ 1. Connect to    │ --> │ 2. Verify/create  │ --> │ 3. Configure         │
 │    AI Core       │     │  the custom metric │     │  optimization params │
 └─────────────────┘     └──────────────────┘     └────────────────────┘
                                                              │
                                                              v
 ┌─────────────────┐     ┌──────────────────┐     ┌────────────────────┐
 │ 6. Register       │ <-- │ 5. Register base  │ <-- │ 4. Load/normalize +  │
 │  base prompt      │     │  dataset artifact  │     │  upload BFCL data     │
 └─────────────────┘     └──────────────────┘     └────────────────────┘
         │
         v
 ┌─────────────────┐     ┌──────────────────┐     ┌────────────────────┐
 │ 7. Configure +    │ --> │ 8. Retrieve the    │ --> │ 9. Compare base vs   │
 │  run + monitor     │     │  optimized prompt   │     │  optimized (live)     │
 └─────────────────┘     └──────────────────┘     └────────────────────┘

A few SAP AI Core terms used throughout:

All REST calls in the Bruno option blocks assume you have already set the collection variables baseUrl (your AI_API_URL, ending in /v2), token (a valid OAuth bearer token), and resourceGroup (your AI Core resource group). Send these headers on every request unless noted:
Authorization: Bearer {{token}}, AI-Resource-Group: {{resourceGroup}}, and Content-Type: application/json.

Create a .env file in your working directory with the credentials from your AI Core service key:

AICORE_CLIENT_ID=<your client id>
AICORE_CLIENT_SECRET=<your client secret>
AICORE_AUTH_URL=<your auth url>
AICORE_BASE_URL=<your base url>
AICORE_RESOURCE_GROUP=<your resource group>

The url in the service key maps to AICORE_AUTH_URL, and serviceurls.AI_API_URL maps to AICORE_BASE_URL. No S3 or AWS credentials are needed — all files are uploaded directly to AI Core’s built-in dataset storage.

The built-in JSON_Match metric does a rigid structural comparison between the model’s JSON and the golden answer. It catches gross formatting errors, but it can’t tell “the model called the wrong tool” apart from “the model got the right tool but phrased an argument slightly differently.” A custom LLM-as-a-judge metric solves this: a second LLM reads the candidate output alongside the golden answer and scores it against criteria you define.

This tutorial uses metric=custom, referencing a metric named tool-call-accuracy:1.0.0 in the genai-optimizations scenario. First verify whether it already exists, then create it if it doesn’t.

Define every constant the run needs. Note the difference between the reference model (gpt-4o:2024-08-06, a teacher the optimizer compares against) and the target model (gemini-2.5-pro:001, the model the final prompt is actually tuned for). The train/test split (25 train, 15 test) mirrors traditional ML: train samples refine candidate prompts, held-out test samples score them so the reported score reflects generalization, not memorization.

The optimizer doesn’t understand BFCL’s native structure — it expects each example as a golden record: {"fields": {"question": ...}, "answer": "<JSON object string>"}. This step is the translation layer. Several helper functions handle it: a robust reader (BFCL v3 ships as concatenated JSON objects with no separators), a tool-schema normalizer (BFCL’s float/dict/any → OpenAI’s number/object/string), a deduplicator for same-named tools with different schemas, and a golden-record builder.

This step is local data processing — there is no REST call, so both option blocks below run the same Python; Bruno users still need these files produced before the upload step.

Serialize the four prepared objects (bfcl_train.json, bfcl_test.json, bfcl_tools.json, bfcl_prompt_template.json) to local files, upload each to the shared remote folder default/datasets/bfcl-optimizer/, then register that folder as a single dataset artifact under the genai-optimizations scenario. The optimizer reads all files from this artifact folder.

Push the deliberately minimal base prompt (system "You are a helpful assistant.", user {{?question}}) to the Prompt Registry. Starting from a weak, generic prompt gives the optimizer maximum room to add structure — JSON-only instructions, tool schemas, reasoning steps — and makes the improvement in Step 9 obvious. {{?question}} is a placeholder that gets replaced by the actual user question at inference time.

This is the heart of the pipeline. Build a configuration from the parameters (base prompt reference, models, dataset filenames, and — crucially — the customMetricId), create an execution from it, then poll until the execution reaches a terminal state.

The metric parameters are mutually exclusive: supply exactly one of customMetricId, optimizationMetric, or field-evaluation metrics. Because metric == "custom" here, you pass customMetricId.

The Prompt Registry has no “get by name” shortcut here, so first list every prompt template to find the id matching your targetPromptMapping output name (bfcl-tool-optimized-custom:0.0.1), then fetch that ID to see its full content. The optimized prompt will be dramatically more detailed than the two-line base — expect a structured-output parser role, strict JSON constraints (no markdown, no backticks), full tool schemas across all functions with per-field normalization rules, a conflict-resolution policy, and internal reasoning steps.

This is the “proof in the pudding” step. Run four multi-tool test questions through both the base and optimized prompts on gemini-2.5-pro and compare. Live inference runs through a deployed orchestration scenario (a running service with its own URL), so first find a deployment with scenario=orchestration and status=RUNNING, then run the comparison.

Because this tutorial creates its own metric, it’s worth asking: does tool-call-accuracy (LLM-as-a-judge) actually change what the optimizer produces versus the built-in JSON_Match? Optimizing the same base prompt and dataset separately under each metric reveals a consistent difference.

The custom-metric output is more verbose in its location and entity normalization — it tends to include both a city and its country/state qualifier (“Tokyo, Japan” vs “Tokyo”) and more reliably populates optional fields like max_prep_time. The JSON_Match output is more minimal — single values per field, fewer optional fields, closer to “just enough to match a golden answer.”

This makes sense: JSON_Match rewards outputs that structurally match the golden answer as closely as possible, pushing the optimizer toward minimal, exact outputs since anything extra risks a mismatch. The LLM-as-a-judge tool-call-accuracy metric instead rewards outputs judged complete and correct by a rubric, giving the optimizer freedom to be thorough without being penalized for not matching character-for-character.

Neither is strictly better in the abstract — it depends on your downstream system. If your tool-calling consumer expects a tight, minimal argument set, JSON_Match may produce a more predictable prompt. If it benefits from richer, more complete extractions, an LLM-as-a-judge metric like tool-call-accuracy may serve you better.

To try JSON_Match yourself, re-run Step 7 with the metric parameter swapped — pass optimizationMetric = JSON_Match instead of customMetricId, and point targetPromptMapping at a new output name so the two optimized prompts don’t overwrite each other.

In the v1 orchestration /completion request, where do the prompt template messages live?

• Directly at the top level of the request body.

• Inside llm_module_config.

• Inside templating_module_config, within orchestration_config → module_configurations.

• Inside input_artifact_bindings.

When configuring a run that uses the custom tool-call-accuracy metric, which parameter must you supply — and which must you omit?

• Supply both customMetricId and optimizationMetric.

• Supply customMetricId and omit optimizationMetric — they are mutually exclusive.

• Supply optimizationMetric set to custom.

• Neither is required; the optimizer picks a metric automatically.