Prompt Optimization with Tool Calling and Response Formatting

Before starting this tutorial, ensure that you:
- Understand the basics of Generative AI workflows in SAP AI Core.
- Are familiar with function calling / tool calling concepts in LLMs.
- Are familiar with creating and managing prompt templates and artifacts in SAP AI Core.
- Understand, at a high level, that there are two complementary ways to push a model toward structured output:
- Prompt optimization — automate the trial-and-error of prompt engineering so the model’s reasoning reliably produces the right structure.
- Response Formatting — an Orchestration Service parameter that constrains the model’s output at the API level, independent of prompt wording.
- Have completed the Quick Start tutorial or equivalent setup for SAP AI Core access.

• Prompt Optimization for tool calling connects the Prompt Registry, AI Core Dataset Storage, and ML Tracking Service to form an end-to-end optimization workflow. Response Formatting sits on the Orchestration Service and is applied at inference time.
• A BFCL v3 dataset is loaded, normalized, and split into train and test sets. All four files (train goldens, test goldens, tool definitions, and prompt template) are uploaded to a shared folder in AI Core’s built-in dataset storage.
• The shared folder is registered as a single artifact under the genai-optimizations scenario.
• The base prompt template is pushed to the Prompt Registry.
• An optimization configuration links the artifact, base prompt, reference model (gpt-4o:2024-08-06), target model (gemini-2.5-pro:001), and the JSON_Match metric.
• During execution, the optimizer iteratively refines the prompt. Metrics are tracked in the ML Tracking Service, and the optimized prompt is saved back to the Prompt Registry.
• Separately, a JSON Schema is built from the unioned tool definitions. At inference time it can be attached as a response_format to the Orchestration call to guarantee the wire format is valid JSON — regardless of prompt wording.
• After completion, the base and optimized prompts are fetched and compared via live inference, both with and without the response_format schema.

For hands-on execution and end-to-end reference, use the accompanying notebook Prompt_Optimization_With_Tool_Calling_And_Response_Formatting.ipynb. It runs the entire pipeline top to bottom — from loading the BFCL dataset and uploading files, through configuration creation, execution, monitoring, and the inference comparison — and includes a Response Formatting primer right after the connection step.

💡 Run the cells in order, top to bottom — later cells depend on variables created earlier (client, configuration_id, execution_id, tool_call_schema, etc.). Configure your .env file and BFCL dataset path before executing.

To use the notebook:
- Download and open Prompt_Optimization_With_Tool_Calling_And_Response_Formatting.ipynb in your preferred environment (e.g., VS Code, JupyterLab).
- Place your BFCL v3 dataset file (e.g., BFCL_v3_parallel_multiple_10tools.json) in the same directory.
- Configure your .env file with your AI Core credentials.
- Execute the cells in order to reproduce the complete prompt optimization with tool calling and response formatting workflow.

Before running the pipeline, define the key configuration constants used throughout the notebook.

• The reference model (REFERENCE_MODEL, e.g. gpt-4o:2024-08-06) acts as a teacher the optimizer compares against while searching for a better prompt.
• The target model(s) (TARGET_MODELS) are the model(s) the final optimized prompt is actually tuned for.
• Train samples (N_TRAIN_SAMPLES = 25) are used to generate and refine candidate prompts; test samples (N_TEST_SAMPLES = 15) are held out and only used to score each candidate, so the reported score reflects genuine generalization.
• JSON_Match does a structural comparison between the candidate’s JSON output and the golden answer — simpler to set up than a custom LLM-as-a-judge metric and well suited to exact structural correctness.

The BFCL v3 dataset contains parallel and multi-tool function calling samples. The notebook uses a robust reader that handles three file formats — JSON array, standard JSONL, and concatenated JSON objects — before normalizing samples into the SAP optimizer golden format. The optimizer expects each example as a golden record: {"fields": {"question": ...}, "answer": "<JSON string>"}.

All four files — train goldens, test goldens, tool definitions, and the prompt template — are serialized locally and then uploaded to a shared folder in AI Core’s dataset storage. The shared folder is then registered as a single artifact.

The base prompt template is pushed to the Prompt Registry. It is intentionally minimal — system "You are a helpful assistant." and user {{?question}} — so the optimizer has maximum room to add structure (JSON-only output instructions, tool schemas, reasoning steps) and you can clearly see the value it adds in the final comparison.

The optimization configuration links the artifact, base prompt, reference model, target model, and metric into one executable setup. Creating a configuration does not run anything yet — think of it as saving a recipe.

⚠️ Note: Model availability and versions (for example, gpt-4o:2024-08-06, gemini-2.5-pro:001) may vary across SAP AI Core tenants. Always verify available models in Generative AI Hub → Models before use.
For the latest updates, refer to SAP Note 3437766 – Model Availability and Support for Generative AI Hub.

After registering the configuration, trigger the optimization execution. The optimizer iteratively refines the base prompt using the train goldens and evaluates candidate prompts against the test goldens using the JSON_Match metric.

After triggering the execution, monitor its status. The execution transitions through UNKNOWN → RUNNING (with progress updates) → COMPLETED. Expect roughly 20–30 minutes for 25 train + 15 test samples. The notebook also fetches logs automatically if the execution fails.

Once the execution completes, the optimized prompt is stored in the Prompt Registry. Retrieve it by its ID to inspect how the optimizer refined the base prompt.

After retrieving both prompt templates, run live inference on the same test questions to compare output quality between the base and optimized prompts — and, additionally, demonstrate what response_format=json_schema adds on top of the optimized prompt.

In this tutorial, you optimized a function-calling prompt using SAP AI Core with a BFCL v3 dataset, and paired it with Response Formatting for API-level structured-output guarantees:

1. Learned Response Formatting — the three response_format modes (text, json_object, json_schema) and how they complement prompt optimization by constraining output at the API level rather than through prompt wording.
2. Loaded and normalized the BFCL v3 dataset — using a robust multi-format reader and normalizing tool definitions to OpenAI ChatCompletions format.
3. Split the dataset into 25 train goldens and 15 test goldens, built a union of all tool definitions, and derived a json_schema response_format from those tools.
4. Uploaded four files (train, test, tools, prompt template) to a shared folder in AI Core’s built-in dataset storage via the /lm/dataset/files endpoint.
5. Registered a dataset artifact linking the shared folder to the genai-optimizations scenario.
6. Pushed the base prompt template (bfcl-tool-base:0.0.1) to the Prompt Registry.
7. Created an optimization configuration with 15 parameter bindings including the JSON_Match metric, reference model (gpt-4o:2024-08-06), and target model (gemini-2.5-pro:001).
8. Triggered and monitored the execution — tracking real-time progress from UNKNOWN through RUNNING to COMPLETED.
9. Retrieved the optimized prompt (bfcl-tool-optimized-gemini25:0.0.1) from the Prompt Registry.
10. Compared base vs optimized prompts via live inference — confirming the optimization WIN — and demonstrated the optimized prompt with and without a response_format schema, showing how prompt optimization and Response Formatting combine for defense-in-depth structured output.