Batch Consumption

What Is Batch Consumption?

Batch consumption lets you process large volumes of LLM inference requests asynchronously in SAP AI Core. You package all your requests into a single JSON Lines (.jsonl) file, upload it to your object store, and submit it as a batch job. SAP AI Core processes the requests in the background and writes all results to your object store in a single output file when complete.

When to Use Batch Consumption

Use batch consumption when:

• You have 50 or more requests to process in one run
• Your workload is not time-critical
• You want to reduce inference costs compared to synchronous calls
• You want SAP AI Core to manage retries and rate limits automatically

Do not use batch consumption when:

• You need real-time responses (for example, a user-facing chat interface)
• Your requests use orchestration pipelines — only native LLM calls are supported

Batch Consumption vs Other Asynchronous Approaches

Key Constraints

• Supports native LLM calls only — orchestration requests are not supported
• All requests in a single batch job must use the same model
• Output lines are not guaranteed to be in the same order as input lines — always use custom_id to match responses to requests

Batch Limits

┌─────────────────────────────────────────────────────────────────────┐
│  Online Retailer — Mixed Product Inputs                              │
│                                                                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌───────────┐  │
│  │ Customer    │  │ Product     │  │ Product     │  │ Product   │  │
│  │ Review      │  │ Image       │  │ Image       │  │ Descrip-  │  │
│  │ (text)      │  │ (base64)    │  │ (URL)       │  │ tion      │  │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └─────┬─────┘  │
└─────────┼────────────────┼────────────────┼───────────────┼─────────┘
          └────────────────┴────────────────┴───────────────┘
                                    │
                                    │ Package as JSONL
                                    ▼
                       ┌────────────────────────┐
                       │  input-batch-mixed.jsonl│
                       │  (4 requests)           │
                       └────────────┬───────────┘
                                    │ Upload
                                    ▼
                       ┌────────────────────────┐
                       │  Object Store          │
                       │  (S3 / Azure / GCS /   │
                       │   Alibaba OSS)         │
                       └────────────┬───────────┘
                                    │ ai:// URI
                                    ▼
                       ┌────────────────────────┐
                       │  SAP AI Core           │
                       │  Batch Job             │
                       │  POST /v1/batches      │
                       └────────────┬───────────┘
                                    │ Async processing
                                    ▼
                       ┌────────────────────────┐
                       │  LLM Provider          │
                       │  (azure-openai)        │
                       └────────────┬───────────┘
                                    │ Results
                                    ▼
                       ┌────────────────────────┐
                       │  Object Store          │
                       │  output/<batch_id>/    │
                       │  output.jsonl          │
                       │  (single file, N lines)│
                       └────────────────────────┘

Generic secrets securely store object store credentials required for batch input and output file access.

Batch jobs require a JSON Lines (.jsonl) file as input. Each line is a self-contained JSON object representing one inference request. Lines must not be separated by commas, and the file must end with a newline after the last line.

Required Fields Per Line

Note: All requests in the file must use the same model.

Supported Input Types

The batch service supports three input types. Each uses a different structure for the body.messages[].content field.

Standard Text Input

The content field is a plain string. Use this for text documents, Q&A, classification, or summarisation.

{
  "role": "user",
  "content": "Your prompt text here."
}

Base64 Encoded Image Input

The content field is an array containing a text prompt and an image as a base64-encoded data URI. Use this when your image is stored locally or is not publicly accessible.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe this image." },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,<base64_string>" } }
  ]
}

Note: Base64 encoding increases the size of your JSONL file. For large images, prefer the image URL approach where possible.

Image URL Input

The content field is an array containing a text prompt and a publicly accessible image URL. The LLM provider fetches the image at processing time.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe this image." },
    { "type": "image_url", "image_url": { "url": "https://your-public-cdn.com/image.png", "detail": "high" } }
  ]
}

Important: The image URL must be reachable by the LLM provider’s servers at processing time — not just from your browser. GitHub raw URLs, internal network URLs, and short-expiry presigned URLs will fail with a 400 error. Use stable public CDN or object store URLs.

The input-batch-text.jsonl file contains three text requests in a single batch job.

request-1

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "What is SAP AI Core and what are its key capabilities?"}]}}

request-2

{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "What is Agentic AI and how does it differ from traditional AI systems?"}]}}

request-3

{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Given the following context about a facility management company:\n---\n Facility Solutions manages over 500 commercial buildings across 12 cities. Their services include HVAC maintenance, cleaning, security, and emergency repairs. They receive an average of 200 service requests daily and prioritise them based on urgency and SLA levels.\n---\nBased on this context, what AI use cases would be most valuable to implement?"}]}}

Note: request-3 demonstrates how to pass a short context passage alongside a question. The context is embedded directly in the user message — no special formatting is required by the batch service.

The input-batch-mixed.jsonl file contains four requests — one per input type.

request-1 — Standard Text

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Analyse the following customer review and return a JSON with keys: \"sentiment\" (`positive`, `neutral`, `negative`), \"category\" (`product_quality`, `delivery`, `customer_service`, `pricing`, `general`), and \"summary\" (one sentence). Return only a valid compact JSON string.\n\nReview:\n---\nI recently purchased the wireless headphones and I am extremely satisfied with the sound quality. The noise cancellation works brilliantly, battery life lasts a full day, and delivery was prompt. Highly recommend!\n---"}]}}

request-2 — Base64 Encoded Image

{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": [{"type": "text", "text": "This is a product image from our retail catalogue. Describe the product, its appearance, colour, and whether the image quality is suitable for a product listing."}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,<base64_encoded_product_image>"}}]}], "max_tokens": 1000}}

request-3 — Image URL

{"custom_id": "request-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": [{"type": "text", "text": "This is a product image from our online catalogue. Describe the product, its key visual attributes, and whether it appears suitable for a retail product listing."}, {"type": "image_url", "image_url": {"url": "https://your-public-cdn.com/product-image.png", "detail": "high"}}]}], "max_tokens": 1000}}

request-4 — Structured Output

{"custom_id": "request-4", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4.1", "messages": [{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Extract the product details from the following description:\n---\nProduct: UltraSound Pro Wireless Headphones. Premium over-ear headphones with 40-hour battery life and active noise cancellation. Available in Midnight Black and Arctic White. Currently in stock. Rated 4.7 out of 5 by over 2,300 customers. Price: $149.99.\n---"}], "response_format": {"type": "json_schema", "json_schema": {"name": "ProductDetailsResponse", "strict": true, "schema": {"type": "object", "properties": {"product_name": {"type": "string"}, "category": {"type": "string"}, "price": {"type": "string"}, "availability": {"type": "string"}, "rating": {"type": "string"}}, "required": ["product_name", "category", "price", "availability", "rating"], "additionalProperties": false}}}}}

Note: For request-2, replace <base64_encoded_product_image> with your actual base64 image string. For request-3, replace the URL with a stable, publicly accessible URL reachable by the LLM provider’s servers.

Upload your input file to your object store. SAP AI Core accesses it using the ai:// URI scheme, which maps to a registered object store secret.

URI Format

ai://<object_store_secret_name>/<file_path>/<file_name>.jsonl

Examples:

ai://batch-consumption/batch_input/input-batch-text.jsonl
ai://batch-consumption/batch_input/input-batch-mixed.jsonl

Note: Decide on your output folder URI now — you will need it in the next step. The output URI must end with /.

Example output URI:

ai://batch-consumption/batch_output/

Once the batch status is COMPLETED, download the output file from your object store.

Output File Location

ai://batch-consumption/batch_output/<batch_id>/output.jsonl

If any individual requests failed, an error file is written alongside:

ai://batch-consumption/batch_output/<batch_id>/error.jsonl

Note: A COMPLETED status means the batch as a whole was processed. Individual requests may still have failed. Always check response.status_code for every line in the output file.

Results: Batch Example 1

The output file contains three response lines — one per text request. Output order is not guaranteed — always use custom_id to match responses to inputs.

{"custom_id": "request-1", "response": {"body": {"choices": [{"message": {"content": "SAP AI Core is a service within the SAP Business Technology Platform that enables lifecycle management of AI functions. Key capabilities include training and deploying AI models at scale, serving LLM inference via the Generative AI Hub, and providing observability for deployed AI workloads."}}]}, "status_code": 200}, "error": null}
{"custom_id": "request-3", "response": {"body": {"choices": [{"message": {"content": "Valuable AI use cases for ProCare include: 1) Automated request triage using NLP to classify and route service requests by urgency and SLA. 2) Predictive maintenance to anticipate equipment failures. 3) Intelligent scheduling to assign technicians based on location and workload. 4) Sentiment analysis on customer communications to proactively identify dissatisfied clients."}}]}, "status_code": 200}, "error": null}
{"custom_id": "request-2", "response": {"body": {"choices": [{"message": {"content": "Agentic AI refers to systems that autonomously plan and execute sequences of actions to achieve a goal — using tools, APIs, or other AI models — with minimal human intervention. Unlike traditional AI that maps a single input to a single output, agentic systems maintain goals across multiple steps, adapt based on intermediate results, and interact with external environments."}}]}, "status_code": 200}, "error": null}

Results: Batch Example 2

The output file contains four response lines — one per input type.

{"custom_id": "request-1", "response": {"body": {"choices": [{"message": {"content": "{\"sentiment\":\"positive\",\"category\":\"product_quality\",\"summary\":\"The customer is highly satisfied with the sound quality, noise cancellation, battery life, and delivery of the wireless headphones.\"}"}}]}, "status_code": 200}, "error": null}
{"custom_id": "request-4", "response": {"body": {"choices": [{"message": {"content": "{\"product_name\":\"UltraSound Pro Wireless Headphones\",\"category\":\"Electronics\",\"price\":\"$149.99\",\"availability\":\"In stock\",\"rating\":\"4.7 out of 5\"}"}}]}, "status_code": 200}, "error": null}
{"custom_id": "request-2", "response": {"body": {"choices": [{"message": {"content": "The image appears to show a small graphical icon rather than a full product photograph. The image resolution is not suitable for a retail product listing. Please provide a high-resolution product photograph."}}]}, "status_code": 200}, "error": null}
{"custom_id": "request-3", "response": {"body": {"choices": [{"message": {"content": "The image shows a pair of over-ear wireless headphones in Midnight Black. The earcups are large and padded with a sleek matte finish. The product is well-lit against a clean white background, making it suitable for a retail product listing."}}]}, "status_code": 200}, "error": null}

Note: status_code: 200 means the request was processed without error. It does not mean the LLM produced the output you expected — always review the content field.

Get Batch Details

Error Codes

Note: Input JSONL files must be encoded as plain UTF-8. Files saved with a Byte-Order-Mark (BOM) — common on Windows — will fail validation. Save explicitly as UTF-8 without BOM.