For the complete documentation index, see llms.txt. This page is also available as Markdown.

Batch Processing at Scale

Process large volumes of LLM inferences, embeddings, and multimodal inputs at scale with Parasail Batch.

Process thousands or millions of inferences offline at 50% off serverless pricing. Batch is ideal for evaluations, data processing, content generation, and any workload that doesn't need real-time responses.

Why batch?

  • 50% off serverless token pricing

  • Additional 50% off for cached tokens

  • Up to 50,000 requests and 1000 MB per input file

  • OpenAI-compatible—use the same batch file format and API

  • Any HuggingFace model—no dedicated deployment needed

  • Resumable—submit in one process, monitor in another

Quickstart

import random
from openai_batch import Batch

with Batch() as batch:
    objects = ["cat", "robot", "coffee mug", "spaceship", "banana"]
    for i in range(100):
        batch.add_to_batch(
            model="NousResearch/DeepHermes-3-Mistral-24B-Preview",
            messages=[{"role": "user", "content": f"Tell me a joke about a {random.choice(objects)}"}]
        )
    result, output_path, error_path = batch.submit_wait_download()
    print(f"Batch completed with status {result.status}")

See the batch quickstart for the full guide.

What you can batch

Task
Example models
Guide

Chat completions

Any HuggingFace transformer

Embeddings

GritLM, GTE-Qwen2

Image understanding

Qwen3-VL-8B-Instruct

Private models

Any private HuggingFace repo

Pricing

Batch is billed at 50% of serverless pricing, based on parameter count. FP8 models incur no additional cost; FP16 models are 30% more. See full pricing details.

Relevant pages

Last updated