Batch Processing at Scale

Process large volumes of LLM inferences, embeddings, and multimodal inputs at scale with Parasail Batch.

Process thousands or millions of inferences offline at 50% off serverless pricing. Batch is ideal for evaluations, data processing, content generation, and any workload that doesn't need real-time responses.

Why batch?

• 50% off serverless token pricing

• Additional 50% off for cached tokens

• Up to 50,000 requests and 1000 MB per input file

• OpenAI-compatible—use the same batch file format and API

• Any HuggingFace model—no dedicated deployment needed

• Resumable—submit in one process, monitor in another

Quickstart

import random
from openai_batch import Batch

with Batch() as batch:
    objects = ["cat", "robot", "coffee mug", "spaceship", "banana"]
    for i in range(100):
        batch.add_to_batch(
            model="NousResearch/DeepHermes-3-Mistral-24B-Preview",
            messages=[{"role": "user", "content": f"Tell me a joke about a {random.choice(objects)}"}]
        )
    result, output_path, error_path = batch.submit_wait_download()
    print(f"Batch completed with status {result.status}")

See the batch quickstart for the full guide.

What you can batch

Task	Example models	Guide
Chat completions	Any HuggingFace transformer	Batch full guide
Embeddings	GritLM, GTE-Qwen2	Batch embedding models
Image understanding	Qwen3-VL-8B-Instruct	Batch image understanding
Private models	Any private HuggingFace repo	Batch private models

Pricing

Batch is billed at 50% of serverless pricing, based on parameter count. FP8 models incur no additional cost; FP16 models are 30% more. See full pricing details.

Relevant pages

• Batch quickstart—comprehensive guide

• Batch helper GitHub repo—library source and README

• File format—JSONL input format

• API reference—OpenAI-compatible batch API

• Priority—batch priority levels

• Troubleshooting