Dedicated Instances
Deploy any Hugging Face model on a private GPU endpoint with full control over hardware, scaling, and latency.
Dedicated Instances are private endpoints that let you run most Hugging Face transformers on our full range of GPUs in a few clicks, at competitive prices. They are ideal for production endpoints that need full control over performance and QoS, and for experimenting with models that aren't available on public serverless endpoints.
When to use Dedicated Instances
Use Dedicated Instances when you need to:
Run a specific or private model that isn't offered on Serverless.
Guarantee predictable latency and throughput for a production workload.
Control the hardware, quantization, scaling, and concurrency of your deployment.
If you just want to call a popular open-source model with no setup, use Serverless instead. For large offline workloads, use Batch.
Advantages
Experiment and deploy from one place—run most Hugging Face transformers by entering the URL.
Private models—run custom models from your private Hugging Face repo with an access token.
Hardware choice—select from a variety of GPU types, from RTX 4090 up to H200, to match your cost and performance needs.
Scale with traffic—add load-balanced replicas as demand grows.
Aggressive latency targets—through GPU selection, model quantization, and speculative decoding.
Pay only when active—no contracts or commitments; pause and resume at any time, and billing only runs while the model is active.
Prerequisites
A Parasail account and an API key (create one here).
A model hosted publicly or privately on Hugging Face. Local models can be loaded by pushing them to a private Hugging Face repo.
Model compatibility
Behind the scenes, we work with a variety of inference stacks to maintain the best model compatibility and performance. For models launched in coordination with the community (for example, Llama models), we typically have 0-day support. Models dropped unannounced can take a few days to a week. If a model you need isn't supported, let us know on Discord or through your support channel and we'll investigate.
We currently support most transformers. Support for diffusion models and more conventional AI models like CNNs or LSTMs is further down our roadmap. Some models aren't supported because they contain local code execution in the model repo; we can whitelist trusted model providers, so contact us if you'd like us to investigate a specific model.
Create a Dedicated Instance
Open the Create New Dedicated Instance page, then configure the deployment:
Deployment Name—a name you choose; you'll use it as the model name when calling the endpoint.
Hugging Face ID / URL—the model to load. You can paste either the full URL or the model ID. For example, for the FP8 version of Qwen2 72B Instruct, use
https://huggingface.co/neuralmagic/Qwen2-72B-Instruct-FP8orneuralmagic/Qwen2-72B-Instruct-FP8. (Hugging Face has a copy button next to each model ID.)Hugging Face Token—load a model from a private Hugging Face repo, ideal for models fine-tuned on private data. See Private Hugging Face Models for how to generate a token.
If the model is supported you'll get a green confirmation message. If it isn't, contact us on Discord or through your support channel.
After selecting your model, choose a hardware service tier, then hit Deploy. To set advanced options, expand the Advanced dropdown.
Once deployed, you're taken to the status page. If the model weights were cached previously, startup takes 3–5 minutes. If the weights must be downloaded from Hugging Face, it can take from 10 minutes to several hours depending on model size.
Advanced options
Speculative Decoding Draft Model Name—set a draft model for faster output with speculative decoding. This is an advanced feature; read Speculative Decoding first.
Replicas—spin up multiple load-balanced replicas for high-throughput production. The hourly cost multiplies by the number of replicas.
Model Mode—most models autodetect, but some support multiple modes from the same weights (for example GritLM, which is both generative and embedding). Use this to force a specific mode if autodetect picks the wrong type.
Scale Down Policy—for experimentation or R&D, set a scale-down policy to shut the model down after a period of inactivity, so a forgotten endpoint doesn't run up a bill. See Auto-Scaling.
Max Concurrent Connections per Replica—when concurrent requests reach this limit, the endpoint returns 429 Too Many Requests until space frees up. This prevents overload and protects quality of service.
Dedicated status page
Once a model is deployed, you can control it and view its metrics on the status page.
Pause—temporarily deactivate the endpoint and pause billing.
Edit—change model settings and redeploy.
Destroy—permanently delete the endpoint.
Next steps
Dedicated Instances Quickstart—deploy and call your first model
Private Hugging Face Models—deploy models from a private repo
Auto-Scaling—scale replicas for production traffic
Management API—create and manage deployments programmatically
Speculative Decoding—reduce latency with a draft model
Last updated