> For the complete documentation index, see [llms.txt](https://docs.parasail.io/parasail-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.parasail.io/parasail-docs/products/overview-1/management-api.md). # Management API The Dedicated Endpoint Management API lets you create and manage Dedicated Instance deployments programmatically. You can use it to check model support, select compatible hardware, create deployments, poll status, update replica counts, pause, resume, and delete deployments. For first-time model and hardware selection, start in the Parasail UI when possible. The UI surfaces compatibility and hardware recommendations that are easier to validate visually before automating the same flow through the API. If you did not create the dedicated endpoint with the API, you can find the `deployment_id` from the Parasail SaaS platform URL. For example, if your deployment status page is `https://www.saas.parasail.io/dedicated/33665`, then the `deployment_id` is `33665`. You can then use this ID for the API commands below. ## Base URLs * **Control API** (for managing deployments):\ `https://api.parasail.io/api/v1` * **OpenAI-compatible inference API** (for inference requests):\ `https://api.parasail.io/v1` ## Authentication All requests to the Control API must include a Bearer token in the header. For example: ```python import httpx api_key = "" control_base_url = "https://api.parasail.io/api/v1" headers = { "Authorization": f"Bearer {api_key}", "Content-Type": "application/json" } client = httpx.Client(base_url=control_base_url, headers=headers) ``` ## Recommended Dedicated Model API Flow Given the large set of combinations of model types and hardware instances, the recommended process for using the Dedicated API follows the UI flow and leverages our performance/memory models and hardware compatibility checkers to ensure smooth operation. The recommended flow is: 1. Check model compatibility with our `/dedicated/support` endpoint. 2. Retrieve a list of supported hardware for that model with our `/dedicated/devices` endpoint. This will include multi-GPU configurations that leverage parallelism. 3. Within that list, set `selected` to `True` for the desired configurations. 4. Submit (`POST`) a deployment payload (including your selected `deviceConfigs`) to `/dedicated/deployments`. 5. Use `/dedicated/deployments` endpoints to monitor, update, pause, resume, and delete deployments. ### 1. Checking Model Compatibility The `/dedicated/support` endpoint can be used to query whether a model is supported. The `modelName` parameter can either be a Hugging Face ID like `Qwen/Qwen2.5-72B-Instruct` or a Hugging Face URL like . Use the `engine` query parameter (for example `VLLM`) when needed. ```python params = { "modelName": "Qwen/Qwen2.5-72B-Instruct", "engine": "VLLM", "modelAccessKey": "" # HF token for privately hosted models } # Make the request response = client.get("/dedicated/support", params=params) if response.status_code == 200: data = response.json() if data.get("supported") is True: print("✅ Model is supported.") else: reason = data.get("errorMessage", "No reason provided.") print(f"❌ Model not supported: {reason}") else: print(f"❌ Request failed with status code {response.status_code}: {response.text}") ``` This step is optional, but is a useful sanity check for API-based flows to ensure the IDs are entered correctly. ### 2. Retrieving Supported Hardware The first step to creating a new dedicated deployment is to select a supported hardware configuration. The `/dedicated/devices` endpoint returns valid hardware configurations and their availability. These configurations can be either single GPUs or multiple GPUs with parallelism enabled (tensor, expert, or sequence). The `selected` flag is set to `false` when returned from `/dedicated/devices`. The recommended flow is to flip `selected` to `true` for your desired configurations, then pass that array to the create deployment endpoint (see next section). The entire structure does not need to be passed to the create dedicated instance endpoint. The required flag shows the parameters that are needed, though it is recommended to ensure compatibility by first calling `/dedicated/devices` and flipping `selected` to true. `/dedicated/devices` returns an array of structures with the following parameters: **DeviceConfigs structure** | Parameter | Type | Required | Description | | ------------------------ | ------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `device` | string | Yes | Device ID. | | `count` | integer | Yes | Number of GPUs in the configuration. Parasail chooses the parallelism strategy for multi-GPU configurations. Use multiple replicas for data parallelism. | | `displayName` | string | No | Human-readable hardware name shown in the UI. | | `cost` | float | No | Estimated hourly cost for the full hardware configuration. | | `estimatedSingleUserTps` | float | No | Estimated output tokens per second for a single user. This estimate may be out of date; do not rely on it for capacity planning. | | `estimatedSystemTps` | float | No | Estimated total output tokens per second for the configuration. This estimate may be out of date; do not rely on it for capacity planning. | | `recommended` | boolean | No | Whether Parasail recommends this configuration for the model. | | `limitedContext` | boolean | No | Whether the full model context may exceed available memory. If true, reduce context length with `contextLengthOverride`. | | `available` | boolean | No | Whether the hardware is currently available for the account and requested count. This field reflects a single replica. | | `selected` | boolean | Yes | Set to `true` on the configuration you want the scheduler to consider when creating the deployment. | Here is example code to retrieve supported hardware, look for a desired configuration, and flip the selected flag to true when it is found: ```python def get_and_select_device_config(client: httpx.Client, model_name: str, desired_device: str, desired_count: int) -> list[dict]: params = { "engine": "VLLM", "modelName": model_name, "modelAccessKey": "" } response = client.get("/dedicated/devices", params=params) if response.status_code != 200: raise RuntimeError(f"Request failed: {response.status_code} - {response.text}") devices = response.json() matched = False for d in devices: d["selected"] = d.get("device") == desired_device and d.get("count") == desired_count matched = matched or d["selected"] if not matched: raise ValueError(f"No matching device config for {desired_device} x{desired_count}") return devices devices = get_and_select_device_config( client, model_name="Qwen/Qwen2.5-72B-Instruct", desired_device="H100SXM", desired_count=2 ) ``` ### 3. Creating a Deployment | Parameter | Type | Required | Description | | ----------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------- | | `deploymentName` | string | Yes | Unique deployment name. Use lowercase letters, numbers, and dashes. | | `modelName` | string | Yes | Hugging Face model ID, such as `Qwen/Qwen2.5-72B-Instruct`. | | `deviceConfigs` | array | Yes | Device configuration array from `/dedicated/devices`, with one or more entries marked `selected: true`. | | `replicas` | integer | Yes | Number of replicas to run. | | `scaleDownPolicy` | string | No | Autoscaling policy, such as `NONE` or `TIMER`. | | `scaleDownThreshold` | integer | No | Idle threshold in milliseconds before scale-down when using a timer policy. | | `draftModelName` | string | No | Advanced draft model setting. | | `engine` | string | No | Inference engine. Defaults to `VLLM`. | | `engineTask` | string | No | Engine task mode. Defaults to `AUTO`. | | `contextLengthOverride` | integer | No | Advanced override for context window length. | | `modelAccessKey` | string | No | Hugging Face token for private models. | Use the `POST /dedicated/deployments` endpoint to create a new model deployment. The response includes an `id` for future operations (status checks, updates, etc.). After the model is created, poll the status to monitor the transition from `STARTING` to `ONLINE` before the endpoint accepts requests. This model also appears on your dedicated status page in the platform UI. This uses the `devices` struct from the previous example: ```python # Example: Creating a deployment deployment_config = { "deploymentName": "myqwen", "modelName": "Qwen/Qwen2.5-72B-Instruct", "engine": "VLLM", # Optional (defaults to VLLM) "deviceConfigs": devices, "replicas": 1, # Optional parameters: # "scaleDownPolicy": "TIMER", # "scaleDownThreshold": 3600000, } response = client.post("/dedicated/deployments", json=deployment_config) if response.status_code == 200: deployment_data = response.json() deployment_id = deployment_data["id"] print(f"Created deployment with ID: {deployment_id}") else: print(f"Failed to create deployment: {response.status_code}, {response.text}") ``` **Response Body Example:** ```json { "id": 12345, "deploymentName": "myqwen", "modelName": "Qwen/Qwen2.5-72B-Instruct", "status": { "status": "STARTING", "statusMessage": "Deployment is being created" }, "externalAlias": "myorganization-myqwen", ... } ``` * **id**: Unique numeric identifier for the deployment. * **status.status**: Current state of the deployment (for example, `STARTING`, `ONLINE`). * **externalAlias**: The string used to reference your model via OpenAI-compatible calls. ### 4. Getting Deployment Status Use the `GET /dedicated/deployments/{deployment_id}` endpoint to retrieve the details of your deployment, including its status. ```python deployment_id = 12345 # from creation step or UI response = client.get(f"/dedicated/deployments/{deployment_id}") if response.status_code == 200: status_data = response.json() model_status = status_data["status"]["status"] # for example "ONLINE", "OFFLINE" model_identifier = status_data["externalAlias"] # alias for OpenAI API calls print(f"Deployment status: {model_status}") print(f"OpenAI model identifier: {model_identifier}") else: print(f"Failed to get deployment: {response.status_code}, {response.text}") ``` **Key fields in the response**: * `status["status"]`: The overall state of the deployment. * `externalAlias`: The string you use in OpenAI Chat/Completion requests (`model=`). Below are the possible values for `status["status"]`: | State | Description | | ------------ | ----------------------------------------------------------------- | | **ONLINE** | Deployment is active and ready to serve requests. | | **STARTING** | Deployment is in the process of starting up. | | **STOPPING** | Deployment is in the process of shutting down. | | **OFFLINE** | Deployment is not running and must be resumed to accept requests. | | **ERROR** | Deployment entered an error state and needs investigation. | | **UNKNOWN** | Deployment state is unknown (for example, stale or missing data). | ### 5. Updating a Deployment To change the configuration of an existing deployment (for example, number of replicas): 1. **GET** the current deployment (as above). 2. Modify the returned JSON structure to include your updated parameters. 3. **PUT** the entire updated JSON back to the same endpoint (`/dedicated/deployments/{deployment_id}`). #### Example: Scaling up the replicas ```python # 1. Fetch the current deployment deployment_id = 12345 response = client.get(f"/dedicated/deployments/{deployment_id}") current_deployment = response.json() # 2. Modify the "replicas" field (for example, from 1 to 2) current_deployment["replicas"] = 2 # 3. Send the updated deployment object update_response = client.put( f"/dedicated/deployments/{deployment_id}", json=current_deployment ) if update_response.status_code == 200: updated_deployment_data = update_response.json() print("Deployment updated successfully.") print(f"New replica count: {updated_deployment_data['replicas']}") else: print(f"Failed to update deployment: {update_response.status_code}, {update_response.text}") ``` ### 6. Pausing and Resuming a Deployment You may want to pause a deployment to reduce costs. Pausing puts the deployment into an `OFFLINE` state and stops billing. When ready, you can resume it to bring it back `ONLINE`. When the resume command is sent, you will need to poll the status to monitor for the transition from `STARTING` to `ONLINE` before the endpoint will accept requests. ```python # Pause the deployment deployment_id = 12345 pause_response = client.post(f"/dedicated/deployments/{deployment_id}/pause") if pause_response.status_code == 200: print("Deployment paused successfully.") else: print(f"Failed to pause: {pause_response.status_code}, {pause_response.text}") # Resume the deployment resume_response = client.post(f"/dedicated/deployments/{deployment_id}/resume") if resume_response.status_code == 200: print("Deployment resumed successfully.") else: print(f"Failed to resume: {resume_response.status_code}, {resume_response.text}") ``` ### 7. Using the Deployed Model via OpenAI-Compatible API Once your deployment is `ONLINE`, you can send inference requests using the **Chat API** at `https://api.parasail.io/v1`. Use the `externalAlias` as the `model` parameter in your OpenAI calls. Example using the OpenAI Python client: ```python from openai import OpenAI openai_client = OpenAI(api_key=api_key, base_url="https://api.parasail.io/v1") # Assume you fetched this alias from /dedicated/deployments/{deployment_id} model_identifier = "parasail-custom-model-id" response = openai_client.chat.completions.create( model=model_identifier, messages=[{"role": "user", "content": "What's the capital of France?"}], ) print(response.choices[0].message.content) ``` **Note**: The `model_identifier` is the `externalAlias` from the deployment status. ### 8. Deleting a Deployment To permanently delete an endpoint, send the `DELETE` request. ```python deployment_id = 12345 delete_response = client.delete(f"/dedicated/deployments/{deployment_id}") if delete_response.status_code == 200: print("Deployment deleted successfully.") else: print(f"Failed to delete deployment: {delete_response.status_code}, {delete_response.text}") ``` ### Example Python Code ```python import httpx import openai from pydantic import BaseModel from typing import Optional import os import time from dotenv import load_dotenv # Load environment variables env_path = os.path.expanduser("~/dev/parasail/.env") if os.path.exists(env_path): load_dotenv(env_path) api_key = os.environ["PARASAIL_API_KEY"] control_base_url = "https://api.parasail.io/api/v1" chat_base_url = "https://api.parasail.io/v1" class deviceConfig(BaseModel): device: str count: int displayName: Optional[str] = None cost: Optional[float] = None estimatedSingleUserTps: Optional[float] = None estimatedSystemTps: Optional[float] = None recommended: Optional[bool] = None limitedContext: Optional[bool] = None available: Optional[bool] = None selected: bool class CreateModelStruct(BaseModel): deploymentName: str modelName: str replicas: int deviceConfigs: list[deviceConfig] engine: Optional[str] = "VLLM" engineTask: Optional[str] = "AUTO" scaleDownPolicy: Optional[str] = None # NONE, TIMER, INACTIVE scaleDownThreshold: Optional[int] = None # time in ms draftModelName: Optional[str] = None contextLengthOverride: Optional[int] = None modelAccessKey: Optional[str] = None def check_compatibility(parasail_client: httpx.Client, model_name: str): params = { "modelName": model_name, "engine": "VLLM", "modelAccessKey": "", # HF token for privately hosted models } response = parasail_client.get("/dedicated/support", params=params) if response.status_code == 200: data = response.json() if data.get("supported") is True: print("✅ Model is supported.") else: reason = data.get("errorMessage", "No reason provided.") raise ValueError(f"❌ Model not supported: {reason}") else: raise RuntimeError( f"❌ Request failed with status code {response.status_code}: {response.text}" ) def get_and_select_device_config( parasail_client: httpx.Client, model_name: str, desired_device: str, desired_count: int, ) -> list[dict]: params = {"engine": "VLLM", "modelName": model_name, "modelAccessKey": ""} response = parasail_client.get("/dedicated/devices", params=params) if response.status_code != 200: raise RuntimeError(f"Request failed: {response.status_code} - {response.text}") devices = response.json() matched = False for d in devices: d["selected"] = ( d.get("device") == desired_device and d.get("count") == desired_count ) matched = matched or d["selected"] if not matched: raise ValueError( f"No matching device config for {desired_device} x{desired_count}" ) return devices def create_deployment( parasail_client: httpx.Client, deployment: CreateModelStruct ) -> dict: response = parasail_client.post( "/dedicated/deployments", content=deployment.model_dump_json() ) assert ( 200 <= response.status_code < 300 ), f"Failed to create deployment: {response.status_code} {response.text}" return response.json() def update_deployment(parasail_client: httpx.Client, deployment: dict) -> dict: assert deployment["id"] is not None, "Deployment ID must be set for update" response = parasail_client.put( f"/dedicated/deployments/{deployment['id']}", json=deployment ) assert ( 200 <= response.status_code < 300 ), f"Failed to update deployment: {response.status_code} {response.text}" return response.json() def pause_deployment(parasail_client: httpx.Client, id: int) -> dict: response = parasail_client.post(f"/dedicated/deployments/{id}/pause") assert ( 200 <= response.status_code < 300 ), f"Failed to pause deployment: {response.status_code} {response.text}" def resume_deployment(parasail_client: httpx.Client, id: int) -> dict: response = parasail_client.post(f"/dedicated/deployments/{id}/resume") assert ( 200 <= response.status_code < 300 ), f"Failed to resume deployment: {response.status_code} {response.text}" def get_deployment(parasail_client: httpx.Client, id: int) -> dict: response = parasail_client.get(f"/dedicated/deployments/{id}") assert ( 200 <= response.status_code < 300 ), f"Failed to get deployment: {response.status_code} {response.text}" return response.json() def is_deployment_offline(parasail_client: httpx.Client, id: int) -> bool: deployment = get_deployment(parasail_client, id) return deployment["status"]["status"] == "OFFLINE" def delete_deployment(parasail_client: httpx.Client, id: int) -> None: response = parasail_client.delete(f"/dedicated/deployments/{id}") assert ( 200 <= response.status_code < 300 ), f"Failed to delete deployment: {response.status_code} {response.text}" def wait_for_status( parasail_client: httpx.Client, id: int, status: str = "ONLINE", timeout_minutes: int = 20, ) -> None: start_time = time.time() timeout_seconds = timeout_minutes * 60 while True: deployment = get_deployment(parasail_client, id) current_status = ( deployment["status"]["status"] if "status" in deployment else "UNKNOWN" ) if current_status == status: print(f"Deployment reached target status: {status}") return elif current_status == "STARTING" or current_status == "STOPPING": elapsed_minutes = (time.time() - start_time) / 60 print( f"Deployment status: {current_status} (elapsed time: {elapsed_minutes:.1f} minutes)" ) else: raise ValueError(f"Unexpected deployment status: {current_status}") if time.time() - start_time > timeout_seconds: raise TimeoutError( f"Deployment status did not change to {status} within {timeout_minutes} minutes" ) time.sleep(30) def test_deployment( parasail_client: httpx.Client, openai_client: openai.Client, id: int ) -> None: deployment = get_deployment(parasail_client, id) assert "status" in deployment, "Deployment status is None" assert ( deployment["status"]["status"] == "ONLINE" ), f"Deployment is not online: {deployment['status']['status']}" chat_completion = openai_client.chat.completions.create( model=deployment["externalAlias"], messages=[{"role": "user", "content": "What's the capital of New York?"}], ) assert len(chat_completion.choices) == 1, "No chat completion choices returned" assert chat_completion.choices[0].message.content, "Empty chat completion response" def main(): # Setup clients model_name = "RedHatAI/Qwen2-72B-Instruct-FP8" parasail_client = httpx.Client( base_url=control_base_url, headers={ "Authorization": "Bearer " + api_key, "Content-Type": "application/json", }, ) openai_client = openai.Client(api_key=api_key, base_url=chat_base_url) check_compatibility(parasail_client, model_name) devices = get_and_select_device_config( parasail_client, model_name=model_name, desired_device="H100SXM", desired_count=2, ) # Create deployment configuration deployment_config = CreateModelStruct( deploymentName="scripting-example-deployment-3", modelName=model_name, deviceConfigs=devices, replicas=1, ) deployment = None try: print("Creating deployment...") deployment = create_deployment(parasail_client, deployment_config) print(f"Created deployment with ID: {deployment['id']}") print("\nWaiting for deployment to come online.") wait_for_status(parasail_client, deployment["id"], status="ONLINE") print("\nTesting deployment with 1 replica...") test_deployment(parasail_client, openai_client, deployment["id"]) print("Initial deployment tests passed successfully") pause_deployment(parasail_client, deployment["id"]) wait_for_status(parasail_client, deployment["id"], status="OFFLINE") resume_deployment(parasail_client, deployment["id"]) wait_for_status(parasail_client, deployment["id"], status="ONLINE") print("\nTesting deployment with 1 replica...") test_deployment(parasail_client, openai_client, deployment["id"]) print("Initial deployment tests passed successfully") deployment = get_deployment(parasail_client, deployment["id"]) print("\nUpdating deployment to 2 replicas...") deployment["replicas"] = 2 deployment = update_deployment(parasail_client, deployment) print("Waiting for deployment to scale up...") wait_for_status(parasail_client, deployment["id"], status="ONLINE") print("\nTesting deployment with 2 replicas...") test_deployment(parasail_client, openai_client, deployment["id"]) print("Scaled deployment tests passed successfully") except Exception as e: print(f"Error: {e}") finally: if deployment: print("\nCleaning up - deleting deployment...") delete_deployment(parasail_client, deployment["id"]) print("Deployment deleted") if __name__ == "__main__": main() ``` ## Next steps * [Dedicated Instances overview](/parasail-docs/products/overview-1.md) * [Auto-Scaling](/parasail-docs/products/overview-1/auto-scaling.md) * [Private Hugging Face Models](/parasail-docs/products/overview-1/private-hf-models.md) * [Dedicated Quickstart](/parasail-docs/quickstart/dedicated.md) * [Authentication API reference](/parasail-docs/api-reference/authentication.md)