​
Getting started

  1. Login to Inferless.com console and copy the CLI keys from Keys section here
  2. Now install Inferless CLI package using this command 
     pip install inferless-cli
  3. Login to Inferless CLI using this command and paste the CLI keys here, you’ll be logged in 
    inferless login

​
Sample model deployment

  1. To deploy a model with Inferless you would ideally require 3 files
    • app.py is a Python file that plays a crucial role in setting up and running models on the Inferless platform. It typically contains a class with three main functions:
    • input_schema.py is a python file specifies the input parameters for your model’s API calls.
    • config.yaml(Runtime) refers to the software and dependencies you can add to your runtime environment to support your model’s specific needs.
  2. To get started with your first deployment we will provide you app.py and input_schema.py. Run this command to download the files 
    inferless scaffold --demo
  3. Now that the files are downloaded. Initialise the model using this command 
    inferless init --name <modelname>
  4. Once the model is initialised. Deploy it using this command 
     inferless deploy --gpu T4
  5. Hurray! You’ve successfully deployed your first model with us

​
Runtime

Runtime in Inferless refers to the environment and configuration in which your model runs. It includes:
  1. System packages
  2. Python packages
  3. Custom shell commands

​
Create and deploy with runtime

  1. Create a new file called inferless_runtime_config.yaml with below data 
    build:
cuda_version: "12.1.1"
python_packages:
    - "accelerate==0.33.0"
    - "torch==2.4.0"
    - "transformers==4.44.0"
    - "diffusers==0.30.0"
  2. Run the below command to create the runtime 
    inferless runtime create --name <runtime_name> --path ./inferless_runtime_config.yaml
  3. Deploy with runtime 
    inferless deploy --gpu t4 --region <region_name> --runtime <runtime_name>

​
Volumes

Volumes in Inferless are NFS-like writable storage spaces that can be connected to multiple replicas simultaneously. They serve several key purposes:
  1. Storing model parameters
  2. Archiving datasets (similar to centralized storage)
  3. Setting up shared caches for collaborative tasks

​
Creating and uploading weights

  1. To create a new volume. Run this command 
    inferless volume create --name <volume_name>
  2. New volume will be created and you will be shown the infer_path where you can store the weights. Make sure you keep this handy
  3. Once the volume is created. Upload the weights using this command and paste the infer_path in the destination 
    inferless volume cp --source <source_path> --destination <remote_path (Infer path)>
  4. Your files will be copied to the server and your volume is ready to be used.
  5. To use these weights, in app.py you can specify the mount path from where the weights can be accessed (Eg: /var/nfs-mount/<volume_name>) 
    from diffusers import StableDiffusionPipeline
import torch
from io import BytesIO
import base64

MODEL_WEIGHTS_DIR =  "/var/nfs-mount/my_volume"

class InferlessPythonModel:    
    def initialize(self):
        self.pipe = StableDiffusionPipeline.from_pretrained(
            "/var/nfs-mount/my_volume",
            use_safetensors=True,
            torch_dtype=torch.float16,
            device_map='auto'
            local_files_only=True
        )
  6. Run the command to create a new model using the above model weights 
    inferless init --name <modelname>
  7. Once the model is initialized. Deploy it using this command (Make sure the path defined inside app.py and here in the command should be same) 
    inferless deploy --gpu t4 --region <region_name> --volume <volume_name> --volume-mount-path <volume_mount_path>