The simplest way to deploy ML models in production

​
Install the Inferless CLI package

pip install --upgrade inferless-cli

​
Login into the inferless workspace

inferless login

A new window will open

Copy the below CLI command

After Login, you will see this message

​
Import a Model

You need to have an app.py in the root folder with below Class as the entrypoint

from diffusers import StableDiffusionPipeline
import torch
from io import BytesIO
import base64

class InferlessPythonModel:
    def initialize(self):
        self.pipe = StableDiffusionPipeline.from_pretrained(
            "stabilityai/stable-diffusion-2-1",
            use_safetensors=True,
            torch_dtype=torch.float16,
            device_map='auto'
        )

    def infer(self, inputs):
        prompt = inputs["prompt"]
        image = self.pipe(prompt).images[0]
        buff = BytesIO()
        image.save(buff, format="JPEG")
        img_str = base64.b64encode(buff.getvalue()).decode()
        return { "generated_image_base64" : img_str }
        
    def finalize(self):
        self.pipe = None

Then create the input_schema.py

# input_schema.py
INPUT_SCHEMA = {
    "prompt": {
        'datatype': 'STRING',
        'required': True,
        'shape': [1],
        'example': ["There is a fine house in the forest"]
    }
}

Once you have this, Run the below command to start a new model import

inferless init

You will see the below prompt

Once init is complete you will see the below files created

./
├── app.py
├── inferless-config.yaml
├── inferless.yaml
└── input_schema.py 
# Depreciated 
├── input.json
└── output.json

  • inferless-runtime-config.yamlThis file will have all the software packages and the Python packages required for the model inferencing.

  • inferless.yamlThis file will have all the configurations required for the deployment. Users can update this file according to their requirements.

  • input_schema.py This file will have all the configurations required for the deployment. Users can update this file according to their requirements.

​
Run a Model Locally

Run a model locally

inferless run

​
Deploy a Model to Inferless

Run the below command to push the model to production

inferless deploy

You will see the below after-deployment

In UI in the Progress Section you will see :

​
Getting the logs

inferless log -i

​
Redeploy the Model with ‘updated code

inferless model redeploy

​
All Options in Inferless CLI

​
Optional Setting :

​
Using Runtimes with CLI

During the model init if you have a custom requirements.txt file you can use that to automatically create the config.yaml

Creating using requirements.txt

Generated file

// Runtime YAML file 
build:
  # cuda_version: we currently support 12.1.1 and 11.8.0.
  cuda_version: 12.1.1
  python_packages:
  - huggingface-hub==0.11.0
  - transformers==4.36.1==6.0.1
  - diffusers==0.24.0

If you don’t have the requirements in the same repo you can build the config.yaml using the below documentation

https://docs.inferless.com/model-import/bring-custom-packages

​
Push the runtime

inferless runtime upload

The CLI will ask you to update the config automatically, or else you can update manually in inferless.yaml

​
Using an existing runtime :

inferless runtime select --id

​
Creating a Volume

inferless volume create

​
List all the volumes

Use this command to get the id of the volume

inferless volume list

​
Using an existing volume

inferless volume select --id

​
Copy data from machine to Volume

copy a file

inferless volume cp -s /path/to/local/file -d infer://region-1/<volume-name>/<folder>/file

copy the entire folder

inferless volume cp -r -s /path/to/local/folder -d infer://region-1/<volume-name>/<folder>

​
List the data in Volume

inferless volume ls -i <volume-id> -p /path

​
Copy data Volume to local machine

inferless volume cp -s infer://region-1/<volume-name>/<folder> -d /path/to/local/file

​
Delete the data in the Volume

inferless volume rm -p infer://region-1/<volume-name>/<folder>

Depreciated - Input / Output Json

  • input.jsonThis file will have the key for the input parameter. Whenever you change the name of the key in the app.py, update it accordingly.

  • output.jsonThis file will have the name of the output key that the def infer the function is going to return.

Update the input.json / Output Json as per your model.

// Input.json 
{
  "inputs": [
    {
      "data": [
        "Image of horse near beach"
      ],
      "name": "prompt",
      "shape": [
        1
      ],
      "datatype": "BYTES"
    }
  ]
}

// Output.json 

{
  "outputs": [
    {
      "data": [
        "/9j/4AAQSkZJRgABAQAAAQABAAD/"
      ],
      "name": "generated_image_base64",
      "shape": [
        1
      ],
      "datatype": "BYTES"
    }
  ]
}
Get Logs - API File structure requirements