Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.inferless.com/llms.txt

Use this file to discover all available pages before exploring further.

Often time you will want to get 1 or more outputs from your model. Inferless allows you to get multiple outputs from your model. In this guide, we’ll walk you through the steps to handle multiple outputs in Inferless. There are 2 scenations for handling multiple outputs:
  1. Singe Dimentional Result Arrays
  2. Multi Dimentional Result Arrays

​
Single Dimentional Result Arrays

In code app.py make sure you return a list for both size = 1 and size > 1 

def infer(self, inputs):
    prompt = inputs["prompt"] 
    results = []
    for each in prompt:
        results.append(processData(each))
    return {"results": results }
in the API response you will get the results as a list of values for example with shape 2 
{ "outputs":[{"name":"results","datatype":"BYTES","shape":[2],"data":[" Making an omelette is a simple process", " Making an omelette is a simple process"]}] }
Limitations: You can only return a single list of type (String/Int/Float/Boolean) in the API response. If you have multiple outputs of for each input you can return a all the different outputs as seperate list. Example : 

def infer(self, inputs):
    prompt = inputs["prompt"] 
    results = []
    for each in prompt:
        # result_dict has 3 keys time , confidence and result
        result_dict.append(processData(each))

    # Flatten the result_dict to get all the values in a single list
    time_list = []
    confidence_list = []
    result_list = []
    for each in result_dict:
        time_list.append(each["time"])
        confidence_list.append(each["confidence"])
        result_list.append(each["result"])


    return {"results": results }
in the API response you will get the results as a list of values for example with shape 2 
{ "outputs":[{"name":"time","datatype":"BYTES","shape":[2],"data":["0.1", "0.2"]},
             {"name":"confidence","datatype":"BYTES","shape":[2],"data":["0.9", "0.8"]},
             {"name":"result","datatype":"BYTES","shape":[2],"data":[" Making an omelette is a simple process", " Making an omelette is a simple process"]}] }

​
Multi Dimentional Result Arrays

If you are returning a multi dimentional array you can return the output as a list of list. Example : 
class InferlessPythonModel:

	def ____init__(self):
		self.model = SentenceTransformer("jinaai/jina-embeddings-v2-base-en", trust_remote_code=True)

	def infer(self, inputs):
		sentences = inputs["sentences"]
		embeddings = self.model.encode(sentences) # embedding are [768]
        # If there are 2 sentences the embeddings will be [2,768]
		return {"results": embeddings }
If you make a API call with 2 sentences you will get the output where data will be of shape [1536] and the shape wil be [2,768] 
curl --location 'https://<>.default.model-v1-dev.inferless.com/v2/models/<model_name>/versions/1/infer' \
          --header 'Content-Type: application/json' \
          --header 'Authorization: Bearer 432424.....3432423' \
          --data '{
    "inputs": [
        {
            "name": "prompt",
            "shape": [
                1
            ],
            "data": [
                "There is a fine house in the forest", "There is a house in the forest"
            ],
            "datatype": "BYTES"
        }
    ]
}'

{ "outputs":[{"name":"results","datatype":"BYTES","shape":[2,768],"data":[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,0.1,0.2,0.3,0.4,0.5,0.6]} ] }
In the clinet you will need to reshape the output to get the original shape. 
import numpy as np
# Get the result 
original_array = result["outputs"][0]['data']
# Reshape the array to 2 rows and 768 columns
reshaped_array = original_array.reshape(result["outputs"][0]['shape'])