This will help you understand how to handle multiple varrible outputs in Client side
Often time you will want to get 1 or more outputs from your model. Inferless allows you to get multiple outputs from your model. In this guide, we’ll walk you through the steps to handle multiple outputs in Inferless.There are 2 scenations for handling multiple outputs:
In code app.py make sure you return a list for both size = 1 and size > 1
Copy
def infer(self, inputs): prompt = inputs["prompt"] results = [] for each in prompt: results.append(processData(each)) return {"results": results }
in the API response you will get the results as a list of values for example with shape 2
Copy
{ "outputs":[{"name":"results","datatype":"BYTES","shape":[2],"data":[" Making an omelette is a simple process", " Making an omelette is a simple process"]}] }
Limitations:You can only return a single list of type (String/Int/Float/Boolean) in the API response.If you have multiple outputs of for each input you can return a all the different outputs as seperate list.Example :
Copy
def infer(self, inputs): prompt = inputs["prompt"] results = [] for each in prompt: # result_dict has 3 keys time , confidence and result result_dict.append(processData(each)) # Flatten the result_dict to get all the values in a single list time_list = [] confidence_list = [] result_list = [] for each in result_dict: time_list.append(each["time"]) confidence_list.append(each["confidence"]) result_list.append(each["result"]) return {"results": results }
in the API response you will get the results as a list of values for example with shape 2
Copy
{ "outputs":[{"name":"time","datatype":"BYTES","shape":[2],"data":["0.1", "0.2"]}, {"name":"confidence","datatype":"BYTES","shape":[2],"data":["0.9", "0.8"]}, {"name":"result","datatype":"BYTES","shape":[2],"data":[" Making an omelette is a simple process", " Making an omelette is a simple process"]}] }
If you are returning a multi dimentional array you can return the output as a list of list.Example :
Copy
class InferlessPythonModel: def ____init__(self): self.model = SentenceTransformer("jinaai/jina-embeddings-v2-base-en", trust_remote_code=True) def infer(self, inputs): sentences = inputs["sentences"] embeddings = self.model.encode(sentences) # embedding are [768] # If there are 2 sentences the embeddings will be [2,768] return {"results": embeddings }
If you make a API call with 2 sentences you will get the output where data will be of shape [1536] and the shape wil be [2,768]
Copy
curl --location 'https://<>.default.model-v1-dev.inferless.com/v2/models/<model_name>/versions/1/infer' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer 432424.....3432423' \ --data '{ "inputs": [ { "name": "prompt", "shape": [ 1 ], "data": [ "There is a fine house in the forest", "There is a house in the forest" ], "datatype": "BYTES" } ]}'
In the clinet you will need to reshape the output to get the original shape.
Copy
import numpy as np# Get the result original_array = result["outputs"][0]['data']# Reshape the array to 2 rows and 768 columnsreshaped_array = original_array.reshape(result["outputs"][0]['shape'])
Assistant
Responses are generated using AI and may contain mistakes.