Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.inferless.com/llms.txt

Use this file to discover all available pages before exploring further.

This doc helps you configure the input / outputs of the ‘infer’ function in the InferlessPythonModule class. This is the interface for inputs between the model and the Inferless platform. You have to define the input_schema.py in your GitHub/GitLab repository this will help us create the Input parameters : For each input, there are 3 fields required and 1 optional field
  • datatype: “STRING”, “BOOL”, “INT8”, “INT16”, “INT32”, “FP16” “FP32”, “UINT8”, “UINT16”, “UINT32”, “UINT64”, “INT64” , “FP64” , “BYTES”, “BF16”
  • shape: The length of the array, If the shape is [1] you will get the variable, if the array > 1 you will get an array, If the length is variable you can put -1
  • required: If the parameter is required in all API calls
  • example( optional ): Sample value for calling the API
In code
Example
def infer(self, inputs):
    prompt = inputs["prompt"] # "There is a fine house in the forest"
    shape = inputs["shape"] # [ 512,1 ]
In input_schema.py
Example
INPUT_SCHEMA = {
    "prompt": {
        'datatype': 'STRING',
        'required': True,
        'shape': [1],
        'example': ["There is a fine house in the forest"]
    },
    'shape': {
        'datatype': 'INT8',
        'required': False,
        'example': [ 512, 1 ],
        'shape': [2]
    },
}

​
Variable Length Inputs

For inputs that can take a variable number of values, you can set the shape fields to [-1]. This indicates that the length of the array is not fixed and can vary. Here’s an example: 
INPUT_SCHEMA = {
    "variable_input": {
        'datatype': 'STRING',
        'required': True,
        'shape': [-1],
        'example': ["value1", "value2", "value3"]
    },
}# Variable Length Inputs

​
Outputs

You can return any dictionary in the return statement of app.py. You don’t need to provide any configuration.

​
Returning Dicts

# Example Return Statement 

return { "label_1" : 0.398 , "label_2" : 0.563, "label_3" : 0.434 }

​
Returning Variable Length Array

# Example Return Statement 
return { "generated_images_base64" : [ img_str1 , img_str2 , img_str3 ] }

​
Returning Dictionary with Variable keys

# Example Return Statement 
dict = {"label_x": 0.4554 , "label_y", 0.3232 }
return { "result": json.dumps(dict) }