Importing from Github

Before you upload your custom model from Github, make sure you go through and follow the prerequisites conditions mentioned below.

​

File structure requirements

The below format should be kept in mind while loading your code to Inferless from GitHub.

1. The mandatory requirement would be an “app.py" file.

2. This file should contain the below functions(Mandatory)

3. def initialize(self)

4. def infer(self, inputs)

5. def finalize(self)

6. You can have other files/modules along with the “app.py" file in case of any dependencies.

7. You should also have a file input_schema.py if not then you will need to define

Github: https://github.com/infer-less/template-method

# app.py - Implement the Load function here for the model 
    def initialize(self):
        self.generator = pipeline("text-generation", model="EleutherAI/gpt-neo-125M",device=0)

    
# Function to perform inference 
    def infer(self, inputs):
        # inputs is a dictionary where the keys are input names and values are actual input data
        # e.g. in the below code the input name is "prompt"
        prompt = inputs["prompt"]
        pipeline_output = self.generator(prompt, do_sample=True, min_length=20)
        generated_txt = pipeline_output[0]["generated_text"]
        # The output generated by the infer function should be a dictionary where keys are output names and values are actual output data
        # e.g. in the below code the output name is "generated_txt"
        return {"generated_text": generated_txt}

# perform any cleanup activity here
    def finalize(self,args):
        self.pipe = None

# input_schema.py
INPUT_SCHEMA = {
    "prompt": {
        'datatype': 'STRING',
        'required': True,
        'shape': [1],
        'example': ["There is a fine house in the forest"]
    }
}

​

Supported Libraries

The below libraries are only supported in the current version. Make sure that the “app.py" requires libraries only from the below list:

• transformers

• torch

• tensorflow

• onnx

• onnxruntime

• numpy

• pandas

• diffusers

• Pillow

• pytesseract

• opencv

In case your use case requires any additional library, feel free to raise a support request with us using the help desk or contact us at support@inferless.com and we would be happy to discuss and take the request on a case-case basis.

Once you have made sure of the above conditions, click here to know more about how to import this model from Github to Inferless.

​

Resources

Click above to open the sample template for reference