Dynamic batching is a feature of Inferless that allows inference requests to be combined by the server so that a batch is created dynamically. Creating a batch of requests typically results in increased throughput. The dynamic batcher should be used for the stateless model.Dynamic batching is enabled and configured independently for each model using the BATCH_SIZE property in the model configuration. These settings control the preferred batch size(s) of the dynamically created batches, the maximum time that requests can be delayed in the scheduler to allow other requests to join the dynamic batch, and queue properties such as batch_window
INPUT_SCHEMA = { "prompt": { 'datatype': 'STRING', 'required': True, 'shape': [1], 'example': ["There is a fine house in the forest"] }}BATCH_SIZE = 4BATCH_WINDOW = 5000 # milliseconds
in app.py
Copy
import jsonimport numpy as npimport torchfrom transformers import pipelineclass InferlessPythonModel: # replace ##task_type## and ##huggingface_name## with appropriate values def initialize(self): self.generator = pipeline("text-generation", model="EleutherAI/gpt-neo-125M",device=0) # Inputs is a list of dictionaries where the keys are input names and values are actual input data # e.g. in the below code the input name is a prompt # Output generated by the infer function should be a List of dictionaries where keys are output names and values are actual output data # e.g. in the below code the output name is generated_txt def infer(self, inputs): output = [] print(" no of inputs to be processed " + str(len(inputs))) for each in inputs: prompt = each["prompt"] pipeline_output = self.generator(prompt, do_sample=True, min_length=20) generated_txt = pipeline_output[0]["generated_text"] print("generated_txt", generated_txt, flush=True) output.append({"generated_text": generated_txt }) return output # perform any cleanup activity here def finalize(self,args): self.pipe = None
In the above configuration, we have set the max_batch_size to 8. This means that the model will try to create a batch of 4 requests. The input and output dimensions are also specified in the configuration file.