Gemma-3-27B-it is a 27-billion-parameter multimodal language model developed by the Gemma team. This model excels in instruction-based tasks, offering superior visual and multilingual capabilities.
Gemma-3-27B-it is a state‑of‑the‑art, 27B vision‑language model from the Gemma team. Built for instruction‑tuned applications, it seamlessly integrates robust language understanding with cutting‑edge visual analysis. Whether it’s detecting intricate visual patterns, parsing complex documents, or analyzing extended video content by highlighting key moments, this model is engineered to act as a versatile visual agent. With capabilities that include generating structured outputs (such as bounding boxes and JSON) and supporting multilingual text within images, it paves the way for innovative interactive chatbots, advanced multimedia content analysis, and beyond.
We are using the transformers to serve the model on a single A100.
We have deployed the model on an A100 GPU(80GB). Here are our observations:
Library | Inference Time | Cold Start Time | Tokens/Sec | Output Tokens Length |
---|---|---|---|---|
transformers | 11.73 sec | 21.64 sec | 10.59 | 128 |
Note: The inference time and cold start time are average values.
We are using the transformers to serve the model on a single A100 (80GB).
Now quickly construct the GitHub/GitLab template, this process is mandatory and make sure you don’t add any file named model.py
.
You can also add other files to this directory.
Using the inferless
Python client and Pydantic, you can define structured schemas directly in your code for input and output, eliminating the need for external file.
When defining an input schema with Pydantic, you need to annotate your class attributes with the appropriate types, such as str
, float
, int
, etc.
These type annotations specifys what type of data each field should contain.
The default
value serves as the example input for testing with the infer
function.
The @inferless.response
decorator helps you define structured output schemas.
infer
FunctionOnce you have annotated the objects you can expect the infer function to receive RequestObjects
as input,
and returns a ResponseObjects
instance as output, ensuring the results adhere to a defined structure.
In the app.py we will define the class and import all the required functions
def initialize
: In this function, you will initialize your model and define any variable
that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference.
def finalize
: This function cleans up all the allocated memory.
This is a mandatory step where we allow the users to upload their custom runtime through inferless-runtime-config.yaml.
You can use the inferless remote-run
(installation guide here) command to test your model or any custom Python script in a remote GPU environment directly from your local machine. Make sure that you use Python3.10
for seamless experience.
To enable Remote Run, simply do the following:
inferless
library and initialize Cls(gpu="A100")
. The available GPU options are T4
, A10
and A100
.initialize
and infer
functions with @app.load
and @app.infer
respectively.my_local_entry
) with @inferless.local_entry_point
.
Within this function, instantiate your model class, convert any incoming parameters into a RequestObjects
object, and invoke the model’s infer
method.From your local terminal, navigate to the folder containing your app.py
and your inferless-runtime-config.yaml
and run:
You can pass the other input parameters in the same way (e.g., --temperature
, --system_prompt
, etc.) as long as your code expects them in the inputs
dictionary.
If you want to exclude certain files or directories from being uploaded, use the --exclude
or -e
flag.
Inferless supports multiple ways of importing your model. For this tutorial, we will use GitHub.
Navigate to your desired workspace in Inferless and Click on Add a custom model
button that you see on the top right. An import wizard will open up.
Once the model is in ‘Active’ status you can click on the ‘API’ page to call the model
Inferless allows you to deploy your model using Inferless-CLI. Follow the steps to deploy using Inferless CLI.
Let’s begin by cloning the model repository:
To deploy the model using Inferless CLI, execute the following command:
Explanation of the Command:
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.
Gemma-3-27B-it is a 27-billion-parameter multimodal language model developed by the Gemma team. This model excels in instruction-based tasks, offering superior visual and multilingual capabilities.
Gemma-3-27B-it is a state‑of‑the‑art, 27B vision‑language model from the Gemma team. Built for instruction‑tuned applications, it seamlessly integrates robust language understanding with cutting‑edge visual analysis. Whether it’s detecting intricate visual patterns, parsing complex documents, or analyzing extended video content by highlighting key moments, this model is engineered to act as a versatile visual agent. With capabilities that include generating structured outputs (such as bounding boxes and JSON) and supporting multilingual text within images, it paves the way for innovative interactive chatbots, advanced multimedia content analysis, and beyond.
We are using the transformers to serve the model on a single A100.
We have deployed the model on an A100 GPU(80GB). Here are our observations:
Library | Inference Time | Cold Start Time | Tokens/Sec | Output Tokens Length |
---|---|---|---|---|
transformers | 11.73 sec | 21.64 sec | 10.59 | 128 |
Note: The inference time and cold start time are average values.
We are using the transformers to serve the model on a single A100 (80GB).
Now quickly construct the GitHub/GitLab template, this process is mandatory and make sure you don’t add any file named model.py
.
You can also add other files to this directory.
Using the inferless
Python client and Pydantic, you can define structured schemas directly in your code for input and output, eliminating the need for external file.
When defining an input schema with Pydantic, you need to annotate your class attributes with the appropriate types, such as str
, float
, int
, etc.
These type annotations specifys what type of data each field should contain.
The default
value serves as the example input for testing with the infer
function.
The @inferless.response
decorator helps you define structured output schemas.
infer
FunctionOnce you have annotated the objects you can expect the infer function to receive RequestObjects
as input,
and returns a ResponseObjects
instance as output, ensuring the results adhere to a defined structure.
In the app.py we will define the class and import all the required functions
def initialize
: In this function, you will initialize your model and define any variable
that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference.
def finalize
: This function cleans up all the allocated memory.
This is a mandatory step where we allow the users to upload their custom runtime through inferless-runtime-config.yaml.
You can use the inferless remote-run
(installation guide here) command to test your model or any custom Python script in a remote GPU environment directly from your local machine. Make sure that you use Python3.10
for seamless experience.
To enable Remote Run, simply do the following:
inferless
library and initialize Cls(gpu="A100")
. The available GPU options are T4
, A10
and A100
.initialize
and infer
functions with @app.load
and @app.infer
respectively.my_local_entry
) with @inferless.local_entry_point
.
Within this function, instantiate your model class, convert any incoming parameters into a RequestObjects
object, and invoke the model’s infer
method.From your local terminal, navigate to the folder containing your app.py
and your inferless-runtime-config.yaml
and run:
You can pass the other input parameters in the same way (e.g., --temperature
, --system_prompt
, etc.) as long as your code expects them in the inputs
dictionary.
If you want to exclude certain files or directories from being uploaded, use the --exclude
or -e
flag.
Inferless supports multiple ways of importing your model. For this tutorial, we will use GitHub.
Navigate to your desired workspace in Inferless and Click on Add a custom model
button that you see on the top right. An import wizard will open up.
Once the model is in ‘Active’ status you can click on the ‘API’ page to call the model
Inferless allows you to deploy your model using Inferless-CLI. Follow the steps to deploy using Inferless CLI.
Let’s begin by cloning the model repository:
To deploy the model using Inferless CLI, execute the following command:
Explanation of the Command:
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.