SOLAR-10.7B, an advanced large language model (LLM) with 10.7 billion parameters, demonstrates superior performance in various natural language processing (NLP) tasks
SOLAR-10.7B, an advanced large language model (LLM) with 10.7 billion parameters, demonstrates superior performance in various natural language processing (NLP) tasks. They have presented a methodology for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications and continued pretraining. They have integrated Mistral 7B weights into the upscaled layers, and finally, continued pre-training for the entire model.
We utilized AutoGPTQ to quantize SOLAR-10.7B-Instruct-v1.0 into a 4-bit quantized GPTQ version. In the inference process, we deployed the quantized model on an A100 GPU (80GB) using vLLM. We also tried deploying via Auto-GPTQ as mentioned in our observations below:
Library | Inference Time | Cold Start Time | Token/Sec | Latency/Token | VRAM Required |
---|---|---|---|---|---|
vLLM (Recommended) | 1.37 sec | 11.69 sec | 111.54 | 8.97 ms | 69.33 GB |
Auto-GPTQ | 27.09 sec | 61.31 sec | 9.82 | 101.98 ms | 5.67 GB |
Quantization techniques reduces the model’s computation cost and memory. It represent the model’s weights and activations in lower precision data-types while trying not to reduce in the accuracy.
We have quantized the model using GPTQ algorithm, GPTQ is a quantization algorithms for LLMs. We have used AutoGPTQ for 4-bit GPTQ quantization.
Install the AutoGPTQ library:
Import
the following libraries, and initialize the model and the tokenizer. For GPTQ calibration phase we are using VMware/open-instruct dataset.
Now you can start the quantization process, it will create a new directory where it will store the quantized model. The quantized model is 5.98 GB, which is approximately 27.85% of the original model 21.47 GB. Here’s the link to our quanitized model.
We are using the vLLM library, which enables you to run LLM on low memory. We deploy a GPTQ 4bit quantized version of the model.
Now quickly construct the GitHub/GitLab template, this process is mandatory and make sure you don’t add any file named model.py
You can also add other files to this directory.
In the app.py we will define the class and import all the required functions
def initialize
: In this function, you will initialize your model and define any variable
that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference. You can also pass custom values for inference and pass it through inputs(dict)
parameter.
def finalize
: This function cleans up all the allocated memory.
This is a mandatory step where we allow the users to upload their custom runtime through inferless-runtime-config.yaml.
You can use the inferless remote-run
(installation guide here) command to test your model or any custom Python script in a remote GPU environment directly from your local machine. Make sure that you use Python3.10
for seamless experience.
To enable Remote Run, simply do the following:
inferless
library and initialize Cls(gpu="A100")
. The available GPU options are T4
, A10
and A100
.initialize
and infer
functions with @app.load
and @app.infer
respectively.my_local_entry
) with @inferless.local_entry_point
.
Within this function, instantiate your model class, convert any incoming parameters into a RequestObjects
object, and invoke the model’s infer
method.From your local terminal, navigate to the folder containing your app.py
and your inferless-runtime-config.yaml
and run:
You can pass the other input parameters in the same way as long as your code expects them in the inputs
dictionary.
If you want to exclude certain files or directories from being uploaded, use the --exclude
or -e
flag.
Inferless supports multiple ways of importing your model. For this tutorial, we will use GitHub.
Navigate to your desired workspace in Inferless and Click on Add a custom model
button that you see on the top right. An import wizard will open up.
Once the model is in ‘Active’ status you can click on the ‘API’ page to call the model
Inferless allows you to deploy your model using Inferless-CLI. Follow the steps to deploy using Inferless CLI.
Let’s begin by cloning the model repository:
To deploy the model using Inferless CLI, execute the following command:
Explanation of the Command:
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.
SOLAR-10.7B, an advanced large language model (LLM) with 10.7 billion parameters, demonstrates superior performance in various natural language processing (NLP) tasks
SOLAR-10.7B, an advanced large language model (LLM) with 10.7 billion parameters, demonstrates superior performance in various natural language processing (NLP) tasks. They have presented a methodology for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications and continued pretraining. They have integrated Mistral 7B weights into the upscaled layers, and finally, continued pre-training for the entire model.
We utilized AutoGPTQ to quantize SOLAR-10.7B-Instruct-v1.0 into a 4-bit quantized GPTQ version. In the inference process, we deployed the quantized model on an A100 GPU (80GB) using vLLM. We also tried deploying via Auto-GPTQ as mentioned in our observations below:
Library | Inference Time | Cold Start Time | Token/Sec | Latency/Token | VRAM Required |
---|---|---|---|---|---|
vLLM (Recommended) | 1.37 sec | 11.69 sec | 111.54 | 8.97 ms | 69.33 GB |
Auto-GPTQ | 27.09 sec | 61.31 sec | 9.82 | 101.98 ms | 5.67 GB |
Quantization techniques reduces the model’s computation cost and memory. It represent the model’s weights and activations in lower precision data-types while trying not to reduce in the accuracy.
We have quantized the model using GPTQ algorithm, GPTQ is a quantization algorithms for LLMs. We have used AutoGPTQ for 4-bit GPTQ quantization.
Install the AutoGPTQ library:
Import
the following libraries, and initialize the model and the tokenizer. For GPTQ calibration phase we are using VMware/open-instruct dataset.
Now you can start the quantization process, it will create a new directory where it will store the quantized model. The quantized model is 5.98 GB, which is approximately 27.85% of the original model 21.47 GB. Here’s the link to our quanitized model.
We are using the vLLM library, which enables you to run LLM on low memory. We deploy a GPTQ 4bit quantized version of the model.
Now quickly construct the GitHub/GitLab template, this process is mandatory and make sure you don’t add any file named model.py
You can also add other files to this directory.
In the app.py we will define the class and import all the required functions
def initialize
: In this function, you will initialize your model and define any variable
that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference. You can also pass custom values for inference and pass it through inputs(dict)
parameter.
def finalize
: This function cleans up all the allocated memory.
This is a mandatory step where we allow the users to upload their custom runtime through inferless-runtime-config.yaml.
You can use the inferless remote-run
(installation guide here) command to test your model or any custom Python script in a remote GPU environment directly from your local machine. Make sure that you use Python3.10
for seamless experience.
To enable Remote Run, simply do the following:
inferless
library and initialize Cls(gpu="A100")
. The available GPU options are T4
, A10
and A100
.initialize
and infer
functions with @app.load
and @app.infer
respectively.my_local_entry
) with @inferless.local_entry_point
.
Within this function, instantiate your model class, convert any incoming parameters into a RequestObjects
object, and invoke the model’s infer
method.From your local terminal, navigate to the folder containing your app.py
and your inferless-runtime-config.yaml
and run:
You can pass the other input parameters in the same way as long as your code expects them in the inputs
dictionary.
If you want to exclude certain files or directories from being uploaded, use the --exclude
or -e
flag.
Inferless supports multiple ways of importing your model. For this tutorial, we will use GitHub.
Navigate to your desired workspace in Inferless and Click on Add a custom model
button that you see on the top right. An import wizard will open up.
Once the model is in ‘Active’ status you can click on the ‘API’ page to call the model
Inferless allows you to deploy your model using Inferless-CLI. Follow the steps to deploy using Inferless CLI.
Let’s begin by cloning the model repository:
To deploy the model using Inferless CLI, execute the following command:
Explanation of the Command:
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.