Llama 3 is an auto-regressive language model, leveraging a refined transformer architecture. The Llama 3 models were trained on 8x more data on over 15 trillion tokens. It has a context length of 8K tokens and increases the vocabulary size of the tokenizer to 128,256 (from 32K tokens in the previous version).
HuggingFaceH4/ultrachat_200k
dataset which is a filtered version of the UltraChat dataset from Huggingface.
For model quantization, we will load the model in a 4-bit format using bitsandbytes.
Finally, when deploying the model on Inferless, you can anticipate the following outcomes.
Library | Inference Time | Cold Start Time | Tokens/Sec |
---|---|---|---|
vLLM | 1.63 sec | 13.30 sec | 78.65 |
HuggingFaceH4/ultrachat_200k
dataset, we will sample 10000 text conversations for a quick run.
We have formatted the data using ChatML as we want our model to follow a specific chat template (ChatML).
AutoTokenizer
.
For ChatML
support, we will use the setup_chat_format()
function in trl
. It will set up the chat_template
of the tokenizer, add special tokens to the tokenizer
and resize the model’s embedding layer to accommodate the new tokens.
Prepare the model for QLoRA training using the prepare_model_for_kbit_training()
.
SFTTrainer
. The SFTTrainer is then created and used to start the fine-tuning process.
model.py
.
def initialize
: In this function, you will initialize your model and define any variable
that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference. You can also pass custom values for inference and pass it through inputs(dict)
parameter.
def finalize
: This function cleans up all the allocated memory.
input_schema.py
in your GitHub/Gitlab repository this will help us create the Input parameters. You can checkout our documentation on Input / Output Schema.
For this tutorial, we have defined a parameter prompt
which is required during the API call. Now lets create the input_schema.py
.
inferless remote-run
(installation guide here) command to test your model or any custom Python script in a remote GPU environment directly from your local machine. Make sure that you use Python3.10
for seamless experience.
inferless
library and initialize Cls(gpu="A100")
. The available GPU options are T4
, A10
and A100
.initialize
and infer
functions with @app.load
and @app.infer
respectively.my_local_entry
) with @inferless.local_entry_point
.
Within this function, instantiate your model class, convert any incoming parameters into a RequestObjects
object, and invoke the model’s infer
method.app.py
and your inferless-runtime-config.yaml
and run:
--temperature
, --max_tokens
, etc.) as long as your code expects them in the inputs
dictionary.
Add a custom model
button that you see on the top right. An import wizard will open up.
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.