Mixtral 8x7B, a high-quality sparse mixture of experts model (SMoE) with open weights. Licensed under Apache 2.0. Mixtral outperforms Llama 2 70B on most benchmarks with 6x faster inference
Library | Bits | Inference Time | Cold Start Time | Token/Sec | Latency/Token | VRAM Required |
---|---|---|---|---|---|---|
PyTorch | 8 bit | 4.94 sec | 11.48 sec | 52.03 | 19.22 ms | 43.78 GB |
vLLM | 4 bit | 6.59 sec | 36.62 sec | 36.39 | 27.47 ms | 65.66 GB |
AutoGPTQ | 4 bit | 38.44 sec | 203.45 sec | 7.11 | 140.76 ms | 22.68 GB |
HQQ | 4 bit | 255.09 sec | 16.19 sec | 5.16 | 193.47 ms | 24.70 GB |
def initialize
: In this function, you will download and initialize your model. Here we have used the model_initialize
function from the get_model script, this loads and initialize the model. You can customize the get_model
script according to your requirements. Also in this function, you can define any variable that you want to use during inference.
def infer
: This function gets called for every request that you send. Here you can define all the steps that are required for the inference. You can also pass custom values for inference and pass it through inputs(dict)
parameters.
def finalize
: This function cleans up all the allocated memory.
Add a custom model
button that you see on the top right. An import wizard will open up.
--gpu A100
: Specifies the GPU type for deployment. Available options include A10
, A100
, and T4
.--runtime inferless-runtime-config.yaml
: Defines the runtime configuration file. If not specified, the default Inferless runtime is used.