Bring custom packages
Custom software and dependencies in your Runtime
Custom Runtime allows you to customize the container to have the software and dependency that you need to run your model.
Here is a sample YAML file that you can write to import. You can follow the required structure of the build. You can specify the System as well as the Python packages that you need to run the model.
cuda_version (Optional):
By default, it will use CUDA 12.1.1. Above are the available options : “12.4.1” / “12.1.1” / “11.8.0”
system_packages :
These are any libraries that you need for the model to run, for eg: “opencv” for image processing or like “ffmpeg” to open the audio files.
python_packages :
These are pip-based libraries that you need in your Python runtime for example torch is required to load PyTorch-based models
run :
These are shell commands executed while building the runtime for example symlink is created here. All commands are executed Sequentially. If you have a package that requires step by step installation you can use run
build:
system_packages:
- "libssl-dev"
- "opencv"
- "ffmpeg"
python_packages:
- "transformers==4.29.0"
- "torch==2.0.1"
- "numpy==1.23.5"
- "pandas==2.0.1"
run:
- "ln -s /usr/local/lib/python3.10/site-packages/torch/lib/lib{nv,cu}* /usr/lib"
Runtime Versions
You can edit the runtime in UI or patch the runtime using CLI to create a new version of the runtime. All your models will remain in the older version you previously deployed with unless you explicitly update them in Model Settings.
You can see all the versions of the runtime by clicking the ‘View Runtime Versions’.
You can look at the packages in the version by clicking on “View”
To update the Runtime of a model you can go to the Model Card and navigate to the Settings Page and select the version of the runtime you want to deploy and click on update.
Method A: Create a new Runtime using Inferless Platform
- Select on Runtime Tab in the left navigation and click on Create Runtime
- Upload the Yaml file created with the required dependencies.
How to use Custom Runtime for the Model
- Go to the Model Import wizard. To use the custom runtime, select it in the
Setup Environment
step.
- Select the Runtime that you have created. Now you all access to all the required software libraries while running the model
Method B: Create a new Runtime using Inferless CLI
Using the Inferless-CLI, inferless runtime
command allows users to list
, select
and upload
the runtimes.
Creating the runtime file
You can create the runtime file in multiple ways:
-
You can create the requirements.txt file and then run the command
inferless init
, continue the process and select the requirements.txt file. This will create theinferless-runtime-config.yaml
, which you can update according to your software and Python libraries requirements. -
You can create the
config.yaml
file and mention all the software packages and Python libraries required for the model inference. And update the runtime file name on theinferless.yaml
.
Upload the Runtime
Once you have created your runtime file, you can now run the command inferless runtime upload
to start the uploading process. First, you are required to pass the file name, and then you need to set a name to your runtime. Update the same to the inferless.yaml file.
Now your runtime is ready to use!