Once you have installed the LEIP SDK you are ready to start optimizing your model.
The following guide will help you get familiar with the SDK and some of its core features. The SDK also provides access to the Latent AI model zoo with sample models and scripts to help get you on your way. If you need further assistance, please contact firstname.lastname@example.org.
The following guide will assume you are running the LEIP SDK container and will reference examples and files installed inside this container.
Preparing a Model
It is highly recommended that you get started with the SDK by first running through a sample model from the Latent AI Model Zoo. This will help you get familiarized with the SDK and ensure you can successfully run through the LEIP workflow from start to finish. However, when using your own trained model please refer to the Supportability Matrix to verify it meets the model requirements.
The following example will use a model from the Latent AI Model Zoo.
Selecting a Zoo Model
leip zoo list command to query the available models and datasets. You will notice that each model includes a model_id and variant_id; you will need to pass those values from the model you choose to the
leip zoo download command in the next step.
# List available models and datasets
leip zoo list
The following command uses the inceptionv3 model and keras-open-images-10-classes variant.
Downloading a Zoo Model
Once you have the model_id and variant_id, you can download an un-frozen model to the local workspace by running the
leip zoo download command. Again, this information was obtained from output of the
leip zoo list command.
# download the model variant to our local workspace
leip zoo download --model_id inceptionv3 --variant_id keras-open-images-10-classes
# notice the directory structure and contents
# The model weights and class names
class_names.txt model.h5 model_schema.json
# The model_schema.json tracks common model attributes
"full_name": "Inception V3",
"description": "Inception V3 is a convolutional neural network developed by Google to perform image classificaiton tasks.",
"type": "Image Classification",
Downloading a Zoo Dataset
You will also need the dataset. The dataset for this model is the open-images-10-classes:
# Download the evaluation dataset
leip zoo download --dataset_id open-images-10-classes --variant_id eval
# All downloads are saved to your workspace
Please see the LEIP Zoo section for more detailed information about accessing and using zoo models and datasets.
Using LEIP Optimize
LEIP Optimize is a state-of-the-art model optimizer that applies post-quantization algorithms to a model and produces a binary representation based on the target specified by the user. The binary is in the form of a shared object file which is loaded into a small runtime for its execution.
leip optimize --help command to obtain the supported options, or view the CLI Reference for LEIP Optimize.
The following example uses asymmetric quantization at 8 bits with a target data type of uint8 and saves the output model to an optimizedModel folder. A calibration is done using files specified in a rep_dataset.txt file. The original model is left untouched.
# Create a representative dataset file, in this case with one example item
echo workspace/datasets/open-images-10-classes/eval/Apple/06e47f3aa0036947.jpg > rep_dataset.txt
# Using the previously downloaded inceptionv3 instance, optimize the model
leip optimize --input_path workspace/models/inceptionv3/keras-open-images-10-classes \
--output_path optimizedModel/ \
--quantizer asymmetric \
--bits 8 \
--rep_dataset rep_dataset.txt \
# After a few minutes, LEIP optimize will complete the process
modelLibrary.so model_schema.json results.json
The command line arguments are documented in LEIP Optimize, but it is worth noting how they are being used here in this example:
--quantizeris set to
leip optimizeto cast the tensors to 8-bit integer representation using per-tensor asymmetric quantization.
During the casting, a file containing the newline-separated paths to sample input files, comprising what is known as the “representative dataset”, is supplied with the
--rep_datasetoption. This tells
leip optimizeto use this input file to run inference on the model to gather the quantization parameters for output tensors. A random image from the dataset that was downloaded above was used here, but in practice the representative dataset file does not need to be from the training or test dataset.
--preprocessoroption specifies the preprocessor callback function name to use during the inference on the representative dataset file.
leip optimize command is run, the folder specified by
--output_path will now contain an 8-bit integer version of the model in the form of a shared object file (
.so). In this file, both the parameters and operations are integer typed.
Using LEIP Evaluate
You can use LEIP to evaluate the accuracy of a model given a dataset and specifying class names and batch size. Run
leip evaluate --help command to obtain the supported options.
The evaluate command is useful to compare the accuracy of a model at different points in the pipeline. For example, you can evaluate the accuracy of the model before and after running optimize. The following example illustrates this.
We will compare the results of the original inceptionv3 model and then the optimized inceptionv3 model that were trained on our OpenImages dataset in this example.
# Evaluate the original model and note the results
leip evaluate --input_path workspace/models/inceptionv3/keras-open-images-10-classes/ \
--test_path workspace/datasets/open-images-10-classes/eval/index.txt \
# Run evaluate again, this time on the previously optimized model, and note the results
leip evaluate --input_path optimizedModel/ \
Additional information about
leip evaluate can be found here.