Classifier Recipe Step One: Training, Evaluating, and Exporting a Model

The default Classifier Recipe supports a number of classifier backbones that have been qualified to ensure that they compress and compile for common hardware targets. We will walk through an example where we train and evaluate some of these models on a small open images dataset. This dataset will enable us to quickly demonstrate each step of the process and give us a decent baseline accuracy for verification purposes.

Later in the tutorial we will provide instructions on adding your own dataset and how to adjust the model and training parameters so that you can adapt the recipe to your needs. We will start with 224x224 image datasets as this is a common configuration that works across all of the currently supported backbones.

Download the Dataset

To get started, download and install the open-images-10-classes dataset:

CODE

cd /latentai
leip zoo download --dataset_id open-images-10-classes --variant_id eval
leip zoo download --dataset_id open-images-10-classes --variant_id train

Train a Model

The machine learning component of a recipe is defined by a configuration file. Additional details of the file will be discussed later, but for now note that the default file we will use is classifier-recipe.yaml and will be passed to the LEIP Machine Learning Applications Framework tool using the --config-name classifier-recipe command line option.

Lets start by training one of the smaller classifier models, timm:gernet_m (See also the list of all classifier models currently supported by LEIP Recipes). We will pass the backbone with the model.module.backbone command to train this model. The open-images-10-classes dataset is already set as the default in the classifier-recipe. Simply use the following command to start the training process:

CODE

# Example training command for timm:gernet_m, with the default classifier dataset:
# af --config-name classifier-recipe model.module.backbone="timm:gernet_m"
#
# The above command will by default store a checkpoint in a time-stamped directory:
# /latentai/artifacts/train/<timestamp>_task_leip_classifier/<checkpoint file>
#
# For this tutorial, we will pass a hard-coded checkpopint filename to simplify
# locating the checkpoint in the later steps.  We can do this by passing the
# callbacks.checkpoint.filename command line option

# Use the following to train the timm:gernet_m backbone classifier:

af --config-name classifier-recipe model.module.backbone="timm:gernet_m" \
  callbacks.checkpoint.filename=/latentai/checkpoint/timm_gernet_m \
  command=train
  
# The checkpoint file will be the callback above with ".ckpt" added.
# Set an env variable to simplify the following steps:

export CHECKPOINT=/latentai/checkpoint/timm_gernet_m.ckpt

Evaluate the Model

Now that the model has been trained, you can evaluate the model from the checkpoint by passing the checkpoint path:

CODE

af --config-name classifier-recipe \
  model.module.backbone=timm:gernet_m \
  +checkpoint=$CHECKPOINT command=evaluate

An evaluation report will be provided in:
/latentai/artifacts/evaluate/my-custom-imagefolder-like-data/val/metrics_report.json

Visualize Predictions

Or you can visualize model predictions on a small set of images:

CODE

af --config-name classifier-recipe \
  model.module.backbone=timm:gernet_m   \
  +checkpoint=$CHECKPOINT command=predict

The images with superimposed labels will be in the directory:

/latentai/artifacts/predictions/my-custom-imagefolder-like-data/validation

Export the Trained Model

Finally, you can use the export command to export the trained model:

CODE

af --config-name classifier-recipe \
  model.module.backbone=timm:gernet_m \
  +checkpoint=$CHECKPOINT command=export

The exported, trained model will be found at:

/latentai/artifacts/export/leip_classifier_batch1_224-224/leip_classifier_timm-gernet_m_1x224x224x10.pt

Next, we will compile and optimize the traced model for evaluation on the target device. If you would like to retrain the model with your own data, we have provided instructions for adding your own data to the recipe.