Getting Started with LEIP Design¶

In this tutorial, we will select, train, and deploy a model optimized for specific hardware using LEIP Design. To make the process quicker and easier, we will use one of the datasets included with LEIP Design.

We will:

1. Select a pre-configured recipe and dataset from the LEIP Design environment.
2. Train the selected model on the Fire and Smoke dataset.
3. Evaluate the trained model’s performance.
4. Optimize and compile the model for target hardware (in this case, CPU).
5. Export the deployable model and stub code.

Environment Setup¶

If you haven't done so already, please ensure that your environment is set up correctly. Please consult the installation guide for step-by-step instructions.

Step 1: Get our pantry¶

To design recipes, we need ingredients. Ingredients live in the pantry. A pantry can have all sorts of ingredients: models, datasets, optimizers, learning rate schedulers, and even training-aware quantization techniques. The code below will download a pantry full of useful ingredients to get started.

In [2]:

from pathlib import Path
import leip_recipe_designer as rd

# Define the workspace path
workspace = Path('./workspace')

# Build the pantry (do not rebuild if it already exists)
pantry = rd.Pantry.build(workspace / "./my_combined_pantry/", force_rebuild=False)

from pathlib import Path import leip_recipe_designer as rd # Define the workspace path workspace = Path('./workspace') # Build the pantry (do not rebuild if it already exists) pantry = rd.Pantry.build(workspace / "./my_combined_pantry/", force_rebuild=False)

2024-10-29 11:35:08,846 | WARNING  | pantry.build-119 | You requested to build a Pantry, but haven't specified the desired execution contexts. Therefore, will use the installed ones ['leip_af', 'leip_forge', 'leip_stub_gen']
  0%|          | 0/155 [00:00<?, ?it/s]/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/leip_recipe_designer/core/pantry_builder/source_readers.py:33: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  with initialize_config_dir(config_dir=str(os.path.abspath(self._configs_folder)), job_name="test_app"):
  1%|          | 1/155 [00:00<00:16,  9.58it/s]/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/leip_recipe_designer/core/pantry_builder/source_readers.py:33: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  with initialize_config_dir(config_dir=str(os.path.abspath(self._configs_folder)), job_name="test_app"):
100%|██████████| 155/155 [00:07<00:00, 20.24it/s]
  0%|          | 0/8 [00:00<?, ?it/s]/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/forge/onnx/calibrate.py:24: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm
WARNING:root:TensorRT timing cache is unset. Set with 'forge.set_tensorrt_time_cache()'.
100%|██████████| 8/8 [00:02<00:00,  3.56it/s]
100%|██████████| 3/3 [00:00<00:00, 14234.06it/s]

We will select a recipe that has been trained across different datasets, providing a good balance between performance and efficiency.

To create this recipe, simply use its recipe ID: 59845. All you need is access to the pantry.

In [3]:

recipe = rd.create.from_recipe_id('59845', pantry=pantry, allow_upgrade=True)

recipe = rd.create.from_recipe_id('59845', pantry=pantry, allow_upgrade=True)

Skipped downloading goldenrecipedb with name "xval_det" and variant "Xval0.3" (0), as it already exists.
This is the Cross-validation volume. Available methods are- 
get_golden_df 
describe_table

To learn how to use the Golden Recipe Database (GRDB) API to explore Golden Recipe volumes and select a recipe from the Golden DataFrame, consult the Explore Golden Volumes guide.

Step 2: Select a dataset¶

We will select the Fire and Smoke dataset, one of our included datasets.

In [4]:

data = rd.helpers.data.get_data_generator_by_name(pantry=pantry, regex_ingredient_name="Fire and smoke")

data = rd.helpers.data.get_data_generator_by_name(pantry=pantry, regex_ingredient_name="Fire and smoke")

This method is used to instantiate a dataset that has already been integrated. In case you want to use your own dataset or look at what else we offer, refer to the Bring Your Own Data (BYOD) guide. Now, replace the dataset in the recipe with the provided one:

In [5]:

rd.helpers.data.replace_data_generator(recipe, data)

rd.helpers.data.replace_data_generator(recipe, data)

Step 3: Set Up the Recipe Before Training¶

Let's assign a logger to monitor training metrics and progress. For this tutorial, we'll use Tensorboard.

In [6]:

recipe.assign_ingredients('loggers', {"my_local_training_log": "Tensorboard"})

recipe.assign_ingredients('loggers', {"my_local_training_log": "Tensorboard"})

Out[6]:

[{'choice_id': 'd139774776112d54355b171d97cff8627840b55995e9d852577242a5ed152a81',
  'choice_name': 'Tensorboard',
  'synonym': 'loggers',
  'parent': 'Full Recipe',
  'slot': 'slot:loggers',
  'path': ['slot:loggers', 'name:my_local_training_log'],
  'name': 'name:my_local_training_log'}]

For this tutorial, we're going to limit training to one epoch and restrict batch training to 10% for faster execution. Training the recipe until convergence will take several minutes—an NVIDIA RTX A4500 GPU took about 15 minutes to train this model.

If you would prefer to wait for the full training to converge instead of downloading the checkpoints, comment out the cell below:

In [7]:

recipe["train.num_epochs"] = 1
recipe["trainer.train_batches_percentage"] = 0.1

recipe["train.num_epochs"] = 1 recipe["trainer.train_batches_percentage"] = 0.1

If the recipe contains any missing elements, we can automatically fill them with default values prior to training. Let's do that now:

In [8]:

recipe.fill_empty_recursively()

recipe.fill_empty_recursively()

Out[8]:

[{'choice_id': '4adc4dcfbc1c805eaff72bb8460c6e5991b3f0e7fa7c779443086d3b85c9051f',
  'choice_name': 'No Quantizer',
  'synonym': 'quantizer',
  'parent': 'TVM Compiler',
  'slot': 'slot:quantizer',
  'queries': ['quantizer'],
  'req_map_key': '',
  'path': ['slot:compiler', 'slot:quantizer']}]

Step 4: Train the selected recipe¶

In [9]:

train_output = rd.tasks.train(recipe)

train_output = rd.tasks.train(recipe)

Executing AF command: 
af --config-dir /tmp/tmpcamt5fj4 --config-name recipe.yaml hydra.job_logging.root.level=50 +command=train

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

model size is  1.0x
init weights...
=> loading pretrained model https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth
Finish initialize NanoDet Head.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!

If you would like to train one or more candidate models, evaluate the results, and pick a winner, consult our Model Training and Optimization guide.

Step 5: Evaluate the model and visualize the predictions¶

Now we can evaluate the model we trained using the candidate recipe.

In [10]:

# Assign a checkpoint ingredient to the recipe for later use (e.g., evaluation)
recipe.assign_ingredients("checkpoint", "Local ckpt file")
recipe['checkpoint.path'] = str(train_output["best_model_path"])

# Evaluate the trained recipe
eval_output = rd.tasks.evaluate(recipe)

# Assign a checkpoint ingredient to the recipe for later use (e.g., evaluation) recipe.assign_ingredients("checkpoint", "Local ckpt file") recipe['checkpoint.path'] = str(train_output["best_model_path"]) # Evaluate the trained recipe eval_output = rd.tasks.evaluate(recipe)

Executing AF command: 
af --config-dir /tmp/tmp9ks35vtv --config-name recipe.yaml hydra.job_logging.root.level=50 +command=evaluate

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

model size is  1.0x
init weights...
=> loading pretrained model https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth
Finish initialize NanoDet Head.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!

In [11]:

print(f"mAP: {eval_output['evaluate.metric_single']:.4f}")

print(f"mAP: {eval_output['evaluate.metric_single']:.4f}")

mAP: 0.0179

Bonus: Detailed Evaluation Metrics¶

In addition to the primary mAP score, we can dive deeper into the evaluation results to get more detailed metrics. These additional metrics can give us better insights into the model's performance across different thresholds and object sizes.

To display detailed evaluation metrics, run the following:

In [12]:

from pprint import pprint

print("Evaluation Metrics:")
pprint(eval_output['evaluate.metric_dict'], indent=2, width=80, compact=True)

from pprint import pprint print("Evaluation Metrics:") pprint(eval_output['evaluate.metric_dict'], indent=2, width=80, compact=True)

Evaluation Metrics:
{ 'classes': [1.0, 2.0],
  'map': 0.017924292013049126,
  'map_50': 0.0824517086148262,
  'map_75': 0.0010657395469024777,
  'map_large': 0.04469352588057518,
  'map_medium': 0.028310101479291916,
  'map_per_class': [0.020685071125626564, 0.015163512900471687],
  'map_small': 0.16993780434131622,
  'mar_1': 0.06800151616334915,
  'mar_10': 0.2026388794183731,
  'mar_100': 0.29363811016082764,
  'mar_100_per_class': [0.2680981457233429, 0.3191780745983124],
  'mar_large': 0.2780952453613281,
  'mar_medium': 0.30813083052635193,
  'mar_small': 0.17619048058986664,
  'metric': 0.017924292013049126}

Next, we will visualize the predictions from the model:

In [13]:

predict_output = rd.tasks.visualize_predictions(recipe)

predict_output = rd.tasks.visualize_predictions(recipe)

Executing AF command: 
af --config-dir /tmp/tmp1ke3mlrh --config-name recipe.yaml hydra.job_logging.root.level=50 +command=predict

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

model size is  1.0x
init weights...
=> loading pretrained model https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth
Finish initialize NanoDet Head.
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!

In [14]:

import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import math
%matplotlib inline

def display_helper(images_path, count=4, image_extension="jpeg"):
    fig = plt.figure(figsize=(20, 10), facecolor='w')
    columns = 4
    rows = math.ceil(count / columns)
    for idx, path in zip(range(rows * columns), Path(images_path).rglob(str("*."+image_extension))):
        img = mpimg.imread(path)
        fig.add_subplot(rows, columns, idx + 1)
        plt.imshow(img)
        plt.axis("off")

display_helper(predict_output["predict.output_directory"], count=8)

import matplotlib.pyplot as plt import matplotlib.image as mpimg import math %matplotlib inline def display_helper(images_path, count=4, image_extension="jpeg"): fig = plt.figure(figsize=(20, 10), facecolor='w') columns = 4 rows = math.ceil(count / columns) for idx, path in zip(range(rows * columns), Path(images_path).rglob(str("*."+image_extension))): img = mpimg.imread(path) fig.add_subplot(rows, columns, idx + 1) plt.imshow(img) plt.axis("off") display_helper(predict_output["predict.output_directory"], count=8)

Recall that for this tutorial, we restricted training. Allowing the recipe to complete training on the entire dataset would result in improved evaluation metrics and more accurate predictions.

Step 6: Compile Your Recipe¶

Next, we will compile the model for your local CPU. To compile artifacts for different hardware targets, refer to the Setting Compiler Targets guide.

In [15]:

recipe['compiler']

recipe['compiler']

VBox(children=(HTML(value='\n<style>\n    .recipe-accordion-style > div[class*="jupyter-widget-Accordion-"] > …

In [16]:

recipe['compiler.output_path'] = './rd_forge_output'
recipe['compiler.force_overwrite'] = True

compiled_model = rd.tasks.compile(recipe)

recipe['compiler.output_path'] = './rd_forge_output' recipe['compiler.force_overwrite'] = True compiled_model = rd.tasks.compile(recipe)

Executing AF command: 
af --config-dir /tmp/tmpwqmbgj5s --config-name recipe.yaml hydra.job_logging.root.level=50 +command=export

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

model size is  1.0x
init weights...
=> loading pretrained model https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth
Finish initialize NanoDet Head.

[11:36:27] /app/src/runtime/threading_backend.cc:343: Warning: more than two frequencies detected! Forced big_count_ to 8
WARNING:autotvm:One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.

If you want to use the LEIP Design API to explore LEIP Optimize's compilation, quantization, and calibration parameters, refer to the Optimize guide.

Step 7: Evaluate Your Compiled Model¶

First, we will add the compiled model back to the recipe to ensure we evaluate the compiled recipe.

In [17]:

recipe.assign_ingredients("checkpoint", "compiled")
recipe['checkpoint.path'] = str(compiled_model['compiler.output_path'])

recipe.assign_ingredients("checkpoint", "compiled") recipe['checkpoint.path'] = str(compiled_model['compiler.output_path'])

Next, we will change the evaluation batch size to match the batch size for which we compiled the model.

In [18]:

# Set the validation batch size to match the compiled batch size
recipe["train.batch_size_val"] = recipe["export.batch_size"]

# Limit the number of evaluation batches to speed up the example evaluation
recipe['evaluation.max_batches'] = 1

# Set the validation batch size to match the compiled batch size recipe["train.batch_size_val"] = recipe["export.batch_size"] # Limit the number of evaluation batches to speed up the example evaluation recipe['evaluation.max_batches'] = 1

Evaluate the compiled model!

In [19]:

hide-output

# Evaluate the compiled recipe
compiled_eval_output = rd.tasks.evaluate(recipe)

# Evaluate the compiled recipe compiled_eval_output = rd.tasks.evaluate(recipe)

Executing AF command: 
af --config-dir /tmp/tmp2e_nfmw8 --config-name recipe.yaml hydra.job_logging.root.level=50 +command=evaluate

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(

model size is  1.0x
init weights...
=> loading pretrained model https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth
Finish initialize NanoDet Head.

[11:37:15] /app/src/runtime/latentai/lre.cpp:149: Setting device from metadata to: 1
[11:37:15] /app/src/runtime/latentai/lre.cpp:177: Attempting to run non-encrypted model

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!

[11:37:16] /app/src/runtime/threading_backend.cc:343: Warning: more than two frequencies detected! Forced big_count_ to 8

In [20]:

# Print the metric (mean Average Precision) after compilation
print(f"mAP after compilation: {compiled_eval_output['evaluate.metric_single']:.4f}")

# Print the metric (mean Average Precision) after compilation print(f"mAP after compilation: {compiled_eval_output['evaluate.metric_single']:.4f}")

mAP after compilation: 0.1586

Step 8: Generate Stub Code¶

In this step, we will generate the necessary stub code for inference, along with the compiled model artifact. This package will include everything required to deploy the model on your target edge hardware.

In [21]:

recipe['model.use_pretrained'] = False
demo_dir = rd.tasks.generate_stub_code(recipe)

recipe['model.use_pretrained'] = False demo_dir = rd.tasks.generate_stub_code(recipe)

/home/sai/miniconda3/envs/design/lib/python3.10/site-packages/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
  warnings.warn(
WARNING:af.core.utils.helpers:Your fire-and-smoke-coco-like dataset's train augmentation has a normalization operation with parameters mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] but it won't be used because Normalization is only applied to the outermost augmentation in the composition of data_generators.
WARNING:af.core.data.generic.coco_reader:
 If a 0-index class is present it must be background. Object classes should start at index 1. Check `label_indexing` parameter for details.

WARNING:af.core.data.generic.coco_reader:
 If a 0-index class is present it must be background. Object classes should start at index 1. Check `label_indexing` parameter for details.

WARNING:af.core.data.sets.composites.composite_mosaic:
 If a 0-index class is present it must be background. Object classes should start at index 1. Check `label_indexing` parameter for details.

2024-10-29 11:37:19,919 | WARNING  | base.copy_sample_image-287 | Could not copy sample image due to error ''CocoDetection' object has no attribute 'dataset'', trying to load
WARNING:leip_stub_gen.core.base:Could not copy sample image due to error ''CocoDetection' object has no attribute 'dataset'', trying to load
2024-10-29 11:37:19,923 | WARNING  | base.copy_sample_image-295 | Could load data-loader to get image due to error 'tuple indices must be integers or slices, not str', cannot include sample image...
WARNING:leip_stub_gen.core.base:Could load data-loader to get image due to error 'tuple indices must be integers or slices, not str', cannot include sample image...

loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!