Skip to content

What to do with a recipe?

A typical flow for a beginner user who wants to train recipes on their data would look like this: Typical BYOD Flow

The below sections briefly go into each part of the multi step process.

This flow is automatable

A big advantage of Recipes is that once you have scripted this process for one recipe, it will be automatable for any number of recipes you wish to train!

Train a Model

In a nutshell, a complete recipe can be trained with a simple command:

from leip_recipe_designer.tasks import train
evaluate_output = train(recipe)
Check out the train task reference API for more details.

Provide My Data: BYOD

For a detection task, your data should be in COCO, PascalVOC, or YOLO formats. See the reference API for Datasets for more details on how to ingest your dataset.

For example:

from leip_recipe_designer.helpers.data import new_pascal_data_generator
new_data = new_pascal_data_generator(...)

If you are modifying an existing recipe, for example one you loaded from our Golden Recipes database, you will have to replace the pre-defined data generator with the newly created one. We have a helper function that does just that:

from leip_recipe_designer.helpers.data import replace_data_generator
replace_data_generator(recipe, new_data)

Save the Best Checkpoint During Training

If you are fine-tuning a model or training from scratch, you can save the best model checkpoint using the Checkpointing callback. We recommend that you always enable this. To see if its enabled, print(recipe["callbacks"]) and see if there is one of type "Checkpointing". If not, you may add it: 

recipe.assign_ingredients("callbacks", {"checkpointing": "Checkpointing"})

This checkpoint monitors your validation loss and saves the best-performing candidate. Additionally you can provide a custom destination or adjust the number of best models to store.

recipe["checkpoint.dirpath"] = "/my/path"
recipe["checkpoint.save_top_k"] = 3
print(recipe["callbacks"]["name:checkpointing"])

Specify a Trained Checkpoint for Further Processing (Evaluate, Visualize Predictions, Export)

Use the following syntax in order to specify an existing checkpoint for continuing training, exporting, evaluating or visualizing:

recipe.assign_ingredients("checkpoint", "local ckpt")
recipe["model.checkpoint"] = <path of .ckpt file>
The path name can be absolute or relative to the current working directory.

If your recipe had enabled the Checkpointing callback, your .ckpt will be under workspace/artifacts/train/[datetime]_task...

Evaluate With a Trained Model Checkpoint

  1. Unless you're looking to evaluate pretrained weights, make sure your checkpoint is specified in your recipe. You can check this via print(recipe["model.checkpoint"]). If you get an error, specify a checkpoint first.
  2. Using the tasks api, call tasks.evaluate.
from leip_recipe_designer.tasks import evaluate
evaluate_output = evaluate(recipe)

The Application Framework will predict and run evaluation metrics over the entire validation set. At the end, you will see a metrics report that will also be exported to workspace/artifacts/evaluate/[data_generator.dataset_name]/metrics_report.json

You can customize the evaluation, see the tasks.evaluate API for more details.

Evaluating compiled models

You can also change what kind of model you are evaluating, from a trained checkpoint, to a compiled LRE artifact, by replacing the "checkpoint" type. See what kind of models you can evaluate with recipe.options("checkpoint")

Visualize Predictions with a Trained Model Checkpoint

  1. Unless you're looking to evaluate pretrained weights, make sure your checkpoint is specified in your recipe. You can check this via print(recipe["model.checkpoint"]). If you get an error, specify a checkpoint first.
  2. Using the tasks api, call tasks.visualize_predictions. 
    from leip_recipe_designer.tasks import visualize_predictions
prediction_output = visualize_predictions(recipe)

The Application Framework runs a prediction over the subset of validation set. When the predictions are calculated, they are visualized on the corresponding images. The result is written to workspace/artifacts/predictions/[data_generator.dataset_name]/[prediction.subset].

You can provide a different subset and adjust its size and other prediction parameters, visit the tasks.visualize_predictions API for more details.

Visualizing predictions of compiled models

You can also change what kind of model you are using to predict, from a trained checkpoint, to a compiled LRE artifact, by replacing the "checkpoint" type. See what kind of models you can use via recipe.options("checkpoint")

Export a Model

  1. Unless you're looking to export pretrained weights, make sure your checkpoint is specified in your recipe. You can check this via print(recipe["model.checkpoint"]). If you get an error, specify a checkpoint first.
  2. Using the tasks api, call tasks.export_model
from leip_recipe_designer.tasks import export_model
export_output = export_model(recipe)

By default, the model is exported to workspace/artifacts/export/[experiment.name]_[model.backbone]_[export.batch_size]x[export.data.height]x[export.data.width].pt. However, you can specify this and other parameters. See the tasks.export_model task API for more information.

Compile and deploy your model

With the upcoming release of LEIP 4.0 you will be able to configure your compilation and deployment parameters via the recipe. Furthermore, you will be able to call tasks.compile(recipe) and tasks.generate_stub_code(recipe) to get a deployable artifact with example wrapping code for your device and environemnt of choice. Cool ha?

Saving and Loading Recipes

Recipes can be stored as strings to be transported and re-run at any point in time. To create the portable string representation of your recipe, simply run:

save_as_sppr.py
sppr_string = recipe.to_sppr()

Then, when you are ready to re-load that recipe, you can use the from_sppr method:

loading_an_sppr.py
from leip_recipe_designer.create import from_sppr
recovered_recipe = from_sppr(sppr_string)
Refer to create.from_sppr()

Note, that reloading a recipe doesn't require a pantry, since a recipe is fully self-contained and has everything it needs to be re-run in the same leip-af version as it was stored when to_sppr() was called.

But, if you need to make any changes to your recipe, or you would like to run in a newer version of the leip-af, you may need to upgrade some ingredients. This can be done automatically by simply passing a pantry to the from_sppr() function and allowing upgrades:

loading_and_upgrading_an_sppr.py
# download the latest pantry
from leip_recipe_designer import Pantry
pantry = Pantry.build("./some_path")

# deserialize the recipe
from leip_recipe_designer.create import from_sppr
upgraded_recipe = from_sppr(sppr_string, pantry, allow_upgrade=True)

Info

SPPR stands for Serialized Packaged Portable Recipe.