Explore Golden Volumes and Select a Recipe¶
In this guide, we will walk through the process of exploring Golden Recipe volumes using the GRDB API and selecting a suitable recipe (Golden Recipe from the Golden DataFrame.
Understanding Golden Recipe Volumes¶
Golden Recipe volumes are collections of recipes that have demonstrated strong performance on specific datasets. Latent AI continuously works to discover better recipes for increasingly diverse datasets, ensuring that you have access to high-performing models.
To begin, let's list the available Golden Recipe volumes:
import leip_recipe_designer as rd
volumes = rd.GoldenVolumes()
volumes
Recipe Volumes: - chemistrylab - bdd100k - insects - wheat - carsimple - xval_det
Each name in the list corresponds to a Golden Recipe Volume. These collections contain recipes that have excelled on particular datasets. By exploring the datasets used to train each volume, you can identify the one that closely matches your own data.
If any of the volumes seem to have data that is very similar to yours, feel free to start with that one. Keep in mind that data similarity is not a well-defined concept, so choose one volume to begin with and iterate as necessary. For more information, consult the guide on GRDB Model Training and Optimization.
volumes["carsimple"]
Skipped downloading goldenrecipedb with name "carsimple" and variant "Carsimplev0.3" (0), as it already exists.
Tab(children=(HTML(value='<table>\n<tbody>\n<tr><td>Description </td><td>'Simple and small dataset …
You can click around on the widgets to explore the data, including the training and validation samples, as well as other metadata such as class distribution and resolution details.
Exploring the Golden DataFrame¶
Golden Recipe Volumes are structured as Pandas DataFrames, allowing you to query and sort them based on your specific performance criteria. For example, you can retrieve the DataFrame for a specific volume like this:
df = volumes["carsimple"].get_golden_df()
df
| id | timestamp | neptune_id | hardware | quantization | calib_method | per_channel | height | width | model_family | ... | od_task_metric | od_inf_rate_ms | od_inf_energy_mJ | od_size_MB | od_gpu_mem_MB | od_cpu_mem_MB | od_unified_mem_MB | od_total_mem_MB | od_avg_cpu_freq_Hz | sppr | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 118984 | 118984 | 2024-10-15 09:38:45.048419 | GRDBCAR-102 | cuda:A4500 | float32 | none | none | 736.0 | 416.0 | Yolo v5 | ... | 0.428954 | 1.401320 | 117.985512 | 16.544411 | 243.000000 | 697.101562 | NaN | 940.101562 | NaN | eJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS... |
| 118985 | 118985 | 2024-10-15 09:38:45.048419 | GRDBCAR-102 | cuda:A4500 | float16 | entropy | False | 736.0 | 416.0 | Yolo v5 | ... | 0.432355 | 1.126059 | 85.763657 | 13.229023 | 218.770279 | 364.319977 | NaN | 583.090256 | NaN | eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... |
| 118986 | 118986 | 2024-10-15 09:38:45.048419 | GRDBCAR-102 | cuda:A4500 | int8 | entropy | False | 736.0 | 416.0 | Yolo v5 | ... | 0.162237 | 1.009040 | 65.303383 | 11.787714 | 222.000000 | 381.723969 | NaN | 603.723969 | NaN | eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... |
| 118987 | 118987 | 2024-10-15 09:38:45.048419 | GRDBCAR-102 | cuda:AGX-Xavier | float32 | none | none | 736.0 | 416.0 | Yolo v5 | ... | 0.428954 | 7.177697 | 0.000000 | 21.236946 | NaN | NaN | 1324.970825 | 1324.970825 | NaN | eJzlvfmX28iRIPy7/wrOjrXq3q/JyhtIvdfjUbf6mm21tS... |
| 118988 | 118988 | 2024-10-15 09:38:45.048419 | GRDBCAR-102 | cuda:AGX-Xavier | float16 | entropy | False | 736.0 | 416.0 | Yolo v5 | ... | 0.432355 | 4.659135 | 0.000000 | 14.740621 | NaN | NaN | 1326.923950 | 1326.923950 | NaN | eJzlvfuX3LaRKPx7/oreu9GVfT93D94gdI43K1t+7bUcXU... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 123024 | 123024 | 2024-10-15 09:38:45.048419 | GRDBCAR-99 | rpi4_8gb | int8 | percentile | False | 512.0 | 512.0 | EfficientDet | ... | 0.035733 | 4296.565918 | NaN | 69.839480 | NaN | 324.141998 | NaN | 324.141998 | 1757.000000 | eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... |
| 123025 | 123025 | 2024-10-15 09:38:45.048419 | GRDBCAR-99 | i9-10920X | int8 | percentile | True | 512.0 | 512.0 | EfficientDet | ... | 0.029460 | 134.523331 | NaN | 40.135960 | NaN | 266.424469 | NaN | 266.424469 | 1621.237564 | eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... |
| 123026 | 123026 | 2024-10-15 09:38:45.048419 | GRDBCAR-99 | rpi4_8gb | int8 | percentile | True | 512.0 | 512.0 | EfficientDet | ... | 0.029460 | 4543.513184 | NaN | 71.299512 | NaN | 326.242004 | NaN | 326.242004 | 1757.000000 | eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... |
| 123027 | 123027 | 2024-10-15 09:38:45.048419 | GRDBCAR-99 | i9-10920X | uint | percentile | False | 512.0 | 512.0 | EfficientDet | ... | 0.030770 | 142.222061 | NaN | 42.585248 | NaN | 234.094345 | NaN | 234.094345 | 1480.753062 | eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... |
| 123028 | 123028 | 2024-10-15 09:38:45.048419 | GRDBCAR-99 | rpi4_8gb | uint | percentile | False | 512.0 | 512.0 | EfficientDet | ... | 0.030770 | 4499.782715 | NaN | 71.369056 | NaN | 324.790039 | NaN | 324.790039 | 1757.000000 | eJztvf2X3LaVIPr7/BWcnXhl70tVgwBBEn1OJitLjuMdy9... |
4045 rows × 26 columns
To get a detailed overview of the metrics and configuration parameters stored in a Golden Recipe Volume, use the describe_table() method. This will display information like model architecture, hardware deployment metrics, memory usage, and inference performance:
volumes["carsimple"].describe_table()
------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- id Index ID for the recipe wrt central db timestamp Time the recipe was added to the DB. neptune_id Neptune AI Id of the model trained. hardware Deployment hardware on which the metrics were calculated. quantization Quantization method used for On Device metrics. calib_method Quantization calibration method used for On Device metrics. per_channel If per-channel quantization was used for On Device metrics. height Height of the input to the model width Width of the input to the model model_family Architecture of the model backbone Backbone used to extract the features. optimizer Optimization algorithm used to adjust weights during training scheduler Learning rate scheduler used to adjust the optimizer learning rate during training if_task_metric Relative accuracy using the coco average on the validation set, computed Pytorch (not on edge hardware). if_inf_size Size of the Pytorch model in bytes in (not on edge hardware). if_inf_macs Number of Multiply-Accumulate Operations during in framework inference (Not on edge hardware) od_task_metric On Device Mean Average Precision using the coco average of the model on the validation set. od_inf_rate_ms On Device Average inference rate in ms of the core artifact, excluding pre- and post processing. Note- post-processing for detector models includes nms and can consume substantial time. od_inf_energy_mJ On Device Energy per inference in mJ measured (Not avaliable for all devices). od_size_MB On Device size in MB of the compiled model artifact (modelLibrary.so) saved to disk. od_gpu_mem_MB On Device GPU Memory usage in MB (Not avaliable for all devices). od_cpu_mem_MB On Device CPU memory usage in MB, i.e. system memory. od_unified_mem_MB On Device GPU and CPU memory useage in MB for devices with shared CPU/GPU memory (eg. Jetsons). (Not avaliable for all devices) od_total_mem_MB On Device combined memory usage in MB CPU + GPU. For CPU, only system memory. For GPU, it is GPU memory + System memory. For devices with combined GPU and CPU (eg. Jetsons), it is combined memory. od_avg_cpu_freq_Hz On Device CPU clock rate in Hz. sppr Serialized Packaged Portable Recipes (SPPR) ready to train on selected data ------------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Filtering for Suitable Recipes¶
In our scenario, we aim to identify candidate recipes that offer a balance between accuracy and fast inference. To achieve this, we will analyze the multiply-accumulate (MAC) operations during inference.
Calculate the 75th Percentile of the Task Metric:
threshold = df["if_task_metric"].quantile(0.75)
Include only those recipes where the task accuracy metric meets or exceeds the calculated threshold and sort the recipes by inference MACs in ascending order:
filtered_df = df[df["if_task_metric"] >= threshold]
sorted_df = filtered_df.sort_values("if_inf_macs")
sorted_df.head()
| id | timestamp | neptune_id | hardware | quantization | calib_method | per_channel | height | width | model_family | ... | od_task_metric | od_inf_rate_ms | od_inf_energy_mJ | od_size_MB | od_gpu_mem_MB | od_cpu_mem_MB | od_unified_mem_MB | od_total_mem_MB | od_avg_cpu_freq_Hz | sppr | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 122834 | 122834 | 2024-10-15 09:38:45.048419 | GRDBCAR-9 | cuda:AGX-Orin | int8 | entropy | False | 320.0 | 256.0 | NanoDet | ... | 0.647767 | 2.243725 | 32.957386 | 9.641616 | NaN | NaN | 2400.000488 | 2400.000488 | 1581.438461 | eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... |
| 122826 | 122826 | 2024-10-15 09:38:45.048419 | GRDBCAR-9 | cuda:A4500 | float32 | none | none | 320.0 | 256.0 | NanoDet | ... | 0.641198 | 1.545569 | 96.821007 | 8.764759 | 373.0 | 702.741089 | NaN | 1075.741089 | NaN | eJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8... |
| 122827 | 122827 | 2024-10-15 09:38:45.048419 | GRDBCAR-9 | cuda:A4500 | float16 | entropy | False | 320.0 | 256.0 | NanoDet | ... | 0.640356 | 0.760797 | 64.680267 | 7.014277 | 224.0 | 446.127960 | NaN | 670.127960 | NaN | eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... |
| 122828 | 122828 | 2024-10-15 09:38:45.048419 | GRDBCAR-9 | cuda:A4500 | int8 | entropy | False | 320.0 | 256.0 | NanoDet | ... | 0.647767 | 0.875939 | 54.021370 | 7.835824 | 213.0 | 368.084015 | NaN | 581.084015 | NaN | eJzlvWu3G7exIPo9v4JnTjSy55rceDSAhtbyZGTJdnKu5e... |
| 122829 | 122829 | 2024-10-15 09:38:45.048419 | GRDBCAR-9 | cuda:AGX-Xavier | float32 | none | none | 320.0 | 256.0 | NanoDet | ... | 0.641198 | 5.011565 | 0.000000 | 9.152363 | NaN | NaN | 1944.831177 | 1944.831177 | NaN | eJzlvf2X27ixIPr7/St098brmX0jNT4IgPA5s1nPeGaS+8... |
5 rows × 26 columns
Selecting a Recipe¶
After filtering and sorting the DataFrame to identify a suitable recipe, the next step is to retrieve the Serialized Packaged Portable Recipe (SPPR) for your chosen model configuration. For instance, from the table above, let’s select the recipe with ID 122834 tailored for cuda:AGX-Orin (Jetson AGX Orin), which employs int8 quantization and entropy calibration.
from pathlib import Path
workspace = Path('./workspace')
pantry = rd.Pantry.build(workspace / "./my_combined_pantry/", force_rebuild=False)
# Select the recipe with the specified ID
recipe_id = 122834
selected_recipe = sorted_df[sorted_df['id'] == recipe_id].iloc[0]
sppr = selected_recipe['sppr']
recipe = rd.create.from_sppr(sppr, pantry=pantry, allow_upgrade=True)
2024-10-29 11:53:04,808 | WARNING | pantry.build-119 | You requested to build a Pantry, but haven't specified the desired execution contexts. Therefore, will use the installed ones ['leip_af', 'leip_forge', 'leip_stub_gen']
By following these steps, you can effectively explore Golden Volumes, filter for recipes that meet your performance needs, and select a recipe that best fits your dataset. Now, you can replace the data ingredient with your dataset and begin training as shown before.
Happy experimenting with your new model!