Detector BYOD: COCO Format Example

Bring Your Own COCO Formatted Data

We will begin with the MS COCO format ingestor. Find the MS COCO-like dataset template file in the Docker container located at /latentai/custom-configs/data/coco-reader-template.yaml

It will look like this:

CODE

defaults:
  - /data/transforms@module.dataset_generator.train_transforms: resize.yaml
  - /data/transforms@module.dataset_generator.valid_transforms: resize.yaml
  - _self_

nclasses: 0 # number of classes in your detection dataset
module:
  _target_: af.core.data.modules.adaptermodule.AdaptorDataModule
  batch_sizes: ${task.batch_sizes}
  num_workers: ${task.num_workers}
  adaptors: ${model.adaptors}
  dataset_generator:
      _target_: af.core.data.generic.coco_reader.CocoReader
      root_path: null # full path to your dataset
      task_family: detection
      train_images_dir: train # path to folder containing training images, relative to root_path
      val_images_dir: validation # path to folder containing only validation images, relative to root_path
      train_annotations_json: annotations/instances_train.json # path to .json file of training annotations, relative to root_path
      val_annotations_json: annotations/instances_val.json # path to .json file of validation annotations, relative to root_path
      label_indexing: 0-indexed-no-background # one of: "0-indexed-no-background", "1-indexed-no-background", "0-indexed-with-background". if dataset contains a background class, it must be at index 0. 
      dataset_name: my-custom-coco-like-data # string - any name for your data

Everything except the root_path is set to match the COCO dataset defaults. The folder structure is expected to be:

CODE

   path/to/mydataset/                
                    |---train
                        |---image1.jpeg
                        |---image2.jpeg
                            ...
                    |---validation
                        |---image8.jpeg
                        |---image9.jpeg
                            ...
                    |---annotations
                        |---instances_train.json
                        |---instances_val.json

You can configure the reader to your dataset specifics by entering the number of classes in your dataset and the dataset_generator fields shown above from line 14 on. Changing those fields with your dataset’s information is enough to load and use your COCO-formatted dataset.

nclasses: number of classes in your dataset
root_path: absolute path to the root directory of the dataset
task_family: detection or segmentation (COCO annotations supports both)
train_images_dir: path to folder containing only training images, relative to root_path
val_images_dir: path to folder containing only validation images, relative to root_path
train_annotations_json: path to .json file of training annotations, relative to root_path
val_annotations_json: path to .json file of validation annotations, relative to root_path
label_indexing: Are the labels 0 indexed or is the first class at index 1? Is there a background class? One of 0-indexed-no-background, 1-indexed-no-background, 0-indexed-with-background. The AF needs this information to decide if it needs to label shift to ensure that if a background class is present it must be at index 0, and other classes start at index 1.
dataset_name: a string that will be used to name any generated artifacts.

You can use the customized YAML file for your dataset as a component by passing its name to the data parameter, like this:

CODE

af --config-name=<recipe_name> data=coco-reader-template \
     command=train task.moniker="BYOD_recipe"

A Concrete BYOD COCO Format Example

SODA10M Dataset Overview

When we look at the SODA10M dataset as an example, we can see that the folder structure does not exactly match the LEIP Recipe defaults:

Download the SODA10 labeled dataset from the website.

Option 1: download and extract via the command line

CODE

pip install gdown
mkdir /latentai/workspace/datasets/soda10m
cd /latentai/workspace/datasets/soda10m
gdown https://drive.google.com/uc?id=1oSJ0rbqNHLmlOOzpmQqXLDraCCQss4Q4
tar -xvf labeled_trainval.tar

# Get back to /latentai
cd /latentai

Option 2: Download via the Soda10M website.

Follow this URL and download SODA10M Trainval
You will download the following file: labeled_trainval.tar
Create a /latentai/soda10m directory in the SDK container
Copy those files there, and then untar them.

Verify that the data has been located in the proper directory:

CODE

# Verify data location matches the directories in this example:
ls /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/
annotations    train     val

Using the Template to Ingest the SODA10M Data

Now we need to create a YAML configuration file for this dataset. Create a new file custom-configs/data/soda10m_config.yaml by copying our template:

CODE

cp /latentai/custom-configs/data/coco-reader-template.yaml \
 /latentai/custom-configs/data/soda10m_config.yaml

Then, modify the fields in the file soda10m_config.yaml to match the needs of this dataset.

Notice the slight difference in annotation files: their instance file, which should now reside in /latentai/soda10m/SSLAD/labeled/annotations, uses instance_train.json instead of the plural instances_train.json. It is not necessary to rename the files. Instead, refer to the example below on how to leverage the flexibility of the Coco Reader’s YAML configuration to account for this detail.

The soda10m_config.yaml will look like this:

CODE

defaults:
  - /data/transforms@module.dataset_generator.train_transforms: resize.yaml
  - /data/transforms@module.dataset_generator.valid_transforms: resize.yaml
  - _self_

nclasses: 6
module:
  _target_: af.core.data.modules.adaptermodule.AdaptorDataModule
  batch_sizes: ${task.batch_sizes}
  num_workers: ${task.num_workers}
  adaptors: ${model.adaptors}
  dataset_generator:
      _target_: af.core.data.generic.coco_reader.CocoReader
      root_path: /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled
      task_family: detection
      train_images_dir: train
      val_images_dir: val
      train_annotations_json: annotations/instance_train.json
      val_annotations_json: annotations/instance_val.json
      label_indexing: 1-indexed-no-background
      dataset_name: soda10m

We can start by visualizing a few samples of the dataset when the YAML configuration is ready.

CODE

af data=soda10m_config command=vizdata

The vizdata command will read the annotations and draw bounding boxes over some of the samples. The sample images will be stored in /latentai/artifacts/vizdata/soda10m/*.

Train

You can now train on this dataset by running the following:

CODE

af --config-name=<recipe_name> data=soda10m_config \
   command=train \
   task.moniker="BYOD_recipe"

Understanding the command:

--config-name=<recipe_name> : Enter the name of the recipe to use. Example --config-name=yolov5. Refer to Advanced AF Options to see a list of all recipes and models that Latent AI supports.
data=soda10m_config : Use the data/soda10m_config.yaml file as the data config for this recipe
- It's important to ensure the .yaml is nested inside an additional data folder within custom-configs
task.moniker="BYOD_recipe" Gives this run a different name so we do not confuse its artifacts with the pretrained model’s artifacts that were generated in the previous sections.

Note where the checkpoint is stored at the end of the training run. The checkpoint will be stored in a path of the form:/latentai/artifacts/train/{date}_{time}_BYOD_recipe/{epoch}.ckpt Locate that file and store it for ease of use in the next steps:

CODE

export SODA_CHECK=<path to .ckpt file>

# Example:
# export SODA_CHECK=/latentai/artifacts/train/2022-08-23_17-25-29_task_BYOD_recipe/epoch-19_step-1840.ckpt

Note: The default recipe parameters are not expected to yield optimal training results for this dataset. This tutorial is only meant to show how to ingest COCO formatted data. Please refer to Advanced AF Options for instructions on how to tune hyperparameters.

Evaluate on the Host Before Exporting

You can now use your newly trained model by passing the checkpoint to the commands you previously used with the pre-trained model. Run the following commands to evaluate:

CODE

af --config-name=<recipe_name> data=soda10m_config \
  command=evaluate \
  task.moniker="BYOD_recipe" \
  +checkpoint=$SODA_CHECK

Refer to Evaluate Options for additional Evaluate command options.

Visualize the Predictions on Host Device

The AF command=predict allows you to predict on a few samples of the data using the trained model checkpoint.

CODE

af --config-name=<recipe_name> data=soda10m_config  \
 command=predict \
 task.moniker="BYOD_recipe"  \
 +checkpoint=$SODA_CHECK

The predicted images will be stored in /latentai/artifacts/predictions/soda10m/*

Left: Ground truth example from the validation set.
Right: Predictions generated by the trained checkpoint.

Refer to Predict Options for additional visualization options.

Export the Model

CODE

af --config-name=<recipe_name> data=soda10m_config \
  command=export \
  task.moniker="BYOD_recipe" \
  +checkpoint=$SODA_CHECK

Your exported model will be saved in:
/latentai/artifacts/export/<some_name>/traced_model.pt

Save the traced_model.pt path as a variable:

CODE

export EXPORTED_MODEL=/latentai/artifacts/export/<some_name>/traced_model.pt

Optimize the Model

You will need to identify a calibration dataset from your BYOD data before you can optimize your trained model. Identifying an ideal calibration dataset is beyond the scope of this tutorial but usually entails selecting a few representative images for the quantizer. If you are targeting a CUDA device, you will need to provide 2N images, where N is your batch size. You will need to provide a path to these images in the file rep_dataset.txt that is in the model recipe directory. We will choose four images in the validation dataset for the SODA10M dataset.

Override the file in /latentai/recipes/<recipe_name>/rep_dataset.txt with paths to samples in the SODA10M dataset:

CODE

/latentai/soda10m/SSLAD-2D/labeled/val/HT_VAL_004979_SH_001.jpg
/latentai/soda10m/SSLAD-2D/labeled/val/HT_VAL_004980_SH_010.jpg
/latentai/soda10m/SSLAD-2D/labeled/val/HT_VAL_004981_GZ_231.jpg
/latentai/soda10m/SSLAD-2D/labeled/val/HT_VAL_004982_SH_010.jpg

If you find that your optimized (INT8) model has substantial accuracy loss from the compiled (Float32) model, try different calibration images. Some image sets will produce outlier results that may adversely affect accuracy.

Please bear in mind that the optimal training dataset may change each time you retrain the model. The above is an example dataset, but you may need to test out different datasets to yield the best results after you have trained your model.

Once you have exported your trained model and determined a calibration dataset, simply compile and optimize it as before providing the traced model as the input path:

CODE

leip pipeline \
  --input_path $EXPORTED_MODEL \
  --output_path /latentai/workspace/output/byod_soda10m \
  --config_path /latentai/recipes/<recipe_name>/pipeline_x86_64_cuda.yaml

Evaluate the SODA10M Example on Host

If you would like to use leip evaluate to evaluate your model, you will need to copy one file to prepare the validation dataset:

CODE

# Some processes will look for the instances file in the data directory:
cp /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/annotations/instance_val2017.json \
  /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/val/

You can now evaluate your compiled, optimized model against the Smoke dataset:

CODE

# Evaluate Float32:
leip evaluate \
  --input_path /latentai/workspace/output/byod_soda10m/Float32-compile/ \
  --test_path /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/val/instance_val.json \
  --dataset_type coco

# Evaluate Int8:
leip evaluate \
  --input_path /latentai/workspace/output/byod_soda10m/Int8-optimize/ \
  --test_path /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/val/instance_val.json \
  --dataset_type coco

Evaluate the SODA10M Example on Target

You will need run leip evaluate in the SDK docker container with inference performed on the device under test in order to evaluate on a remote target device. You will first need to setup your target by installing Latent AI Object Runner (LOR). You will then evaluate using the LRE objects created by leip pipeline.

CODE

# Substitute the IP address of your target device for <IP_ADDR>
# The default port for LOR is 50051

# Evaluating Float32:
leip evaluate \
  --input_path /latentai/workspace/output/byod_soda10m/Float32-package \
  --host <IP_ADDR> --port 50051 \
  --test_path /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/val/instance_val.json \
  --dataset_type coco

# Evaluating Int8:
leip evaluate \
  --input_path /latentai/workspace/output/byod_soda10m/Int8-package \
  --host <IP_ADDR> --port 50051 \
  --test_path /latentai/workspace/datasets/soda10m/SSLAD-2D/labeled/val/instance_val.json \
  --dataset_type coco