Iterative Knowledge Distillation for Automatic Check-Out
- Publication
- IEEE Transactions on Multimedia 2021Journal
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@article{retailproductdpsnet2021,
title = {{Iterative Knowledge Distillation for Automatic Check-Out}},
author = {Libo Zhang and Dawei Du and Congcong Li and Yanjun Wu and Tiejian Luo},
year = {2021},
journal = {IEEE Transactions on Multimedia},
url = {https://doi.org/10.1109/tmm.2020.3037502},
doi = {10.1109/tmm.2020.3037502}
}
The retail product domain includes several branches, one of which is automatic check-out.
Automatic Check-Out (ACO) is the full name of this task. Let’s examine related research.
Problem Definition
With the gradual digital transformation of the retail industry, systematically recognizing the types and quantities of products purchased by customers has become key to improving customer experience and reducing labor costs. However, ACO differs significantly from typical object detection tasks in both problem structure and data characteristics.
Taking the RPC dataset as an example, the training data are captured on a turntable, each image containing a single product from various viewpoints, with clean backgrounds and stable lighting; whereas the test data simulate real checkout scenes, stacking multiple products on a white board shot from a fixed top-down view, with cluttered backgrounds, heavy occlusions, and uneven lighting.
- This domain shift—from “single product display” to “stacked scenes”—is the most common challenge faced in ACO tasks.
Besides domain shift, another difficulty in ACO is the asymmetry of annotation resources.
In practical applications, model training can only rely on manually annotated images, while the complexity of scenes during testing is far greater than in training. Even though datasets like RPC provide multi-level annotations, in real scenarios test images are often “unlabeled” or “weakly labeled,” limiting the extensibility of supervised learning in ACO.
Some studies attempt to use synthetic data, such as CycleGAN-based techniques, to reduce the domain gap; however, the effect is limited, and training data may still have quality issues.
Severe distribution misalignment, unequal supervision signals, and highly occluded, densely overlapping test scenes persist.
The authors of this paper aim to find an effective training process amidst these difficulties.
Solution
The overall model training framework is shown above. Facing the huge gap between the source domain (single-product images) and the target domain (checkout scenes), the authors first propose a data augmentation strategy aimed at simulating real scenes and training adaptation to complexity.
Data Augmentation
This augmentation process can be divided into three stages: Background Removal → Pose Pruning → Synthetic Image Generation.
-
Background Removal
In the RPC dataset, training images are taken on a turntable. Although bounding boxes are provided, much background noise remains. Directly compositing these images would cause the model to learn the “wrong background.”
Therefore, the authors use a coarse-to-fine image processing method for background removal:
The specific steps are:
- Edge Detection: Use an edge detector to find object contours, filtering out low-confidence areas.
- Morphological Processing: Use dilation and erosion operations to remove noise and fill broken regions.
- Edge Smoothing: Apply median filtering to further eliminate jagged edges.
- Visual Saliency Model: Finally, use a saliency detection network (pretrained on MSRA-B and fine-tuned with coarse masks) to generate clean object masks.
This transforms the original images containing backgrounds and turntables into clean product images, which can be used for subsequent synthetic training samples.
-
Pose Pruning
Not all single-product poses are suitable for checkout scenes. For example, a water bottle photographed from a low upward angle looks unnatural in checkout images.
To automatically select “reasonable poses,” the authors design a simple but practical metric:
- : The mask area of category at viewpoint .
- A small relative mask area is used as a heuristic to discard poses; it is not a universal test of whether a pose is physically natural.
Only poses passing this threshold (greater than some value ) are kept for synthesis.
-
Checkout Scene Synthesis
Once clean masks and reasonable poses are available, the next step is to synthesize simulated checkout scenes.
The operation proceeds as follows:
- Randomly place multiple products: Select several filtered products, randomly rotate them (0–360°) and scale them (0.4–0.7), then paste onto a clean background.
- Avoid excessive occlusion: Each product is occluded at most 50%.
- Lighting and Shadow Handling: Although the positions are reasonable, synthetic images lack real-world lighting variation. Therefore, CycleGAN style transfer is applied to make images closer to real test scenes.
The figure below shows examples of the synthesized checkout scenes:
DPSNet
Through data augmentation, the model gains preliminary ability to simulate real checkout images, but this alone is insufficient to bridge the semantic gap with real test images. The authors propose a model architecture named DPSNet (Dual Pyramid Scale Network) that attempts to learn shared feature representations from two different views:
- Object Detection
- Object Counting
First, the Detection View aims to locate objects. It outputs a set of bounding boxes in the form , where:
- : object class
- : object center coordinates
- : object width and height
This is a typical object detection output providing spatial localization and classification.
The Counting View predicts a density map whose spatial sum estimates the number of objects. Although the formulation permits C channels, the final setting uses C=1: it counts all products together, not each SKU separately. The detection branch still predicts product classes.
DPSNet uses an FPN to construct multiscale features through from the backbone, allowing both views to use features from different levels.
Feature fusion proceeds as follows:
- Upsample low-resolution, high-level features step by step to , combining information across scales.
- Downsample the high-resolution feature to , adding spatial detail.
The fused counting features pass through 3×3 and 1×1 convolutions to produce the density map. The detection head uses multiscale features and fully connected layers to predict candidate box classes and locations.
Iterative Knowledge Distillation
Although the model already has prediction capability, its stable performance on the target domain cannot be guaranteed.
The authors adapt to unlabeled target/test images with iterative pseudo-labeling. This is a transductive setting: test images participate in adaptation, although their labels are withheld. It must be distinguished from evaluation on completely untouched test inputs. The process is illustrated below:
The overall workflow is as follows:
-
Consistency Check
For each unlabeled test image, DPSNet performs:
- Detection view D: predicts object locations and classes
- Counting view C: generates a density map estimating the total number of items (C=1)
Then, it checks whether the two views are “consistent”:
- If the total count of high-confidence detections agrees with the rounded integral of the density map, the image is considered consistent. With , this checks the total item count. Agreement is a reliability heuristic.
- These “consistent samples” are added to the training data with generated pseudo labels
-
Iterative Distillation
In each training iteration, DPSNet will:
- Use the current model to predict on the test set
- Perform consistency check → select reliable samples → add to training
- Retrain with this new batch of pseudo labels
- Repeat this process multiple times
-
Instance-Level Refinement
Finally, after several rounds of distillation, the authors add a refinement step:
- For images that were entirely discarded, if some objects have high confidence, retain those pseudo-labels and fill the low-confidence object regions with background pixels in the training image; simply dropping their labels would leave unlabeled foreground objects
- Remove the counting head, then use these repaired “semi-reliable” images to fine-tune the detection network
The figure below shows three common types of discarded cases:
The top row shows three original checkout images; the bottom row shows their corresponding images after low-confidence object regions are masked. A/B/C mark the three columns’ ignored regions: A) a product that looks similar to neighboring products, B) an occluded product, C) a non-product item.
Through the cycle of “consistency → distillation → refinement,” knowledge gradually permeates from the source domain to the target domain, strengthening progressively at both image and instance levels, ultimately enhancing model stability on real test scenes.
Loss Function
To ensure DPSNet learns domain-specific information stably during iterative training, the authors designed loss functions for the two views and combined them as the overall training objective.
The loss overview is:
The counting view learns , a spatial count density. Its integral approximates the object count; its values are not class-presence probabilities bounded by one.
- Loss type: L2 loss (squared Euclidean distance)
- Intuition: The closer the predicted density map is to the true density distribution, the better the model estimates the item count. The final setting estimates a total rather than per-category counts.
Ground-truth density maps are constructed by:
- Placing Gaussian kernels at each object center
- Summing the kernels within each chosen channel grouping; C=1 combines all objects
- Producing a density map whose height and width are each 1/8 those of the input, such as 100×100 for an 800×800 image
The detection view’s goal is to predict class and bounding box locations for each object:
- Classification loss (Cross-Entropy): error between predicted class and ground-truth class
- Regression loss (Smooth L1): bounding box coordinate error, computed only for true objects ()
The indicator function means:
- Foreground objects: compute both classification and regression loss
- Background: only classification loss to avoid false positives
The regression targets , represent parameterized bounding box coordinates (e.g., center and size). The Smooth L1 loss provides stable gradients for small errors, improving training stability.
Discussion
RPC Dataset Benchmark
The experimental results of DPSNet are detailed above, compared with two baselines: Reference [1] is the original checkout detection system paper; Reference [8] is the subsequent improved DPNet.
Each method underwent training with two data versions: one using only CycleGAN-processed Render images, and another adding Copy-Paste synthetic images to form the Syn+Render training set. DPSNet also added an Instance+Render setup to evaluate the impact of instance-level pseudo-labels.
We start with the core metric Checkout Accuracy (cAcc), a highly practical measure:
- A prediction is considered “completely successful” only if all product categories and quantities in the entire checkout image are exactly correct. It is highly sensitive to errors and effectively reveals the model’s reliability in real scenarios.
The original Render-only setup (using CycleGAN images) achieves just 45.60% cAcc. Despite style transfer, the semantic gap between synthetic and real scenes remains large.
Adding Copy-and-Paste synthetic images raises cAcc to 56.68%, reflecting improved data quantity and diversity, aiding generalization. However, such synthetic images may introduce noise like incorrect occlusions, unnatural poses, or inconsistent backgrounds, possibly leading to “more diverse but messier” data.
DPNet combines dual-view (detection and counting) learning with the selection of reliable pseudo-labels from test data for training. Even trained only on Render, DPNet pushes cAcc up to 77.91%, breaking the baseline ceiling.
DPNet trained on Syn+Render further improves to 80.51%. This demonstrates target-domain adaptation under the stated protocol; it is not the first use of self-training or unlabeled target data.
DPSNet advances DPNet with two key innovations:
- Architecture: Introduces pyramid-scale feature fusion, enabling detection and counting views to operate synchronously over multi-level features, beyond coarse stacking.
- Training: Adopts iterative knowledge distillation, progressively selecting reliable test images and further filtering and refining at the instance level.
“DPSNet (Render)” reaches 86.54% average cAcc, compared with 85.98% for Syn+Render. Adding instance-level samples brings Instance+Render to the table’s best 88.14%. Copy-Paste synthesis did not improve cAcc in this setting.
Beyond cAcc, DPSNet also excels on other metrics:
- ACD (Average Counting Distance): Measures the per-image L1 error across category counts; Instance+Render scores 0.17 in the average mode, below DPNet (Syn+Render) at 0.34.
- mCIoU (mean Category IoU): Compares predicted and true category counts; Instance+Render reaches 98.66% in the average mode, 1.33 percentage points above DPNet (Syn+Render) at 97.33%.
- mmAP (mean mean Average Precision, IoU=0.50–0.95): Instance+Render reaches 79.75% in the average mode, compared with 77.04% for DPNet (Syn+Render).
These complementary metrics portray DPSNet’s improved coordination and precision between detection and counting tasks.
An interesting observation is that DPSNet trained only with Render slightly outperforms Syn+Render (86.54% > 85.98%). In this data and training setting, adding Copy-Paste synthetic images did not improve cAcc; the table alone cannot separate the effects of data volume, distribution, and pseudo-labeling.
This invites reconsideration of:
“Where do truly valuable training signals come from—large-scale simulation or understanding and filtering of real distributions?”
For DPSNet in this setting, the reliable instance-level test samples bring a measurable gain.
Below are inference examples on the RPC dataset, ordered from easy to hard samples top to bottom.
Ablation Study
- Density Map Dimension Design
In the counting view, the third dimension of the density map, , represents the number of categories simultaneously estimated.
The authors compared the following three settings:
| Density Category (C) | Description | cAcc |
|---|---|---|
| C = 1 | Counting head estimates the total item count | 86.54% |
| C = 17 | Using 17 super-categories | 80.05% |
| C = 200 | Full fine-grained classes | 77.22% |
Across these three settings, higher yields lower cAcc, worse counting MAE, and fewer selected pseudo-labels. The authors attribute this to harder convergence with more finely divided density channels.
The C=1 setting provides category-agnostic counts and is selected for the final model. Category recognition remains the detector’s task.
- Impact of Feature Representation
From the table, using only the pyramid-scale architecture (without co-learning) performs worse than DPNet (70.08% < 70.80%). This indicates that architectural advances without corresponding training strategies may not translate into performance gains.
With co-learning, DPSNet’s performance surpasses the baseline, increasing cAcc to 79.35%. This shows the potential of the pyramid-scale design can only be unlocked through interactive dual-view learning.
If detection (object localization) and counting (density estimation) are trained separately, the model may fall into local optima. For example, an object may be accurately localized but have density mispredicted, or vice versa.
With the dual pyramid scale architecture held fixed, adding collaborative learning raises cAcc from 70.08% to 79.35%. The 70.80% result is a different configuration without the dual pyramid scale architecture, so it is not the starting point for this single-variable comparison.
Iterative learning then raises cAcc from 79.35% to 86.54%; adding instance-level samples brings it to 88.14%.
The core idea is that the model no longer depends on external labels but extracts reliable signals from its own consistent predictions and iteratively trains itself with these.
Conclusion
DPSNet addresses two core challenges in ACO—cross-domain distribution mismatch and limited labeled data—through pyramid-scale feature representation, multi-view co-learning, and optional knowledge distillation with iterative self-training.
This study reminds us that in label-scarce real-world scenarios, unlabeled target data are a potential adaptation resource under a declared transductive protocol. Stable pseudo-labeling strategies can progressively convert them into learning resources. Moreover, compared to style transfer, semantic and structural consistency learning is more critical for cross-domain adaptation; scenarios difficult to cover with a single task can be complemented through multi-task designs.
From an engineering perspective, although counting cannot finely distinguish semantics, it provides additional supervision during training that effectively improves overall performance and is worth practical consideration.