Back to paper library
arXiv 2022CVPR 2023Text recognitionPaper notes

SIGA

Glyph Attention Mechanism

Original paper

Self-supervised Implicit Glyph Attention for Text Recognition

Paper authors

Tongkun Guan · Chaochen Gu · Jingzheng Tu …

Show all 8 authors

Tongkun Guan · Chaochen Gu · Jingzheng Tu · Xue Yang · Qi Feng · Yudi Zhao · Xiaokang Yang · Wei Shen

Publication
CVPR 2023Conference
First submitted to arXiv
arXiv:2203.03382
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionsiga2022,
  title = {{Self-supervised Implicit Glyph Attention for Text Recognition}},
  author = {Tongkun Guan and Chaochen Gu and Jingzheng Tu and Xue Yang and Qi Feng and Yudi Zhao and Xiaokang Yang and Wei Shen},
  year = {2022},
  eprint = {2203.03382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.03382}
}
On this page

While the integration of language models has become a mainstream approach in this field, some researchers continue to explore alternative methods to improve Scene Text Recognition (STR).

In this paper, the authors focus on studying “glyphs” and achieve promising results.

Problem Definition

We have reviewed multiple papers on STR in the past.

Historically, attention mechanisms in this field have been categorized into two primary types:

  1. Implicit Attention: This approach calculates attention weights across all positions in one-dimensional or two-dimensional space, based on sequence-level text annotations. However, it has a drawback: potential misalignment issues.
  2. Supervised Attention: This method uses character bounding box annotations to generate character segmentation maps. Its drawback is the need for substantial character-level annotated data, which can be memory-intensive when dealing with a large number of character classes.

Neither method seems to be ideal, does it?

The authors decided to approach the problem from the perspective of “glyphs.” Each character has a unique glyph, and there are inherent patterns and relationships between these glyphs. Leveraging these patterns could potentially enhance text feature extraction.

Based on this idea, the authors proposed the "Self-Supervised Implicit Glyph Attention (SIGA)" mechanism. This approach generates glyph structures through self-supervised text segmentation and implicit attention alignment, refining the accuracy of attention during training.

In essence, among the two main approaches mentioned above, the authors chose the first—implicit attention—and further enhanced it with a self-supervised mechanism.

So, how exactly does this work? Let’s continue to explore.

Solution

Model Architecture

The diagram above illustrates the model architecture of SIGA.

The overall architecture appears somewhat complex, so let’s break it down into its main components for easier understanding.

Backbone

This is perhaps the simplest part. ResNet, SVTR, or Vision Transformer (ViT) can be used as the feature extractor here.

Once the input image passes through the feature extractor, it generates a feature map, which is then fed into the subsequent branches of the model.

Glyph Pseudo-Label Construction

This part, referred to as Glyph Pseudo-label Construction (GPC), is the core of SIGA.

The first step here involves self-supervised text segmentation, highlighted by the red box in the diagram.

The authors employ K-means clustering on the internal structure of the text image to directly obtain pseudo-labels for text masking, bypassing the need for time-consuming pixel-level annotation.

The clustering algorithm is configured to split the image into two classes: foreground and background. According to the authors' experiments, this method effectively clusters the morphological structures of glyphs in most text images.

Following this, the authors use a text segmentation network to extract foreground features from the self-supervised segmentation results.

Although not explicitly depicted in the diagram, the authors detail the implementation in the paper. Specifically, they define the outputs of Conv 0, Block 0, and Block 1 in the ResNet model as P0P_0, P1P_1, and P2P_2, respectively. A top-down pyramid structure is applied to integrate these features:

{O2=ϕ(P2),O1=ϕ([T(O2,s1),P1]),O0=ϕ([T(O1,s0),P0]),\begin{cases} O_2 = \phi(P_2), \\ O_1 = \phi([T(O_2, s_1), P_1]), \\ O_0 = \phi([T(O_1, s_0), P_0]), \end{cases}

where ϕ(⋅)\phi(\cdot) represents a two-layer convolution with Batch Normalization and ReLU activation, T(⋅)T(\cdot) denotes a 2x upsampling operation on OkO_k, sks_k denotes its resolution (i.e., Hk×WkH_k \times W_k), and [⋅][\cdot] indicates channel-wise concatenation.

Finally, a binary convolutional layer is used to generate the text segmentation mask SmS_m from O0O_0.

The authors optimize the text segmentation network by matching SmS_m to the pseudo-label SplS_{pl} using a binary cross-entropy loss LinsL_{ins}, enabling it to recognize glyph morphological structures even in challenging text images.

The outcome resembles the following:

Here, (a) is the original image, (b) is the pseudo-label generated via clustering, and (c) is the prediction from the text segmentation network.

Notably, the segmentation network’s predictions occasionally surpass the pseudo-label in quality!

Implicit Attention Alignment

We’re still looking at the same diagram, but focusing on a different section: attention alignment, indicated by the red box.

In the decoding unit, implicit attention weights α={αt}t=1,...,T\alpha = \{\alpha_t\}_{t=1,...,T} are used to capture significant character dependencies in the encoded sequence.

However, the temporal information in the decoder can be "disturbed by other information" in subsequent decoding steps, causing an alignment drift issue.

An orthogonality penalty discourages different decoding steps from attending to the same area. It encourages distinct attention regions.

The process for this method is as follows:

  1. Calculate the attention inner-product sum ScorS_{cor}:

    The authors sum the pairwise inner products of attention vectors from different decoding steps and penalize that sum. This encourages different steps to use different visual regions; the later constraint against the text mask is still needed to align them with the characters.

    For three characters, the penalty encourages the attention vectors to concentrate on separate regions.

  2. Generate Character Saliency Maps SsalS_{sal}:

    Next, each attention vector αt\alpha_t contributes to a character saliency map. Using a one-dimensional linear interpolation function ξ\xi, each αt\alpha_t distribution is mapped to fit the image width WW, producing βt\beta_t. A nonlinear activation function σ\sigma then maps βt\beta_t to [0,1][0,1]. The authors multiply each σ(βt)\sigma(\beta_t) elementwise by the text segmentation mask SmS_m and sum the results across decoding steps to form SsalS_{sal}.

    For example, if we have an image with a width of 10 pixels, the attention vector αt\alpha_t is transformed into βt\beta_t by interpolation, generating positional weights that indicate where the character lies, such as between pixels 3 and 5, meaning the model considers the character to be located within this range.

  3. Optimize the Loss Function LseqL_{seq}:

    • LcorL_{cor}: This loss is the attention inner-product sum ScorS_{cor}. Minimizing it discourages attention vectors from different steps from overlapping and helps reduce alignment drift.
    • LdifL_{dif}: This loss measures the difference between SmS_m (the segmentation network’s predicted text mask) and SsalS_{sal} (the generated character saliency map). Minimizing LdifL_{dif} enables the model to bring the aggregate attention-derived mask closer to the predicted foreground mask.

    For example, if SmS_m marks character foreground between pixels 5 and 8 along the image width, SsalS_{sal} should cover that region too. This is consistency between two model-derived signals, rather than supervision from manually labeled character masks. Errors in the segmentation prediction can therefore affect the resulting supervision.

Glyph Pseudo-Label Construction

As shown in the image above, the aligned attention weights βt\beta_t are combined elementwise with the text segmentation mask SmS_m to generate each character's glyph pseudo-label. The operation selects foreground regions (the character shapes) according to each character's attention position.

Assuming SgtS_{gt} as the glyph pseudo-label, it is constructed through the following concatenation operation:

Sgt=[1−Sm,1[β1≥δ]⋅Sm,...,1[βT≥δ]⋅Sm]S_{gt} = [1 - S_m, 1[\beta_1 \geq \delta] \cdot S_m, ..., 1[\beta_T \geq \delta] \cdot S_m]

where:

  • 1−Sm1 - S_m denotes the background regions in SmS_m.
  • 1[βt≥δ]⋅Sm1[\beta_t \geq \delta] \cdot S_m selects the character's foreground region, retaining positions whose attention value is at least the threshold δ\delta.
  • [⋅][ \cdot ] represents channel-wise concatenation, so SgtS_{gt} contains multiple channels, each representing different character positions within the text.

In the experiments, the threshold δ\delta was set to 0.05. A foreground pixel is retained in that character's SgtS_{gt} channel when its βt\beta_t value is at least 0.05. This helps filter out low-confidence positions.

In the testing phase, the glyph pseudo-label construction module is removed, meaning that during testing, the model relies directly on trained weights for prediction without generating glyph pseudo-labels.

Glyph Attention Network (GLAN)

The original term for this is Glyph Attention Network (GLAN).

Remember the mathematical expression we discussed earlier?

To recap, here it is:

The output O0O_0 was used in the previous section to predict the text foreground mask, which then helped construct the glyph pseudo-label.

Now, O1O_1 and O2O_2 will be used for the Glyph Attention Network (GLAN).

The glyph attention maps generated by GLAN have a fixed length and are channel-independent of character classes. Each specific channel corresponds to a particular order of glyph attention.

The approach here is to predict the glyph attention map SgamS_{gam} by applying several convolutional layers to features OkO_k. The channel count for SgamS_{gam} is set to Ns=1+MN_s = 1 + M, where 1 represents the background class, and MM represents the maximum number of characters in a text image (e.g., set to 26).

Next, the constructed glyph pseudo-label SgtS_{gt} is used as a supervisory signal. A combined loss function of multi-class Dice loss and cross-entropy loss is applied to improve the segmentation performance of the Glyph Attention Network.

The formulas are as follows:

{Ldice=1L∑j=2L+1(1−2∑i=1n(ωj,iωj,i∗)∑i=1nωj,i+∑i=1nωj,i∗),Lcel=−1n∑i=1n(ρilog⁡(∑j=2M+1ωj,i∗)+(1−ρi)log⁡(1−∑j=2M+1ωj,i∗)),Lseg=Ldice+Lcel,\begin{cases} L_{dice} = \frac{1}{L} \sum_{j=2}^{L+1} \left( 1 - \frac{2 \sum_{i=1}^n (\omega_{j,i} \omega^*_{j,i})}{\sum_{i=1}^n \omega_{j,i} + \sum_{i=1}^n \omega^*_{j,i}} \right), \\ L_{cel} = -\frac{1}{n} \sum_{i=1}^n \left( \rho_i \log\left(\sum_{j=2}^{M+1} \omega^*_{j,i}\right) + (1 - \rho_i) \log\left(1 - \sum_{j=2}^{M+1} \omega^*_{j,i}\right) \right), \\ L_{seg} = L_{dice} + L_{cel}, \end{cases}

where ωj,i\omega_{j,i} and ωj,i∗\omega^*_{j,i} are the confidence scores of pixel pip_i for the jj-th channel in the glyph pseudo-label SgtS_{gt} and glyph attention map SgamS_{gam}, respectively, and ρi\rho_i is the confidence score of pixel pip_i in SmS_m. LL represents the number of characters in the text image.

Attention-based Character Fusion Module

This module, termed the Attention-based Character Fusion Module, is shown within the red box in the above figure.

As mentioned, in the decoding step tt, the visually aligned "glimpse" gtg_t and glyph feature Ik,tI_{k,t} represent two distinct character feature representations.

Since their contributions to Scene Text Recognition (STR) might vary across different text images, the authors dynamically fuse the sequential feature Ik,tI_{k,t} with gtg_t to enrich the semantic information for character recognition. This dynamic adjustment of the weighting between Ik,tI_{k,t} and gtg_t based on the characteristics of the current image allows the model to extract more useful information from both feature sources.

This design ensures that the model can flexibly combine visual and glyph features during character recognition, leading to more precise recognition results.

Discussion

Comparison with Other Methods

Language-independent text recognition methods mainly rely on visual information to recognize text.

The table above compares SIGAR with existing language-independent models, grouped by backbone type. Their input sizes and other settings still differ.

  • CNN-based Methods

    SIGAR leads on several splits of the seven context benchmarks; other methods remain ahead on SVT, IC15-2077, and SP.

    Compared to supervised attention methods (e.g., CA-FCN and TextScanner), SIGAR does not require additional character-level annotations. Where both sides report results in the table, its accuracy is 2.0 to 11.8 percentage points higher.

    When compared to implicit attention methods, SIGAR also shows superior performance, surpassing the second-best results on benchmark datasets such as IIIT (0.3 percentage points), IC03-860 (0.7 percentage points), IC03-867 (0.8 percentage points), IC13-857 (0.6 percentage points), IC13-1015 (0.5 percentage points), and IC15-1811 (1.1 percentage points).

    The authors also used SVTR as the backbone. SIGAS achieved higher accuracy in four of the six reported columns, with a sample-weighted mean improvement of 0.63 percentage points; it was lower on IC13-857 and CT.

  • Transformer-based Methods

    SIGAT exhibits a significant advantage, setting new best results on benchmark datasets including IIIT, SVT, IC03-860, IC03-867, IC13-857, IC13-1015, and CT.

    These results indicate that introducing glyph attention enables SIGA to successfully extract more discriminative visual features, resulting in excellent performance on context-based benchmark datasets.


Language models can refine visual output through semantic inference, reducing prediction errors and enhancing overall recognition accuracy on context benchmarks.

As shown in the table above, when further compared to language-related models, SIGAR achieved competitive results on most standard benchmark datasets. Even without semantic inference, SIGAT achieved the best accuracy on six out of eight benchmarks.

SIGAT’s performance on the SVT, IC13-857, SP, and CT benchmarks improved by 1.5, 0.4, 1.2, and 0.9 percentage points, respectively. This indicates that even a purely visual model can perform well on context benchmarks.

Visualization Analysis

In the figure above, images in (a) and (b) are generated by the representative implicit attention method ABINet and by SIGA, respectively.

SIGA’s approach can perceive intricate glyph structure details, especially when the glyphs in the text image are very blurry (as in the fifth set). SIGA can even degrade gracefully to a similar attention form as other STR models.

Conclusion

This paper studies how glyph structure can improve STR, offering a new perspective for attention-based text recognition models.

By combining self-supervised text segmentation and implicit attention alignment, SIGA automatically generates glyph pseudo-labels as attention supervision for the model. This approach overcomes the traditional reliance on character-level annotations, significantly improving the precision of glyph structure alignment and focus. This advancement could encourage future research to consider more visual details of text to enhance model recognition capabilities.