dominant hand canonicalization for improving transfer learning in sign language recognition
Simplified write-up of a manuscript in preparation.
Out of the estimated 300 sign languages in use currently, very few have enough labelled data to train a recogniser accurate enough for a practical implementation. For example, American Sign Language has a corpus with tens of thousands of clips, whereas Sinhala Sign Language only has a few thousand. A large majority of languages also have no clips at all. A relatively obvious choice here is transfer learning: training an encoder on a relatively well-resourced sign language, then adapt it to a low-resourced one with a handful of examples per word.
In brief, setup consisted of a CTR-GCN (ICCV 2021), used unmodified from the official repo, trained on AUTSL (a Turkish Sign Language corpus with 31,610 clips of 226 different signs, by 37 signers). The trained model is then adapted with k=1, 5 or 10 examples per word to six target languages: Bangla, Russian, Argentinian, Indian, American, and Sinhala. Of course, that explanation hides all of the interesting inner workings, so we will take a closer look at the architecture and setup shortly.
From video to graph
First of all, we need to establish the conversion from a video to an input for the CTR-GCN (which works on a graph input). Every clip is first run through MediaPipe Holistic, Google's pose estimation stack, which detects a person and regresses the image coordinates of anatomical 'landmarks'. This produces 33 body points, 21 points per hand (wrist + 4 joints per finger), and a face mesh. From that point, I keep 49 of those landmarks - nose, shoulders, elbows, wrists and both full hands, discarding the face entirely (for reasons that will soon be clear). Another concept to note whilst we are looking at the architecture, is that each hand occupies a fixed block of nodes (meaning that the network has individual parameters for each hand). If you are familiar with sign languages, and the fact that some signers are left-hand dominant vs right-hand dominant, you may already start to see an issue with this.
A skeleton is naturally a graph: joints are vertices, bones are edges. A graph convolutional neural network generalises the convolution to this graph setting. Where an image convolution aggregates pixels with their spatial neighbours, a graph convolution aggregates each joint's features with those of its anatomical neighbours. The ST-GCN (Yan et al., 2018) is the ancestor of every model in this project, and stacks such spatial layers alternately with ordinary 1-D temporal convolutions along the frame axis. The model 'mixes' over the body, then over time, repeatedly, and then uses a global-average-pool into one embedding per clip. Put more simply, the network uses spatial (space) and temporal (time) relationships to produce a single summary vector. A CTR-GCN (Channel-wise Topology Refinement, Chen et al., ICCV 2021) is the same system with one upgrade: it is allowed to 'notice' two components which are moving in a related way (e.g. two wrists moving in the same direction for a symmetrical sign) and to temporarily treat them as neighbours too. As such, the system has the same fixed skeleton, with an additional clip-specific correction.
Adaptation
Training the donor is ordinary supervised classification (35 epochs, SGD, batch 256). The interesting part is the adaptation. Given a target language, and a budget of k clips per word, I built an 'episode': k support clips per word from one set of signers, and a disjoint test set from other signers. Note that changing the signers between train and test sets is crucial, as the model otherwise has the ability to recognize the signer instead of the sign. Two types of adaptation were looked at:
# 1. FULL FINE-TUNE: re-train everything on the k x N support clips
net.load_state_dict(donor_weights, strict=False) # keep encoder, new head
for epoch in range(30):
loss = CE(net(support_clips), support_labels) # every weight moves
# 2. PROTOTYPE: no training at all - nearest class-centroid in embedding space
E_support = embed(support_clips) # frozen donor encoder
prototypes = [E_support[labels == c].mean(0) for c in classes]
prediction = cosine_nearest(embed(test_clip), prototypes)The full fine-tuning asks if the donor is a strong initialisation (meaning the weights that are set as default before training, as opposed to random weights). The prototypes ask if the donor's embedding space is already organized such that the same-word clips cluster. If a transfer effect shows up under both, we can be almost entirely sure that the effect is a property of the representation.
A fifty-year-old observation
The next part requires some small amount of contextual understanding of sign language linguistically. Sign language phonology (since Stokoe, 1960) decomposes signs into parameters (e.g. handshape, location, movement) the same way that spoken phonology decomposes syllables into phonemes. Battison (1978) added that one hand, the dominant hand, is the active articulator, while the other is typically heavily restricted. Further, no sign language distinguishes two words by which physical hand articulates them. By consequence, a left-handed signer's vocabulary is simply a mirror image of a right-handed signer's. The dominant/non-dominant hand split can be compared to vocal pitch - it carries identity, but no lexical contrast.
If the above theory holds, a sign language recogniser should be free to erase this dimension entirely. If you recall the fact that each hand has its own node block and the network has per node parameters, the same sign performed right-dominantly or left-dominantly is processed by different parameters. Thus, the network actually ends up learning every sign twice (left handed or right handed). AUTSL is split roughly 57/43 between the two, which means that over the 31,610 training clips, the evidence from each sign is nearly cut in half.
The fix is very simple, and only about eight lines of code.
BODYSW = [0, 2, 1, 4, 3, 6, 5] # nose fixed; swap L/R body pairs
PERM = BODYSW + list(range(28, 49)) + list(range(7, 28)) # ...and swap the two hand blocks
v = np.abs(np.diff(X, axis=frames)) # per-frame movement
eL = v[..., 7:28].sum() # total motion, left hand
eR = v[..., 28:49].sum() # total motion, right hand
if eL > 1.2 * eR: # left-dominant clip
X = X[..., PERM] # relabel: left block <-> right block
X[x_channel] *= -1 # reflect across the body midlineNote that the threshold is 1.2 rather than 1, so that symmetric two-handed signs, where the hands move near equally, are left alone, instead of being flipped on a coin toss. I later dismiss concerns about the original donor training differently due to this change - it differs by just under 0.2 of a standard deviation.
Results
The swap is applied at donor training time and at adaptation time, against donors identical in every other respect. The headline gain is +5.88 accuracy points under full fine-tuning - a donor-level difference of means (four standard donors against three mirrored), where each donor’s value is an average over the same 90 evaluation cells: all six target languages × k ∈ {1, 5, 10} × five evaluation seeds. The prototype arm of the same comparison shows +4.57. The effect is positive in all 6 languages tested. The below figure shows the relative accuracy gain.
Please note that statistical testing has been omitted from this post to reduce confusion on an already quite complicated system (but it will be included in the final paper). Regardless, one important piece of evidence that ties together this experiment's narrative quite nicely is the result of testing the in-language transform. The mirrored donors score 74.97% on held-out Turkish signers; the standard donors score 75.78%. That difference when compared to the seed noise is a comfortable null, which is exactly what the theory predicts: within one corpus, the network has enough data to learn every sign twice, and the effect only comes into play when transferred, as the representation becomes more robust. Published in-language ablations of exactly this transform report it as worthless. In-language, that holds.
In the interest of keeping this post concise and not too technically dense, I have left out the large amount of research into choosing the architecture, and my other tests on trying linguistically sophisticated interventions. One example of this was my testing with teaching the network phonology directly with auxiliary heads. I will try to make another post on those results, which despite being predominantly nulls, or negative, can teach us a lot about the extent to which we can use linguistic methods in this type of recognition task.
An accidental discovery?
While stress-testing the headline result, I calculated the fraction of clips in each corpus where MediaPipe simply fails to find a hand for most of the clip.
This was a result that I could not find in the majority of literature that I looked at. This is surprising, as it would seem relevant, even linguistically, as it is directly linked to whether signs are one or two-handed.
One-handed signs often leave the other hand resting, and a resting hand is lost by MediaPipe's tracking. Thus, the missingness encodes how many hands a sign uses, which is a lexical property. Any statistic comparing two hands silently inherits that structure (including the transform that is mentioned as the fix).
To make the claim more precise, I used the intraclass correlation coefficient (ICC). Take every clip's measured 'which hand leads' value and ask: how much of its variance is explained by which word is being signed, versus which signer is signing? As expected, which hand leads is 87% based on the signer. This confirms Battison's aforementioned claims. To audit my own work, I ran the same experiments on only clips where the entirety of both hands were tracked for the whole time. In short, the gain survives (+5.33 -> +4.15) and is still positive everywhere (this audit ran on five of the six languages with a single evaluation seed, so the within-pair drop is the comparable quantity).
Scaling runs
If the mechanism is 'the network learns every sign twice', more data per sign should reduce the need for the transform (and should reduce it to 0 with enough data). Subsampling the donor corpus:
The gain actually grows with donor size. If mirroring only recovered the halved evidence per sign, enough data should make it redundant - and in-language, it does. The transfer results point at something structural instead: without canonicalisation, the network stores every sign as a fusion of two mirror-image clusters, and handedness survives as a large nuisance direction in the embedding. A k-shot support set rarely covers both variants, so test clips of the opposite dominance land far from their prototype. Erase that direction and the whole representation is spent on the sign itself - an advantage that compounds, rather than washes out, as the donor gets richer. It also explains the in-language null: with a trained head and enough data, two clusters per sign cost nothing; it is few-shot transfer that pays for them. The caveat is that the canonicalisation is only as trustworthy as the dominance heuristic it bootstraps from - which is exactly why the missingness audit above matters.
A more direct probe of the mechanism is to occlude one hand block at a time and watch where the damage lands. On left-dominant clips, removing the left block costs the standard donor 5.02 logits against 2.91 for the right block; for the mirrored donor the peak moves to the right block, 5.05 against 2.35. Same clips, same occlusion, and the dependence follows the canonical block simply because the donor was trained on canonicalised input. The standard donor never shows the reverse pattern on right-dominant clips, which I read as the model absorbing a prior from the corpus itself: 45.6% of its training clips are left-dominant against 7.8% right-dominant. That group is also only 45 clips, so I treat it as unresolved rather than as evidence either way.
Key takeaways
- Be both aware and careful of what preprocessing steps like trackers miss. In this case we saw that pose data is not in fact a perfect ground truth, but instead the output of a fallible instrument. We also saw that failure modes are not just random issues with the instrument, but instead are linguistically correlated.
- Measure noise floor before your effect. This reflection is particularly important, and relevant when working with improving models that have a low baseline accuracy, or high variance in accuracy when small changes are made.
- One of the most boring and dismissed transforms beat out all of the interesting linguistic modifications. Battison's fifty-year-old observation and an architectural fact about node blocks allowed us to make a real improvement to the algorithm.
Referenced papers for further reading
Stokoe, W. C. (1960). Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. Studies in Linguistics, Occasional Papers 8.
Battison, R. (1978). Lexical Borrowing in American Sign Language. Linstok Press.
Snell, J., Swersky, K. & Zemel, R. (2017). Prototypical Networks for Few-shot Learning. NeurIPS 2017. arXiv:1703.05175.
Yan, S., Xiong, Y. & Lin, D. (2018). Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. AAAI 2018. arXiv:1801.07455.
Lugaresi, C. et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172.
Sincan, O. M. & Keles, H. Y. (2020). AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods. IEEE Access 8. arXiv:2008.00932.
Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y. & Hu, W. (2021). Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition. ICCV 2021. arXiv:2107.12213.