Publications
Our published Work!
2024
Shi, Wei-Pei; Nordling, Torbjörn E M
Combining old school autoencoder with Cotracker for improved skin feature tracking Proceedings Article
In: The 19th IEEE Conference on Industrial Electronics and Applications (ICIEA 2024), IEEE, Kristiansand, Norway, 2024.
Abstract | BibTeX | Tags: autoencoder, Convolutional neural network, Cotracker, Deep feature encoder, human motion assessment, Skin feature tracking, Transformer
@inproceedings{Shi2024ICIEA,
title = {Combining old school autoencoder with Cotracker for improved skin feature tracking},
author = {Wei-Pei Shi and Torbjörn E M Nordling},
year = {2024},
date = {2024-08-01},
urldate = {2024-08-01},
booktitle = {The 19th IEEE Conference on Industrial Electronics and Applications (ICIEA 2024)},
publisher = {IEEE},
address = {Kristiansand, Norway},
series = {IEEE Conference on Industrial Electronics and Applications (ICIEA 2024)},
abstract = {Abstract—Background: Skin feature tracking enables quantification of human
motion in an explainable way, making it suitable for clinical assessments.
Accuracy is crucial, but no study has investigated state-of-the-art deep neural
network-based point tracking models such as Cotracker. Cotracker jointly tracks
points and has been shown to have better 3-pixel accuracy than five other
state-of-the-art deep learning methods on the two most commonly used datasets
for evaluation of single target point tracking. In 2021, Chang and Nordling
introduced the Deep Feature Encoder (DFE) and demonstrated skin feature tracking
so accurate that the errors cannot be excluded to stem from the manual labeling
of the videos based on a χ2-test.
Problem: How accurately can different methods track skin features and how to avoid
the intrinsic weaknesses of the methods?
Methods: We use videos of the Unified Parkinson’s Disease Rating Scale postural
tremor test recorded at two hospitals for benchmarking. DFE utilizes the encoder
part of an autoencoder consisting of a five-layer convolutional neural network
trained to reproduce skin crops without supervision. The residual squared error
of the latent features of the encoder is then compared with crops to obtain a
predicted position. We also propose Cotracker DFE, using Cotracker to obtain an
approximate position and subsequently cropping a small area that is fed to DFE
to obtain a position predicted with a lower mean pixel error.
Results: The mean Euclidean distance errors of Cotracker, DFE, and Cotracker-DFE
are 1.2, 0.8, and 0.8 pixels, respectively. DFE requires time-consuming computations,
making it 35 times slower than Cotracker.
Conclusion: The old school DFE provided more accurate skin feature tracking, while
combining DFE with Cotracker provides the best overall performance, circumventing
the lack of labeled data and computational resources required to fine-tune Cotracker.},
howpublished = {The 19th IEEE Conference on Industrial Electronics and Applications (ICIEA 2024), in Kristiansand, Norway 05-08 August 2024},
keywords = {autoencoder, Convolutional neural network, Cotracker, Deep feature encoder, human motion assessment, Skin feature tracking, Transformer},
pubstate = {published},
tppubtype = {inproceedings}
}
Abstract—Background: Skin feature tracking enables quantification of human
motion in an explainable way, making it suitable for clinical assessments.
Accuracy is crucial, but no study has investigated state-of-the-art deep neural
network-based point tracking models such as Cotracker. Cotracker jointly tracks
points and has been shown to have better 3-pixel accuracy than five other
state-of-the-art deep learning methods on the two most commonly used datasets
for evaluation of single target point tracking. In 2021, Chang and Nordling
introduced the Deep Feature Encoder (DFE) and demonstrated skin feature tracking
so accurate that the errors cannot be excluded to stem from the manual labeling
of the videos based on a χ2-test.
Problem: How accurately can different methods track skin features and how to avoid
the intrinsic weaknesses of the methods?
Methods: We use videos of the Unified Parkinson’s Disease Rating Scale postural
tremor test recorded at two hospitals for benchmarking. DFE utilizes the encoder
part of an autoencoder consisting of a five-layer convolutional neural network
trained to reproduce skin crops without supervision. The residual squared error
of the latent features of the encoder is then compared with crops to obtain a
predicted position. We also propose Cotracker DFE, using Cotracker to obtain an
approximate position and subsequently cropping a small area that is fed to DFE
to obtain a position predicted with a lower mean pixel error.
Results: The mean Euclidean distance errors of Cotracker, DFE, and Cotracker-DFE
are 1.2, 0.8, and 0.8 pixels, respectively. DFE requires time-consuming computations,
making it 35 times slower than Cotracker.
Conclusion: The old school DFE provided more accurate skin feature tracking, while
combining DFE with Cotracker provides the best overall performance, circumventing
the lack of labeled data and computational resources required to fine-tune Cotracker.
motion in an explainable way, making it suitable for clinical assessments.
Accuracy is crucial, but no study has investigated state-of-the-art deep neural
network-based point tracking models such as Cotracker. Cotracker jointly tracks
points and has been shown to have better 3-pixel accuracy than five other
state-of-the-art deep learning methods on the two most commonly used datasets
for evaluation of single target point tracking. In 2021, Chang and Nordling
introduced the Deep Feature Encoder (DFE) and demonstrated skin feature tracking
so accurate that the errors cannot be excluded to stem from the manual labeling
of the videos based on a χ2-test.
Problem: How accurately can different methods track skin features and how to avoid
the intrinsic weaknesses of the methods?
Methods: We use videos of the Unified Parkinson’s Disease Rating Scale postural
tremor test recorded at two hospitals for benchmarking. DFE utilizes the encoder
part of an autoencoder consisting of a five-layer convolutional neural network
trained to reproduce skin crops without supervision. The residual squared error
of the latent features of the encoder is then compared with crops to obtain a
predicted position. We also propose Cotracker DFE, using Cotracker to obtain an
approximate position and subsequently cropping a small area that is fed to DFE
to obtain a position predicted with a lower mean pixel error.
Results: The mean Euclidean distance errors of Cotracker, DFE, and Cotracker-DFE
are 1.2, 0.8, and 0.8 pixels, respectively. DFE requires time-consuming computations,
making it 35 times slower than Cotracker.
Conclusion: The old school DFE provided more accurate skin feature tracking, while
combining DFE with Cotracker provides the best overall performance, circumventing
the lack of labeled data and computational resources required to fine-tune Cotracker.
Chang, Jose Ramon; Nordling, Torbjörn E. M.
Unsupervised Skin Feature Tracking with Deep Neural Networks Journal Article
In: arXiv preprint, 2024.
Abstract | Links | BibTeX | Tags: autoencoder, Cotracker, feature matching, feature tracking, image registration, Lucas-Kanade method, PIPs, SIFT, SURF
@article{chang2021skin,
title = {Unsupervised Skin Feature Tracking with Deep Neural Networks},
author = {Jose Ramon Chang and Torbjörn E. M. Nordling},
url = {https://arxiv.org/abs/2405.04943},
year = {2024},
date = {2024-05-08},
journal = {arXiv preprint},
publisher = {Cornell University},
abstract = {Facial feature tracking is essential in imaging ballistocardiography for accurate heart rate estimation and enables motor degradation quantification in Parkinson's disease through skin feature tracking. While deep convolutional neural networks have shown remarkable accuracy in tracking tasks, they typically require extensive labeled data for supervised training. Our proposed pipeline employs a convolutional stacked autoencoder to match image crops with a reference crop containing the target feature, learning deep feature encodings specific to the object category in an unsupervised manner, thus reducing data requirements. To overcome edge effects making the performance dependent on crop size, we introduced a Gaussian weight on the residual errors of the pixels when calculating the loss function. Training the autoencoder on facial images and validating its performance on manually labeled face and hand videos, our Deep Feature Encodings (DFE) method demonstrated superior tracking accuracy with a mean error ranging from 0.6 to 3.3 pixels, outperforming traditional methods like SIFT, SURF, Lucas Kanade, and the latest transformers like PIPs++ and CoTracker. Overall, our unsupervised learning approach excels in tracking various skin features under significant motion conditions, providing superior feature descriptors for tracking, matching, and image registration compared to both traditional and state-of-the-art supervised learning methods.},
keywords = {autoencoder, Cotracker, feature matching, feature tracking, image registration, Lucas-Kanade method, PIPs, SIFT, SURF},
pubstate = {published},
tppubtype = {article}
}
Facial feature tracking is essential in imaging ballistocardiography for accurate heart rate estimation and enables motor degradation quantification in Parkinson's disease through skin feature tracking. While deep convolutional neural networks have shown remarkable accuracy in tracking tasks, they typically require extensive labeled data for supervised training. Our proposed pipeline employs a convolutional stacked autoencoder to match image crops with a reference crop containing the target feature, learning deep feature encodings specific to the object category in an unsupervised manner, thus reducing data requirements. To overcome edge effects making the performance dependent on crop size, we introduced a Gaussian weight on the residual errors of the pixels when calculating the loss function. Training the autoencoder on facial images and validating its performance on manually labeled face and hand videos, our Deep Feature Encodings (DFE) method demonstrated superior tracking accuracy with a mean error ranging from 0.6 to 3.3 pixels, outperforming traditional methods like SIFT, SURF, Lucas Kanade, and the latest transformers like PIPs++ and CoTracker. Overall, our unsupervised learning approach excels in tracking various skin features under significant motion conditions, providing superior feature descriptors for tracking, matching, and image registration compared to both traditional and state-of-the-art supervised learning methods.
Subscribe To Our Newsletter
Nordling Lab News will provide you with irregular updates on our research and topics of interest.