Overview

Background and Goal

The Kyoto Collection of Human Embryo stored at the Congenital Anomaly Research Center, Graduate School of Medicine, Kyoto University is one of the precious and most unique collections in the world.

While serial tissue sections allow detailed observation of organogenesis under a microscope, it is difficult to determine the original three-dimensional shape of a single specimen because it is spread into hundreds of sections. In addition, physical distortion occurs in the sectioning processes, and the original three-dimensional shape cannot be restored by simply stacking digitized section images.

Therefore, this study addresses the issue of restoring the original 3D shape by compensating for the physical distortion.

The process of human development is still unknown

• Major organs have developed two months after fertilization (usually complete by the time pregnancy is detected)
• Development processes occur in the body and are difficult to observe

Serial section specimen of human embryo

Produced about 50 years ago; new collection is difficult

✓ Easy to discriminate organs and tissues in detail using microscopy

✗ Difficulty in understanding 3D morphology

3D reconstruction from serial sections

- Hundreds of sections or more from a single specimen

- Physical distortion occurs during sectioning

Approach

To cope with large deformations, we took a feature-based approach that estimates the keypoint matches between image pair. The problem is to estimate the deformation field caused by sectioning from the keypoint matches.
In general, only a small number of keypoint matches are obtained, and accurate estimation of the deformation field is not trivial. In this study, we focus on “geometric transformations” that can be estimated from several keypoint matches obtained in a local region.

Instead of the vector field representation, we considered the field of geometrical transformations that change smoothly in space as a latent mathematical structure in real space, and work on its modeling. For example, by using similarity transformations and rigid transformations, geometric properties such as isotropy and isometry are expected to be preserved.

Method

This project aims to develop sparse modeling techniques to model geometric transformation fields in a data-driven manner from a small number of samples.

Only multiplicative operations are allowed for geometric transformations to preserve their geometric properties. Since conventional sparse modeling methods are formulated based on additive operations, if applied as is, the geometric properties of geometric transformations are lost. For example, linear combination of geometric transformations should be formulated according to Lie algebra.
Therefore, we extend the conventional sparse modeling according to the Lie algebra and reformulate it to fit the Lie group, which geometric transformations belongs to. We also developed a method to optimize the entire registration of hundreds of serial sections to minimize the deformation in each image, and to reconstruct the original 3D shape.

We have developed a new method that enables us to understand the morphological changes from limited information, and demonstrated its effectiveness for a wide range of objects from medical image processing to meteorological data.

Established Theory

A new analysis method that can express complex deformations from limited samples by changing the focus from "point to point" to "region to region".

When trying to estimate the deformation of all pixels from a small number of keypoint matches, the coordinates of the red dots are the same in Figures A - C and Figures A’ - C’. The conventional vector field, which captures changes from “point to point” cannot distinguish them.

However, focusing on the region around the red dots, it can be seen that the rotation is different in Figures B and C. By using a method that captures changes from “region to region”, it is possible to express the transformation of all A to C differently.

Conventional method: Vector field

Capture position changes as "point to point" = addition

Displacement＝translation
\( x + dx \)：Operation system is addition

Samples cannot distinguish A-C

Our method: geometric transformation field

Capture position changes as "region to region"=multiplication

Geometric transformation＝rigid/similarity transformation
Translation (=displacement)＋rotation motion
\( qx \)：Operation system is multiplication

A, B and C are different samples

Developed methods for Geometric Transformation Fields

• Sparse Regression Analysis
• Graph Optimization
• Spectral Analysis

Results

Result 1　Non-rigid Registration of Image Pair

Sparse Regression Analysis \(\times\) 2D rigid transformations

Derive the deformation field from sparse keypoint matches in an image pair

Result 2　3D Reconstruction from Serial Sections

Graph Optimization \(\times\) 2D rigid transformations

We perform global optimization of the non-rigid registration for hundreds of serial sections to reconstruct a 3D model while minimizing distortions in each slice

Movie: Input Movie: Result

Applications

Application 1　Advection Modeling on Globe

Sparse Regression Analysis \(\times\) 3D rotations

Estimating the global advection from the cloud motion observed from satellite images

Movie

Application 2　Registration of CT images

Sparse Regression Analysis \(\times\) 3D similarity transformations

Estimating the lung deformation between inspiration and expiration

Movie

Future Prospects

• Integration with deep learning techniques, other statistical analysis, and signal processing techniques
• Investigating the guideline for the selection of geometric transformation class (rigid/similarity transformations, etc.) according to the application target
• Exploring other applications