Estimating manifold dimension by inversion error

Abstract

High-dimensional data encountered in scientific and engineering applications often lies on or near a low-dimensional manifold embedded in the ambient space. Estimating the intrinsic dimensionality of such manifolds is a fundamental problem in data analysis, machine learning, and scientific computing. We introduce a novel method for estimating manifold dimension based on the inversion error of local coordinate systems. The method constructs local parametric models and measures the error incurred when inverting local maps as a function of assumed dimensionality. The true manifold dimension corresponds to the dimensionality at which inversion error is minimized. We demonstrate the method on synthetic manifolds of known dimension and on neural data from the olfactory system, where the method recovers known dimensionality constraints.

Introduction

Dimensionality reduction and manifold learning have become central to modern data analysis. Methods such as PCA, Isomap, and LLE rely on an accurate estimate of intrinsic dimensionality, yet estimating this quantity robustly remains challenging, particularly for noisy or sparsely sampled data.

We propose a geometrically motivated approach: if data truly lies on a d-dimensional manifold, then a local d-dimensional parametrization should invert (reconstruct) points with low error, while parametrizations of higher or lower dimension should fail. This inversion error criterion provides a natural and robust estimator of intrinsic dimension.

Key Findings

• Inversion error exhibits a clear minimum at the true manifold dimension on synthetic datasets.
• The method is robust to noise and outperforms correlation-dimension estimates on sparse data.
• Applied to locust olfactory neural data, the method identifies low-dimensional structure consistent with known biology.

Cite This Article