Workshop: Generative Models in Science and Machine Learning

I will consider the generative problem of sampling from an unknown distribution for which only a sufficiently large number of training samples are available. The general approach is that of plug & play Langevin dynamics where the required data-driven drift term is approximated using Schrödinger bridges. A key bottleneck of this approach is the exponential dependence of the required training samples on the dimension, d, of the ambient state space. I will discuss a localization strategy which exploits conditional independence of conditional expectation values. Localization thus replaces a single high-dimensional Schrödinger bridge problem by d low-dimensional Schrödinger bridge problems over the available training samples. In this context, a connection to multi-head self attention transformer architectures is established. As for the original Schrödinger bridge sampling approach, the localized sampler is stable and geometric ergodic. The sampler also naturally extends to conditional sampling and to Bayesian inference. I will demonstrate the performance of the proposed scheme through experiments on a Gaussian problem with increasing dimensions and several problems involving inferring stochastic processes from given time-series.

In this talk, we explore generative diffusion models through the lens of time-reversed stochastic processes. Starting from the stochastic theory of time-reversed Markov processes — a classical concept in probability — we show how random time horizons and Doob's h-transform allow a process to terminate exactly at a desired sampling distribution, at a random time chosen to minimize the expected runtime. This perspective leads to a new class of generative diffusion models that maintain a time-homogeneous structure for both the noising and denoising phases. The resulting dynamics automatically adapt the number of sampling steps to the current noise level, enabling more efficient sampling. For data with low intrinsic dimensionality, the termination condition reduces to a simple first-hitting rule, offering new insights into the manifold hypothesis. The method is also well-suited for natural conditioning, providing the foundation for the subsequent talk by Jan Kallsen. Joint work with Jan Kallsen, Claudia Strauch, and Lukas Trottner (arXiv:2501.19373).

We consider the problem of recovering a latent signal from its noisy observation. The unknown law of the signal and in particular its support are accessible only through a large sample of i.i.d. training data. We further assume the support to be a low-dimensional submanifold of a high-dimensional Euclidean space. As a filter or denoiser we suggest an estimator of the metric projection of the observation on the submanifold. For its computation we study an auxiliary semiparametric model where the observation is obtained by adding isotropic Laplace noise to the signal. Using score matching in a corresponding diffusion model, we obtain an estimator of the Bayesian posterior in this setup. Our main theoretical result shows that, in the limit of high dimensions, this posterior is concentrated near the desired metric projection of the observation on the signal submanifold. Based on joint work with Sören Christensen, Claudia Strauch, and Lukas Trottner.

Denoising diffusion probabilistic models (DDPMs) represent a recent advance in generative modelling that has delivered state-of-the-art results across many domains of applications. Despite their success, a rigorous theoretical understanding of the error within DDPMs, particularly the non-asymptotic bounds required for the comparison of their efficiency, remain scarce. Making minimal assumptions on the initial data distribution, allowing for example the manifold hypothesis, in this talk I will present explicit non-asymptotic bounds on the forward diffusion error in total variation (TV), expressed as a function of the terminal time T. The key idea is to parametrise multi-modal data distributions in terms of the distance $R$ to their furthest modes and consider forward diffusions with additive and multiplicative noise. Our analysis rigorously proves that, under mild assumptions, the canonical choice of the Ornstein-Uhlenbeck (OU) process cannot be significantly improved in terms of reducing the terminal time $T$ as a function of $R$ and error tolerance $\epsilon>0$. Motivated by data distributions arising in generative modelling, we also establish a cut-off like phenomenon (as $R\to\infty$) for the convergence to its invariant measure in TV of an OU process, initialized at a multi-modal distribution with maximal mode distance $R$. This joint work with Miha Brešar (CUHK-Shenzhen) is to appear in the Annals of Applied Probability.

[TBA]

Probabilistic models that approximate a distribution using a dynamical system that transports particles from a source distribution to the target have seen rapid development and adoption in recent years. Indeed, diffusion models and continuous normalising flows show success in generative modelling for various domains, but they can also be trained to sample intractable Bayesian posterior distributions, where no samples are available but an unnormalised target density can be queried. In this talk, I will first survey past work (by me and others) on fitting stochastic dynamical systems that sample from a distribution when no target samples are available using deep reinforcement learning methods. However, the underlying dynamical system — not only its end-time marginal distribution — is an interesting object in its own right. I will thus proceed to present recent work that extends the mentioned approaches to the Schrödinger bridge problem — that is, inference of stochastic dynamics minimising a certain transport cost — between a pair of distributions without access to data. Time permitting, I will conclude with ongoing work on learning-based methods for dynamic optimal transport in a multimarginal setting, which allows inference of (deterministic) dynamics from sparse observations. Applications include sampling Boltzmann densities of molecular conformations, inverse problems and conditional generation under pretrained generative model priors, and single-cell RNA sequence modelling.

In this talk, I will survey some recent progresses in the field of entropic optimal transport (a.k.a. Schrödinger problem). More precisely I will focus on results providing theoretical guarantees of convergence for the most popular algorithms employed to compute approximate solutions in machine learning applications, namely Sinkhorn’s algorithm and the iterated Markovian fitting algorithm. In particular, I will develop the connection between the regularity of entropic potentials, stability of optimal solutions and exponential convergence of the above mentioned algorithms in the number of iterations. This is partially based on joint work with A.Chiarini, A.Durmus, M.Gentiloni, G.Greco, L.Tamanini.

Flow Matching and associated generative models have recently attracted significant interest due to their simulation-free training via a straightforward least squares criterion and its flexible underlying ordinary differential equation framework. The cheap generation of new samples opens the door to efficient distribution estimation, an essential component of forecasting tasks such as weather prediction. In this talk, we first adapt the Flow Matching method to smooth conditional density estimation. We show that the resulting estimator can be related to the classical Nadaraya-Watson estimator. Then, we bridge the gap between proper scoring rules, the established method of evaluating predictions, and the mean integrated squared risk in density estimation. Building on this, we demonstrate an anisotropic rate of convergence for the Flow Matching estimator with respect to a spectral family of proper scoring rules which includes the popular energy score. The talk is based on joint work with Lea Kunkel.

We link conditional generative modelling to quantile regression. We propose a suitable loss function and derive minimax convergence rates for the associated risk under smoothness assumptions imposed on the conditional distribution. To establish the lower bound, we show that nonparametric regression can be seen as a sub-problem of the considered generative modelling framework. Finally, we discuss extensions of our work to generate data from multivariate distributions. This is joint work with Petr Zamolodtchikov.

In this talk I will introduce causal generative models, a novel class of deep generative models that do not only accurately fit observational data but can also provide accurate estimates to interventional and counterfactual queries. Specifically I will discuss how to use deep learning architectures to design such models so that with a single generative model, we can provide estimate of a large number of causal queries with theoretical guarantees, while keeping assumptions practical. I will finally discuss the open challenges of designing such causal generative models.

[TBA]

Building upon the regularity of the score, we will discuss the convergence rates achievable through denoising score matching. Then, we will discuss the distributional stability of the overall SGM pipeline, and conclude on the minimax estimation rates for the underlying distribution.