2023

Granted Santander-UAM Scholarship. Uncertainty Estimation in LLM at Cambridge University.

Computational and Biological Learning Lab, University of Cambridge

2021

Granted FPI-UAM Scholarship. Competitive Predoctoral Contract for Training Research Personnel

Department of Computer Science, Autonomous University of Madrid

2020

Research Collaboration Scholarship

Department of Computer Science, Autonomous University of Madrid

2020

Granted Highest Mark on Bachelor's Thesis, 10/10. Statistical Models with Variational Methods

Department of Computer Science and Faculty of Science, University of Granada

Implicit processes (IPs) are a generalization of Gaussian processes (GPs). IPs may lack a closed-form expression but are easy to sample from. Examples include, among others, Bayesian neural networks or neural samplers. IPs can be used as priors over functions, resulting in flexible models with well-calibrated prediction uncertainty estimates. Methods based on IPs usually carry out function-space approximate inference, which overcomes some of the difficulties of parameter-space approximate inference. Nevertheless, the approximations employed often limit the expressiveness of the final model, resulting, e.g., in a Gaussian predictive distribution, which can be restrictive. We propose here a multi-layer generalization of IPs called the Deep Variational Implicit process (DVIP). This generalization is similar to that of deep GPs over GPs, but it is more flexible due to the use of IPs as the prior distribution over the latent functions. We describe a scalable variational inference algorithm for training DVIP and show that it outperforms previous IP-based methods and also deep GPs. We support these claims via extensive regression and classification experiments. We also evaluate DVIP on large datasets with up to several million data instances to illustrate its good scalability and performance. selected = false

Ensembles are widely used in machine learning and, usually, provide state-of-the-art performance in many prediction tasks. From the very beginning, the diversity of an ensemble has been identified as a key factor for the superior performance of these models. But the exact role that diversity plays in ensemble models is poorly understood, specially in the context of neural networks. In this work, we combine and expand previously published results in a theoretically sound framework that describes the relationship between diversity and ensemble performance for a wide range of ensemble methods. More precisely, we provide sound answers to the following questions: how to measure diversity, how diversity relates to the generalization error of an ensemble, and how diversity is promoted by neural network ensemble algorithms. This analysis covers three widely used loss functions, namely, the squared loss, the cross-entropy loss, and the 0-1 loss; and two widely used model combination strategies, namely, model averaging and weighted majority vote. We empirically validate this theoretical analysis with neural network ensembles.

Model selection in machine learning (ML) is a crucial part of the Bayesian learning procedure. Model choice may impose strong biases on the resulting predictions, which can hinder the performance of methods such as Bayesian neural networks and neural samplers. On the other hand, newly proposed approaches for Bayesian ML exploit features of approximate inference in function space with implicit stochastic processes (a generalization of Gaussian processes). The approach of Sparse Implicit Processes (SIP) is particularly successful in this regard, since it is fully trainable and achieves flexible predictions. Here, we expand on the original experiments to show that SIP is capable of correcting model bias when the data generating mechanism differs strongly from the one implied by the model. We use synthetic datasets to show that SIP is capable of providing predictive distributions that reflect the data better than the exact predictions of the initial, but wrongly assumed model.

Variational Linearized Laplace Approximation for Bayesian Deep Learning (Under Review) [pre-print]

Uncertainty estimation on pre-trained Deep Learning models using Variational Inference and LLA.

PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime [pre-print]

Explaining deep learning techniques (weight-decay, overparameterization, data-augmentation) using Large Deviation Theory

If there is no underfitting, there is no Cold Posterior Effect [pre-print]

Misspecification leads to Cold Posterior Effect (CPE) only when the resulting Bayesian posterior underfits.

PAC-Bayes-Chernoff Bounds for Unbounded Losses [pre-print]

PAC-Bayes version of the Chernoff bound which solves the open problem of optimizing the free parameter on many PAC-Bayes bounds.