kernel

Kernel Methods

Table of Contents

• Regressors

• Types

• Linear Algebra

• Utilities

Regressors

Common

linear_operator_learning.kernel.predict(num_steps, fit_result, kernel_YX, kernel_Xin_X, obs_train_Y)

Predicts future states given initial values using a fitted regressor.

Parameters:

• num_steps (int) – Number of steps to predict forward (returns the last prediction)

• fit_result (FitResult) – FitResult object containing fitted U and V matrices

• kernel_YX (ndarray) – Kernel matrix between output data and input data (or inducing points for Nystroem)

• kernel_Xin_X (ndarray) – Kernel matrix between initial conditions and input data (or inducing points for Nystroem)

• obs_train_Y (ndarray) – Observable evaluated on output training data (or inducing points for Nystroem)

Return type:

ndarray

Shape:

kernel_YX: \((N, N)\), where \(N\) is the number of training data, or inducing points for Nystroem.

kernel_Xin_X: \((N_0, N)\), where \(N_0\) is the number of inputs to predict.

obs_train_Y: \((N, *)\), where \(*\) is the shape of the observable.

Output: \((N, *)\).

linear_operator_learning.kernel.eig(fit_result, K_X, K_YX)

Computes the eigendecomposition of a regressor.

Parameters:

• fit_result (FitResult) – Fit result as defined in linear_operator_learning.kernel.structs.

• K_X (ndarray) – Kernel matrix of the input data.

• K_YX (ndarray) – Kernel matrix between the output data and the input data.

Return type:

EigResult

Shape:

K_X: \((N, N)\), where \(N\) is the sample size.

K_YX: \((N, N)\), where \(N\) is the sample size.

Output: U, V of shape \((N, R)\), svals of shape \(R\) where \(N\) is the sample size and \(R\) is the rank of the regressor.

linear_operator_learning.kernel.evaluate_eigenfunction(eig_result, which, K_Xin_X_or_Y)

Evaluates left or right eigenfunctions of a regressor.

Parameters:

• eig_result (EigResult) – EigResult object containing eigendecomposition results

• which (Literal['left', 'right']) – String indicating “left” or “right” eigenfunctions

• K_Xin_X_or_Y (ndarray) – Kernel matrix between initial conditions and input data (for right eigenfunctions) or output data (for left eigenfunctions)

Shape:

eig_result: U, V of shape \((N, R)\), svals of shape \(R\) where \(N\) is the sample size and \(R\) is the rank of the regressor.

K_Xin_X_or_Y: \((N_0, N)\), where \(N_0\) is the number of inputs to predict and \(N\) is the sample size.

Output: \((N_0, R)\)

Reduced Rank

linear_operator_learning.kernel.reduced_rank(kernel_X, kernel_Y, tikhonov_reg, rank, svd_solver='arnoldi')

Fits the Reduced Rank estimator from Kostic et al.[1].

Parameters:

• kernel_X (ndarray) – Kernel matrix of the input data.

• kernel_Y (ndarray) – Kernel matrix of the output data.

• tikhonov_reg (float) – Tikhonov (ridge) regularization parameter.

• rank (int) – Rank of the estimator.

• svd_solver (Literal[ "arnoldi", "full" ], optional) – Solver for the generalized eigenvalue problem. Defaults to “arnoldi”.

Return type:

FitResult

Shape:

kernel_X: \((N, N)\), where \(N\) is the number of training data.

kernel_Y: \((N, N)\).

linear_operator_learning.kernel.nystroem_reduced_rank(kernel_X, kernel_Y, kernel_Xnys, kernel_Ynys, tikhonov_reg, rank, svd_solver='arnoldi')

Fits the Nyström Reduced Rank estimator from Meanti et al.[2].

Parameters:

• kernel_X (ndarray) – Kernel matrix of the input inducing points.

• kernel_Y (ndarray) – Kernel matrix of the output inducing points.

• kernel_Xnys (ndarray) – Kernel matrix between the input data and the input inducing points.

• kernel_Ynys (ndarray) – Kernel matrix between the output data and the output inducing points.

• tikhonov_reg (float) – Tikhonov (ridge) regularization parameter.

• rank (int) – Rank of the estimator.

• svd_solver (Literal[ "arnoldi", "full" ], optional) – Solver for the generalized eigenvalue problem. Defaults to “arnoldi”.

Return type:

FitResult

Shape:

kernel_X: \((N, N)\), where \(N\) is the number of training data.

kernel_Y: \((N, N)\).

kernel_Xnys: \((N, M)\), where \(M\) is the number of Nystroem centers (inducing points).

kernel_Ynys: \((N, M)\).

linear_operator_learning.kernel.rand_reduced_rank(kernel_X, kernel_Y, tikhonov_reg, rank, n_oversamples=5, optimal_sketching=False, iterated_power=1, rng_seed=None, precomputed_cholesky=None)

Fits the Randomized Reduced Rank Estimator from Turri et al.[3].

Parameters:

• kernel_X (ndarray) – Kernel matrix of the input data

• kernel_Y (ndarray) – Kernel matrix of the output data

• tikhonov_reg (float) – Tikhonov (ridge) regularization parameter

• rank (int) – Rank of the estimator

• n_oversamples (int, optional) – Number of Oversamples. Defaults to 5.

• optimal_sketching (bool, optional) – Whether to use optimal sketching (slower but more accurate) or not.. Defaults to False.

• iterated_power (int, optional) – Number of iterations of the power method. Defaults to 1.

• rng_seed (int | None, optional) – Random Number Generators seed. Defaults to None.

• precomputed_cholesky (optional) – Precomputed Cholesky decomposition. Should be the output of cho_factor evaluated on the regularized kernel matrix.. Defaults to None.

Return type:

FitResult

Shape:

kernel_X: \((N, N)\), where \(N\) is the number of training data.

kernel_Y: \((N, N)\).

Principal Component Regression

linear_operator_learning.kernel.pcr(kernel_X, tikhonov_reg=0.0, rank=None, svd_solver='arnoldi')

Fits the Principal Components estimator.

Parameters:

• kernel_X (ndarray) – Kernel matrix of the input data.

• tikhonov_reg (float, optional) – Tikhonov (ridge) regularization parameter. Defaults to 0.0.

• rank (int | None, optional) – Rank of the estimator. Defaults to None.

• svd_solver (Literal[ "arnoldi", "full" ], optional) – Solver for the generalized eigenvalue problem. Defaults to “arnoldi”.

Return type:

FitResult

Shape:

kernel_X: \((N, N)\), where \(N\) is the number of training data.

linear_operator_learning.kernel.nystroem_pcr(kernel_X, kernel_Y, kernel_Xnys, kernel_Ynys, tikhonov_reg=0.0, rank=None, svd_solver='arnoldi')

Fits the Principal Components estimator using the Nyström method from Meanti et al.[2].

Parameters:

• kernel_X (ndarray) – Kernel matrix of the input inducing points.

• kernel_Y (ndarray) – Kernel matrix of the output inducing points.

• kernel_Xnys (ndarray) – Kernel matrix between the input data and the input inducing points.

• kernel_Ynys (ndarray) – Kernel matrix between the output data and the output inducing points.

• tikhonov_reg (float, optional) – Tikhonov (ridge) regularization parameter. Defaults to 0.0.

• rank (int | None, optional) – Rank of the estimator. Defaults to None.

• svd_solver (Literal[ "arnoldi", "full" ], optional) – Solver for the generalized eigenvalue problem. Defaults to “arnoldi”.

Return type:

FitResult

Shape:

kernel_X: \((N, N)\), where \(N\) is the number of training data.

kernel_Y: \((N, N)\).

kernel_Xnys: \((N, M)\), where \(M\) is the number of Nystroem centers (inducing points).

kernel_Ynys: \((N, M)\).

Types

class linear_operator_learning.kernel.structs.FitResult

Return type for kernel regressors.

class linear_operator_learning.kernel.structs.EigResult

Return type for eigenvalue decompositions of kernel regressors.

Linear Algebra Utilities

linear_operator_learning.kernel.linalg.weighted_norm(A, M=None)

Weighted norm of the columns of A.

Parameters:

• A (ndarray) – 1D or 2D array. If 2D, the columns are treated as vectors.

• M (ndarray or LinearOperator, optional) – Weigthing matrix. the norm of the vector \(a\) is given by \(\langle a, Ma \rangle\) . Defaults to None, corresponding to the Identity matrix. Warning: no checks are

• operator. (performed on M being a PSD)

Returns:

(ndarray or float) – If A.ndim == 2 returns 1D array of floats corresponding to the norms of the columns of A. Else return a float.

linear_operator_learning.kernel.linalg.stable_topk(vec, k_max, rcond=None, ignore_warnings=True)

Takes up to k_max indices of the top k_max values of vec. If the values are below rcond, they are discarded.

Parameters:

• vec (ndarray) – Vector to extract the top k indices from.

• k_max (int) – Number of indices to extract.

• rcond (float, optional) – Value below which the values are discarded. Defaults to None, in which case it is set according to the machine precision of vec’s dtype.

• ignore_warnings (bool) – If False, raise a warning when some elements are discarted for being below the requested numerical precision.

linear_operator_learning.kernel.linalg.add_diagonal_(M, alpha)

Add alpha to the diagonal of M inplace.

Parameters:

• M (ndarray) – The matrix to modify inplace.

• alpha (float) – The value to add to the diagonal of M.

General Utilities

linear_operator_learning.kernel.utils.topk(vec, k)

Get the top k values from a Numpy array.

Parameters:

• vec (ndarray) – A 1D numpy array

• k (int) – Number of elements to keep

Returns:

values, indices – top k values and their indices

linear_operator_learning.kernel.utils.sanitize_complex_conjugates(vec, tol=10.0)

This function acts on 1D complex vectors. If the real parts of two elements are close, sets them equal. Furthermore, sets to 0 the imaginary parts smaller than tol times the machine precision.

Parameters:

• vec (ndarray) – A 1D vector to sanitize.

• tol (float, optional) – Tolerance for comparisons. Defaults to 10.0.

[1]

Vladimir Kostic, Pietro Novelli, Andreas Maurer, Carlo Ciliberto, Lorenzo Rosasco, and Massimiliano Pontil. Learning dynamical systems via koopman operator regression in reproducing kernel hilbert spaces. Advances in Neural Information Processing Systems, 35:4017–4031, 2022.

[2] (1,2)

Giacomo Meanti, Antoine Chatalic, Vladimir R Kostic, Pietro Novelli, Massimiliano Pontil, and Lorenzo Rosasco. Estimating koopman operators with sketching to provably learn large scale dynamical systems. In Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL: https://openreview.net/forum?id=GItLpB1vhK.

[3]

Giacomo Turri, Vladimir Kostic, Pietro Novelli, and Massimiliano Pontil. A randomized algorithm to solve reduced rank operator regression. 2023. URL: https://arxiv.org/abs/2312.17348, doi:10.48550/ARXIV.2312.17348.