The usefulness of infinite-dimensional vector spaces is that often concepts in finite dimensions extend to infinite dimensions; this allows us to reason about infinite-dimensional problems using finite-dimensional intuition. (This is explained in detail in my primer on RKHS theory, for example.) Nevertheless, infinite-dimensional spaces exhibit behaviours not found in finite dimensions, and working effectively with infinite-dimensional spaces requires understanding how they differ from finite dimensional spaces.
This article discusses several key differences between a Gaussian measure on Euclidean space and on the separable Hilbert space , which can be thought of as an infinite-dimensional generalisation of Euclidean space. (Recall consists of all square-summable sequences.)
A Gaussian Measure on
Let denote the canonical basis vectors of , that is, is the sequence of all zeros except for the th term which is unity.
Essentially, we wish to construct a Gaussian random variable on element-wise by first generating a collection of real-valued independent random variables and then setting , or in other words, . We assume .
It is essential for to decay to zero sufficiently fast, for otherwise, the sequence will have a non-zero probability of not lying in . We require to hold with probability one. Since the standard deviation gives the “typical” size of a random realisation of , it is not surprising that the requirement be . (Throughout, curious readers are invited to consult an introductory book on infinite-dimensional analysis for more details. For example, in general, one starts with a linear operator that will serve as the covariance matrix, and requires to be of trace class. Here, I am simply taking to be the “diagonal matrix” .)
It turns out that the above procedure can be made rigorous; provided , there is indeed a Gaussian random variable on such that the are independent Gaussian random variables .
To any (Borel-measurable) subset , we define the Gaussian measure as the probability that lies in . (To a mathematician, this is putting the cart before the horse, but nevertheless, it is a convenient way to think when it comes to having to evaluate in certain situations.)
Since we insisted all the be positive, the measure is non-zero on any open subset of . Note too that .
Some Possibly Surprising Facts
Given that the following do not hold in finite dimensions, they are initially counter-intuitive. The remainder of the article aims to give sufficient explanations to “improve” our intuition, thereby making the following facts unsurprising.
- Given a Gaussian measure on , there exists a subset and an element such that
- and are disjoint;
- ; and
Why should these facts be surprising? By analogy with the finite-dimensional case, one would expect that unless is pretty much all of then it would not be possible for . Indeed, is non-zero on any open subset of , hence must be dense in ? So how can the translated version be disjoint from ? And how can the “large” set , having measure 1, go to the “small” set having measure 0, simply by a translation?
For concreteness, choose .
The subset will be taken to be the limit of a monotonically increasing sequence of subsets . The subsets will be taken to be “rectangles” of the form where the will be strictly positive.
Since the variance of the goes to zero, we hope to be able to choose the so that they decay to zero in , for fixed , while ensuring has measure close to unity. This gives us a chance of constructing an which is not the whole of yet has measure equal to unity. The rate of decay is too fast because the probability that lies between and does not depend on ; if this probability is then would be the measure of the rectangle. This motivates trying a slower rate of decay: . (The numerator is there to grow the rectangles so the probability of lying in goes to unity as , and the is for later convenience.)
The probability that lies between is conveniently expressed in terms of the error function as . Hopefully, for all , and as , so that . This is indeed the case.
[Here is the tedious calculation to prove the claims. Let . A well-known bound on the complementary error function is . Therefore, where . Note and when and . Provided is finite, will be strictly positive, as required. Now, , hence it suffices to prove is finite. The ratio test for absolute convergence involves the ratio . Since , it suffices to show in order to conclude that is finite. Now, whenever , as required. To show , we need to show . An earlier calculation shows , so that where . It is clear that this can be made arbitrarily close to zero by choosing large.]
- Had we been in finite dimensions, the sets would have been open. Here, they are not open.
Since the as , there is no open ball centred at the origin, with positive radius , that is contained in . Indeed, for a given radius and , an can be found such that , showing is not fully contained within the ball.
- Let . The point is not in .
We may be led to believe this is surprising: as gets bigger, all the sides of the rectangle get bigger, hence we may expect that it will grow to be the whole of . However, as explained presently, in infinite dimensions, the order in which limits are taken matters, and the above argument is invalid. It will be seen that while the sides of the rectangle do grow, they grow too slowly.
On the one hand, it is true that if we fix an , then we can find an sufficiently large so that the th side of the rectangle contains , the th term of . Mathematically, this is true because . However, this is only really telling us that the “truncated” approximations are all in , but analytically, we know that if is not closed then the limit of these approximations need not lie in . Figuratively speaking, even though an ordinary towel can be stuffed in a suitcase by sequentially pushing in the bits that are hanging out, this reasoning is not valid if the towel were infinitely big; after each push, there may still be an infinite portion of the towel hanging out. Instead, we must think directly of the as suitcases of increasing size, and ask if the towel fits entirely inside one of these suitcases.
The reason why does not lie in any of the , and hence is not in , is that the terms of decrease more slowly to zero than the sides of the rectangles. For a fixed , there will be a sufficiently large so that , thereby showing for all .
- But how can not be in if every dimension of the suitcase is expanding with ?
The sides of the suitcase are aligned with the canonical basis vectors . However, the form a Schauder basis, not a Hamel basis, so it is not true that “every dimension of the suitcase is expanding”. For example, the point does not lie in the span of the . (The span of the consists of all sequences with only a finite number of non-zero terms.) The argument given earlier readily extends to show that, for all , the point does not lie in any of the suitcases . So while the suitcases are getting bigger in the directions , they always have zero width in the direction . (It is true that the distance from to goes to zero as , but that just means is in the closure of .)
This is another phenomenon unique to the infinite-dimensional case. Recall that . The th side of is the interval from to . We know as (the sides of the suitcase decrease much faster than the ). And normalised by the standard deviation of also goes to infinity, i.e., . So the probability that lies inside the interval from to goes to zero. While the finite product of positive numbers is positive, the infinite product of positive numbers that decay to zero is zero: there is zero probability that a randomly chosen point will lie in .
- The sets and are disjoint.
Again, this is a consequence of the order of the limits. For a fixed , the th edge of the suitcase decays to zero faster than . So for sufficiently large, the th edges of and do not overlap, hence and are disjoint.
While the above gave one specific example, the general theory goes as follows. Given a Gaussian measure on with mean zero and covariance , the shifted Gaussian measure is either equivalent to (meaning the two measures are absolutely continuous with respect to each other), or the two measures are singular. The Cameron-Martin space is the space of all for which the two measures are equivalent. There is a simple expression for this space: it is , the image of under the operator .