Why should all the poles be inside the unit circle for a discrete-time digital system to be stable? And why must we care about regions of convergence (ROC)? With only a small investment in time, it is possible to gain a very clear understanding of exactly what is going on — it is not complicated if learnt step at a time, and there are not many steps.
Step 1: Formal Laurent Series
Despite the fancy name, Step 1 is straightforward. Digital systems operate on sequences of numbers: a sequence of numbers goes into a system and a sequence of numbers comes out. For example, if denotes an input sequence then an example of a discrete-time system is .
How can a specific input sequence be written down, and how can the output sequence be computed?
The direct approach is to specify each individual , for example, for , , , and for . Direct substitution then determines the output: , and so forth.
The system has a number of nice properties, two of which are linearity and time invariance. This is often expressed in textbooks by saying the system is LTI, where the L in LTI stands for Linear and the TI stands for Time Invariant. When first learning about discrete-time digital systems, usually attention is restricted to linear time-invariant systems, and indeed, it is for such systems that the Z-transform proves most useful.
Any LTI system has the property that the output is the convolution of the input with the impulse response. For example, the impulse response of is found by injecting the input and for . This impulse as input produces the output , and for . It is common to denote the impulse response using the letter , that is, , and for .
By the definition of convolution, the convolution of a sequence with a sequence is . If , and for then and this is equal to ; we have verified in this particular case that the output of the original system is indeed given by convolving the input with the impulse response.
Let’s be honest: writing , and for is tedious! An alternative is to use a formal Laurent series to represent a sequence. Here, “formal” means that we do not try to evaluate this series! A Laurent series just means instead of limiting ourselves to polynomials in a dummy variable , we allow negative powers of , and we allow an infinite number of non-zero terms. (The variable has nothing to do with time; it will be replaced in Step 3 by where it will be referred to as the Z-transform. To emphasise, is just a dummy variable, a placeholder.)
Precisely, the sequence , and for can be encoded as ; isn’t that much more convenient to write? In general, an arbitrary sequence can be encoded as the formal series . The sole purpose of the term is as a placeholder for placing the coefficient . In other words, if we are given the formal series then we immediately know that that is our secret code for the sequence that consists of all zeros except for the -5th term which is 2, the 0th term which is 3 and the 7th term which is 1.
Being able to associate letters with encoded sequences is convenient, hence people will write to mean , , and for . Do not try to evaluate for some value of though – that would make no sense at this stage. (Mathematically, we are dealing with a commutative ring that involves an indeterminant . If you are super-curious, see Formal Laurent series.)
Now, there is an added benefit! If the input to our system is and the impulse response is then there is a “simple” way to determine the output: just multiply these two series together! The multiplication rule for formal Laurent series is equivalent to the convolution rule for two sequences. You can check that (corresponding to etc) really is the output of our system if the input were for , , , and for .
- A formal Laurent series provides a convenient encoding of a discrete-time sequence.
- Convolution of sequences corresponds to multiplication of formal Laurent series.
- Do not think of formal Laurent series as functions of a variable. (They are elements of a commutative ring.) Just treat the variable as a placeholder.
- In particular, there is no need (and it makes no sense) to talk about ROC (regions of convergence) if we restrict ourselves to formal Laurent series.
Step 2: Analytic Functions
This is where the fun begins. We were told not to treat a formal Laurent series as a function, but it is very tempting to do so… so let’s do just that!🙂 The price we pay though is that we do need to start worrying about ROC (regions of convergence) for otherwise we could end up with incorrect answers.
To motivate further our desire to break the rules, consider the LTI system . Such a system has “memory” or an “internal state” because the current output depends on a previous output and hence the system must somehow remember previous values of the output. We can assume that in the distant past the system has been “reset to zero” and hence the output will be zero up until the first time the input is non-zero. The impulse response is determined by setting to 1 and observing the output: , , and in general for . Encoding this using a formal Laurent series gives the alternative representation .
There is a more “efficient” representation of based on the Taylor series expansion . Indeed, if we were to break the rules and treat as a function, we might be tempted to write instead of . It turns out that, with some care, it is mathematically legitimate to do this. (Furthermore, Step 3 will explain why it is beneficial to go to this extra trouble.)
We know we can encode a sequence using a formal Laurent series, and we can reverse this operation to recover the sequence from the formal Laurent series. In Step 2 then, we just have to consider when we can encode a formal Laurent series as some type (what type?) of function, meaning in particular that it is possible to determine uniquely the formal Laurent series given the function.
Power series (and Laurent series) are studied in complex analysis: recall that every power series has a (possibly zero) radius of convergence, and within that radius of convergence, a power series can be evaluated. Furthermore, within the radius of convergence, a power series (with real or complex coefficients) defines a complex analytic function. The basic idea is that a formal Laurent series might (depending on how quickly the coefficients die out to zero) represent an analytic function on a part of the complex plane.
If the formal Laurent series only contains non-negative powers of then it is a power series and from complex analysis we know that there exists an , the radius of convergence (which is possibly zero or possibly infinite), such that the sum converges if and the sum diverges if . In particular, is an analytic function on the open disk .
If the formal Laurent series only contains non-positive powers, that is, , then we can consider it to be a power series in and apply the above result. Since , there exists a (possibly zero or possibly infinite) such that is an analytic function on . [From above, the condition would be which is equivalent to hence defining verifies the claim.]
In the general case, a formal Laurent series decomposes as . Both sums must converge if it is to define an analytic function, hence in the general case, a formal Laurent series defines an analytic function on a domain of the form .
The encoding of a sequence as an analytic function is therefore straightforward in principle: given a formal Laurent series , determine the constants and , and provided the region is non-empty, we can use the analytic function to encode the sequence . We must specify the ROC together with whenever we wish to define a sequence in this way; without knowing the ROC, we do not know the domain of definition of , that is, we would not know for what values of does describe the sequence we want. It is possible for a function to describe different sequences depending on the ROC! An example of this is now given.
If then we have seen that . But wait! This is not entirely true. If then certainly .Yet if then . The function for does not describe the series . It looks like a perfectly good function though, so what series does it describe?
The region is unbounded. (In fact, it is an open disc centred at the point at infinity.) This motivates us to replace by so that the domain becomes and we can attempt looking for a power series in . Precisely, . Therefore, the single function actually encodes two different series depending on whether we treat it as a function on or a function on .
Readers remembering their complex analysis will not find this bizarre because a holomorphic (i.e., complex differentiable) function generally requires more than one power series to represent it. A power series about a point is only valid up until a pole is encountered, after which another point must be chosen and a power series around that point used to continue the description of the function. When we change points, the coefficients of the power series will generally change. In the above example, the first series is a power series around the origin while the second series is a power series around the point at infinity and therefore naturally has different coefficients. (Thankfully there are only these two possibilities to worry about: a power series about a different point would look like and is not of the form we described in Step 1. Only when or does the power series match up with the formal Laurent series in Step 1.)
While it might seem that introducing analytic functions is an unnecessary complication, it actually makes certain calculations simpler! Such a phenomenon happened in Step 1: we discovered convolution of sequences became multiplication of Laurent series (and usually multiplication is more convenient than convolution). In Step 3 we will see how the impulse response of can be written down immediately.
Step 3: The Z-transform
Define and . Clearly, the constraint implies a relationship between and , but how can we determine this relationship?
Note that is actually an infinite number of constraints, one for each . The first step is to think of these as a single constraint on the whole sequences and . We can do this either intuitively or rigorously, arriving at the same answer either way.
Intuitively, think of a table (my lack of WordPress skills prevents me from illustrating this nicely). The top row looks like . The next row looks like . The last row looks like . Stack these three rows on top of each other so they line up properly: the purpose of the table is to line everything up correctly! Then another way of expressing is by saying that the first row of the table is equal to the second row of the table plus the third row of the table, where rows are to be added elementwise.
Rigorously, what has just happened is that we are treating a sequence as a vector in an infinite-dimensional vector space: just like is a vector in , we can think of as a vector in . Each of the three rows of the table described above is simply a vector.
To be able to write the table compactly in vector form, we need some way of going from the vector to the shifted-one-place-to-the-right version of it, namely . In linear algebra, we know that a matrix can be used to map one vector to another vector provided the operation is linear, and in fact, shifting a vector is indeed a linear operation. In abstract terms, there exists a linear operator that shifts a sequence one place to the right: .
Letting and denote the vectors and respectively, we can rigorously write our system as . This is precisely the same as what we did when we said the first row of the table is equal to the second row of the table plus the third row of the table.
Our natural instinct is to collect like terms: if denotes the identity operator (think of a matrix with ones on the diagonal), so that , then is equivalent to , that is, . This equation tells us the output as a function of the input — but how useful is it? (Does the inverse exist, and even if it does, how can we evaluate it?)
While in some cases the expression might actually be useful, often it is easier to use series rather than vectors to represent sequences: we will see that the linear operator is equivalent to multiplication by , which is very convenient to work with!
Precisely, since and represent exactly the same sequence, it should be clear that the equation is equivalent to the equation . Indeed, returning to the table, corresponds to the first row, corresponds to the last row, and corresponds to the second row.
Now, from Step 2, we know that we can think of and as analytic functions (provided we are careful about the ROC). Therefore, we feel entitled to continue manipulating to obtain . Since by definition the impulse response satisfies , we seem to have immediately found the impulse response of the original system; at the very least, it agrees with the longer calculation performed at the start of Step 2.
Remark: Actually, the only way to justify rigorously that the above manipulation is valid is to check the answer we have found really is the correct answer. Indeed, to be allowed to perform the manipulations we must assume the existence of a domain on which both and are analytic. If we assume is “nice” then, under that assumption, prove that is “nice”, that does not mean is nice. It is like saying “if it is raining then we can show it really is raining, therefore it must be raining right now!”. A rigorous justification would look something like the following. First we must make some assumption about the input sequence for otherwise we have no idea what ROC to use. If we are interested in inputs that are uniformly bounded and which are zero up until time 0 then we can safely assume that is analytic on . Since is non-zero whenever , we know is analytic on . Therefore will be analytic on . This means that can be expanded as a power series in a neighbourhood of the origin, and the coefficients of that power series are what we believe the output of the system will be. To check this really is the output of the system, it suffices to show for . This is straightforward: as required, where every manipulation can be seen to be valid for (there are no divisions by zero or other bad things occurring).
The remark above shows it is (straightforward but) tedious to verify rigorously that we are allowed to perform the manipulations we want. It is much simpler to go the other way and define a system directly in terms of and its ROC. Then, provided the ROC of overlaps with the ROC of , the output is given by on the overlap, which can be correctly expanded as a Laurent series with regard to the ROC, and the output sequence read off.
All that remains is to introduce the Z-transform and explain why engineers treat as a “time shift operator”.
The Z-transform is simply doing the same as what we have been doing, but using instead of . Why rather than ? Just convention (and perhaps a bit of convenience too, e.g., it leads to stable systems being those with all poles inside the unit circle, rather than outside). In fact, sometimes is used instead of , hence you should always check what conventions an author or lecturer has decided to adopt.
The rule that engineers are taught is that when you “take the Z-transform” of then you replace by , by and by . The reason this rule works was justified at great length above: recall that as an intermediate mental step we can think of the input and output as vectors, and this led to thinking of them instead as series, because multiplication by will then shift the sequence one place to the right. Thus, is asserting that the sequence is equal to the sequence plus a half times the sequence shifted to the right by one place, which is equivalent to the original description .
Step 4: Poles and Zeros
A straightforward but nonetheless rich class of LTI systems can be written in the form . Importantly, this class is “closed” in that if you connect two such systems together (the output of one connects to the input of the other) then the resulting system can again be written in the same way (but generally with larger values of ). Another important feature of this class of systems is that they can be readily implemented in hardware.
Applying the Z-transform to such a system shows that the impulse response in the Z-domain is a rational function. Note that the product of two rational functions is again a rational function, demonstrating that this class of systems is closed under composition, as stated above. Most of the time, it suffices to work with rational impulse responses.
If is rational (and in reduced form — no common factors) then the zeros of are the solutions of the polynomial equation while the poles are the solutions of the polynomial equation . [More generally, if is analytic, then the zeros are the solutions of and the poles are the solutions of .] Note that a pole of is a zero of its inverse: if we invert a system, poles become zeros and zeros become poles.
Poles are important because poles are what determine the regions of convergence and hence they determine when our manipulations in Step 3 are valid. This manifests itself in poles having a physical meaning: as we will see, the closer a pole gets to the unit circle, the less stable a system becomes.
Real-world systems are causal: the output cannot depend on future input. The impulse response of a causal system is zero for . Therefore, the formal Laurent series (Step 1) representation of has no negative powers of . Its region of convergence will therefore be of the form for some . (If then the system would blow up!) Since , the ROC of a causal transfer function will therefore be of the form for some . If is rational then it has only a finite number of poles, and it suffices to choose to be the largest of the magnitudes of the poles.
Let’s look at an unstable system: . This system is clearly unstable because its impulse response is readily seen to be . We put in a bounded signal ( and for ) yet obtain an unbounded output ( for ). Taking the Z-transform gives , so that . For , this gives us the correct series representation .
If we put a signal that starts at time zero (i.e., for ) into a causal system we will get an output even if the system is unstable. This is because the input will have a ROC of the form and has a ROC of the form , so will have a ROC .
If we put a signal that started at time into our system, then there might be no solution! (Intuitively, it means the system will have to have blown up before reaching time 0.) For example, with ROC represents the signal for and for . We cannot form the product because there is no value of for which both and are valid: one’s ROC is while the other’s is .
There are many different definitions of a sequence being bounded. Three examples are: 1) there exists an such that, for all , ; 2) ; and 3) . Out of these three, the easiest to detect whether is bounded given only is the second one: if the ROC of includes then, by definition (see a complex analysis textbook), is absolutely convergent for , meaning , hence because implies for all . This type of boundedness is called “bounded in “. For the reverse direction, recall that the radius of convergence is such that a power series in will converge for all and diverge for all . Therefore, if the boundary of the largest possible ROC of does not contain the unit circle then must be unbounded in . (The boundary case is inconclusive: the power series has a ROC yet is bounded. On the other hand, has a ROC and is unbounded.)
A sequence is bounded in if the largest possible ROC of includes the unit circle . A sequence is unbounded in if the closure of the largest possible ROC does not include the unit circle.
If the ROC of the transfer function includes the unit circle, then an -bounded input will produce an -bounded output. Indeed, the ROC of both and will include the unit circle, therefore, the ROC of will include the unit circle and will be -bounded.
If all the poles of a causal are within the unit circle then the ROC will include the unit circle, hence a bounded input will produce a bounded output. Engineers therefore call a system stable if all the poles of are inside the unit circle.
Note that it follows from earlier remarks that if has a pole outside the unit circle then the impulse response will be unbounded. If has a pole on the unit circle then the impulse response will resonant at the corresponding frequency indefinitely. For example, if where then for . This does not die away because for all . By writing we get and therefore we think of this as a resonance at frequency . Furthermore, if where , then and we see that there is a damped resonance at frequency . As the pole gets closer to the unit circle (that is, ) the impulse response takes longer to die out. Engineers care about pole placement!
- Although often engineers can get away with just following recipes, it is safer in the long run and easier to remember the recipes if you have spent a bit more time to learn the underlying rationale for when and why the rules work.
- Linear algebra and complex analysis are considered core subjects of an undergraduate mathematics degree because they have such broad applicability: here we saw them play integral roles in discrete-time signal processing.
- This is because the “mathematical behaviour” of Laurent series matches the “physical behaviour” of discrete-time linear time-invariant systems.
- There is a close relationship between the Z-transform and the Fourier transform: just make the substitution . This is only allowed if the ROC contains the unit circle. The concept of a pole does not explicitly appear in the Fourier domain because, rather than permit to be anywhere in the complex plane, the substitution means we are only looking at the behaviour of on the unit circle.
- When it comes to system design, understanding where the poles and zeros are can be more productive than trying to understand the whole frequency spectrum.
The following hints at why the quintic equation cannot be solved using radicals. It follows the approach in the first part of Ian Stewart’s book “Galois Theory”. If time permits, a future post will summarise the approach in V. B. Alekseev’s book “Abel’s Theorem in Problems and Solutions”. Another candidate is Klein’s book “Lectures on the Icosahedron and the Solution of Equations of the Fifth Degree”.
To set the stage, consider the generic quadratic polynomial equation . Solving it is the same as factorising it: given the coefficients find the roots such that .
The first observation is the prominent role played by symmetric polynomials. Note . Here, and are called elementary symmetric polynomials. They are symmetric because swapping and does not change the product . This “annoying” ambiguity turns out to be a friend; the presence of symmetry forces certain relationships to be true, as will be seen presently.
Solving a quadratic equation is equivalent to the following. Given the values of the symmetric polynomials and (up to sign, these are equivalent to the coefficients and in ), find the roots .
Given , we can readily evaluate any symmetric function of the roots that we wish; this is a property of elementary symmetric polynomials that is not too difficult to prove. For example, since is symmetric, it must be capable of being written as a polynomial in . Trial and error leads to finding that , verifying the claim in this instance.
The first insight is that this is not an all or nothing affair. Although is not symmetric, it still behaves nicely when are swapped: . [If we were trying to solve cubic or quartic equations, we would look for polynomial expressions of the roots that took on only two or three distinct values as the roots were permuted.] Moreover, since , this implies is symmetric, a fact that is also self-evident. Therefore, although we cannot express as a polynomial function of , we can do the next best thing: write as the square root of a polynomial in .
Specifically, , and from earlier, we saw that , therefore .
At this point, the reader should pause to recognise that we have “rediscovered” the discriminant of a quadratic . (In our case, , and , hence .)
It is now straightforward to derive the well-known formula for solving a general quadratic equation since we have reduced it to solving linear equations: we know that and .
Why is it impossible to derive a similar type of formula for solving a quintic equation?
It comes down to a property of the symmetry group of five elements: it behaves differently from the symmetry groups on two, three or four elements, which is why formulae exist for solving polynomials of degree less than 5 but not for those of degree 5. (Generic polynomials of higher degrees also have no closed-form solutions expressible in terms of radicals. Note that this does not mean a formula cannot be given whatsoever; for example, degree 5 polynomials can be solved using elliptic functions.)
The remainder of this note endeavours to shed some light on why there is this remarkable connection between symmetry groups and solutions of polynomial equations.
Solving the generic quintic equation amounts to inverting the map where are the elementary symmetric polynomials. While it is possible to show that no formula exists for this inverse that uses only ordinary arithmetic operations (addition, subtraction, multiplication and division) and radicals (taking square roots, cube roots etc), we will prove a weaker but nonetheless insightful result. In his book, Stewart refers to this as showing no solution “by Ruffini radicals” exists. (Ruffini proved this result, without realising the stronger result does not follow from it.)
The type of formulae we will permit are of the same form as that used to solve the quadratic equation. We start off with known values for ; indeed, these are given by which are the coefficients of the quintic equation under consideration. At each step, we wish to determine more information about the roots by finding a rational function of the whose nth power is a rational function of the already known quantities, for some integer . At the first step, this means finding a rational function of the whose nth power is equal to a rational function of . Taking an nth root then gives us a new quantity that can be used in subsequent steps, and so forth. (Ruffini’s oversight was not realising that a priori it might be useful to compute nth roots of other quantities; the above description only permits an nth root to be computed if the answer is expressible as a rational function of the roots. For example, we are not allowed to take the nth root of because the nth root of cannot be written as a rational function of the roots . Abel was able to prove that computing nth roots of other quantities did not help, thereby deducing the stronger result from Ruffini’s result.)
Consider step 1. A rational function is sought such that, for some integer , the th power of is a rational function of the elementary symmetric polynomials , that is, where are rational. Without loss of generality, we may assume is a prime number. (If we wanted a sixth root, we could first take a cube root then, in step 2, take a square root.)
To emphasise the dependence on the roots, we have . Can be chosen relatively freely, or does this constraint impose severe restrictions on the possible choices of the rational functions ?
Here comes the crux of the matter. Despite the seemingly unwieldy equation , the symmetry of the right-hand side (irrespective of what actually is) forces and to be essentially unique. Permuting the roots does not change the right-hand side, therefore permuting the roots cannot change the left-hand side either. It might change , but it will not change . [Actually, if it did not change then would be directly expressible as a rational function of the and hence we can ignore this case, which would correspond to .]
Recall momentarily the quadratic case, where . Swapping caused the sign of to change, but that was it. Here we have, for example, that , therefore, , or in other words, there exists an nth root of unity such that . [To be clear, is a complex number satisfying .]
Different permutations of the roots will lead to possibly different values of . Sure, some might equal 1, but there will be at least one permutation such that the corresponding does not equal one. (If this were not true then would be symmetric and hence directly expressible as a rational function of the , hence we could ignore this step as no radical would need computing.)
The mapping from the permutation of the roots to the value of is a (nontrivial) group homomorphism. It cannot be arbitrary, and in particular, we will see that it forces to equal 2.
Denote the set of all permutations of by . This set naturally has a group structure: multiplication corresponds to applying one permutation after another, so to speak. Precisely, it is a symmetric group.
Denote by the set of all th roots of unity; must lie in . This set also has a standard group structure given by ordinary multiplication: if you multiply two th roots of unity you get another th root of unity. In fact, it is what is known as a cyclic group (and is isomorphic to , the abelian group of integers modulo ).
As mentioned above, the fact that is symmetric induces a function, call it , from to . Precisely, if is a permutation of the then ; recall from earlier that this ratio will be an element of . Crucially, has structure: because . That is, is compatible with the group structure and hence is a group homomorphism. Furthermore, from earlier observations, we know is nontrivial: there exists at least one permutation such that .
There is only a single possible nontrivial group homomorphism from to a cyclic group of prime order!
While is not a very small group (it has 120 elements), its group structure is what prevents more such group homomorphisms from existing. [Since is a nontrivial homomorphism then the kernel of must be a proper normal subgroup of . This proper normal subgroup must also contain every commutator because is abelian. It can be shown the only such subgroup is what is known as the alternating subgroup , implying .]
At this juncture, the group structure of has forced to be (up to a constant) the square-root of the discriminant: . [I have omitted the proof of this as it is not the main focus; I mentioned it only because it helps keep the outline as concrete as possible, and indeed, it follows closely the situation encountered at the start when the quadratic polynomial was being considered.] Going into Step 2, we know not only the values of the elementary symmetric polynomials but also the value of . It turns out though that we get blocked at Step 2 from going any further. The reasoning is similar to above, but with replaced by a smaller group.
Precisely, at Step 2, we want to find a rational function and a prime number such that is equal to a rational function of not only the but also the new quantity we found in Step 1, namely . This time it is no longer true that must be symmetric in because the new quantity is not symmetric. However, if we only look at what are known as even permutations then will remain invariant. The alternating subgroup of consists precisely of the even permutations. In particular, we therefore know that if is in then . This is the same as in Step 1 except has been replaced by . [We did not need to know that Step 1 produced . A simpler group-theoretic argument mentioned parenthetically above implies that the kernel of in Step 1 is and hence that whatever new function of the roots is determined in Step 1, it will be invariant with respect to permutations belonging to .]
Exactly as in Step 1, the invariance of forces the existence of a nontrivial group homomorphism, this time from to a cyclic group of order . And once again, the group structure of limits the possibilities.
There are no nontrivial group homomorphisms from to a cyclic group of prime order.
The above (which is a relatively straightforward result in group theory) immediately implies that there is no formula of the kind specified earlier for solving a quintic polynomial.
Galois theory goes well beyond the basic argument sketched here. It can be used to conclude that a generic quintic (or any higher-degree polynomial for that matter) has no solution in terms of radicals (rather than use Abel’s result to strengthen Ruffini’s result, as alluded to earlier, a cleaner proof is possible). It can also be used to conclude that there exist particular quintic polynomials whose roots cannot be expressed using radicals; this does not follow from the generic case because the generic case sought a single formula for all quintic polynomials, whereas a priori there might exist a different formula for each quintic equation. (Certainly, some quintic equations can be solved using radicals.)
The usefulness of infinite-dimensional vector spaces is that often concepts in finite dimensions extend to infinite dimensions; this allows us to reason about infinite-dimensional problems using finite-dimensional intuition. (This is explained in detail in my primer on RKHS theory, for example.) Nevertheless, infinite-dimensional spaces exhibit behaviours not found in finite dimensions, and working effectively with infinite-dimensional spaces requires understanding how they differ from finite dimensional spaces.
This article discusses several key differences between a Gaussian measure on Euclidean space and on the separable Hilbert space , which can be thought of as an infinite-dimensional generalisation of Euclidean space. (Recall consists of all square-summable sequences.)
A Gaussian Measure on
Let denote the canonical basis vectors of , that is, is the sequence of all zeros except for the th term which is unity.
Essentially, we wish to construct a Gaussian random variable on element-wise by first generating a collection of real-valued independent random variables and then setting , or in other words, . We assume .
It is essential for to decay to zero sufficiently fast, for otherwise, the sequence will have a non-zero probability of not lying in . We require to hold with probability one. Since the standard deviation gives the “typical” size of a random realisation of , it is not surprising that the requirement be . (Throughout, curious readers are invited to consult an introductory book on infinite-dimensional analysis for more details. For example, in general, one starts with a linear operator that will serve as the covariance matrix, and requires to be of trace class. Here, I am simply taking to be the “diagonal matrix” .)
It turns out that the above procedure can be made rigorous; provided , there is indeed a Gaussian random variable on such that the are independent Gaussian random variables .
To any (Borel-measurable) subset , we define the Gaussian measure as the probability that lies in . (To a mathematician, this is putting the cart before the horse, but nevertheless, it is a convenient way to think when it comes to having to evaluate in certain situations.)
Since we insisted all the be positive, the measure is non-zero on any open subset of . Note too that .
Some Possibly Surprising Facts
Given that the following do not hold in finite dimensions, they are initially counter-intuitive. The remainder of the article aims to give sufficient explanations to “improve” our intuition, thereby making the following facts unsurprising.
- Given a Gaussian measure on , there exists a subset and an element such that
- and are disjoint;
- ; and
Why should these facts be surprising? By analogy with the finite-dimensional case, one would expect that unless is pretty much all of then it would not be possible for . Indeed, is non-zero on any open subset of , hence must be dense in ? So how can the translated version be disjoint from ? And how can the “large” set , having measure 1, go to the “small” set having measure 0, simply by a translation?
For concreteness, choose .
The subset will be taken to be the limit of a monotonically increasing sequence of subsets . The subsets will be taken to be “rectangles” of the form where the will be strictly positive.
Since the variance of the goes to zero, we hope to be able to choose the so that they decay to zero in , for fixed , while ensuring has measure close to unity. This gives us a chance of constructing an which is not the whole of yet has measure equal to unity. The rate of decay is too fast because the probability that lies between and does not depend on ; if this probability is then would be the measure of the rectangle. This motivates trying a slower rate of decay: . (The numerator is there to grow the rectangles so the probability of lying in goes to unity as , and the is for later convenience.)
The probability that lies between is conveniently expressed in terms of the error function as . Hopefully, for all , and as , so that . This is indeed the case.
[Here is the tedious calculation to prove the claims. Let . A well-known bound on the complementary error function is . Therefore, where . Note and when and . Provided is finite, will be strictly positive, as required. Now, , hence it suffices to prove is finite. The ratio test for absolute convergence involves the ratio . Since , it suffices to show in order to conclude that is finite. Now, whenever , as required. To show , we need to show . An earlier calculation shows , so that where . It is clear that this can be made arbitrarily close to zero by choosing large.]
- Had we been in finite dimensions, the sets would have been open. Here, they are not open.
Since the as , there is no open ball centred at the origin, with positive radius , that is contained in . Indeed, for a given radius and , an can be found such that , showing is not fully contained within the ball.
- Let . The point is not in .
We may be led to believe this is surprising: as gets bigger, all the sides of the rectangle get bigger, hence we may expect that it will grow to be the whole of . However, as explained presently, in infinite dimensions, the order in which limits are taken matters, and the above argument is invalid. It will be seen that while the sides of the rectangle do grow, they grow too slowly.
On the one hand, it is true that if we fix an , then we can find an sufficiently large so that the th side of the rectangle contains , the th term of . Mathematically, this is true because . However, this is only really telling us that the “truncated” approximations are all in , but analytically, we know that if is not closed then the limit of these approximations need not lie in . Figuratively speaking, even though an ordinary towel can be stuffed in a suitcase by sequentially pushing in the bits that are hanging out, this reasoning is not valid if the towel were infinitely big; after each push, there may still be an infinite portion of the towel hanging out. Instead, we must think directly of the as suitcases of increasing size, and ask if the towel fits entirely inside one of these suitcases.
The reason why does not lie in any of the , and hence is not in , is that the terms of decrease more slowly to zero than the sides of the rectangles. For a fixed , there will be a sufficiently large so that , thereby showing for all .
- But how can not be in if every dimension of the suitcase is expanding with ?
The sides of the suitcase are aligned with the canonical basis vectors . However, the form a Schauder basis, not a Hamel basis, so it is not true that “every dimension of the suitcase is expanding”. For example, the point does not lie in the span of the . (The span of the consists of all sequences with only a finite number of non-zero terms.) The argument given earlier readily extends to show that, for all , the point does not lie in any of the suitcases . So while the suitcases are getting bigger in the directions , they always have zero width in the direction . (It is true that the distance from to goes to zero as , but that just means is in the closure of .)
This is another phenomenon unique to the infinite-dimensional case. Recall that . The th side of is the interval from to . We know as (the sides of the suitcase decrease much faster than the ). And normalised by the standard deviation of also goes to infinity, i.e., . So the probability that lies inside the interval from to goes to zero. While the finite product of positive numbers is positive, the infinite product of positive numbers that decay to zero is zero: there is zero probability that a randomly chosen point will lie in .
- The sets and are disjoint.
Again, this is a consequence of the order of the limits. For a fixed , the th edge of the suitcase decays to zero faster than . So for sufficiently large, the th edges of and do not overlap, hence and are disjoint.
While the above gave one specific example, the general theory goes as follows. Given a Gaussian measure on with mean zero and covariance , the shifted Gaussian measure is either equivalent to (meaning the two measures are absolutely continuous with respect to each other), or the two measures are singular. The Cameron-Martin space is the space of all for which the two measures are equivalent. There is a simple expression for this space: it is , the image of under the operator .
We currently are running several reading groups for engineering students and one is on Lie Groups and Representation Theory. When elementary group theory was introduced yesterday, not surprisingly, there were many questions concerning normal subgroups. Like many areas of mathematics, the traditional learning paradigm is to “just memorise the definitions and later things will make sense”. While it is true that later things will make sense — once a critical mass of ideas and definitions has been accumulated, the connections between them can be seen, and the structure becomes self-supporting — we should still strive to make things as intuitive as possible right from the start. Below is one such attempt. It tries to introduce group theory based around several examples, one example at a time. Particular emphasis is placed on explaining normal and non-normal subgroups. Key points will be written in bold. (Caveat: The below captures what first came to mind and is likely to be unduly verbose.)
Free Group with Two Generators
Group theory came from the study of “transformations” of a space, and often, it is useful to think of each element of a group as something which “acts on” or “transforms” something. Later, we will consider permutation groups, which precisely fit this description. (In the reading group, we consider rotation groups, which also fit this description precisely.) Here, we consider a different group first, giving two equivalent definitions of it.
The algebraic definition is that our group consists of all “words” that can be written using the letters , , and , with the cancellation rule that whenever and appear next to each other, they can be omitted from the word, and the same for and . Besides the “empty word”, denoted , the group contains elements such as , , and so forth. The group operation (often written as multiplication) is just concatenation, so that times is .
Since is the same as by definition of our cancellation rule (later, a physical interpretation will be given), note that times equals .
The axioms of a group require the group operation to be associative, and it is clear that concatenation is associative, i.e., the order of doing the multiplications does not matter: ( (word one) times (word two)) times (word three) gives the same as (word one) times ((word two) times (word three)). For example, .
There must also be an identity element. In this case, this is the empty word , and it is clear that adding a “blank” to the start or the end of a word leaves the word unchanged, hence the empty word really is the identity element. (Time precludes me from writing down the group axioms and verifying them in detail.)
Lastly, to verify is a group, it must be shown that every element has an inverse. This is intuitively clear; for example, the inverse of is because . This example shows the inverse can be constructed one letter at a time, going from right to left, writing down the inverse of each letter. (Note that the blank word also results if the inverse had been multiplied on the left rather than the right: . If this were not true, then would not be a group, i.e., the axioms require that each element have a left inverse which is also a right inverse.)
While the above algebraic definition shows that this group, known as the free group with two generators, is easy to work with algebraically, it lacks a physical interpretation. This is now remedied by thinking of the free group with two generators as the fundamental group of the plane with two holes removed; don’t worry if that does not make any sense as I will explain the concept intuitively.
Imagine the floor of a room has an X marked on it, and to the left of the X is a vertical pole sticking out of the ground, and to the right there is another pole. A spider starts at X and can be commanded to do four things. Command “a” will make the spider walk from X clockwise around the pole on the left and return to X, all the while leaving a thin web behind him. (If you prefer, think of a robot laying down a rope.) Command “a inverse” will make the spider leave X, walk anti-clockwise around the pole on the left then return to X. Notice that if the spider does “a” then ”a inverse” the web can be shrunk or contracted back to X, so “a” followed by “a inverse” is the equivalent of “do nothing”, because we want to treat two web configurations as the same if one can be deformed into the other without breaking the web. (Technically, the web is allowed to pass through itself so we do not need to worry about knots forming; we simply care about what happens as the web shrinks to as short a length as possible.) Note that “a” followed by “a” gives a configuration (denoted ) that is different from just instructing the spider to do “a” (denoted ); the former has the web encircling the pole on the left twice while the latter has the web encircling the pole only once, and no amount of jiggling of the web will transform one configuration into the other.
Similarly, commands “b” and “b inverse” can be defined to mean walk clockwise or anti-clockwise around the pole on the right, respectively. The reader should pause to ensure the algebraic definition and this spider-web definition agree intuitively by doing several thought experiments and visualising the resulting web configuration.
A subgroup can be thought of as what results if the spider is no longer allowed to carry out arbitrary commands but can only carry out certain sequences of commands and their inverses. For example, perhaps the spider can only carry out the command and its inverse . Then the resulting subgroup consists of . (This subgroup is isomorphic to the group of integers.) Another subgroup is obtained by simply allowing the spider to carry out command “a” or its inverse. (This subgroup is also isomorphic to the group of integers.)
Let denote a subgroup of . Recall the definition of normality: is a normal subgroup of if, for every and , the so-called conjugate is in . This definition will have been motived several times before the end of this article, however, for now, we start with a high level if not somewhat vague explanation: a normal subgroup is one whose elements produce actions that do not get “tangled up” very much with all the other actions, as now discussed.
If is the subgroup generated by , meaning the elements of are precisely then is not normal. For example, cannot be written as an element of because there is no way for the actions of and to be exchanged: if first the spider is given the command to wrap a web around the right pole clockwise, then apply a command from , then wrap a web around the right pole anti-clockwise, there is no way for the web around the right pole to “cancel out” and leave a web that is only wrapped around the left pole; the subgroup is not normal.
Why is normality interesting? Before giving the standard explanation (i.e., the left cosets of a subgroup form a quotient group if and only if the subgroup is normal, and essentially equivalently, a subgroup is the kernel of a group homomorphism if and only if it is normal), a high level pragmatic explanation is proffered: when we are writing down sequences of group operations, we want to be able to simplify things, so we need some way of being allowed to interchange operations. In the simplest of cases, there are so-called Abelian groups which, by definition, are commutative: for all and in the group. The simplest examples of Abelian groups are vector spaces with the usual vector addition being treated as the group operation (and scalar multiplication is forgotten about, except for multiplication by which is the inverse group operator). In an Abelian group, , hence every subgroup is normal. This is the ultimate in “interchangeability”.
A normal subgroup has a property similar to, but a bit weaker than, full commutativity, in the following sense. If is normal, and , then although may not equal , I can achieve the next best thing: I can find a such that . That is, I have been able to interchange the action with an action from provided I do not mind using a possibly different action from . (Note that this follows immediately from the definition of normality: where is guaranteed to be in by normality. Note that can be replaced by in the definition of normality without changing anything.)
At this point, the reader should be comfortable in believing that being normal is a special property not shared by every subgroup, and if a subgroup is normal, then it is a useful thing, because it allows one to manipulate group operations to achieve simplifications.
The Symmetric Group on Three Letters
Consider three distinct balls arranged in a row. A robot can be asked to change their ordering: one action might be to swap the left-most ball with the right-most ball. The set of all distinct actions forms a group , called the symmetric group on three letters, with group multiplication taken to mean “first perform action then perform action ”, which again is a form of concatenation. Googling will bring up more than enough information about symmetric groups, so here I just wish to emphasise thinking in terms of each element corresponding to asking a robot to perform an action.
Let denote the subgroup of formed from the action “interchange the left-most and middle balls”. Denote this action by , and note that it is its own inverse. (I am using cycle notation.) This is another example of a non-normal subgroup: consider , with the notation meaning, first swap the left and right balls, then swap the left and middle balls, then swap the left and right balls. If the balls (red, green, blue) start off in the order RGB, then applying (13) gives BGR, applying (12) gives GBR and applying gives RBG. So overall, it transformed RGB to RBG. Note that the right ball has been changed, yet no command from can change the right ball, hence is not normal. This again is an illustration of the vague intuition that non-normal subgroups perform actions that tangle other actions up.
Besides the empty group and the whole group, it can be shown that the only other normal subgroup of is the so-called alternating group formed from all the even permutations.
Gaining further insight requires looking more closely at applications of normality. Perhaps the most germane to look at is whether a certain way of partitioning a group into equivalence classes leads to a compatible group structure on the equivalence classes. Recall is the subgroup of generated by (12). It turns out that any subgroup leads to a partition of the original group: where is a left coset. Crucially, for two distinct , either , or . (Intuitively why this is so will be explained later. For now, an algebraic proof is given: Assume there exist such that . For any , define , so that . The result follows if we can show belongs to . Note which is in because , and are all in .)
If we treat a vector space as a group, and take for a linear subspace, then would correspond to the affine subspaces obtained by translations of ; perhaps this is the simplest way of visualising left cosets. (Vector spaces form Abelian groups, hence all their subgroups are normal, and simply considering vector spaces as groups does not lead to insight into non-normal subgroups.)
It can be deduced fairly readily that the elements of are (), (12), (13), (23), (123), (132). (There are six elements because the left ball can be placed in one of three positions, the middle ball can be placed in one of two positions, and having done so, there is only one spot remaining for the right ball, i.e., there are possible re-arrangements.)
The cosets and are both just because the elements of are () and (12).
The coset consists of the elements (13) and (13)(12). The latter transforms RGB into BRG and hence is equivalent to (123). So the cosets and are the same; they contain the elements (13) and (123) of .
The coset consists of the elements (23) and (23)(12) = (132). Hence the cosets and are the same; they contain the elements (23) and (132) of .
This is therefore seen to partition into three sets called equivalence classes: .
Consider multiplying two elements from different equivalence classes of the partition. For example, (13) from the second equivalence class times (23) from the third equivalence class gives (13)(23)=(132), which lies in the third equivalence class. However, (123) from the second equivalence class times (132) from the third equivalence class gives (), which lies in the first equivalence class. Therefore, there is no consistent way of defining what it means to multiply the second equivalence class with the third equivalence class, since choosing different elements to represent each equivalence class can lead to different answers. (This should be compared with modulo arithmetic, where things work nicely.) Since is not normal, it tangles things up too much.
What is required for the equivalence classes to have a group structure? If are such that and , that is, and represent the same equivalence class, and the same for and , then we want to be the same as . This must be true in the special case when . If then must be an element of . (If then we can find such that , so that .) So a necessary condition is for when . Recall from earlier that if were normal then we could interchange the order in which and are multiplied, albeit by having to change to a different element of . Thus, if is normal then for some , hence , as required. With a little more care, it can be shown that a necessary and sufficient condition for left cosets to have a group structure, is for the subgroup to be normal. One way to see why normality suffices is by writing because, by normality, will equal . (This notation requires some justification which I do not elaborate on; treat it as a mnemonic, if you prefer.)
Why do Cosets form a Partition?
For Abelian groups, such as vector spaces, it is relatively clear why left cosets form equivalence classes. Why does this hold for arbitrary groups, when multiplication can tangle things up?
Let be a subgroup of . First note two things: if then , and if then . Proving the former is left as an easy exercise. The latter holds because, if is in , and if is in , then must also be in .
We can now appeal to symmetry. Groups were invented to study symmetry, and in the first instance, this means often we understand what is going on by shifting things back to the “origin”, meaning the identity element of the group. If we want to understand what and look like relative to each other, then we can premultiply them by . The key point is that this “transformation” is one-to-one and preserves structure, so we are allowed to do this without altering anything important. Thus, comparing with is essentially the same as comparing with . But we know from earlier that either is an element of , in which case and hence , or else is not an element of , in which case and are disjoint, and hence and are also disjoint.
Given that reproducing kernel Hilbert spaces (RKHS) behave more like Euclidean space than more general Hilbert spaces, it is somewhat surprising they are not better known. I have uploaded to arXiv a primer on reproducing kernel Hilbert spaces that differs from other introductory material in a number of ways.
It is perhaps the first to start by considering finite-dimensional spaces rather than jump straight to infinite-dimensional spaces. Surprisingly much can be gained by considering the finite-dimensional case first. It also does not assume any prior knowledge about Hilbert spaces and goes to great lengths to explain clearly about closed and non-closed subspaces and completions of spaces. (e.g., It is explained that a space which is not complete can be thought to have a ‘scratched surface’.)
Whereas other material refers to a RKHS theory as providing a “coordinate-free approach”, I argue that the presence of a Hilbert space already provides a coordinate-free approach, with the added benefit of a reproducing kernel Hilbert space being a canonical coordinate system associated to each point in the Hilbert space.
The primer explains that RKHS theory studies the configuration of an inner product space sitting inside a function space; it describes an extrinsic geometry because how the space is oriented inside the larger function space matters. This way of thinking leads to being able to determine when RKHS theory might be relevant to a problem at hand; if the solution of the problem changes continuously as the configuration changes continuously then it is possible the solution might be expressible in terms of the kernel in a nice way.
How RKHS theory can be used to solve linear equations is discussed along with some novel descriptions and extensions, as is the role of RKHS theory in signal detection.
My colleague contributed substantially by describing some of the recent work in the statistical learning theory literature, including explaining how RKHS theory can be used for testing for independence, for nonlinear filtering, and so forth.
This note is a fresh attempt at introducing tensor products in a simple and intuitive way. The setting is a vector space on which we wish to define a multiplication. Importantly, we do not wish to limit ourselves to a multiplication producing an element of . For example, we would like to think of and as two examples of multiplying elements of . The first multiplication, , is a map from to . The second multiplication, , is a map from to . In fact, since is well-defined even if and have different dimensions, the most general setting is the following.
Let and be fixed vector spaces agreed upon at the start. Assume someone has defined a multiplication rule but has not told us what it is. Here, is a function and is a vector space, both of which are unknown to us. For example, if and , a possible choice might be where .
What does it mean for to be a rule for multiplication? We wish it to have certain properties characteristic of multiplication. In a field, multiplication is involved in a number of axioms, including the distributive law . Since and are vector spaces, potentially distinct from each other, not all the axioms for multiplication in a field make sense in this more general setting. (For instance, commutativity makes no sense if .) The outcome is that is declared to be a rule for multiplication if it is bilinear: , and .
Not knowing does not prevent us from working with it; we simply write it as and are free to manipulate it according to the aforementioned rules. We do the equivalent all the time when we prove general results about a metric; we say “let be a metric” and then make use of only the axioms of a metric, thereby ensuring our derivation is valid regardless of which metric is actually being represented by .
While the tensor product is more than this, in the first instance, it suffices to treat it merely as an unspecified rule for multiplication. (It is specified, but no harm comes from forgetting about this in the first instance.) Rather than write it is more convenient to write , and thinking of as multiplication reminds us that .
The first point is that we can treat as an unspecified multiplication, simplify expressions by manipulating it in accordance with the rules for multiplication, and at the end of the day, if we are eventually told the rule , we can evaluate its simplified expression. In computer science parlance, this can be thought of as lazy evaluation.
Whereas there are many metrics “incompatible” with each other, the rules for multiplication are sufficiently rigid that there exists a “most general multiplication”, and all other multiplications are special cases of it, as explained presently. The tensor product represents this most general multiplication possible.
To consider a specific case first, take and to be . We already know two possible multiplications are and . The claim is that the latter represents the most general multiplication possible. This means, among other things, that it must be possible to write in terms of , and indeed it is: . Note that trace is a linear operator. The precise claim is the following. No matter what is, we can pretend it is , work with this definition, then at the very end, if we are told what actually is, we can obtain the true answer by applying a particular linear transform. At the risk of belabouring the point, if we wish to simplify then we can first pretend is to obtain . Later when we are told we can deduce that the linear map required to convert from to is . Applying this linear map to yields and this is indeed the correct answer because it equals .
Readers wishing to think further about the above example are encouraged to consider that the most general multiplication returning a real-valued number is of the form for some matrix . How can be obtained from ? What about for multiplication rules that return a vector or a matrix?
The general situation works as follows. Given vector spaces and , two things can be constructed, both of which use the same symbol for brevity. We need a space for our most general multiplication, and we denote this by . We also need a rule for multiplication and we denote this by . By definition, is an element of . (Just like not all matrices can be written as , there are elements in that cannot be written as . They can, however, be written as linear combinations of such terms.)
At this point, I leave the reader to look up elsewhere the definition of a tensor product and relate it with the above.
This short note presents a very simple and intuitive derivation that explains why the likelihood ratio is used for hypothesis testing.
Recall the standard setup: observed is a realisation of a random variable (or vector or process) generated either with probability density or .
The observation space is to be partitioned into two regions, one region labelled zero and the other unity. If lies in the region labelled zero then is assumed to have been generated under . The rough aim is to choose the partition so that if this “game” is repeated many times — choose to be 0 or 1; repeatedly generate samples from and count how many times they fall in the two regions — then the ratio of “correct outcomes” to “incorrect outcomes” is maximised, regardless of whether or was used. On average, we wish to be correct many more times than we are wrong.
Since there is a trade-off between being correct when and when , Neyman-Pearson’s underlying criterion is to fix the probability of being wrong when and maximise the probability of being correct when . (This part is assumed to be familiar to the reader.)
Mathematically, the region of the observation space must be found such that 1) the probability of the observation lying in under is some fixed (and small) value, while 2) the probability of the observation lying in under is as large as possible. (This region will be the region labelled unity.) This is a constrained optimisation problem involving a region of space. Pleasantly then, there is a simple way of solving it.
The trick is to imaging the observation space as broken up into many small tiles fitting together to cover the observation space. (If the observation space is drawn as a square then imagine dividing it into hundreds of small squares.) With this quantisation, finding the region (up to quantisation error) means finding which of the tiles we want to be part of the region .
Think of starting with being the empty set and growing it by adding tiles one at a time. The first tile we add will do two things: it will give us a which we want to keep as small as possible and it will give us a which we want to maximise. (At this point, some readers may wish to re-tile the region with tiles of possibly differing sizes, so that the probability under of landing on any given tile is the same fixed constant regardless of which tile was chosen.) It is intuitively clear (and even clearer after re-tiling) that the very first tile we should pick is the one which will make as large as possible. If has not exceeded the limit then we are allowed to pick another tile, and we pick the next best tile, that is, out of the tiles remaining, we want the tile that maximises the ratio . Each time, we greedily pick the best remaining tile that will help us make as large as possible while not letting grow too quickly.
As the size of the tiles shrinks to zero, the “limit” is the likelihood ratio for the “infinitesimal” tile centred at . Choosing the region for some constant corresponds precisely to choosing the tiles with the largest ratios; every tile with ratio better than is included.
The above is a hand-waving argument and is not the best on which to base a rigorous proof. Often this is the case though: an intuitive derivation is subsequently verified by a less intuitive but more elementary mathematical argument. An elementary proof can be found in the Wikipedia.