ADAPTIVE BAYESIAN INFERENCE ON THE MEAN OF AN INFINITE DIMENSIONAL NORMAL

DISTRIBUTION

By Eduard Belitser and Subhashis Ghosal^{∗}

Leiden University and University of Minnesota

We consider the problem of estimating the mean of an infinite dimensional Gaussian distribution from the Bayesian perspective. Under assumption that the unknown true mean satisfies a “smoothness condition”, we first derive the convergence rate of the posterior distribution for a certain choice of prior, relating this to the minimax counterpart result. Next, we consider the situation complicated by the fact that instead of one single model we have a sequence of nested models parameterized by a “smoothness” parameter q ranging over some countable set. The parameter q is unknown and, besides estimation of the mean, we encounter the problem of selecting a model. The Bayesian approach can handle this uncertainty in the model selection simply by replacing model choice with mixing over various models. We show that, under some conditions, the resulting hierarchical prior leads to the same convergence rate of the posterior as if we had a single model.

∗Most of the work was done when the second author was affiliated with the Free University of Amsterdam.

AMS 1991 subject classification. Primary 62G20; secondary 62C10, 62G05.

Key words and phrases. Adaptive Bayes procedure, convergence rate, minimax risk, posterior distribution, selection.

Running head. Adaptive Bayesian Inference.

1. Introduction

Suppose we observe an infinite dimensional random vector X = (X_{1}, X_{2}, …), where X_{i}’s are independent, X_{i} has distribution N(θ_{i}, n^{−1}), i = 1, 2, …, and θ = (θ_{1}, θ_{2}, …) ∈ ℓ_{2}, i.e., ∑_{i=1}^{∞} θ_{i}^{2} < ∞. The parameter θ is unknown and the goal is to make inference about θ. Let P_{θ} stand for the distribution of X; here and throughout, we suppress the dependence of X and P_{θ} on n. We shall write ∥ · ∥ for the ℓ_{2}-norm throughout the paper.

We also note that X may be thought of as the sample mean vector of the first n observations of an i.i.d. sample Y_{1}, Y_{2}, …, each taking values in ℝ^{∞} and distributed like Y = (Y_{1}, Y_{2}, …), where Y_{i}’s are independent, Y_{i} has distribution N(θ_{i}, 1), i = 1, 2, …, respectively. Since the sample mean is sufficient, these two formulations are statistically equivalent. To avoid complicated notations, we shall generally work with the first formulation. However, the latter interpretation will help us apply the general theory of posterior convergence developed by Ghosal, Ghosh and van der Vaart (2000) and meaningfully talk about almost sure convergence with respect to P_{θ}^{∞}, where P_{θ} now stands for the distribution of Y above.

The interest in the infinite dimensional normal model is partly due to its equivalence with the prototypical white noise model:

dX_{ϵ}(t) = f(t)dt + ϵ dW (t), 0 ≤ t ≤ 1,

where f(·) ∈ L_{2}([0, 1]) is the unknown signal, W (t) is a standard Wiener process and ϵ > 0 is a small parameter. The statistical estimation problem is to recover the signal f(t), based on the observation X_{ϵ}(t). The above model arises as the limiting experiment in some curve estimation problems such as in density estimation [Nussbaum (1996), Klemelä and Nussbaum (1998)] and regression model [Brown and Low (1996)], where ϵ = n^{−1/2} and n is the sample size.

Suppose that the functions {ϕ_{i}}_{i=1}^{∞} form an orthonormal basis in L_{2}[0, T]. Under the assumption that the unknown signal f belongs to the Hilbert space L_{2}[0, T], we can reduce this problem to the problem of estimating the mean θ = (θ_{1}, θ_{2}, …) ∈ ℓ_{2} of an infinite dimensional normal distribution. Indeed,

X_{i} = θ_{i} + ϵ ξ_{i}, i = 1, 2, …,

where X_{i} = ∫_{0}^{T} ϕ_{i}(t)dX_{ϵ}(t) and ξ_{i} = ∫_{0}^{T} ϕ_{i}(t)dW (t), so that the ξ_{i}’s are independent standard normal random variables. In so doing, we arrive at our original infinite dimensional Gaussian shift experiment with ϵ = n^{−1/2}. The signal f(t) is recovered from the basis expansion f(t) = ∑_{i=1}^{∞} θ_{i}ϕ_{i}(t) and the map relating f and θ is an isometric isomorphism.

Coming back to the infinite normal model, if a Bayesian analysis is intended, one assigns a prior to θ and looks at the posterior distribution. Diaconis and Freedman (1997) and Freedman (1999) considered the independent normal prior N(0, τ_{i}^{2}) for θ_{i}, i = 1, 2, …, and comprehensively studied one infinite dimensional functional ∥θ − θ̂∥^{2}, where θ̂ is the Bayes estimator, both from the Bayesian and the frequentist perspectives. As a consequence of their main results, they established the frequentist consistency of the Bayes estimator for all θ ∈ ℓ_{2} if ∑_{i=1}^{∞} τ_{i}^{2} < ∞. Another peculiar implication is that the Bernstein-von Mises theorem fails to hold for this simple infinite dimensional problem.

This estimation problem was first studied in the minimax setting by Pinsker (1980). The minimax frequentist counterpart is as follows: if the unknown infinite dimensional parameter θ is assumed to belong to an ellipsoid Θ_{q}(Q) = {θ : ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} ≤ Q}, then the exact asymptotics of the minimax quadratic risk over the ellipsoid Θ_{q}(Q) is

(1.1)  lim_{n→∞} inf ; θ^{}ˆsup_{θ∈Θ_{q}(Q)} n^{2q/(2q+1)}E_{θ}∥θ̂ − θ∥^{2} = Q^{1/(2q+1)}γ(q) ,

where γ(q) = (2q + 1)^{1/(2q+1)}(q/(q + 1))^{2q/(2q+1)} is the Pinsker constant [Pinsker (1980)] and E_{θ} denotes the expectation with respect to the probability measure P_{θ} generated by X given θ.

Zhao (1998b) considered the independent normal prior with τ_{i}^{2} = τ_{i}^{2}(q) = i^{−(2q+1)}, i = 1, 2, … [considered also in Cox (1993) and Diaconis and Freedman (1997)], and showed that the posterior mean attains the minimax rate n^{−q/(2q+1)}. We write Π_{q} to denote this prior. In turn, we show in Theorem 2.1 that the posterior distribution Π_{q}(·|X) obtained from the prior Π_{q}, also converges at the rate n^{−q/(2q+1)} in the sense that, for any sequence M_{n} → ∞,

Π_{q}{n^{q/(2q+1)}∥θ − θ_{0}∥ > M_{n}| X} → 0, (1.2)

in P_{θ_{0}}-probability as n → ∞, where the true value of the parameter θ_{0} belongs to the linear subspace Θ_{q} = {θ : ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} < ∞}. It is long known that for finite dimensional models, posterior distribution converges at the classical n^{−1/2} rate, but for infinite dimensional models, results have only been obtained recently by Ghosal, Ghosh and van der Vaart (2000) and Shen and Wasserman (1998).

Our main goal in the present paper is to study a more complicated situation when the smoothness parameter q is not known, using an adaptive Bayesian approach to the problem. Precisely, instead of one single model θ ∈ Θ_{q}, we have a nested collection of models θ ∈ Θ_{q}, q ∈ Q. An adaptive Bayesian approach consists in constructing a prior that satisfies (1.2) and is not based on knowledge of q, that is the same prior works irrespective of the smoothness condition. Putting a prior on smoothness parameter q leads to mixing over candidate models. Mixture models are attractive since they can accommodate various ranges of models. So, we consider a mixture of Π_{q}’s over q, i.e., a prior of the form ∑ λ_{q}Π_{q}, where λ_{q} > 0 and ∑ λ_{q} = 1 and the summation is over taken some countable set. We show that, the resulting hierarchical prior leads to the same convergence rate of posterior as if we had a single model. Such a technique was also used by Ghosal and van der Vaart (1999) in the context of adaptive Bayesian density estimation.

The problem of adaptive estimation in minimax sense were first studied by Efromovich and Pinsker (1984). They proposed an estimator of θ which attains the asymptotics (1.1) without knowledge of the smoothness parameter q. Their method is based on adaptive determining optimal damping coefficients in an orthogonal series estimator.

The organization of the paper is as follows. In the next section we first study the case when the smoothness parameter is known. Then we formulate the main result of the paper concerning more complicated situation when the smoothness parameter is not known. The proof of the main result is based on several auxiliary lemmas presented in this section as well, among which the Selection Lemma is of interest on its own. At the end of the section we discuss the case of continuous parameter spectrum and a uniformity issue. Sections 3 and 4 present the proofs of the Selection Lemma and the other auxiliary lemmas respectively.

2. Main results

From now on, all symbols O and o refer to the asymptotics n → ∞ unless otherwise specified. By ⌊x⌋, we shall mean the greatest integer less than or equal to x.

Recall that we have independent observations X_{i} distributed as N(θ_{i}, n^{−1}), i = 1, 2, . . .. Suppose that the unknown parameter θ = (θ_{1}, θ_{2}, …) belongs to the set Θ_{q} = {θ : ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} < ∞}. First we study a non-adaptive version of the problem.

2.1. Known smoothness

Suppose that the smoothness parameter q is known. Let θ_{0} = (θ_{10}, θ_{20}, …) denote the true value of the parameter so that we have θ_{0} ∈ Θ_{q}. We apply Bayesian analysis with the prior Π_{q}: θ_{i}’s are independent and normally distributed with means zero and variances i^{−(2q+1)} respectively, so we use here our prior knowledge of q which corresponds to non-adaptive situation. It follows immediately that the posterior distribution of θ given X is defined as follows: θ_{i}’s are independent and normally distributed with means nX_{i}/(n + i^{2q+1}) and variances (n + i^{2q+1})^{−1} respectively. As it should be, the posterior distribution of θ_{i} depends on X only through X_{i}. Zhao (1998a) showed that the posterior mean θ̂ = (θ̂_{1}, θ̂_{2}, …), given by θ̂_{i} = nX_{i}/(n + i^{2q+1}), i = 1, 2, …, converges to θ_{0} at the rate n^{−q/(2q+1)}. The following theorem shows that the posterior distribution also converges at this rate.

Theorem 2.1. For any sequence M_{n} → ∞, (1.2) holds.

Proof. Applying Chebyshev’s inequality, the posterior probability in (1.2) is at most

M_{n}^{−2}n^{2q/(2q+1)} ∑_{i=1}^{∞} ((θ̂_{i} − θ_{i0})^{2} + var(θ_{i}|X_{i})) . (2.1)

It suffices to show that the P_{θ_{0}}-expectation of the above expression tends to zero. Now, var(θ_{i}|X_{i}) = (n + i^{2q+1})^{−1} and it is easy to see that

∑_{i=1}^{∞} var(θ_{i}|X_{i}) = ∑_{i=1}^{∞} (n + i^{2q+1})^{−1} = O(n^{−2q/(2q+1)}). (2.2)

To see this, let k = ⌊n^{1/(2q+1)}⌋ and split the sum into two terms, sums over the ranges {i : 1 ≤ i ≤ k} and {i : i > k} respectively. The first term is then bounded by k/n and the second by ∑_{i=k+1}^{∞} i^{−(2q+1)} ≤ (2q)^{−1}k^{−2q}. Both terms are thus of the order O(n^{−2q/(2q+1)}) and (2.2) follows.

For the first term in (2.1), note that E_{θ_{0}} θ̂_{i} = nθ_{i0}/(n + i^{2q+1}) and therefore

E_{θ_{0}}(θ̂_{i} − θ_{i0})^{2} = var_{θ_{0}}(θ̂_{i}) + (E_{θ_{0}} θ̂_{i} − θ_{i0})^{2}
= \frac{n^{2} var_{θ_{0}}(X_{i})}{(n + i^{2q+1})^{2}} + (\frac{nθ_{i0}}{n + i^{2q+1}} − θ_{i0})^{2}
= \frac{n}{(n + i^{2q+1})^{2}} + \frac{i^{4q+2}θ_{i0}^{2}}{(n + i^{2q+1})^{2}} .

Next we observe that

∑_{i=1}^{∞} \frac{n}{(n + i^{2q+1})^{2}} = O(n^{−2q/(2q+1)}) (2.3)

and

∑_{i=1}^{∞} \frac{i^{4q+2}θ_{i0}^{2}}{(n + i^{2q+1})^{2}} = O(n^{−2q/(2q+1)}). (2.4)

Equation (2.3) follows since the left hand side (LHS) of (2.3) is bounded by the expression in the middle of (2.2). For (2.4), we again split the sum into two sums over {i : 1 ≤ i ≤ k} and {i : i > k}, with k = ⌊n^{1/(2q+1)}⌋. Since ∑_{i=1}^{∞} i^{2q}θ_{i0}^{2} < ∞, the first sum is then bounded by

\frac{k^{2q+2}}{n^{2}} ∑_{i=1}^{k} i^{2q}θ_{i0}^{2} = O(n^{−2q/(2q+1)})

and the second by

(k + 1)^{−2q} ∑_{i=k+1}^{∞} i^{2q}θ_{i0}^{2} = O(n^{−2q/(2q+1)}) .

By combining all these assertions, the result follows easily.

Remark 2.1. One can [essentially by the same computations as in Zhao (1998b)] show that the above prior Π_{q} is the best choice among independent normal priors with power-variance structure in the sense that the resulting convergence rate of the posterior is the fastest one. Indeed, if θ_{i} ∼ N(0, i^{−2p}), then var(θ_{i}|X_{i}) = (n + i^{2p})^{−1} so that ∑_{i=1}^{∞} var(θ_{i}|X_{i}) = O(n^{−1+1/(2p)}) and var_{θ_{0}}(θ̂_{i}) = O(n^{−1+1/(2p)}) for p > 1/2, otherwise both are infinite. Let ∑_{i=1}^{∞} i^{2q}θ_{i0}^{2} = C_{q}. Then some algebra gives

(E_{θ_{0}} θ̂_{i} − θ_{i0})^{2} = ∑_{i=1}^{∞} \frac{i^{4p}θ_{i0}^{2}}{(n + i^{2p})^{2}} ≤ C_{q} max_{i} \frac{i^{4p−2q}}{(n + i^{2p})^{2}} = O(n^{−q/p})

for p > q/2, otherwise (E_{θ_{0}} θ̂_{i} − θ_{i0})^{2} = O(n^{−1}). Therefore, similar to Theorem 2.1, one can derive the convergence of the posterior at the rate n^{α} with α = min{1 − 1/(2p), q/p} for p > 1/2. Then the best possible choice is max_{p>1/2} min{1−1/(2p), q/p} = 2q/(2q +1) with optimal p = (2q + 1)/2.

Remark 2.2. Interestingly, as Zhao (1998b) pointed out, the above prior, although being the right prior for θ in a sense (see Remark 2.1 above), has a peculiar property. Both the prior and the posterior assign zero probability to Θ_{q}. This is an easy consequence of the criterion for summability of a random series. It is also interesting to note that summability of ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} is barely missed by the prior since ∑_{i=1}^{∞} i^{2p}θ_{i}^{2} < ∞ almost surely for any p < q; therefore q = sup{p : ∑_{i=1}^{∞} i^{2p}θ_{i}^{2} < ∞ almost surely}. To fix this, Zhao (1998a) proposed a compound prior ∑_{k=1}^{∞} w_{k}π_{k}, where under π_{k}, θ_{i}’s are independent with distribution N(0, i^{−(2q+1)}) for i = 1, …, k, and degenerate at 0 otherwise, and w_{k}’s are weights bounded from below by a sequence exponentially decreasing in k. She showed that on the one hand, the posterior mean attains the minimax rate and on the other hand assigns probability one to Θ_{q}.

2.2. Discrete spectrum adaptation

So far our approach relies heavily on the fact that we know the smoothness, i.e., we have chosen a single model as the “correct” one from the collection of possible models corresponding to different choices of q > 0. In general, when the parameter specifying the model is not chosen correctly, this may lead to suboptimal or inconsistent procedures, since further analysis does not take into account the possibility of other models.

The prior Π_{q} used in Theorem 2.1 is appropriate only if we know the smoothness parameter q correctly, i.e., we know that θ_{0} ∈ Θ_{q} and the assertion holds for no larger value of q. If actually θ_{0} ∈ Θ_{q′} for some q′ > q, then we lose in terms of the rate of convergence since we could have obtained the rate n^{−q′/(2q′+1)} by using the prior Π_{q′}. On the other hand, if θ_{0} ∈ Θ_{q′}, q′ < q only, then the model is misspecified. In general, one expects inconsistency in this type of situations. In this case however, since all the models are dense in ℓ_{2}, the posterior is nevertheless consistent. Still, there is a loss in rate of convergence again — we only have the rate n^{−q′/(2q+1)} instead of the rate n^{−q′/(2q′+1)} achievable by the prior Π_{q′}. The rate n^{−q′/(2q+1)} may be obtained by proceeding as in the proof of Theorem 2.1. All the estimates other than (2.4) remain valid, and it is easily seen that the LHS of (2.4) is O(n^{−q′/(2q+1)}).

Therefore, we intend to present a prior Π that achieves the posterior rate of convergence n^{−q/(2q+1)} at θ_{0} whenever θ_{0} ∈ Θ_{q} for different values of the smoothness parameter q. Let Q = {…, q_{−1}, q_{0}, q_{1}, …} be a countable subset of the positive real semiaxis arranged in an increasing sequence, i.e., 0 < … < q_{−1} < q_{0} < q_{1} < . . .. We show that there is a prior Π for θ such that whenever θ_{0} ∈ Θ_{q}, q ∈ Q, the assertion (1.2) holds.

We may think of q as another unknown parameter. So, instead of one single model, we have a sequence of nested models parameterized by a “smoothness” parameter q ranging over some countable set. Bayesian approach can handle this uncertainty in the model selection simply by replacing model choice with mixing over various models, i.e., by putting a prior on this “hyperparameter” q. The resulting prior Π is therefore the following two-level hierarchical prior: given q, the θ_{i}’s are independent normally distributed with zero means and variances i^{−(2q+1)} respectively; q is distributed according to a probability measure λ. Let λ_{m} = λ(q = q_{m}) and henceforth, we shall write Π_{m} for Π_{q_{m}}. We thus have Π = ∑_{m=−∞}^{∞} λ_{m}Π_{m}. Throughout, we shall assume that λ_{m} > 0 for all m.

Without loss of generality, we may assume that θ_{0} ∈ Θ_{q_{0}}. Otherwise, we simply relabel. For the sake of brevity, denote r_{n} = r_{n}(q_{0}) = n^{q_{0}/(2q_{0}+1)}.

Theorem 2.2. For any sequence M_{n} → ∞, the posterior probability

Π{r_{n}∥θ − θ_{0}∥ > M_{n}|X} → 0 (2.5)

in P_{θ_{0}}-probability as n → ∞.

Since q serves as a parameter, we can talk about its posterior distribution. The idea behind the proof is first to show that the posterior probability of q being smaller than q_{0} is small for large n in P_{θ_{0}}-probability, i.e., probability of selecting a less smooth model from the posterior is small. This effectively reduces the prior to the form ∑_{m≥0} λ_{m}Π_{m}. For such a prior we then show that the posterior probability of {θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B} is small for a sufficiently large B. Therefore the effective parameter space is further reduced to {θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B} for which the general theory developed by Ghosal, Ghosh and van der Vaart (2000) applies.

The following lemma, the Selection Lemma, shows that the conditional probability of misspecifying the model in favor of a less smooth model (than the true one) converges to zero.

Lemma 2.1 (Selection Lemma). There exist integer N and a constant c > 0 such that for any m < 0 and n > N,

E_{θ_{0}}Π(q = q_{m}|X) ≤ \frac{λ_{m}}{λ_{0}} exp [ − cn^{1/(q_{0}+q_{m}+1)}] (2.6)

and therefore Π(q < q_{0}|X) → 0 (2.7)

in P_{θ_{0}}-probability as n → ∞.

The proof of the Selection Lemma is somewhat lengthy and involves some messy calculations. We defer it to the next section.

Remark 2.3. Interestingly, with the second interpretation of the model as described in the introduction, we can also conclude that the above convergence holds in the almost sure sense. We, however, do not utilize this fact further.

According to the Selection Lemma, the posterior mass is asymptotically concentrated on q ≥ q_{0}. Introduce λ̄_{m} = λ_{m}/ ∑_{j=0}^{∞} λ_{j}, and Π̄ = ∑_{m=0}^{∞} λ̄_{m}Π_{m}. The next lemma shows that the posterior Π̄(·|X) is effectively concentrated on the set {θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B} for a large B. The proof of this lemma is postponed to Section 4.

Lemma 2.2.

(2.8)  lim_{B→∞} sup_{n≥1} E_{θ_{0}} Π̄{θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X} = 0.

Finally, the following lemma shows convergence of posterior at rate r_{n}^{−1}. In the proof which is given in Section 4, we exploit the general theory of posterior convergence developed by Ghosal, Ghosh and van der Vaart (2000).

Lemma 2.3. For any B > 0 and M_{n} → ∞,

lim_{n→∞} E_{θ_{0}} Π̄{θ : r_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B|X} = 0. (2.9)

Proof of Theorem 2.2. Clearly,

Π{r_{n}∥θ − θ_{0}∥ > M_{n}|X} = Π{r_{n}∥θ − θ_{0}∥ > M_{n}, q ≥ q_{0}|X} + Π{r_{n}∥θ − θ_{0}∥ > M_{n}, q < q_{0}|X} = Π̄{r_{n}∥θ − θ_{0}∥ > M_{n}|X}Π(q ≥ q_{0}|X) + Π{r_{n}∥θ − θ_{0}∥ > M_{n}, q < q_{0}|X}

≤ Π̄{r_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X}

+ Π̄{r_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B|X} + Π(q < q_{0}|X)

≤ Π̄{ ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X} + Π̄{r_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B|X}

+ Π(q < q_{0}|X).

Given ϵ > 0, B can be chosen sufficiently large to make the first term less that ϵ by Lemma 2.2. For this B, apply Lemma 2.3 to the second term. The last term goes to zero in P_{θ_{0}}-probability by the Selection Lemma.

Remark 2.4. In place of Π_{m}, Zhao’s (1998b) mixture prior described above in Remark 2.2 may also possibly be used. However, we do not pursue this approach for the following reasons. First, this will further complicate the already enough complicated expressions. Secondly, we think that the property that Π_{m} assigns zero probability to Θ_{q} is not as bad as it might first appear since, after all, Θ_{q} is not closed, and its closure, the whole of ℓ_{2}, obviously receives the whole mass. Indeed, ℓ_{2} is the support of Π_{m}. Finally, the criterion that “Θ_{q} should receive the whole mass” loses much of its original motivation in the adaptive context.

2.3. Continuous spectrum adaptation

Suppose now that q_{0} ∈ Q, where Q is an arbitrary subset of the positive semiaxis ℝ_{+}. Since our basic approach relies on the fact that the smoothness parameter can only take countably many possible values, we proceed as follows. Take a countable dense (in Q) set Q^{∗}, Q^{∗} ⊂ ℝ_{+}, and put a prior λ on it such that λ(q = s) > 0 for each s ∈ Q^{∗}. The true q_{0} may not belong to Q^{∗}, but for any q′ < q_{0} there exists q^{∗} ∈ Q^{∗} such that q′ < q^{∗} < q_{0}. Since θ_{0} ∈ Θ_{q_{0}} implies that θ_{0} ∈ Θ_{q^{∗}}, one would like to derive the analogue of Theorem 2.2, with q^{∗} and r_{n}^{∗} = r_{n}(q^{∗}) = n^{q^{∗}/(2q^{∗}+1)} instead of q_{0} and r_{n} respectively. This would mean in fact that, by a slight sacrifice in the convergence rate, the main result holds for the case of continuous parameter spectrum as well. However, the Selection Lemma fails, although the rest of the proof carries through. This is because q^{∗} is now not strictly separated from the the set {q ∈ Q^{∗} : q < q^{∗}}. It is intuitively clear that the requirement of being strictly separated stems from the fact that the closer possible values of smoothness parameter are, the more difficult it is to distinguish between them.

We can fix the Selection Lemma by a further slight sacrifice in the convergence rate. Indeed, for any q′ < q_{0} we pick q̄ ∈ Q^{∗} such that q′ < q̄ < q^{∗} < q_{0}. Denote λ_{s} = λ(q = s) for each s ∈ Q^{∗} and r̄_{n} = n^{q̄/(2q̄+1)}, the prior Π̄ = ∑_{q∈Q^{∗},q≥q̄} λ̄_{q}Π_{q} with λ̄_{s} = λ_{s}/ ∑_{q∈Q^{∗},q≥q̄} λ_{q}. Then, similar to the proof of Theorem 2.2, we obtain the following bound:

Π{r̄_{n}∥θ − θ_{0}∥ > M_{n}|X}

≤ Π̄{ ∑_{i=1}^{∞} i^{2q̄}θ_{i}^{2} > B|X} + Π̄{r̄_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q̄}θ_{i}^{2} ≤ B|X}

+ Π̄(q < q̄|X).

By Lemmas 2.2 and 2.3 with q_{0} replaced by q̄, the first two terms on the right hand side (RHS) of the last inequality can be handled in the same way as in Theorem 2.2. The last term goes to zero in P_{θ_{0}}-probability by the Selection Lemma by noting that θ_{0} ∈ Θ_{q^{∗}} and the set {q : q < q̄} is separated from q^{∗}. It follows therefore that

Π{r̄_{n}∥θ − θ_{0}∥ > M_{n}| X} → 0 ,

in P_{θ_{0}}-probability as n → ∞. Since q̄ could be chosen arbitrarily close to q_{0} from the support of the prior, this means that given any δ > 0, the posterior converges at least at the rate n^{−(q_{0}/(2q_{0}+1))+δ} in ℓ_{2}.

2.4. Uniformity over ellipsoids

It is of some interest to know to what extent the posterior convergence in Theorems 2.1 and 2.2 (and the continuous spectrum version of Theorem 2.2) is uniform over θ_{0}. For Theorem 2.1, it is immediately seen from the proof that (1.2) holds uniformly over an ellipsoid, i.e., for any Q ≥ 0 and M_{n} → ∞,

sup_{θ_{0}∈Θ_{q}(Q)} E_{θ_{0}}Π_{q}{n^{q/(2q+1)}∥θ − θ_{0}∥ > M_{n}| X} → 0. (2.10)

For Theorem 2.2, such a proposition is much more delicate. One needs to check uniformity in each of the Lemmas 2.1, 2.2 and 2.3. Going through the proof of Lemma 2.2, it is easily seen that

(2.11)  lim_{B→∞} sup_{θ_{0}∈Θ_{q_{0}}(Q)} sup_{n≥1} E_{θ_{0}} Π̄{θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X} = 0.

For the uniform version of Lemma 2.3, we need to show that for any B > 0,

(2.12)  lim_{n→∞} sup_{θ_{0}∈Θ_{q_{0}}(Q)} E_{θ_{0}} Π̄{θ : r_{n}∥θ − θ_{0}∥ > M_{n}, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B|X} = 0.

To this end, we first note that (4.11) in Theorem 4.1 could have been stated as an inequality

E_{θ_{0}}Π\{P ∈ P̄ : H(P_{0}, P) ≥ Mϵ_{n}|Y_{1}, Y_{2}, …, Y_{n}\} ≤ Ce^{−cnϵ_{n}^{2}} + \frac{1}{nϵ_{n}^{2}} ,

where C and c are positive constants depending on c_{1}, c_{2} and c_{3} [see Ghosal, Ghosh and van der Vaart (2000)]. One would therefore obtain (2.12) whenever it will be possible to choose the same c_{1}, c_{2} and c_{3} for different θ_{0}’s. The entropy equation (4.9) does not depend on the true value θ_{0}, so c_{1} is free of θ_{0}. It is possible to choose the same c_{2} and c_{3} for θ_{0} belonging to the ellipsoid Θ_{q}(Q) by the estimates given by Lemmas 4.2 and 4.3. Therefore it remains to consider the uniform version of the Selection Lemma.

However, (2.7) in the Selection Lemma does not seem to hold uniformly over θ_{0} ∈ Θ_{q_{0}}(Q) for a given Q > 0. This is because for θ_{0} ∈ Θ_{q_{0}}(Q), the LHS of (3.7) may tend to zero arbitrarily slowly making the integer N_{3} defined there dependent on θ_{0}. Nevertheless, one can see that for a given θ^{∗} = (θ_{1}^{∗}, θ_{2}^{∗}, …), it is possible to choose an ϵ > 0 such that

(2.13)  lim_{n→∞} sup_{θ_{0}∈E_{q_{0}}(θ^{∗},ϵ)} E_{θ_{0}}Π(q < q_{0}|X) → 0,

where E_{q_{0}}(θ^{∗}, ϵ) = {θ_{0} : ∑_{i=1}^{∞} i^{2q_{0}}(θ_{i0} − θ_{i}^{∗})^{2} < ϵ}. Combining all these above assertions, we conclude that the conclusion of Theorem 2.2 is uniform over small ellipsoids, i.e., given any θ^{∗}, there exists ϵ > 0 such that

sup_{θ_{0}∈E_{q_{0}}(θ^{∗},ϵ)} E_{θ_{0}}Π_{q}{n^{q/(2q+1)}∥θ − θ_{0}∥ > M_{n}| X} → 0. (2.14)

As a result, the uniformity holds over all compact sets in the topology generated by the norm (∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2})^{1/2} on Θ_{q_{0}}.

Similarly, one can formulate the uniform version of posterior convergence for the case of continuous spectrum.

3. Proof of the Selection Lemma

In this section, we prove the Selection Lemma. We begin with a couple of preliminary lemmas.

We recall that given θ, X_{i}’s are independent and distributed as N(θ_{i}, n^{−1}) respectively, and given q = q_{m}, θ_{i}’s are independent and distributed as N(0, i^{−(2q_{m}+1)}) respectively. Therefore, given q = q_{m}, the marginal distribution of X is given by the countable product of N(0, n^{−1} + i^{−(2q_{m}+1)}), i = 1, 2, . . .. Let us denote this measure by P_{m}. Let P_{λ} = ∑_{m=−∞}^{∞} λ_{m}P_{m}, the marginal distribution of X when q is distributed according to λ. The true distribution of X is however P_{θ_{0}}, the countable product of N(θ_{i0}, n^{−1}), i = 1, 2, . . ..

Lemma 3.1. For any n ≥ 1 and θ_{0} ∈ ℓ_{2}, P_{λ} and P_{θ_{0}} are mutually absolutely continuous.

Proof. It suffices to show that for any m, P_{m} and P_{θ_{0}} are mutually absolutely continuous.

Abbreviate q_{m} by q. Recall that the affinity between two probability densities f(x) and g(x) on the real line is defined by A(f, g) = ∫ \sqrt{f(x)g(x)}dx. The affinity between two normal densities N(μ_{1}, σ_{1}^{2}) and N(μ_{2}, σ_{2}^{2}) is therefore given by

[1 − \frac{(σ_{1} − σ_{2})^{2}}{σ_{1}^{2} + σ_{2}^{2}} ]^{1/2} exp \{ − \frac{(μ_{1} − μ_{2})^{2}}{4(σ_{1}^{2} + σ_{2}^{2})}\}.

Since both P_{m} and P_{θ_{0}} are product measures, according to Kakutani’s criterion [see, e. g., Williams (1991, page 144)], we need only to show that the infinite product

∏_{i=1}^{∞} A(N(0, n^{−1} + i^{−(2q+1)}), N(θ_{i0}, n^{−1}))

does not diverge to zero. Clearly it is sufficient to verify the convergence of the infinite products

∏_{i=1}^{∞} \{1 − \frac{((n^{−1} + i^{−(2q+1)})^{1/2} − n^{−1/2})^{2}}{2n^{−1} + i^{−(2q+1)}} \} (3.1)

and

∏_{i=1}^{∞} exp \{ − \frac{θ_{i0}^{2}}{4(2n^{−1} + i^{−(2q+1)})}\}. (3.2)

A product of the form ∏_{i=1}^{∞}(1 − a_{i}), a_{i} > 0, i = 1, 2, …, converges if ∑_{i=1}^{∞} a_{i} converges. Also, (a − b)^{2} ≤ a^{2} − b^{2} for a ≥ b ≥ 0. Therefore, (3.1) converges as

∑_{i=1}^{∞} \frac{i^{−(2q+1)}}{2n^{−1} + i^{−(2q+1)}} ≤ \frac{n}{2} ∑_{i=1}^{∞} i^{−(2q+1)} < ∞.

The product in (3.2) also converges since

∑_{i=1}^{∞} \frac{θ_{i0}^{2}}{8n^{−1} + 4i^{−(2q+1)}} ≤ \frac{n}{8} ∑_{i=1}^{∞} θ_{i0}^{2} < ∞.

The lemma is proved.

The following lemma estimates the posterior probability of the mth model.

Lemma 3.2. For any integers m and n ≥ 1,

E_{θ_{0}}Π(q = q_{m}|X) ≤ \frac{λ_{m}}{λ_{0}} exp [\frac{1}{2} ∑_{i=1}^{∞} \frac{(i^{−(2q_{0}+1)} − i^{−(2q_{m}+1)})(i^{−(2q_{0}+1)} − θ_{i0}^{2})}{i^{−2(q_{0}+q_{m}+1)} + 2n^{−1}i^{−(2q_{0}+1)} + n^{−2}}]. (3.3)

Proof. By the martingale convergence theorem [see, e.g., Williams (1991, page 109)],

Π(q = q_{m}|X) = lim_{k→∞} Π(q = q_{m}|X_{1}, …, X_{k}) a.s. [P_{λ}].

Therefore, by Lemma 3.1, Π(q = q_{m}|X_{1}, …, X_{k}) converges, as k → ∞, to Π(q = q_{m}|X) a.s. [P_{θ_{0}}]. Since these, being probabilities, are bounded by 1, the convergence also takes place in L_{1}(P_{θ_{0}}). By Bayes theorem,

Π(q = q_{m}|X_{1}, …, X_{k})

= \frac{λ_{m} ∏_{i=1}^{k}(n^{−1} + i^{−(2q_{m}+1)})^{−1/2} exp [ − \frac{1}{2} ∑_{i=1}^{k}(n^{−1} + i^{−(2q_{m}+1)})^{−1}X_{i}^{2}]}{∑_{l=−∞}^{∞} λ_{l} ∏_{i=1}^{k}(n^{−1} + i^{−(2q_{l}+1)})^{−1/2} exp [ − \frac{1}{2} ∑_{i=1}^{k}(n^{−1} + i^{−(2q_{l}+1)})^{−1}X_{i}^{2}]}

≤ \frac{λ_{m} ∏_{i=1}^{k}(n^{−1} + i^{−(2q_{m}+1)})^{−1/2} exp [ − \frac{1}{2} ∑_{i=1}^{k}(n^{−1} + i^{−(2q_{m}+1)})^{−1}X_{i}^{2}]}{λ_{0} ∏_{i=1}^{k}(n^{−1} + i^{−(2q_{0}+1)})^{−1/2} exp [ − \frac{1}{2} ∑_{i=1}^{k}(n^{−1} + i^{−(2q_{0}+1)})^{−1}X_{i}^{2}]}.

Put a_{i} = (n^{−1} + i^{−(2q_{m}+1)})^{−1} − (n^{−1} + i^{−(2q_{0}+1)})^{−1}. Exploiting the independence of X_{i}’s under P_{θ_{0}} and using

E( exp [ − \frac{α}{2}X^{2}]) = (1 + ασ^{2})^{−1/2} exp [\frac{μ^{2}}{2σ^{2}} \{(1 + ασ^{2})^{−1} − 1\}]

for X distributed as N(μ, σ^{2}) and α > −σ^{−2}, we obtain

E_{θ_{0}}Π(q = q_{m}|X)

= lim_{k→∞} E_{θ_{0}}Π(q = q_{m}|X_{1}, …, X_{k})

≤ \frac{λ_{m}}{λ_{0}} lim sup_{k→∞} ∏_{i=1}^{k} \{(\frac{n^{−1} + i^{−(2q_{0}+1)}}{n^{−1} + i^{−(2q_{m}+1)}} )^{1/2}E_{θ_{0}}( exp [ − \frac{a_{i}}{2} X_{i}^{2}])\}

= \frac{λ_{m}}{λ_{0}} ∏_{i=1}^{∞} \{(\frac{n^{−1} + i^{−(2q_{0}+1)}}{n^{−1} + i^{−(2q_{m}+1)}} )^{1/2}(1 + \frac{a_{i}}{n} )^{−1/2} exp [\frac{n}{2}((1 + \frac{a_{i}}{n} )^{−1} − 1)θ_{i0}^{2}]\}

= \frac{λ_{m}}{λ_{0}} ∏_{i=1}^{∞} \{(1 + \frac{a_{i}i^{−(2q_{0}+1)}}{1 + \frac{a_{i}}{n}} )^{1/2} exp [ − \frac{a_{i}}{2(1 + \frac{a_{i}}{n})}θ_{i0}^{2}]\}

≤ \frac{λ_{m}}{λ_{0}} exp [\frac{1}{2} ∑_{i=1}^{∞} \frac{a_{i}}{1 + \frac{a_{i}}{n}} (i^{−(2q_{0}+1)} − θ_{i0}^{2})]

= \frac{λ_{m}}{λ_{0}} exp [\frac{1}{2} ∑_{i=1}^{∞} \frac{(i^{−(2q_{0}+1)} − i^{−(2q_{m}+1)})(i^{−(2q_{0}+1)} − θ_{i0}^{2})}{i^{−2(q_{0}+q_{m}+1)} + 2n^{−1}i^{−(2q_{0}+1)} + n^{−2}}].

The last three steps follow from some algebra and the inequality 1 + x ≤ e^{x} for all x.

Remark 3.1. By exactly the same arguments, we can also conclude that for any l,

E_{θ_{0}}Π(q = q_{m}|X) ≤ \frac{λ_{m}}{λ_{l}} exp [\frac{1}{2} ∑_{i=1}^{∞} \frac{(i^{−(2q_{l}+1)} − i^{−(2q_{m}+1)})(i^{−(2q_{l}+1)} − θ_{i0}^{2})}{i^{−2(q_{l}+q_{m}+1)} + 2n^{−1}i^{−(2q_{l}+1)} + n^{−2}}]. (3.4)

We are now in a position to prove the Selection Lemma.

Proof of the Selection Lemma. Introduce the notations

S_{1} = ∑_{i=1}^{∞} \frac{(i^{−(2q_{0}+1)} − i^{−(2q_{m}+1)})i^{−(2q_{0}+1)}}{i^{−2(q_{0}+q_{m}+1)} + 2n^{−1}i^{−(2q_{0}+1)} + n^{−2}}

and

S_{2} = − ∑_{i=1}^{∞} \frac{(i^{−(2q_{0}+1)} − i^{−(2q_{m}+1)})θ_{i0}^{2}}{i^{−2(q_{0}+q_{m}+1)} + 2n^{−1}i^{−(2q_{0}+1)} + n^{−2}} .

We shall show that there exists an N not depending on m, m < 0, such that for n > N

S_{1} ≤ −\frac{1}{12}n^{1/(q_{0}+q_{m}+1)} (3.5)

and

S_{2} ≤ \frac{1}{24}n^{1/(q_{0}+q_{m}+1)}. (3.6)

Note that

i^{−(2q_{0}+1)} − i^{−(2q_{m}+1)} ≤ \{ \begin{matrix}0 & for all i, \\ −\frac{1}{2}i^{−(2q_{m}+1)} & for i > I, \end{matrix}

where I = 2^{1/2(q_{0}−q_{−1})}. Also if n > N_{1} = 2^{(q_{0}+q_{−1}+1)/(q_{0}−q_{−1})}, then for i ≤ n^{1/(q_{0}+q_{m}+1)}, the first term in the common denominator in the expressions for S_{1} and S_{2} dominates the other two terms. Piecing these facts together, we see that the terms in S_{1} are less than or equal to −1/6 for I < i ≤ n^{1/(q_{0}+q_{m}+1)} and less than or equal to zero in general. Thus

S_{1} ≤ −\frac{1}{6}(⌊n^{1/(q_{0}+q_{m}+1)}⌋ − I) ≤ −\frac{1}{12}n^{1/(q_{0}+q_{m}+1)}

if n > N_{2} = (2I + 2)^{2q_{0}+1}.

Let C_{0} = max(1, ∑_{i=1}^{∞} i^{2q_{0}}θ_{i0}^{2}). Now for any k

S_{2} ≤ ∑_{i=1}^{∞} \frac{i^{−(2q_{m}+1)}θ_{i0}^{2}}{i^{−2(q_{0}+q_{m}+1)} + 2n^{−1}i^{−(2q_{0}+1)} + n^{−2}}

≤ ∑_{i=1}^{k} i^{2q_{0}+1}θ_{i0}^{2} + n^{2} ∑_{i=k+1}^{∞} i^{−(2q_{m}+1)}θ_{i0}^{2}

≤ kC_{0} + n^{2}(k + 1)^{−(2q_{0}+2q_{m}+1)} ∑_{i=k+1}^{∞} i^{2q_{0}}θ_{i0}^{2}.

The first term can be made less than or equal to (48)^{−1}n^{1/(q_{0}+q_{m}+1)} by choosing k = ⌊(48C_{0})^{−1}n^{1/(q_{0}+q_{m}+1)}⌋. Since k + 1 ≥ (48C_{0})^{−1}n^{1/(2q_{0}+1)}, the second term is also less than or equal to (48)^{−1}n^{1/(q_{0}+q_{m}+1)} for n > N_{3}, where N_{3} is the smallest n such that

∑_{i≥(48C_{0})^{−1}n^{1/(2q_{0}+1)}} i^{2q_{0}}θ_{i0}^{2} < (48C_{0})^{−2(2q_{0}+1)}. (3.7)

Note that N_{1}, N_{2}, N_{3} work for all m < 0. Thus for n > N = max(N_{1}, N_{2}, N_{3}), (3.5) and (3.6) follow. Equation (2.6) now follows from Lemma 3.2. Equation (2.7) is an immediate consequence of (2.6).

4. Proof of Lemmas 2.2 and 2.3

To simplify notations, we drop the overhead bars from λ_{m}’s and Π and simply write Π = ∑_{m=0}^{∞} λ_{m}Π_{m}, where ∑_{m=0}^{∞} λ_{m} = 1.

Proof of Lemma 2.2. By Chebyshev’s inequality,

Π{θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X} ≤ B^{−1} ∑_{i=1}^{∞} i^{2q_{0}}E(θ_{i}^{2}|X).

Now

E(θ_{i}^{2}|X) = ∑_{m=0}^{∞} Π(q = q_{m}|X)E(θ_{i}^{2}|X, q = q_{m})

= ∑_{m=0}^{∞} Π(q = q_{m}|X)\{\frac{n}{(n + i^{2q_{m}+1})^{2}} + \frac{n^{2}X_{i}^{2}}{(n + i^{2q_{m}+1})^{2}} \}

≤ \frac{n}{(n + i^{2q_{0}+1})^{2}} + \frac{n^{2}X_{i}^{2}}{(n + i^{2q_{0}+1})^{2}} .

Therefore

E_{θ_{0}}Π{θ : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} > B|X}

≤ B^{−1} ∑_{i=1}^{∞} i^{2q_{0}}\{\frac{n}{(n + i^{2q_{0}+1})^{2}} + \frac{n^{2}E_{θ_{0}}X_{i}^{2}}{(n + i^{2q_{0}+1})^{2}} \}

≤ B^{−1}\{2 ∑_{i=1}^{∞} \frac{ni^{2q_{0}}}{(n + i^{2q_{0}+1})^{2}} + ∑_{i=1}^{∞} i^{2q_{0}}θ_{i0}^{2} \frac{n^{2}}{(n + i^{2q_{0}+1})^{2}} \}.

It therefore suffices to show that the sums on the above display are finite and bounded in n. The second sum is clearly so because the numerator is less than the denominator in the fraction and θ_{0} ∈ Θ_{q_{0}}. We split the first sum into sums over ranges {i : 1 ≤ i ≤ k} and {i : i > k}, where k = ⌊n^{1/(2q_{0}+1)}⌋. Then

∑_{i=1}^{∞} \frac{ni^{2q_{0}}}{(n + i^{2q_{0}+1})^{2}} ≤ n^{−1} ∑_{i=1}^{k} i^{2q_{0}} + n ∑_{i=k+1}^{∞} i^{−2q_{0}−2} ≤ \frac{n^{−1}k^{2q_{0}+1} + n(k + 1)^{−2q_{0}−1}}{2q_{0} + 1} ,

which is bounded by 2/(2q_{0} + 1).

To prove Lemma 2.3, it is more convenient to interpret the model in terms of n i.i.d. observations Y_{1}, Y_{2}, …, Y_{n}, where each Y_{j} is distributed like Y = (Y_{1}, Y_{2}, …) with distribution P_{θ} and Y_{i}’s are independently distributed as N(θ_{i}, 1), i = 1, 2, . . .. Note that here we distinguish between this P_{θ} and the distribution of X. We need some preparatory lemmas.

Lemma 4.1. For any θ, θ_{0} ∈ ℓ_{2},

(i) P_{θ} is absolutely continuous with respect to P_{θ_{0}};

(ii) ∑_{i=1}^{∞}(Y_{i} −θ_{i0})(θ_{i} −θ_{i0}) converges a.s. [P_{θ_{0}}] and in L_{2}(P_{θ_{0}}), has mean 0 and variance ∥θ − θ_{0}∥^{2};

(iii) \frac{dP_{θ}}{dP_{θ_{0}}} (Y) = exp [ ∑_{i=1}^{∞}(Y_{i} − θ_{i0})(θ_{i} − θ_{i0}) − \frac{1}{2}∥θ − θ_{0}∥^{2}];

(iv) − ∫ log \frac{dP_{θ}}{dP_{θ_{0}}} (y)dP_{θ_{0}}(y) = \frac{1}{2}∥θ − θ_{0}∥^{2};

(v) ∫ ( log \frac{dP_{θ}}{dP_{θ_{0}}} (y))^{2}dP_{θ_{0}}(y) = ∥θ − θ_{0}∥^{2} + \frac{1}{4}∥θ − θ_{0}∥^{4};

(vi) The Hellinger distance H(θ_{0}, θ) = ( ∫ ([(dP_{θ}/dP_{θ_{0}})(y)]^{1/2} − 1)^{2}dP_{θ_{0}}(y))^{1/2} satis-

fies H(θ_{0}, θ) ≤ \frac{1}{2}∥θ − θ_{0}∥. If ∥θ − θ_{0}∥ ≤ 1, H(θ_{0}, θ) ≥ e^{−1/16}∥θ − θ_{0}∥/2.

Proof. As P_{θ} and P_{θ_{0}} are product measures and the infinite product of affinities of their respective components converges to exp[−\frac{1}{8}∥θ − θ_{0}∥^{2}], (i) follows from Kakutani’s criterion as in the proof of Lemma 3.1. For (ii), consider the mean zero martingale S_{k} = ∑_{i=1}^{k}(Y_{i} − θ_{i0})(θ_{i} − θ_{i0}) and note that sup_{k≥1} ES_{k}^{2} = ∥θ − θ_{0}∥^{2} < ∞. Martingale convergence theorem [see, e.g., Williams (1991, page 109)] now applies. For (iii), note that on the sigma-field generated by (Y_{1}, Y_{2}, …, Y_{k}), the Radon-Nikodym derivative is given by

exp [ ∑_{i=1}^{k}(Y_{i} − θ_{i0})(θ_{i} − θ_{i0}) − \frac{1}{2} ∑_{i=1}^{k}(θ_{i} − θ_{i0})^{2}].

The rest follows from (ii) and the martingale convergence theorem. Assertions (iv) and (v) are immediate consequences of (ii) and (iii). For (vi), note that H^{2}(θ_{0}, θ) = 2 − 2E_{θ_{0}}((dP_{θ}/dP_{θ_{0}})(Y))^{1/2}. To evaluate E_{θ_{0}}((dP_{θ}/dP_{θ_{0}})(Y))^{1/2}, consider the martingale

S_{k} = exp [\frac{1}{2} ∑_{i=1}^{k}(Y_{i} − θ_{i0})(θ_{i} − θ_{i0}) − \frac{1}{8} ∑_{i=1}^{k}(θ_{i} − θ_{i0})^{2}].

Clearly S_{k} is positive, has unit expectation and bounded second moment, so its limit (dP_{θ}/dP_{θ_{0}})^{1/2}(Y) exp[\frac{1}{8}∥θ − θ_{0}∥^{2}] has also expectation 1. This implies that H^{2}(θ_{0}, θ) = 2 (1 − exp[−\frac{1}{8}∥θ − θ_{0}∥^{2}]) . The first assertion in (vi) now follows from the inequality 1 − e^{−x} ≤ x and the second from the mean value theorem.

The following lemma estimating the probability of a ball under product normal measure from below will be used in calculation of certain prior probabilities. The result appeared as Lemma 5 in Shen and Wasserman (1998). Its simple proof is included for completeness.

Lemma 4.2. Let W_{1}, …, W_{N} be independent random variables, with W_{i} having distribution N(−ξ_{i}, i^{−2d}), d > 0. Then

Pr { ∑_{i=1}^{N} W_{i}^{2} ≤ δ^{2}} ≥ 2^{−N/2}e^{−dN} exp [ − ∑_{i=1}^{N} i^{2d}ξ_{i}^{2}] Pr { ∑_{i=1}^{N} V_{i}^{2} ≤ 2δ^{2}N^{2d}}, (4.1)

where V_{1}, …, V_{N} are independent standard normal random variables.

Proof. Using (a + b)^{2} ≤ 2(a^{2} + b^{2}), N! ≥ e^{−N}N^{N} and the change of variable v_{i} = \sqrt{2}N^{d}w_{i}, we obtain

Pr { ∑_{i=1}^{N} W_{i}^{2} ≤ δ^{2}}

= ∫_{∑_{i=1}^{N} w_{i}^{2}≤δ^{2}} ∏_{i=1}^{N} \{(2π)^{−1/2}i^{d} exp [ − \frac{1}{2}i^{2d}(w_{i} + ξ_{i})^{2}]\}dw_{1} … dw_{N}

≥ (2π)^{−N/2}(N!)^{d} ∫_{∑_{i=1}^{N} w_{i}^{2}≤δ^{2}} exp [ − ∑_{i=1}^{N} i^{2d}(w_{i}^{2} + ξ_{i}^{2})]dw_{1} … dw_{N}

≥ (2π)^{−N/2}(N!)^{d}e^{− ∑_{i=1}^{N} i^{2d}ξ_{i}^{2}} ∫_{∑_{i=1}^{N} w_{i}^{2}≤δ^{2}} exp [ − N^{2d} ∑_{i=1}^{N} w_{i}^{2}]dw_{1} … dw_{N}

≥ (N!)^{d}e^{− ∑_{i=1}^{N} i^{2d}ξ_{i}^{2}}2^{−N/2}N^{−dN}(2π)^{−N/2} ∫_{∑_{i=1}^{N} v_{i}^{2}≤2N^{2d}δ^{2}} e^{− \frac{1}{2} ∑_{i=1}^{N} v_{i}^{2}}dv_{1} … dv_{N}

≥ 2^{−N/2}e^{−dN}e^{− ∑_{i=1}^{N} i^{2d}ξ_{i}^{2}} Pr { ∑_{i=1}^{N} V_{i}^{2} ≤ 2N^{2d}δ^{2}}.

Lemma 4.3. There exist positive constants c_{1}, c_{2} such that for sufficiently small ϵ > 0,

Π{θ : ∥θ − θ_{0}∥ ≤ ϵ} ≥ c_{1}λ_{0} exp [ − c_{2}ϵ^{−1/q_{0}}]. (4.2)

Proof. Clearly the LHS of (4.2) is bounded from below by λ_{0}Π_{0}{θ : ∑_{i=1}^{∞}(θ_{i} − θ_{i0})^{2} ≤ ϵ^{2}}. Now by independence, for any N,

Π_{0}{θ : ∑_{i=1}^{∞}(θ_{i} − θ_{i0})^{2} ≤ ϵ^{2}}

≥ Π_{0}{θ : ∑_{i=1}^{N}(θ_{i} − θ_{i0})^{2} ≤ ϵ^{2}/2} Π_{0}{θ : ∑_{i=N+1}^{∞}(θ_{i} − θ_{i0})^{2} ≤ ϵ^{2}/2}.

Also

∑_{i=N+1}^{∞}(θ_{i} − θ_{i0})^{2} ≤ 2 ∑_{i=N+1}^{∞} θ_{i}^{2} + 2 ∑_{i=N+1}^{∞} θ_{i0}^{2}. (4.3)

The second sum in (4.3) is less than or equal to

2N^{−2q_{0}} ∑_{i=N+1}^{∞} i^{2q_{0}}θ_{i0}^{2} ≤ 2N^{−2q_{0}}C_{0} < \frac{ϵ^{2}}{4}

whenever N > N_{1} = (8C_{0})^{1/2q_{0}}ϵ^{−1/q_{0}}, where C_{0} = ∑_{i=1}^{∞} i^{2q_{0}}θ_{i0}^{2}. By Chebyshev’s inequality, the first sum on the RHS of (4.3) is less than ϵ^{2}/4 with probability at least

1 − \frac{8}{ϵ^{2}} ∑_{i=N+1}^{∞} E_{Π_{0}}(θ_{i}^{2}) = 1 − \frac{8}{ϵ^{2}} ∑_{i=N+1}^{∞} i^{−(2q_{0}+1)} ≥ 1 − \frac{4}{q_{0}N^{2q_{0}}ϵ^{2}} > \frac{1}{2}

if N > N_{2} = (8/q_{0})^{1/2q_{0}}ϵ^{−1/q_{0}}. To bound Π_{0}{θ : ∑_{i=1}^{N}(θ_{i} − θ_{i0})^{2} ≤ ϵ^{2}/2} we apply

Lemma 4.2 with d = q_{0}+ \frac{1}{2}, ξ = θ_{i0} and δ^{2} = ϵ^{2}/2. Note that by the central limit theorem, the factor Pr\{∑_{i=1}^{N} V_{i}^{2} ≤ 2δ^{2}N^{2d}\} on the RHS of (4.1) is at least \frac{1}{4} if 2δ^{2}N^{2d} > N and N is large, i.e., if N > N_{3} = ϵ^{−1/q_{0}} and N > N_{4}. Choosing N = max(N_{1}, N_{2}, N_{3}, N_{4}) and noting that ∑_{i=1}^{N} i^{2q_{0}+1}θ_{i0}^{2} ≤ NC_{0}, (4.2) is obtained.

The final lemma gives the entropy estimates. Recall that for a totally bounded subset S of a metric space with a distance function d, the ϵ-packing number D(ϵ, S, d) is defined to be the largest integer m such that there exist s_{1}, …, s_{m} ∈ S with d(s_{j}, s_{k}) > ϵ for all j, k = 1, 2, …, m, j ≠ k. The ϵ-entropy is log D(ϵ, S, d). The result is possibly obtainable through Fourier transformation from Theorem 5.2 of Birman and Solomjak (1967), where a bound for the entropy of Sobolev balls was obtained. However, the exact correspondence is somewhat unclear because of the possible fractional value of the index q. Below, we present a simple proof which could be of some independent interest as well.

Lemma 4.4. For all ϵ > 0,

log D(ϵ, \{θ : ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} ≤ B\}, ∥ · ∥) ≤ \frac{1}{2}(4B)^{1/(2q)} log(6(2e)^{2q})ϵ^{−1/q}. (4.4)

Proof. Denote Θ_{q}(B) = {θ : ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} ≤ B}. If ϵ^{2} > 4B, then

∑_{i=1}^{∞} θ_{i}^{2} ≤ ∑_{i=1}^{∞} i^{2q}θ_{i}^{2} ≤ B < \frac{ϵ^{2}}{4} ,

and hence for any two θ, θ′ ∈ Θ_{q}(B), ∥θ − θ′∥ < ϵ, implying that D(ϵ, Θ_{q}(B)) = 1. Thus (4.4) is trivially satisfied for such ϵ. We shall therefore assume that ϵ^{2} ≤ 4B.

Let θ_{1}, …, θ_{m} ∈ Θ_{q}(B) be such that ∥θ_{i} − θ_{j}∥ > ϵ, j, k = 1, 2, …, m, j ≠ k. For an integer N, consider the set

Θ_{q,N}(B) = {(θ_{1}, …, θ_{N}, 0, …) : ∑_{i=1}^{N} i^{2q}θ_{i}^{2} ≤ B} .

With θ̄ = (θ_{1}, … θ_{N}, 0 …), note that for any θ ∈ Θ_{q}(B),

∥θ − θ̄∥^{2} = ∑_{i=N+1}^{∞} θ_{i}^{2} ≤ (N + 1)^{−2q}B ≤ \frac{ϵ^{2}}{4}

if N = ⌊(4B)^{1/(2q)}ϵ^{−1/q}⌋. For this choice of N, we have

1 ≤ N ≤ (4B)^{1/(2q)}ϵ^{−1/q} < N + 1 ≤ 2N. (4.5)

Therefore, for j, k = 1, 2, …, m, j ≠ k,

ϵ^{2} < ∥θ_{j} − θ_{k}∥^{2} = ∥θ̄_{j} − θ̄_{k}∥^{2} + ∥(θ_{j} − θ̄_{j}) − (θ_{k} − θ̄_{k})∥^{2} ≤ ∥θ̄_{j} − θ̄_{k}∥^{2} + 2\frac{ϵ^{2}}{4} ,

implying that ∥θ̄_{j} − θ̄_{k}∥ > ϵ/\sqrt{2}. The set Θ_{q,N}(B) may be viewed as an N-dimensional ellipsoid and θ̄_{j}’s as N-dimensional vectors in ℝ^{N} with the usual Euclidean distance. For t = (t_{1}, …, t_{N}) and δ > 0, let B(t, δ) stand for the Euclidean ball with radius δ centered at t. The balls B(θ̄_{j}, ϵ/2\sqrt{2}) are clearly disjoint. Note that if t ∈ Θ_{q}(B) and t′ = (t′_{1}, …, t′_{N}) ∈ B(t, ϵ/2\sqrt{2}), we have

∑_{i=1}^{N} i^{2q}t_{i}^{′2} ≤ 2 ∑_{i=1}^{N} i^{2q}t_{i}^{2} + 2 ∑_{i=1}^{N} i^{2q}(t′_{i} − t_{i})^{2} ≤ 2B + 2N^{2q} \frac{ϵ^{2}}{8} ≤ 3B

by (4.5), and hence

⋃_{j=1}^{m} B(θ̄_{j}, \frac{ϵ}{2\sqrt{2}}) ⊂ Θ_{q,N}(3B). (4.6)

Let V_{N} stand for the volume of the unit ball in ℝ^{N}. Note that the volume of the ellipsoid {t : ∑_{i=1}^{N} a_{i}^{2}t_{i}^{2} ≤ A} is given by V_{N}A^{N/2}/ ∏_{i=1}^{N} a_{i}. Bounding the volume of ⋃_{j=1}^{m} B(θ̄_{j}, ϵ/2\sqrt{2}) by that of Θ_{q,N}(3B), we obtain

m(\frac{ϵ^{2}}{8} )^{N/2}V_{N} ≤ (3B)^{N/2}V_{N} ∏_{i=1}^{N} i^{−q} = (3B)^{N/2}V_{N}(N!)^{−q}. (4.7)

Since N! ≥ N^{N}e^{−N} for all N ≥ 1, we arrive at the estimate

m ≤ (\frac{24Be^{2q}}{N^{2q}ϵ^{2}} )^{N/2}. (4.8)

From (4.5), we also have N^{2q}ϵ^{2} ≥ 2^{2−2q}B. Substituting in (4.8), the required bound is easily obtained.

To prove of Lemma 2.3, we shall use the following variation of of Theorem 2.1 of Ghosal, Ghosh and van der Vaart (2000).

Theorem 4.1. Suppose we have i.i.d. observations Y_{1}, Y_{2}, … from a distribution P having a density p (with respect to some sigma-finite measure) belonging to a class P. Let H(P_{0}, P) denote the Hellinger distance between P_{0} and P. Let Π be a prior on P and P̄ ⊂ P. Let ϵ_{n} be a positive sequence such that ϵ_{n} → 0 and nϵ_{n}^{2} → ∞ and suppose that log D(ϵ_{n}, P̄, H) ≤ c_{1}nϵ_{n}^{2} (4.9)

and

Π\{P : − ∫ ( log \frac{p}{p_{0}} )dP_{0} ≤ ϵ_{n}^{2}, ∫ ( log \frac{p}{p_{0}} )^{2}dP_{0} ≤ ϵ_{n}^{2}\} ≥ c_{2}e^{−c_{3}nϵ_{n}^{2}} (4.10)

for some constants c_{1}, c_{2}, c_{3}, where p_{0} stands for the density of P_{0}. Then for a sufficiently large constant M, the posterior probability

Π{P ∈ P̄ : H(P_{0}, P) ≥ Mϵ_{n}|Y_{1}, Y_{2}, …, Y_{n}} → 0 (4.11)

in P_{0}^{n}-probability.

Proof of Lemma 2.3. Take P = {P_{θ} : θ ∈ ℓ_{2}} and P̄ = {P_{θ} : ∑_{i=1}^{∞} i^{2q_{0}}θ_{i}^{2} ≤ B}. By part (vi) of Lemma 4.1, the Hellinger distance H(P_{θ_{0}}, P_{θ}) is equivalent to the ℓ_{2} distance ∥θ_{0} − θ∥, so in (4.9) and in the conclusion, the former may be replaced by the latter. Lemma 4.4 shows that (4.9) is satisfied by a multiple of n^{−q_{0}/(2q_{0}+1)}. For (4.10), note that by parts (iv) and (v) of Lemma 4.1, the set

\{θ : −E_{θ_{0}}( log \frac{dP_{θ}}{dP_{θ_{0}}} (Y)) ≤ ϵ^{2}, E_{θ_{0}}( log \frac{dP_{θ}}{dP_{θ_{0}}} (Y))^{2} ≤ ϵ^{2}\}

contains {θ : ∥θ−θ_{0}∥^{2} ≤ ϵ^{2}/2} for ϵ < 1. Therefore by Lemma 4.3, (4.10) is also satisfied by a multiple of n^{−q_{0}/(2q_{0}+1)}. The result thus follows.

References

[1] Birman, M. S. and Solomjak, M. Z. (1967). Piecewice-polynomial approximation of functions of the class W_{p}. Sov. Math. Sbornik 73 295–317.

[2] Brown, L. D. and Low, M. G. (1996). Asymptotic equivalence of nonparametric regression and white noise. Ann. Statist. 24 2384–2398.

[3] Cox D. (1993). An analysis of Bayesian inference for nonparametric regression. Ann. Statist. 21 903–923.

[4] Diaconis, P. and Freedman, D. (1997). On the Bernstein-von Mises theorem with infinite dimensional parameters,. Technical Report 492, Department of Statistics, University of California.

[5] Efromovich, S. Y. and Pinsker, M. S. (1984). Learning algorithm for nonparametric filtering. Automat. Remote Control 11 1434–1440.

[6] Freedman, D. (1999). On the Bernstein-von Mises theorem with infinite dimensional parameters. Ann. Statist. 27 1119–1140.

[7] Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. To appear.

[8] Ghosal, S. and van der Vaart, A. W. (1999). On Bayesian adaptation, Preprint.

[9] Klemelä, J. and Nussbaum, M. (1998). Constructive asymptotic equivalence of density estimation and Gaussian white noise. Technical Report 53, Humbolt University, Berlin.

[10] Nussbaum, M. (1996). Asymptotic equivalence of density estimation and Gaussian white noise. Ann. Statist. 24 2399–2430.

[11] Pinsker, M. S. (1980). Optimal filtration of square-integrable signals in Gaussian noise. Problems of Information Transmission 16, 120–133.

[12] Shen, X. and Wasserman, L. (1998). Rates of convergence of posterior distributions. Preprint.

[13] Williams, D. (1991). Probability with Martingales. Cambridge University Press, Cambridge.

[14] Zhao, L. H. (1998a). A hierarchical Bayesian approach in nonparametric function estimation. Preprint.

[15] Zhao, L. H. (1998b). Bayesian aspects of some nonparametric problems. Preprint.

Mathematical Institute Leiden University Niels Bohrweg 1 2333 CA Leiden The Netherlands E-mail: belitser@math.leidenuniv.nl School of Statistics 376 Ford Hall 224 Church St. SE University of Minnesota Minneapolis, MN 55455 U. S. A. E-mail: ghosal@stat.umn.edu