PALEY–WIENER THEOREMS FOR THE DUNKL TRANSFORM

MARCEL DE JEU

Abstract. We conjecture a geometrical form of the Paley–Wiener theorem for the Dunkl transform and prove three instances thereof, by using a reduction to the one-dimensional even case, shift operators, and a limit transition from Opdam’s results for the graded Hecke algebra, respectively. These Paley– Wiener theorems are used to extend Dunkl’s intertwining operator to arbitrary smooth functions.

Furthermore, the connection between Dunkl operators and the Cartan motion group is established. It is shown how the algebra of radial parts of invariant differential operators can be described explicitly in terms of Dunkl operators. This description implies that the generalized Bessel functions coincide with the spherical functions. In this context of the Cartan motion group, the restriction of Dunkl’s intertwining operator to the invariants can be interpreted in terms of the Abel transform. We also show that, for certain values of the multiplicities of the restricted roots, the Abel transform is essentially inverted by a differential operator.

1. Introduction and overview

In recent times the study of special functions associated with root systems has developed to a considerable degree. Starting with a number of conjectures by Macdonald, and the work of Heckman and Opdam on multivariable hypergeometric functions in the late 1980’s, the development of the theory was greatly enhanced by the introduction of rational Dunkl operators by Dunkl [7]. Through various intermediate steps of generalization, these operators can even be said to have ultimately provided crucial building blocks for Cherednik’s work on double affine Hecke algebras and the q-Macdonald conjectures.

Originally, before the introduction of Dunkl operators, the idea when studying special functions related to root systems was to consider root multiplicities in the theory of spherical functions on Lie groups as parameters, and then to develop a theory for Weyl group invariant objects in this more general situation, without the aid of the presence of the group. It was this point of view which underlay the Macdonald conjectures and which led Heckman and Opdam to the development of their theory of hypergeometric functions in higher dimension. One of the main technical problems in this context is the description of the generalized radial parts of invariant differential operators. Apart from an explicit formula for the generalized radial part of the Laplacian—an expression which was in fact the starting point for Heckman and Opdam—the other operators remain somewhat intangible.

2000 Mathematics Subject Classification. Primary 33C52; Secondary 43A32, 33C80, 22E30.

Key words and phrases. Dunkl operator, Paley–Wiener theorem, graded Hecke algebra, Cartan motion group, spherical function, multivariable Bessel function.

This problem disappeared when Dunkl found the operators which have come to bear his name. They are parameterized deformations of the ordinary derivatives, involving a finite reflection group, for which it is still relatively easy to study the spectral problem and develop the theory of the corresponding Fourier transformation— the Dunkl transform. The invariant part of the theory then answers questions in a generalized theory of spherical functions—for the Cartan motion group, to be precise—as described above. The operators under consideration being explicitly given, it is actually easier to study the general context of Dunkl operators, and then specialize to invariant objects later on, than it is to restrict oneself to the invariants from the outset. The same holds true for the modification of Dunkl operators into Cherednik operators, which gives rise to a representation of the graded Hecke algebra [26] and the non-invariant “envelope” of the work of Heckman and Opdam on hypergeometric functions.

Quite remarkably, the theory of spherical functions in analysis on Lie groups can in a number of situations thus be regarded as only the invariant part of a general non-invariant theory for Dunkl-type operators. At the time of writing, it is unknown to the author whether there is an underlying reason for this phenomenon.

In view of all this, these Dunkl operators and their modifications have attracted considerable attention in various areas of mathematics and mathematical physics during the last decade. To get an impression of their influence on the development of special functions associated with root systems—also in the general non-invariant context—we refer to, e.g., [17], [27], [13], [29]. For their use in the study of integrable quantum many body systems of Calogero–Moser–Sutherland type we refer to [5] and the bibliography therein.

In this paper, we are mainly concerned with the further development of the general theory of the Dunkl transform, notably with the Paley–Wiener theorem. In addition, we describe the relation between Dunkl operators and the Cartan motion group. We will now turn to an overview of the contents of this paper with regard to these two subjects.

The first substantial results for the Dunkl transform, i.e., the Plancherel theorem and inversion theorem, were obtained by Dunkl [10] and the author [20]. Two Paley–Wiener theorems were established in [21, Chapter 3] (unpublished), of which this paper is an extension. Whereas the proof of the inversion theorem in [20] is partly a formal argument, based on various symmetry properties of the eigenfunctions of Dunkl’s operators, it will become apparent below that quite some more work is required in Paley–Wiener theory, due to the lack of adequate asymptotic results. We will establish three Paley–Wiener theorems, each of these being a special case of a conjectured geometrically more precise general Paley–Wiener theorem, which can be found below as Conjecture 4.1. Each version requires a different technique. The first and most general version relies on a reduction to the one-dimensional even case—where asymptotic results are available—and some non-trivial results from general representation theory and from the representation theory of the orthogonal group. The second version, which is proved for a discrete set of parameters only, is established through shift operators. Finally, the third version, which we prove for Weyl groups, follows from a limit transition from results of Opdam [26]. This limit transition has some interest in itself, and for a discrete set of parameters its validity has also been established by Ben Saïd and Ørsted, cf. [2] and [3].

Turning to the Cartan motion group, for which the material is taken from [21], we will show how the algebra of radial parts of invariant differential operators can be described explicitly in terms of Dunkl operators with suitable values of the parameters. From this description it is easy to see that the invariant components of the eigenfunctions of the Dunkl operators are precisely the spherical functions. We can then also describe Dunkl’s intertwiner operator, or rather its restriction to the invariants, in terms of the Abel transform. For certain multiplicities, the results on shift operators imply that this Abel transform is essentially inverted by a differential operator.

The organization of this paper is as follows. In Section 2 we establish the necessary general definitions and notations, and we recall some previous results. Section 3 contains a number of useful formulas. We do not claim any originality in particular for these formulas, but we believe that the proofs are considerably simpler than the ones in the existing literature. The conjectured Paley–Wiener theorem can be found in Section 4, as can the proven three instances of it which were described above. These Paley–Wiener theorems are then used in Section 5 to extend Dunkl’s intertwining operator to the smooth functions. To conclude with, Section 6 contains the details of the connection between Dunkl operators and the Cartan motion group, including the results on the Abel transform.

2. Notations and previous results

Let 𝔞 be a real vector space of finite dimension N which is equipped with an inner product ( . , .), inducing a Lebesgue measure dx on 𝔞. We let 𝔞_{ℂ} = 𝔞 ⊗_{ℝ} ℂ denote the complexification of 𝔞 and we extend the form ( . , .) to a bilinear form on 𝔞_{ℂ}, again denoted by ( . , .). Both 𝔞 and 𝔞_{ℂ} can be identified with their duals via ( . , .); for ξ in 𝔞 or 𝔞_{ℂ} the corresponding linear functional is then denoted by ξ^{∗}. Define the orthogonal group O(N) = O(𝔞, ( . , .)). The norm | . | which is induced on 𝔞 by ( . , .) extends to an O(N)-invariant norm on 𝔞_{ℂ}, also denoted by | . |. There is a natural action of O(N) on functions:

(g · f)(x) = f(g^{−1}x).

Let G ⊂ O(N) be a finite (real) reflection group with corresponding root system R. We may and will assume that (α, α) = 2 for all α ∈ R. If α ∈ R, then r_{α} is the orthogonal reflection in the hyperplane perpendicular to α. We choose and fix a positive system R_{+} in R.

A function k : R ↦ ℂ is called a multiplicity function if k is G-invariant. We write k ≥ 0 if all values of k are non-negative, with analogous notations Re k ≥ 0 and k > 0. We let K denote the vector space of multiplicity functions on R.

Let k ∈ K and ξ ∈ 𝔞. Then the corresponding Dunkl operator T_{ξ}(k) is defined by

T_{ξ}(k) = ∂_{ξ} + ∑_{α∈R_{+}} k_{α}(α, ξ)M_{(α^{∗})^{−1}}(1 − r_{α}).

Here ∂_{ξ} (ξ ∈ 𝔞) is the (unnormalized) directional derivative operator; we have used the notation M_{f} for pointwise multiplication by a function f.

The definition of the operators is independent of the choice of the positive system. The T_{ξ}(k) leave the polynomials P invariant, mapping the homogeneous polynomials P_{n} of degree n into P_{n−1}. Furthermore, the spaces C^{∞}(𝔞) of smooth functions, D(𝔞) of compactly supported smooth functions and S(𝔞) of rapidly decreasing smooth functions are also invariant [20, Lemma 2.1].

Quite remarkably, the T_{ξ}(k) form a commutative family: T_{ξ}(k)T_{η}(k) = T_{η}(k)T_{ξ}(k) for all ξ, η ∈ 𝔞 and all k ∈ K, as was proved by Dunkl [7], see [11] for a different proof. As a consequence of the commutativity the map ξ ↦ T_{ξ}(k) extends to P; for p ∈ P the corresponding operator is then denoted by T_{p}(k). The important k-Laplacian T_{| . |^{2}}(k) is denoted by Δ_{k}.

For k ≥ 0, Dunkl [8] has constructed a linear isomorphism V_{k} : P ↦ P, homogeneous of degree 0, such that V_{k}1 = 1 and T_{ξ}(k)V_{k} = V_{k}∂_{ξ} (ξ ∈ 𝔞). An alternative approach for more general k can be found in [11]. A concrete description of this intertwiner operator V_{k} is presently still unknown, with the exception of the onedimensional case [8] and the case A_{2} [12]. Significant abstract results were obtained by Rösler [28], who showed, amongst others, that V_{k} is for k ≥ 0 a positive operator which can be described in terms of measures.

Let λ ∈ 𝔞_{ℂ} and consider the simultaneous eigenfunction problem

(2.1)  T_{ξ}(k)f = (λ, ξ)f (ξ ∈ 𝔞).

This problem was studied first by Dunkl [9] for k ≥ 0; later Opdam [25] treated the general case. One more definition is needed to state the general result: a multiplicity function k ∈ K is said to be singular if the simultaneous kernel of {T_{ξ}(k)}_{ξ∈𝔞} in P is non-trivial, i.e., if it properly contains the constants. We will use the self-evident notations K^{sing} and K^{reg}.

The set K^{sing} has been determined in all cases [11]. Some information about the simultaneous kernel for singular multiplicities can also be found in [loc.cit.]; for a detailed account in the case of the symmetric group we refer to [15]. We will mainly be concerned with multiplicities satisfying Re k ≥ 0. Such multiplicities are regular, as is most easily seen by considering the operator ∑_{i=1}^{N} e_{i}^{∗}T_{e_{i}}(k) for an orthonormal basis {e_{1}, …, e_{N}} of 𝔞 [11].

With this notation, the result from [25] for the eigenfunction problem is as follows.

Theorem 2.1. For all k ∈ K^{reg} the eigenfunction problem (2.1) has a 1-dimensional solution space for all λ ∈ 𝔞_{ℂ}. This space contains a (unique) function Exp_{G}(λ, k, .) such that Exp_{G}(λ, k, 0) = 1. Furthermore, Exp_{G}(λ, k, .) extends to a holomorphic function on 𝔞_{ℂ}, and

Exp_{G}( . , . , .) : 𝔞_{ℂ} ↦ ℂ

is a meromorphic function with poles precisely in K^{sing}.

In order to be able to define the corresponding Fourier transform—the Dunkl transform—for sufficiently general functions, one needs non-trivial bounds for the eigenfunctions. It is shown in [20] that for Re k ≥ 0 one has

(2.2)  |Exp_{G}(λ, k, z)| ≤ \sqrt{|G|} exp(max_{g∈G} Re (gλ, z)) (λ, z ∈ 𝔞_{ℂ}),

in particular

(2.3)  |Exp_{G}(iλ, k, x)| ≤ \sqrt{|G|} (λ, x ∈ 𝔞).

If k ≥ 0 then the constant \sqrt{|G|} in (2.2) and (2.3) can in fact be improved to 1, as a consequence of Rösler’s results on the intertwiner operator [28].

Following Dunkl, we define for Re k ≥ 0 the G-invariant complex-valued weight function w_{k} = ∏_{α∈R_{+}} |α^{∗}|^{2k_{α}}. The T_{ξ}(k) are anti-symmetric with respect to this weight function: if f ∈ S(𝔞) and g is smooth such that both g and T_{ξ}(k)g are of at most polynomial growth, then

(2.4)  ∫_{𝔞}(T_{ξ}(k)f) gw_{k} dx = − ∫_{𝔞} f (T_{ξ}(k)g)w_{k} dx.

In view of (2.3), the Dunkl transform for Re k ≥ 0 is meaningfully defined on L_{1}(𝔞, |w_{k}(x)|dx) as

D_{k}f(λ) = \frac{1}{c_{k}} ∫_{𝔞} f(x)Exp_{G}(−iλ, k, x)w_{k}(x) dx (λ ∈ 𝔞, f ∈ L_{1}(𝔞, |w_{k}(x)|dx)).

Its alleged inverse is

E_{k}f(x) = \frac{1}{c_{k}} ∫_{𝔞} f(λ)Exp_{G}(iλ, k, x)w_{k}(λ) dλ (x ∈ 𝔞, f ∈ L_{1}(𝔞, |w_{k}(λ)| dλ)).

Here the normalizing constant c_{k} is the Mehta integral, which is defined as

c_{k} = ∫_{𝔞} ψw_{k} dx.

There is a closed expression for this integral; this former Macdonald conjecture has been proved by Opdam, first for Weyl groups [24] and later for finite reflection groups in general [25]. For our needs it suffices to know that c_{k} ≠ 0 if Re k ≥ 0, which can be proved by more elementary means [20, Corollary 4.17].

The first results for the transform, and notably a version of the Plancherel theorem, were obtained by Dunkl [10]. Later on, a more systematic study was undertaken in [20]. The main properties and results are as follows.

Theorem 2.2. Let Re k ≥ 0 and ξ ∈ 𝔞. Then:

(1) D_{k}T_{ξ}(k)f = M_{iξ^{∗}}D_{k}f (f ∈ S(𝔞)).

(2) E_{k}T_{ξ}(k)f = −M_{iξ^{∗}}E_{k}f (f ∈ S(𝔞)).

(3) D_{k}M_{iξ^{∗}}f = −T_{ξ}(k)D_{k}f (f ∈ S(𝔞)).

(4) E_{k}M_{iξ^{∗}}f = T_{ξ}(k)E_{k}f (f ∈ S(𝔞)).

(5) D_{k} is a linear homeomorphism of S(𝔞), with inverse E_{k}.

(6) If f ∈ L_{1}(𝔞, |w_{k}(x)|dx) and D_{k}f ∈ L_{1}(𝔞, |w_{k}(x)|dx) then D_{k}E_{k}f = E_{k}D_{k}f= f a.e.

(7) If k ≥ 0 then D_{k} maps L_{1}(𝔞, w_{k}(x) dx)∩L_{2}(𝔞, w_{k}(x) dx) into L_{2}(𝔞, w_{k}(x) dx), isometrically with respect to the two-norm corresponding to w_{k}(x) dx, and extends uniquely from L_{1}(𝔞, w_{k}(x) dx) ∩ L_{2}(𝔞, w_{k}(x) dx) to a unitary operator on L_{2}(𝔞, w_{k}(x) dx).

Remark 2.3. The definition of Dunkl operators can be generalized to complex reflection groups [14].

3. Formularium

In this section we establish some useful formulas. Some of them are known from the work of Dunkl, who based his proofs to a large extent on the existence of a k-harmonic decomposition for polynomials [6, Theorem 1.7]. However, with the benefit of hindsight we can simplify some of the original proofs considerably, by systematically exploiting the following commutator relation [7, Proposition 2.2]:

(3.1)  [M_{ξ^{∗}}, \frac{Δ_{k}}{2} ] = −T_{ξ}(k) (ξ ∈ 𝔞).

To start, we first note that Δ_{k} : P ↦ P is homogeneous of degree −2, hence locally nilpotent. This enables us to define the linear automorphisms e^{± \frac{Δ_{k}}{2}} of P. These automorphisms will play an important part in what follows, together with the Gaussian ψ(x) = exp(−|x|^{2}/2). It is known [10] [20] that the Gaussian is an eigenfunction of both D_{k} and E_{k} with eigenvalue 1.

Lemma 3.1. Let ξ ∈ 𝔞. Then for arbitrary k we have

(1) in End_{ℂ}(P): [M_{ξ^{∗}} , e^{− \frac{Δ_{k}}{2}} ] = T_{ξ}(k) ∘ e^{− \frac{Δ_{k}}{2}} ;
(2) in Hom_{ℂ}(P, S(𝔞)): T_{ξ}(k) ∘ M_{ψ} ∘ e^{− \frac{Δ_{k}}{2}} = −M_{ψ} ∘ e^{− \frac{Δ_{k}}{2}} ∘ M_{ξ^{∗}}.

Proof. The first part follows immediately from (3.1) and the obvious fact that [Δ_{k}, T_{ξ}(k)] = 0. As to the second part, note that T_{ξ}(k) ∘ M_{ψ} = M_{ψ} ∘ T_{ξ}(k) − M_{ψ} ∘ M_{ξ^{∗}}, as a consequence of the G-invariance of ψ. Hence

T_{ξ}(k) ∘ M_{ψ} ∘ e^{− \frac{Δ_{k}}{2}} = M_{ψ} ∘ T_{ξ}(k) ∘ e^{− \frac{Δ_{k}}{2}} − M_{ψ} ∘ M_{ξ^{∗}} ∘ e^{− \frac{Δ_{k}}{2}}

= M_{ψ} ∘ T_{ξ}(k) ∘ e^{− \frac{Δ_{k}}{2}} − M_{ψ} ∘ \{[M_{ξ^{∗}} , e^{− \frac{Δ_{k}}{2}} ] + e^{− \frac{Δ_{k}}{2}} ∘ M_{ξ^{∗}}\}

= M_{ψ} ∘ T_{ξ}(k) ∘ e^{− \frac{Δ_{k}}{2}} − M_{ψ} ∘ \{T_{ξ}(k) ∘ e^{− \frac{Δ_{k}}{2}} + e^{− \frac{Δ_{k}}{2}} ∘ M_{ξ^{∗}}\}

= −M_{ψ} ∘ e^{− \frac{Δ_{k}}{2}} ∘ M_{ξ^{∗}}.
□

Repeated application of the second part of Lemma 3.1 yields the formula

(3.2)  T_{p}(k) \{(e^{− \frac{Δ_{k}}{2}} q) ψ\} = (−1)^{deg p} \{e^{− \frac{Δ_{k}}{2}} (pq)\} ψ,

for homogeneous p ∈ P and arbitrary q ∈ P. Taking q = 1 this implies, together with Theorem 2.2, the following result, which is equivalent to [10, Proposition 2.1].

Corollary 3.2. If p ∈ P is homogeneous and Re k ≥ 0, then

(1) D_{k}(pψ) = (−i)^{deg p} (e^{− \frac{Δ_{k}}{2}} p) ψ.

(2) E_{k}(pψ) = i^{deg p} (e^{− \frac{Δ_{k}}{2}} p) ψ.

The second part of Lemma 3.1 also enables us to reprove the symmetry of a bilinear form which was introduced by Dunkl [9], as follows. For p, q ∈ P, put (p, q)_{k} = (T_{p}(k)q) (0). Although it is not obvious from the definition, this bilinear form on P is actually symmetric. This symmetry follows from the following generalization by Dunkl [loc.cit.] of a result of Macdonald:

(3.3)  (p, q)_{k} = \frac{1}{c_{k}} ∫_{𝔞} (e^{− \frac{Δ_{k}}{2}} p) (e^{− \frac{Δ_{k}}{2}} q) ψw_{k} dx (Re k ≥ 0, p, q ∈ P),

in which the right hand side is obviously symmetric. In order to re-establish (3.3), denote the right hand side by [p, q]_{k}. Now (2.4) and Lemma 3.1 imply that

[p, T_{ξ}(k)q]_{k} = \frac{1}{c_{k}} ∫_{𝔞} (e^{− \frac{Δ_{k}}{2}} p) (T_{ξ}(k)e^{− \frac{Δ_{k}}{2}} q) ψw_{k} dx

= \frac{1}{c_{k}} ∫_{𝔞} −T_{ξ}(k) \{(e^{− \frac{Δ_{k}}{2}} p) ψ\} (e^{− \frac{Δ_{k}}{2}} q) w_{k} dx

= \frac{1}{c_{k}} ∫_{𝔞} \{e^{− \frac{Δ_{k}}{2}} (M_{ξ^{∗}}p)\} (e^{− \frac{Δ_{k}}{2}} q) ψw_{k} dx

= [M_{ξ^{∗}}p, q]_{k}.

But the other form ( . , .)_{k} also has this property: (M_{ξ^{∗}}p, q)_{k} = (p, T_{ξ}(k)q)_{k}, as a direct consequence of its definition. Since it is easy to see that (1, q)_{k} = [1, q]_{k}, an induction with respect to deg p then proves that (p, q)_{k} = [p , q]_{k}, which is (3.3).

The following proposition will be used in the reduction of the proof of the Paley– Wiener theorem 4.10 to the one-dimensional even case.

Proposition 3.3. If Re k, Re k′ ≥ 0, then

E_{k′}D_{k} (pψ) = (e^{− \frac{Δ_{k′}}{2}} e^{\frac{Δ_{k}}{2}} p) ψ (p ∈ P).

Proof. We may assume that p is homogeneous. Then Corollary 3.2 implies that

E_{k′}D_{k} (pψ) = (−i)^{deg p}E_{k′} [\{∑_{n=0}^{∞} \frac{1}{n!} (−\frac{Δ_{k}}{2} )^{n} p\} ψ]

= (−i)^{deg p} [∑_{n=0}^{∞} \frac{i^{deg p−2n}}{n!} e^{− \frac{Δ_{k′}}{2}} \{(−\frac{Δ_{k}}{2} )^{n} p\}] ψ

= (e^{− \frac{Δ_{k′}}{2}} e^{\frac{Δ_{k}}{2}} p) ψ.

□

To conclude this section, we re-establish a formula of Heckman [16] which expresses the pivotal rôle of the k-Laplacian Δ_{k}:

(3.4)  T_{p}(k) = \frac{1}{n!} (ad \frac{Δ_{k}}{2} )^{n} M_{p},

for p ∈ P_{n}. The proof in [loc.cit.] is based on representation theory for sl(2), but it can also be seen directly, as follows. We may assume that M_{p} = ∏_{i=1}^{n} M_{ξ_{i}^{∗}}. Then

(ad \frac{Δ_{k}}{2} )^{n} M_{p} = ∑ ; ∑ j_{i}=n \frac{n!}{j_{1}! ⋯ j_{n}!} \{(ad \frac{Δ_{k}}{2} )^{j_{1}} M_{ξ_{1}^{∗}}\} ∘ ⋯ ∘ \{(ad \frac{Δ_{k}}{2} )^{j_{n}} M_{ξ_{n}^{∗}}\} .

But (ad Δ_{k})^{2} M_{ξ_{i}^{∗}} = 0 as a consequence of (3.1) and the commutativity of the T_{ξ}(k), so the only surviving term in the summation is the one with j_{1} = … = j_{n} = 1. Using (3.1) once more, this proves (3.4) .

4. Paley–Wiener theorems

In this section, we conjecture a geometrical form of the Paley–Wiener theorem for the Dunkl transform, and present several theorems to support it.

Establishing notation, for S ⊂ 𝔞 we let D(S) denote the smooth compactly supported functions with support contained in S. If S is compact and non-empty, then we define the indicator I_{S} : 𝔞 ↦ ℝ as I_{S}(x) = max_{y∈S}(x, y) for x ∈ 𝔞. For such S, let H_{S} be the functions on 𝔞_{ℂ} of Paley–Wiener type corresponding to S, i.e., those entire functions with the property that for each integer M ≥ 0 there exists a constant γ_{M} such that |f(λ)| ≤ γ_{M}(1 + |λ|)^{−M} exp I_{S}(Im λ) for all λ ∈ 𝔞_{ℂ}. We then conjecture the following.

Conjecture 4.1 (Paley–Wiener conjecture). Let G be a finite reflection group. If Re k ≥ 0, and if S is a non-empty G-invariant compact convex subset of 𝔞, then D_{k} is a linear isomorphism between D(S) and H_{S}.

There is some evidence supporting this conjecture:

• The inversion theorem and [20, Corollary 4.10] show that D_{k} is an injective map from D(S) into H_{S}.

• For k = 0 the statement holds [36, Exercise 29.5].

• If S is a ball centered at the origin, then a reduction to the one-dimensional even case, where asymptotics can be used, enables us to establish the statement as Theorem 4.10 below.

• If the k_{α} are all strictly positive integers, then the statement can be established, using shift operators, as Theorem 4.11 below.

• If G is a Weyl group, and if S is the intersection of convex hulls of orbits, then a limit transition from results of Opdam establishes the statement as Theorem 4.15 below.

The main obstacle for a possible proof of the conjecture along the usual lines, using a contour shift, is the absence of adequate asymptotic results for the Dunkl kernel. There are some asymptotic results available [30], but these fall far short of what is needed. It is to be expected that better results could be obtained if more were known about Rösler’s representing measures [28], but as yet these remain elusive. But even if much stronger asymptotic results became available, the proof of Theorem 4.4 below seems to suggest that additional monodromy arguments may then still be necessary.

Remark 4.2. In [38], a proof of Conjecture 4.1 if k ≥ 0 and ∑_{α∈R_{+}} k_{α} > 0 is presented. We feel, however, that this proof is not complete and that Conjecture 4.1 is at the time of writing still open.

4.1. The case of arbitrary G and Re k ≥ 0. Throughout this section, B_{R} will denote the closed ball in 𝔞 with radius R and the origin as center. The space D(B_{R}) carries the usual Fréchet topology of uniform convergence of all derivatives.

Our approach of the Paley–Wiener theorem for such sets—under the assumption that Re k ≥ 0—consists of three steps, cf. the proof of Theorem 4.10. First, we prove the result for even functions in one dimension. Second, it is shown that this implies the theorem for radial functions in arbitrary dimension. In the third step we finally prove that the result for radial functions implies the theorem for general smooth functions with support in a closed ball B_{R}.

Remark 4.3. The special rôle of radial functions in the theory has been noted by several authors [21] [39] [31] [37], e.g., the Dunkl transform of a radial function is again radial [31]. In Paley–Wiener theory this phenomenon is encountered once more, when reducing the radial case in arbitrary dimension to the even case in one dimension. The one-dimensional even case can be handled by either Weyl fractional integral operators [37], or as in our approach below, which uses classical results for the asymptotics of Bessel functions combined with a contour shift. The final step, from the radial to the general case, and for which a reference to [21] was given in [37], is in a sense the deepest, since it ultimately rests on the general theory of representations of compact groups in Fréchet spaces.

Starting with the proof, the one-dimensional even case is settled in the following theorem, where, as usual, invariance is denoted by superscripts.

Theorem 4.4. Let 𝔞 be one-dimensional and Re k ≥ 0. If f ∈ H_{B_{R}}^{ℤ_{2}}, then E_{k}f ∈ D(B_{R})^{ℤ_{2}}.

Proof. Let f be as in the statement. Then obviously E_{k}f is an even rapidly decreasing smooth function. We have to show that

∫_{−∞}^{∞} f(λ)Exp_{ℤ_{2}}(iλ, k, x)|λ|^{2k} dλ = 0

if x > R. Fix such x. From [10] we have

Exp_{ℤ_{2}}(iλ, k, x) = Γ(k + \frac{1}{2}) (\frac{λx}{2})^{\frac{1}{2} −k} \{J_{k− \frac{1}{2}} (λx) + iJ_{k+ \frac{1}{2}} (λx)\} .

Using the invariance of f we see that we are left with showing that

(4.1)  ∫_{−∞}^{∞} f(λ) (λx)^{\frac{1}{2} −k} J_{k− \frac{1}{2}} (λx)|λ|^{2k} dλ = 0.

This expression makes it obvious that there are two obstructions for a direct application of the classical argument of shifting the contour to infinity. First, the Bessel function has exponential growth in both the positive and negative imaginary directions, and second the weight function |λ|^{2k} is in general not the restriction of a holomorphic function on the upper or lower half plane. So we proceed indirectly.

Recall the definition of the Bessel functions of the third kind:

H_{ν}^{(1)} = \frac{J_{−ν} − e^{−νπi}J_{ν}}{i sin νπ} ,
H_{ν}^{(2)} = \frac{J_{−ν} − e^{νπi}J_{ν}}{−i sin νπ} ,

so that

J_{ν} = \frac{H_{ν}^{(1)} + H_{ν}^{(2)}}{2} .

If ν is an integer then a limit has to be taken. In our case, this occurs if k is half-integer, but by continuity in k we may and will assume that this is not the case.

For our purposes, the important property of these functions is the asymptotic behaviour [1, 9.2.7]:

(4.2)  H_{ν}^{(1)}(z) = \sqrt{\frac{2}{πz}} e^{i(z− \frac{νπ}{2} − \frac{π}{4} )} (1 + O (\frac{1}{z})) ,

valid if −π < arg z < 2π (which is to be interpreted in the sense of analytic continuation). Note that the range of validity of this asymptotic development contains the entire upper half plane and that (in contrast to the ordinary Bessel function) this Hankel function has exponential decrease in the positive imaginary direction.

Define ϕ^{(1)}, ϕ^{(2)} : (0, ∞) ↦ ℂ by

ϕ^{(1)}(λ) = (λx)^{\frac{1}{2} −k}H_{k− \frac{1}{2}}^{(1)} (λx)λ^{2k}

and

ϕ^{(2)}(λ) = (λx)^{\frac{1}{2} −k}H_{k− \frac{1}{2}}^{(2)} (λx)λ^{2k}.

Let ϕ_{c}^{(1)} denote the analytic continuation of ϕ^{(1)} from (0, ∞) to ℂ \ {it | t ≤ 0}. If one recalls that J_{ν}(z) = z^{ν} J̃_{ν}(z) with J̃_{ν} entire and ℤ_{2}-invariant, one notes that ϕ_{c}^{(1)}(λ) remains bounded as λ → 0 in ℂ \ {it | t ≤ 0}, since Re k ≥ 0. A small computation will also make it clear that ϕ_{c}^{(1)}(λ) = ϕ^{(2)}(−λ) for λ < 0.

Using the invariance of f and the weight function, we then compute as follows.

∫_{−∞}^{∞} f(λ) (λx)^{\frac{1}{2} −k}J_{k− \frac{1}{2}} (λx)|λ|^{2k} dλ =

= 2 ∫_{0}^{∞} f(λ) (λx)^{\frac{1}{2} −k} J_{k− \frac{1}{2}} (λx)λ^{2k} dλ

= ∫_{0}^{∞} f(λ) (λx)^{\frac{1}{2} −k} (H_{k− \frac{1}{2}}^{(1)} (λx) + H_{k− \frac{1}{2}}^{(2)} (λx)) λ^{2k} dλ

= ∫_{0}^{∞} f(λ) (ϕ^{(1)}(λ) + ϕ^{(2)}(λ)) dλ

= lim_{ϵ↓0} {∫_{ϵ}^{∞} f(λ)ϕ^{(1)}(λ) dλ + ∫_{ϵ}^{∞} f(λ)ϕ^{(2)}(λ) dλ}

= lim_{ϵ↓0} {∫_{ϵ}^{∞} f(λ)ϕ^{(1)}(λ) dλ + ∫_{−∞}^{−ϵ} f(−λ)ϕ^{(2)}(−λ) dλ}

= lim_{ϵ↓0} {∫_{ϵ}^{∞} f(λ)ϕ_{c}^{(1)}(λ) dλ + ∫_{−∞}^{−ϵ} f(λ)ϕ_{c}^{(1)}(λ) dλ} .

Now fϕ_{c}^{(1)} is holomorphic on ℂ \ {it | t ≤ 0}, and, since x > R, it has exponential decrease in the positive imaginary direction, as a consequence of (4.2). The classical argument therefore establishes (4.1), with a minor modification involving a semicircle of radius ϵ around 0 in the upper half plane and using that fϕ_{c}^{1} is bounded around 0. □

Next, we proceed with the reduction of the general radial case to the onedimensional even case. To this end, fix x_{0} ≠ 0 in 𝔞, and define the map Res_{x_{0}} : S(𝔞)^{O(N)} ↦ S(ℝ)^{ℤ_{2}} by restricting to the line passing through x_{0}:

(Res_{x_{0}} f)(s) = f (s\frac{x_{0}}{|x_{0}|}) (s ∈ ℝ, f ∈ S(𝔞)^{O(N)}) .

The following proposition, which implies the Paley-Wiener theorem for radial functions (as will become apparent in the proof of Theorem 4.10), involves the Dunkl transform for general 𝔞 and ℝ both at the same time. We therefore add a subscript ℤ_{2} in the latter case for clarity. Let S(𝔞)^{O(N)} denote the O(N)-invariants in S(𝔞).

Proposition 4.5. Suppose Re k ≥ 0. Let f ∈ S(𝔞)^{O(N)} and put γ = ∑_{α∈R_{+}} k_{α}. Then

Res_{x_{0}} E_{k}D_{0}f = E_{γ,ℤ_{2}}D_{0,ℤ_{2}}Res_{x_{0}} f

for all nonzero x_{0} ∈ 𝔞.

Proof. Let S_{0}(𝔞) = {pψ | p ∈ P}, with ψ denoting the Gaussian as in Section 3. It is known that S_{0}(𝔞) is dense in S(𝔞), see [33, p. 263] for this particular result, or [22] for a general framework for this type of problems. Note that the canonical projection from S(𝔞) onto S(𝔞)^{O(N)} is continuous as a consequence of the closed graph theorem, implying that S_{0}(𝔞)^{O(N)} is dense in S(𝔞)^{O(N)}. It is therefore, by linearity and continuity, sufficient to prove the proposition for a function f of the form |x|^{2q}ψ, where q is a non-negative integer. This can be done using Proposition 3.3 and an identity for Laguerre polynomials. Recall the definition:

L_{n}^{(α)}(x) = ∑_{m=0}^{n} \frac{(−1)^{m}}{m!} \binom{n + α}{n − m}x^{m}.

Then the following identity holds [1, 22.12.6]:

(4.3)  ∑_{m=0}^{n} L_{m}^{(α)}(x)L_{n−m}^{(β)}(y) = L_{n}^{(α+β+1)}(x + y).

It is known [9, Proposition 3.9] that e^{\frac{Δ_{0}}{2}} |x|^{2q} = 2^{q}q!L_{q}^{(N/2 −1)} (−|x|^{2}/2) and e^{− \frac{Δ_{k}}{2}} |x|^{2q} = (−2)^{q}q!L_{q}^{(N/2 +γ−1)} (|x|^{2}/2). Using Proposition 3.3 we therefore find

E_{k}D_{0} (|x|^{2q}ψ) = (e^{− \frac{Δ_{k}}{2}} e^{\frac{Δ_{0}}{2}} |x|^{2q}) ψ

= 2^{q}q! ∑_{m=0}^{q}(−1)^{m}\binom{N/2 + q − 1}{q − m}L_{m}^{(N/2 +γ−1)} (|x|^{2}/2) ψ

= (−2)^{q}q! ∑_{m=0}^{q} \binom{−N/2 − m}{q − m}L_{m}^{(N/2 +γ−1)} (|x|^{2}/2) ψ.

The special case n = q, y = 0, α = N/2 + γ − 1 and β = −N/2 − q of (4.3) then shows that

E_{k}D_{0} (|x|^{2q}ψ) = (−2)^{q}q!L_{q}^{(γ−q)} (|x|^{2}/2) ψ.

Curiously enough, the dimension N has dropped out, and the proposition follows immediately from this observation. □

It follows easily from this result that the Dunkl transform of a radial function is again radial, retrieving this result from [31].

Let D_{0}(B_{R}) = Span{T_{p}(0)f | p ∈ P, f ∈ D(B_{R})^{O(N)}}. The following density result is crucial in the step from the radial to the general case.

Proposition 4.6. D_{0}(B_{R}) is dense in D(B_{R}).

This proposition implies a special case of [19, Cor. 7.8, p. 310], but the latter result does not seem to imply the proposition. The proof of Proposition 4.6 is based on representation theory for compact groups in general and O(N) in particular; we recall a few facts to start with.

Let π : K ↦ Aut(E) be a strongly continuous representation of the compact group K in the automorphism group of a Fréchet space E. A vector e ∈ E is Kfinite if the orbit of e under K spans a finite dimensional subspace of E. If δ ∈ K̂ (the unitary dual of K), then a vector e ∈ E is K-finite of type δ if the span of the orbit of e decomposes into finitely many copies of δ. For δ ∈ K̂, let E_{δ} denote the subspace of K-finite vectors of type δ. By [18, Lemma 1.9, p. 396] the subspace ⨁_{δ∈K̂} E_{δ} of K-finite vectors is dense in E.

The natural action of O(N) on D(B_{R}) is such a strongly continuous representation. Hence Proposition 4.6 will be implied by the density of the K-finite vectors, as mentioned above, once we have proved the following result.

Proposition 4.7. D_{0}(B_{R}) is the space of O(N)-finite vectors in D(B_{R}).

Obviously, any element of D_{0}(B_{R}) is O(N)-finite, but the reverse appears to be harder. The proof of Proposition 4.7 below uses some Fourier analysis; for clarity of notation, we will temporarily denote the ordinary Fourier transform by F rather than D_{0}. We need the following classical result in the representation theory of O(N).

Theorem 4.8. Let H_{n} be the harmonic polynomials in P_{n}. Any p ∈ P_{n} has a unique decomposition of the form

p = ∑_{l=0}^{[ \frac{n}{2} ]} |x|^{2l}p_{l}

with p_{l} ∈ H_{n−2l}. The map π_{n,l} : P_{n} ↦ H_{n−2l} that sends p to p_{l} is O(N)- equivariant. Furthermore, the representations of O(N) on the spaces H_{n} (n = 0, 1, …) are irreducible and mutually inequivalent.

As a further preparation for the proof of Proposition 4.7 we need the following result.

Lemma 4.9. Let δ ∈ Ô(N) and f ∈ D(B_{R}). Suppose that the orbit of f under O(N) spans a copy V_{δ} of δ in D(B_{R}). Then there exist a harmonic homogeneous polynomial p of type δ and scalars λ_{m} (m = 0, 1, …), such that F(f)(z) = p(z) ∑_{m=0}^{∞} λ_{m}(z, z)^{m} (z ∈ ℂ^{N}), where the series converges uniformly on compact subsets of ℂ^{N}.

Proof. By the Paley–Wiener theorem the Fourier transform F(f) is an entire function on ℂ^{N}. F commutes with the O(N)-action, so there is a copy of δ in the space of functions of Paley–Wiener type. Since the action of O(N) on entire functions is homogeneous (in the sense that the action is on each homogeneous component of the power series development around 0 separately), we conclude that δ also occurs in P. By Theorem 4.8 there exists a unique copy, H_{l} say, of δ as the harmonics of some homogeneous degree.

For n ≥ 0, let π_{n,l} denote the map P_{n} ↦ H_{l} defined by the decomposition in Theorem 4.8. Also, for n ≥ 0 and g ∈ V_{δ}, let F_{n}(g) denote the component of homogeneous degree n in the power series development of F(g) around 0. Consider the O(N)-equivariant maps π_{n,l} ∘ F_{n} : V_{δ} ↦ H_{l} (n ≥ 0). Since F(f) ≠ 0, there exists n_{0} such that π_{n_{0},l} ∘ F_{n_{0}} is non-zero, hence an isomorphism. But then for all n the map (π_{n,l} ∘ F_{n}) ∘ (π_{n_{0},l} ∘ F_{n_{0}})^{−1} : H_{l} ↦ H_{l} must be equal to a multiple λ_{n} of the identity. Hence f(z) = ∑_{m=0}^{∞} λ_{m}(z, z)^{m}p(z) (z ∈ ℂ^{N}) for some nonzero p ∈ H_{l}. Since there exist z ∈ ℂ^{N} with p(z) ≠ 0 and |z| arbitrarily large, the factor p(z) can be brought outside the summation. □

Proof of Proposition 4.7. As remarked before, D_{0}(B_{R}) is certainly contained in the space of O(N)-finite vectors in D(B_{R}). Conversely, let f ∈ D(B_{R}) be an O(N)- finite vector. We may assume that the orbit of f spans a copy of δ in D(B_{R}) for some δ ∈ Ô(N). Applying Lemma 4.9 one obtains a factorization F(f) = ph of F(f) in a polynomial p and an entire O(N)-invariant function h. We proceed to show that h is in fact in H_{B_{R}}. By [32, Lemma 8.3] there exists a constant A > 0 such that

(4.4)  |h(z)| ≤ A ∫_{T^{N}} |F(f)(z + w)| dσ_{N}(w) (z ∈ ℂ^{N}),

where T^{N} = {(e^{iθ_{1}}, …, e^{iθ_{N}}) | θ_{j} ∈ ℝ (j = 1, …, N)} is the N-torus and σ_{N} is the normalized Haar measure on T^{N}. There exist constants γ_{M} (M = 0, 1, 2, …) such that

(4.5)  |F(f)(z)| ≤ γ_{M}(1 + |z|)^{−M}e^{R|Im z|} (z ∈ ℂ^{N}).

Now note that

\frac{1 + |z|}{1 + |z + w|} ≤ 1 + \frac{|w|}{1 + |z + w|} ≤ 1 + \sqrt{N} (z ∈ ℂ^{N}, w ∈ T^{N}).

Since in addition |Im (z + w)| ≤ |Im z| + \sqrt{N} for z ∈ ℂ^{N} and w ∈ T^{N}, we conclude from (4.4) that h also satisfies estimates as in (4.5), i.e., h ∈ H_{B_{R}}. Hence F^{−1}(h) is in D(B_{R}), and in fact F^{−1}h ∈ D(B_{R})^{O(N)} since h is O(N)-invariant. This implies that f ∈ D_{0}(B_{R}), as required. □

This concludes the proof of Proposition 4.6.

We can now prove the Paley–Wiener theorem of this section by putting the pieces together.

Theorem 4.10 (Paley–Wiener theorem, first version). Let G be a finite reflection group and suppose that Re k ≥ 0. Then the Dunkl transform D_{k} is a linear isomorphism between D(B_{R}) and H_{B_{R}}, for all R ≥ 0.

Proof. In view of the inversion theorem, all that needs to be done is to show that E_{k}g ∈ D(B_{R}) for all g ∈ H_{B_{R}}. To this end, start by noting that E_{k}D_{0}f ∈ D(B_{R}) if f ∈ D(B_{R})^{O(N)}, as an immediate consequence of Proposition 4.5 and Theorem 4.4. But then we also have E_{k}D_{0}f ∈ D(B_{R}) for f ∈ D_{0}(B_{R}), since E_{k} ∘ D_{0} ∘ ∂_{ξ} = T_{ξ}(k) ∘ E_{k} ∘ D_{0}. From Proposition 4.6 and continuity we can then conclude that E_{k}D_{0}f ∈ D(B_{R}) for f ∈ D(B_{R}). By the Paley–Wiener theorem for D_{0}, we see that E_{k}g ∈ D(B_{R}) for all g ∈ H_{B_{R}}, as required. □

4.2. The case of arbitrary G and strictly positive integral k. If all k_{α} are strictly positive integers, Conjecture 4.1 can be established from the corresponding theorem for the Fourier transform, by showing that Exp_{G} is then in fact an elementary function. This can be seen by starting from Heckman’s work on shift operators [16].

We introduce some notation. Let R = ⋃_{i=1}^{m} S_{i} be the decomposition of R into G-orbits, and put S_{i,+} = S_{i} ∩ R_{+}. Define π_{i} = ∏_{α∈S_{i,+}} α^{∗}, and let Res denote restriction to the G-invariant functions. Then we have the following results [16], valid for p ∈ P^{G}:

(4.6)  Res (M_{π_{i}}^{−1}T_{π_{i}}(k)) Res (T_{p}(k)) = Res (T_{p}(k + 1_{S_{i}})) Res (M_{π_{i}}^{−1}T_{π_{i}}(k)) ,
(4.7) Res (T_{π_{i}}(k)M_{π_{i}}) Res (T_{p}(k + 1_{S_{i}})) = Res (T_{p}(k)) Res (T_{π_{i}}(k)M_{π_{i}}) ,

where (k + 1_{S_{i}})_{α} = k_{α} + 1 if α ∈ S_{i} and (k + 1_{S_{i}})_{α} = k_{α} if α ∉ S_{i}.

The generalized Bessel kernel J_{G} is defined for k ∈ K^{reg} by

(4.8)  J_{G}(λ, k, x) = \frac{1}{|G|} ∑_{g∈G} Exp_{G}(λ, k, gx)

and is (up to multiples) the unique G-invariant solution of the Bessel system

T_{p}(k)f = p(λ)f (p ∈ P^{G}).

It follows immediately from (4.6), (4.7) and the uniqueness of invariant solutions of the Bessel system that there exist complex constants c_{1,2}(λ, k) such that

M_{π_{i}}^{−1}T_{π_{i}}(k)J_{G}(λ, k, .) = c_{1}(λ, k)J_{G}(λ, k + 1_{S_{i}}, .)

and

T_{π_{i}}(k)M_{π_{i}}J_{G}(λ, k + 1_{S_{i}}, .) = c_{2}(λ, k)J_{G}(λ, k, .).

Evaluation at zero yields c_{2}(λ, k) = (π_{i}, π_{i})_{k}, where ( . , .)_{k} is the bilinear form from Section 3.

c_{1}(λ, k)(π_{i}, π_{i})_{k}J_{G}(λ, k, .) = c_{1}(λ, k)T_{π_{i}}(k)M_{π_{i}}J_{G}(λ, k + 1_{S_{i}}, .)
= T_{π_{i}}(k)M_{π_{i}}M_{π_{i}}^{−1}T_{π_{i}}(k)J_{G}(λ, k, .)
= π_{i}^{2}(λ)J_{G}(λ, k, .).

Hence

(4.9)  π_{i}^{2}(λ)J_{G}(λ, k + 1_{S_{i}}, .) = (π_{i}, π_{i})_{k}M_{π_{i}}^{−1}T_{π_{i}}(k)J_{G}(λ, k, .).

Suppose then that the (fixed) value of k on S_{i} is k_{i}, a strictly positive integer. Repeated application of (4.9) yields

w_{k}(λ)J_{G}(λ, k, .) = ∏_{i=1}^{m} ∏_{n=1}^{k_{i}}(π_{i}, π_{i})_{(n−1)1_{S_{i}} +∑_{j<i} k_{j}1_{S_{j}}} · (4.10)

· ∏_{i=1}^{m} ∏_{n=1}^{k_{i}} M_{π_{i}}^{−1}T_{π_{i}} ((n − 1)1_{S_{i}} + ∑_{j<i} k_{j}1_{S_{j}} ) \frac{1}{|G|} ∑_{g∈G} e^{(gλ, .)}.

We conclude from (4.10) that there exist functions p_{g}(λ, x) on 𝔞_{ℂ}×𝔞_{ℂ}^{reg} (we suppress the dependence on our fixed k) which are polynomials in λ with rational functions in x as coefficients (with poles with respect to x along the singular hyperplanes, if any) such that

w_{k}(λ)J_{G}(λ, k, x) = ∑_{g∈G} p_{g}(λ, x)e^{(gλ,x)} (λ ∈ 𝔞_{ℂ}, x ∈ 𝔞^{reg}),

where as usual 𝔞^{reg} = {x ∈ 𝔞 | (α, x) ≠ 0 ∀α ∈ R} and 𝔞_{ℂ}^{reg} is defined similarly. Now the first operator acting in (4.10) is T_{π_{1}}(0), showing that the p_{g} are divisible by π_{1}(λ). Since the numbering of the S_{i} is arbitrary we see that the p_{g} are in fact divisible by π(λ) = ∏_{i=1}^{m} π_{i}(λ). Thus

(4.11)  w_{k}(λ)J_{G}(λ, k, x) = π(λ) ∑_{g∈G} p′_{g}(λ, x)e^{(gλ,x)} (λ ∈ 𝔞_{ℂ}, x ∈ 𝔞^{reg}),

where the p′_{g} are of the same form as the p_{g}. It is here that we have used that the k_{α} are strictly positive integers, in order to obtain the factor π(λ) in the right hand side of (4.11) which is needed below.

There is a similar expression for Exp_{G}, which we obtain using a result of Opdam [25] describing how Exp_{G} may be obtained from J_{G}. According to [loc.cit., 6.4-6.6] there exists a rational function Q(λ, x) on 𝔞_{ℂ} × 𝔞_{ℂ} such that:

• (λ, x) ↦ π(λ)Q(λ, x) is a polynomial on 𝔞_{ℂ} × 𝔞_{ℂ};
• T_{Q(λ, .)}(k)J_{G}(λ, k, .) = Exp_{G}(λ, k, .) (λ ∈ 𝔞_{ℂ}^{reg}, x ∈ 𝔞_{ℂ}).

Using this and (4.11) we see that, for λ ∈ 𝔞_{ℂ}^{reg} and x ∈ 𝔞^{reg},

(4.12)  w_{k}(λ)Exp_{G}(λ, k, x) = T_{Q(λ, .)}w_{k}(λ)J_{G}(λ, k, .)

= T_{Q(λ, .)}π(λ) ∑_{g∈G} p′_{g}(λ, x)e^{(gλ,x)}

= T_{π(λ)Q(λ, .)} ∑_{g∈G} p′_{g}(λ, x)e^{(gλ,x)}

= ∑_{g∈G} p′′_{g}(λ, x)e^{(gλ,x)},

where again the p′′_{g} are of the same form as the p_{g}. But then (4.12) is by continuity in fact valid for λ ∈ 𝔞_{ℂ} (rather than λ ∈ 𝔞_{ℂ}^{reg}) and x ∈ 𝔞^{reg}. This observation allows us to establish Conjecture 4.1 for strictly positive integral multiplicities.

Theorem 4.11 (Paley–Wiener theorem, second version). Let G be a finite reflection group and suppose that the k_{α} are all strictly positive integers. If S is a non-empty G-invariant compact convex subset of 𝔞, then D_{k} is a linear isomorphism between D(S) and H_{S}.

Proof. Suppose f ∈ H_{S}, x ∈ 𝔞, x ∉ S. In view of the inversion theorem, the only thing that remains to be proved is:

(4.13)  ∫_{𝔞} f(λ)Exp_{G}(iλ, k, x)w_{k}(λ) dλ = 0.

If x ∈ 𝔞^{reg}, then (4.12) allows us to write this integral as

∑_{g∈G} ∫_{𝔞} f(λ)p′′_{g}(iλ, x)e^{i(λ,g^{−1}x)} dλ,

where the p′′_{g} are polynomials in λ with rational functions in x as coefficients (with poles with respect to x along the singular hyperplanes, if any). Hence the geometrical form of the Paley–Wiener theorem for the Fourier transform [36, Exercise 29.5] shows that each of the summands is zero. Then (4.13) holds for all x ∉ S by continuity. □

4.3. The case where G is a Weyl group and Re k ≥ 0. In the case where G is a Weyl group and Re k ≥ 0, a Paley–Wiener theorem for intersections of convex hulls of orbits can be obtained by a limit transition from the results of Opdam on Cherednik operators for k ≥ 0 [26]. If every non-empty G-invariant compact convex set were the intersection of the convex hulls of orbits, then this would establish Conjecture 4.1 for Weyl groups. However, this intersection property does not hold, as the two-dimensional example of the closed unit ball and A_{1} × A_{1} shows. Thus, Conjecture 4.1 remains open even for Weyl groups, and this example also shows that in this context neither one of the Theorems 4.10 above and 4.15 below implies the other.

We start by collecting the relevant results from [26].

Suppose that R is an integral root system, and let k : R ↦ ℂ be a multiplicity function. We will assume that R is reduced, which is sufficient for our purposes. Choose a system R_{+} of positive roots, and define the Cherednik operator D_{ξ}, for ξ ∈ 𝔞_{ℂ}, as

D_{ξ}(k) = ∂_{ξ} + ∑_{α∈R_{+}} k_{α}(α, ξ)M(1 − e^{−α^{∗}})^{−1}(1 − r_{α}) − (ρ(k), ξ),

where ρ(k) = \frac{1}{2} ∑_{α∈R_{+}} k_{α}α. The Cherednik operators are not equivariant, and their definition is dependent on the choice of R_{+}. For fixed multiplicity they form a commutative family, and together with G they then generate an algebra of operators which is an isomorphic copy of the graded Hecke algebra, corresponding to the choice of R_{+} and k, as it is defined by Lusztig [23]. We refer to [4] for this isomorphism, or to [26]. Aside, we remark that the D_{ξ}(k) certainly depend on the length of the roots, in contrast to the Dunkl operators, but it will become apparent that this is innocent for our purposes.

The Dunkl kernel Exp_{G} can be obtained from the eigenfunctions of the Cherednik operators by taking a suitable limit. To see this, we recall a weakened version of some of Opdam’s results [26, c.f. Theorem 3.15 and Proposition 6.1].

Theorem 4.12. For all k ≥ 0 there exists an open neighborhood U of 0 ∈ 𝔞 and a holomorphic function C_{G}( . , k, .) : 𝔞_{ℂ} × (𝔞 + iU) ↦ ℂ with the following properties:

(1) C_{G}(λ, k, 0) = 1 for all λ ∈ 𝔞_{ℂ};

(2) D_{ξ}(k)C_{G}(λ, k, .) = (λ, ξ)C_{G}(λ, k, .) for all λ, ξ ∈ 𝔞_{ℂ};

(3) if λ ∈ 𝔞_{ℂ}, and z = x + iy with x ∈ 𝔞 and y ∈ U, then

(4.14) |C_{G}(λ, k, z) ≤ \sqrt{|G|}e^{− min_{g∈G}(g Im λ,y)+max_{g∈G}(g ρ(k),y)+max_{g∈G}(g Re λ,x)}.

The estimate in (4.14) enables us to prove the following limit transition.

Theorem 4.13. Let k ≥ 0 be fixed, and suppose that S ⊂ 𝔞_{ℂ} × 𝔞_{ℂ} is a non-empty compact subset. Then, as ϵ → 0 through the complex numbers, we have

(4.15)  lim_{ϵ→0} C_{G}(ϵ^{−1}λ, k, ϵz) = Exp_{G}(λ, k, z),

uniformly for (λ, z) ∈ S. Here it is understood that |ϵ| is taken sufficiently small, so that C_{G}(ϵ^{−1}λ, k, ϵz) is defined for all (λ, z) ∈ S.

Proof. Suppose, to the contrary, that there exist η > 0, a non-empty compact set S ⊂ 𝔞_{ℂ} × 𝔞_{ℂ}, and a complex sequence ϵ_{n} → 0 such that, for all n,

(4.16)  sup_{(λ,z)∈S} |C_{G}(ϵ_{n}^{−1}λ, k, ϵ_{n}z) − Exp_{G}(λ, k, z)| ≥ η.

Take an open neighborhood of S of the form V_{1} × V_{2}, with V_{1}, V_{2} ⊂ 𝔞_{ℂ} open balls centered at the origin, and define ϕ_{n} : V_{1} × V_{2} ↦ ℂ by ϕ_{n}(λ, z) = C_{G}(ϵ_{n}^{−1}λ, k, ϵ_{n}z), discarding a finite number of ϵ_{n} if necessary. From (4.14) one sees that the sequence {ϕ_{n}}_{n=1}^{∞} is uniformly bounded on V_{1} × V_{2}. By Montel’s theorem we may therefore assume, passing to a subsequence if necessary, that the ϕ_{n} converge uniformly on compact subsets of V_{1} × V_{2} to a holomorphic function ϕ_{∞} on V_{1} × V_{2}.

Now a small computation shows, for fixed λ ∈ V_{1} and for arbitrary ξ ∈ 𝔞_{ℂ}, that as functions on V_{2} we have

\{∂_{ξ} + ϵ_{n} ∑_{α∈R_{+}} k_{α} \frac{(α, ξ)}{1 − e^{−ϵ_{n}(α, .)}} (1 − r_{α})\} ϕ_{n}(λ, .) = \{(ξ, λ) + ϵ_{n}(ρ(k), ξ)\} ϕ_{n}(λ_{,} .).

Passing to the limit one obtains

\{∂_{ξ} + ∑_{α∈R_{+}} k_{α} \frac{(α, ξ)}{(α, .)}(1 − r_{α})\} ϕ_{∞}(λ, .) = (ξ, λ)ϕ_{∞}(λ_{,} .).

This is the defining equation for the Dunkl kernel, which has a local solution space spanned by Exp_{G}(λ, k, .). Since obviously ϕ_{∞}(λ, 0) = 1, we conclude that ϕ_{∞}(λ, z) = Exp_{G}(λ, k, z) for all (λ, z) ∈ V_{1} × V_{2}. But this implies that the ϕ_{n} converge uniformly to Exp_{G}( . , k, .) on the compact set S ⊂ V_{1} × V_{2} after all, which contradicts (4.16). □

In the end, we will only need the special case where λ ∈ i𝔞 of the following corollary. Note that the function involved is defined for all ϵ under consideration.

Corollary 4.14. Let k ≥ 0. Then, as ϵ → 0 through the real numbers, we have lim_{ϵ→0} C_{G}(ϵ^{−1}λ, k, ϵx) = Exp_{G}(λ, k, x) for all λ ∈ 𝔞_{ℂ} and all x ∈ 𝔞.

We will now invoke Opdam’s Paley–Wiener theorem. To this end, fix x ∈ 𝔞, let co(G · x) denote the convex hull of the orbit of x, and fix y ∉ co(G · x). Suppose that f ∈ H_{co(G·x)}. Then, if k_{α} > 0 for all α, the Paley–Wiener theorem for the Cherednik transform [26, Theorem 8.6.(2)] implies that

∫_{𝔞} f(λ)C_{G}(iλ, k, y) ∏_{a∈R_{+}} (1 − \frac{k_{α}(α, α)}{2i(λ, α)}) ∏_{α∈R_{+}} |\frac{Γ (\frac{2i(λ,α)}{(α,α)} + k_{α})}{Γ (\frac{2i(λ,α)}{(α,α)})}|^{2} dλ = 0,

where the integrand should be read as 0 for all λ ∈ 𝔞^{sing}. Now note, for real ϵ ≠ 0, that the map λ ↦ f(ϵλ) is an element of H_{co(G·ϵx)}. Since ϵy ∉ co(G · ϵx), we have

∫_{𝔞} f(ϵλ)C_{G}(iλ, k, ϵy) ∏_{a∈R_{+}} (1 − \frac{k_{α}(α, α)}{2i(λ, α)}) ∏_{α∈R_{+}} |\frac{Γ (\frac{2i(λ,α)}{(α,α)} + k_{α})}{Γ (\frac{2i(λ,α)}{(α,α)})}|^{2} dλ = 0.

After a change of variables it follows that (4.17)

∫_{𝔞} f(λ)C_{G}(iϵ^{−1}λ, k, ϵy) ∏_{a∈R_{+}} (1 − \frac{ϵk_{α}(α, α)}{2i(λ, α)}) ∏_{α∈R_{+}} |ϵ^{k_{α}} \frac{Γ (\frac{2iϵ^{−1}(λ,α)}{(α,α)} + k_{α})}{Γ (\frac{2iϵ^{−1}(λ,α)}{(α,α)})}|^{2} dλ = 0,

for all ϵ > 0. Now it follows from a well-known result for the Gamma function [34, p. 151] that, for each a ∈ ℝ, there exist constants M_{1}(a), M_{2}(a), M̃_{1}(a), M̃_{2}(a) ≥ 0 such that, for all s ∈ ℝ,

(4.18)  |\frac{Γ(is + a)}{Γ(is)}| ≤ M_{1}(a)|s|^{a} + M_{2}(a),
(4.19)  |(1 − \frac{a}{is}) \frac{Γ(is + a)}{Γ(is)}| ≤ M̃_{1}(a)|s|^{a} + M̃_{2}(a).

Furthermore, for all s, a ∈ ℝ, s ≠ 0, one has

(4.20)  lim_{ϵ↓0} |ϵ^{a} \frac{Γ(iϵ^{−1}s + a)}{Γ(iϵ^{−1}s)}| = |s|^{a}.

From (4.18) and (4.19) one sees that the dominated convergence theorem applies in (4.17) as ϵ ↓ 0, since f ∈ S(𝔞) and |C_{G}(iϵ^{−1}λ, k, ϵy)| ≤ \sqrt{|G|} by (4.14). In the limit one thus obtains from (4.20) and Corollary 4.14 that

(4.21)  ∫_{𝔞} f(λ)Exp_{G}(iλ, k, y)w_{k}(λ) dλ = 0,

for k > 0, x ∈ 𝔞, f ∈ H_{co(G·x)}, and y ∉ co(G · x).

It is now an easy matter to prove the following.

Theorem 4.15 (Paley–Wiener theorem, third version). Let G be a Weyl group and suppose that Re k ≥ 0. If S ⊂ 𝔞 is an intersection of convex hulls of G-orbits, then the Dunkl transform is a linear isomorphism between D(S) and H_{S}.

Proof. As in the proof of Theorem 4.11, in view of the inversion theorem all that needs to be done is to prove that E_{k} : H_{S} ↦ S(𝔞) has its image contained in D(S). To this end, first assume that S is the convex hull of one orbit and that k > 0. In that case, (4.21) is just the required result. But then (4.21) actually holds for Re k > 0 by analytic continuation in k, and the case Re k ≥ 0 follows from this again by continuity. The result for an intersection of convex hulls of orbits follows trivially from the result for the convex hull of one orbit. □

Remark 4.16. The type of limit transition above has also been introduced by Ben Saïd and Ørsted [3]. The approach in [loc.cit.] starts from the invariant case, using shift operators, and then proceeds to the general case. The use of shift operators thus restricts the validity of the proofs to integral multiplicities. The approach to this transition in the present paper appears to be somewhat simpler, since Corollary 4.14 is seen to hold directly in the general case and for more general multiplicities.

5. Extending the intertwining operator

In this section we return to the general situation of an arbitrary reflection group and Re k ≥ 0. With the aid of the Paley–Wiener theorems, we will define a linear isomorphism V_{k} : C^{∞}(𝔞) ↦ C^{∞}(𝔞), such that T_{ξ}(k)V_{k} = V_{k}∂_{ξ} (ξ ∈ 𝔞), and which extends the intertwiner operator as it has originally been constructed for polynomials by Dunkl for k ≥ 0 [8]. The polynomial case for general regular k was considered in [11] and an extension of the intertwiner operator to an algebra of real analytic functions was constructed in [9] for k ≥ 0. The extension to C^{∞}(𝔞) for Re k ≥ 0 as in this section was first established in [21]. The case where k ≥ 0 and ∑_{α∈R_{+}} k_{α} > 0 was later on also considered in [37], using Rösler’s representing measures which are presently only known to exist when k ≥ 0.

Recall that the symmetry S, defined by Sf(x) = f(−x), relates E_{k} and D_{k} by SD_{k} = D_{k}S = E_{k} [20, Lemma 4.3]. The Paley–Wiener theorem 4.10 therefore implies that D_{k}E_{0} = SE_{k}E_{0} maps D(𝔞) into itself. This map is easily seen to be continuous in the standard inductive limit topology (e.g., as a consequence of the closed graph theorem) and it is actually a homeomorphism since the inverse is D_{0}E_{k}. Thus the transpose (D_{k}E_{0})^{t} : D′(𝔞) ↦ D′(𝔞) is a homeomorphism in the weak^{∗}-topology, leaving S′(𝔞) invariant as a consequence of part (5) in Theorem 2.2.

For the formulation of the following theorem, we let B′_{R} denote the open ball of radius R, centered at the origin.

Theorem 5.1. Let G be a finite reflection group and suppose that Re k ≥ 0. The map W_{k} : C^{∞}(𝔞) ↦ D′(𝔞), defined by

W_{k}f = \frac{c_{0}}{c_{k}} (D_{k}E_{0})^{t}(fw_{k}),

is actually a linear automorphism of C^{∞}(𝔞). Let V_{k} : C^{∞}(𝔞) ↦ C^{∞}(𝔞) be its inverse. Then both W_{k} and V_{k} commute with the G-action and the following hold:

(1) W_{k}T_{ξ}(k) = ∂_{ξ}W_{k} (ξ ∈ 𝔞).

(2) W_{k}1 = 1.
(3) W_{k}P_{n} = P_{n}.
(4) (a) If f ∈ C^{∞}(𝔞) vanishes on B′_{R}, then W_{k}f vanishes on B′_{R}.

(b) If U is an open G-invariant convex subset of 𝔞, and f ∈ C^{∞}(𝔞) van-

ishes on U, then V_{k}f vanishes on U.

(5) If f ∈ C^{∞}(𝔞) ∩ L_{1}(𝔞, |w_{k}(x)|dx), D_{k}f ∈ L_{1}(𝔞, |w_{k}(x)|dx), and x ∈ 𝔞, then

W_{k}f(x) = \frac{c_{0}}{c_{k}} E_{0}(w_{k}D_{k}f)(x) = \frac{1}{c_{k}} ∫_{𝔞} D_{k}f(λ) exp(i(λ, x))w_{k}(λ) dλ.

(6) If f ∈ C^{∞}(𝔞) ∩ L_{1}(𝔞, dx), D_{0}f ∈ L_{1}(𝔞, dx), and x ∈ 𝔞, then

V_{k}f(x) = \frac{c_{k}}{c_{0}} E_{k}(w_{k}^{−1}D_{0}f) = \frac{1}{c_{0}} ∫_{𝔞} D_{0}f(λ)Exp_{G}(ix, k, λ) dλ.

Proof. As a preparation, we recall from [20, Lemma 4.13] that ∫_{𝔞}(D_{k}f)gw_{k} dx = ∫_{𝔞} f(D_{k}g)w_{k} dx for f, g ∈ L_{1}(𝔞, |w_{k}(x)|dx), and similarly for E_{k}.

Suppose T ∈ D′(𝔞) and supp T ∩ B′_{R} = ∅. Then supp (D_{k}E_{0})^{t}T ∩ B′_{R} = ∅. Indeed, let ψ ∈ D(B′_{R}). Then ⟨(D_{k}E_{0})^{t}T, ψ⟩ = ⟨T, D_{k}E_{0}ψ⟩ = ⟨T, E_{k}D_{0}ψ⟩ = 0 by the Paley–Wiener theorem 4.10.

Now let f ∈ C^{∞}(𝔞). Fix R > 0 and choose ϕ ∈ D(𝔞) such that ϕ = 1 on B′_{R}. Then supp (ϕfw_{k} − fw_{k}) ∩ B′_{R} = ∅, hence by the above we have for arbitrary ψ ∈ D(B′_{R}):

(5.1)  ⟨(D_{k}E_{0})^{t}(fw_{k}), ψ⟩ = ⟨(D_{k}E_{0})^{t}(ϕfw_{k}), ψ⟩
= ∫_{𝔞} ψ(x)E_{0} (D_{k}(ϕf)w_{k}) (x) dx.

Note that D_{k}(ϕf) ∈ S(𝔞), so E_{0} (D_{k}(ϕf)w_{k}) is smooth. Since the left hand side in (5.1) does not depend on ϕ, the restriction of E_{0} (D_{k}(ϕf)w_{k}) to B′_{R} is apparently also independent of the choice of ϕ. Hence we can unambiguously define a smooth function representing the distribution (D_{k}E_{0})^{t}(fw_{k}), by choosing for x ∈ 𝔞 any B′_{R} such that x ∈ B′_{R}, any ϕ ∈ D(𝔞) such that ϕ = 1 on B′_{R}, and putting ((D_{k}E_{0})^{t}(fw_{k})) (x) = E_{0} (D_{k}(ϕf)w_{k}) (x). This shows that W_{k} maps C^{∞}(𝔞) into itself. It follows from the remarks before the statement of the theorem that W_{k} is injective.

In order to see that W_{k} is also surjective we consider (D_{0}E_{k})^{t}. For (D_{0}E_{k})^{t} a geometrically more precise property with respect to supports can be proved than for (D_{k}E_{0})^{t}. To be precise: if U is an open G-invariant convex subset of 𝔞, and T ∈ D′(𝔞) is such that supp T ∩ U = ∅, then supp (D_{0}E_{k})^{t}T ∩ U = ∅. Indeed, if ψ ∈ D(U), then ⟨(D_{0}E_{k})^{t}T, ψ⟩ = ⟨T, D_{0}E_{k}ψ⟩ = ⟨T, E_{0}D_{k}ψ⟩ = 0 by the geometrical form of the Paley–Wiener theorem for the ordinary Fourier transform—note that D_{k}ψ is of Paley–Wiener type corresponding to the G-invariant compact convex set co(G · supp ψ) ⊂ U. Let R > 0 and f ∈ C^{∞}(𝔞) and choose ϕ as above. Taking the open ball B′_{R} as U, we then compute as follows for arbitrary ψ ∈ D(B′_{R}):

(5.2) ⟨(D_{0}E_{k})^{t}f, ψ⟩ = ⟨(D_{0}E_{k})^{t}(ϕf), ψ⟩
= ∫_{𝔞} D_{0}(ϕf)(λ)(E_{k}ψ)(λ) dλ

= c_{k}^{−1} ∫_{𝔞} D_{0}(ϕf)(λ) {∫_{𝔞} ψ(x)Exp_{G}(ix, k, λ)w_{k}(x) dx} dλ

= ∫_{𝔞} {c_{k}^{−1} ∫_{𝔞} D_{0}(ϕf)(λ)Exp_{G}(ix, k, λ) dλ} ψ(x) w_{k}(x) dx

Now the function g_{ϕ} : x ↦ c_{k}^{−1} ∫_{𝔞} D_{0}(ϕf)(λ)Exp_{G}(ix, k, λ) dλ is smooth on 𝔞 since the derivatives of Exp_{G}(i . , k, λ) are of polynomial growth in λ [20, Corollary 3.7] and D_{0}(ϕf) ∈ S(𝔞). The g_{ϕ} patch together, as before, to a smooth function g on 𝔞 which satisfies (D_{0}E_{k})^{t}f = gw_{k} by construction. Taking inverses we find (D_{k}E_{0})^{t}(gw_{k}) = f, showing that W_{k} is surjective, as required.

It is easy to check that W_{k} and hence its inverse V_{k} commute with the G-action, using an invariant function ϕ in the construction, and noting that D_{k} and E_{0} have this property.

Continuing, we define the Dunkl operator T′_{ξ}(k) on D′(𝔞) by ⟨T′_{ξ}(k)T, ψ⟩ = −⟨T, T_{ξ}(k)ψ⟩. We have the compatibility T′_{ξ}(k)(fw_{k}) = (T_{ξ}(k)f)w_{k} for f ∈ C^{∞}(𝔞). Hence

⟨(D_{k}E_{0})^{t}((T_{ξ}(k)f)w_{k}), ψ⟩ = ⟨(T_{ξ}(k)f)w_{k}, D_{k}E_{0}ψ⟩
= ⟨T′_{ξ}(k)(fw_{k}), D_{k}E_{0}ψ⟩
= −⟨fw_{k}, D_{k}E_{0}∂_{ξ}ψ⟩
= ⟨∂_{ξ}((D_{k}E_{0})^{t}(fw_{k})), ψ⟩,

proving part (1).

For part (2) and part (3) we consider D_{k}^{t} : S′(𝔞) ↦ S′(𝔞). By the inversion theorem we have D_{k}^{t}w_{k} = c_{k}δ, hence (D_{k}E_{0})^{t}w_{k} = c_{k}E_{0}^{t}δ = c_{k}c_{0}^{−1}, proving part (2). Furthermore, one easily verifies the relation D_{k}^{t}(M_{ξ^{∗}}T) = iT′_{ξ}(k)D_{k}^{t}T for T ∈ S′(𝔞) and ξ ∈ 𝔞. Thus for p ∈ P_{n} we have D_{k}^{t}(pw_{k}) = c_{k}i^{n}T′_{p}(k)δ. By the homogeneity of the T_{ξ}(k), there exists a differential operator D̃ with constant coefficients and of homogeneous degree n such that c_{k}i^{n}T′_{p}(k)δ = D̃δ. Hence (D_{k}E_{0})^{t}(pw_{k}) = E_{0}^{t}D̃δ is in P_{n}, as stated in part (3).

Part (4) follows from the properties regarding the supports of distributions which we have used above.

The parts (5) and (6) follow by performing the computations in (5.1) and (5.2) with ϕ = 1, which is validated by the integrability conditions. □

Remark 5.2.

(1) It is easily seen that the validity of Conjecture 4.1 would imply that the open ball in part (4)(a) for W_{k} could be replaced by an open G-invariant convex set, as for V_{k} in part (4)(b). In fact, if all multiplicities are strictly positive integers, then we already know that this stronger statement actually holds, in view of Theorem 4.11. Likewise, if the groups is a Weyl group, then we known that this stronger statement holds for the interior of an intersection of convex hulls of orbits, in view of Theorem 4.15.

(2) For k ≥ 0, part (4)(b) also follows from the properties of Rösler’s representing measures, cf. [37].

6. Connection with the Cartan motion group

If G is a Weyl group W, then, for certain multiplicities, the W -invariant part of the theory for Dunkl operators has an interpretation in terms of the Cartan motion group, as we will explain in this section. The starting point is Heckman’s result

(6.1)  T_{p}(k) = \frac{1}{n!} (ad \frac{Δ_{k}}{2} )^{n} M_{p},

for p ∈ P_{n}, cf. Section 3. A double application of this result enables us to give an explicit description of the radial parts in terms of Dunkl operators. From this description we can then conclude that the generalized Bessel functions coincide with the restriction of the spherical functions to 𝔞. This identification, in turn, enables us to identify the restriction to the invariants of the operators W_{k} and V_{k} from Theorem 5.1 in terms of (the flat analogue of) the Abel transform. Furthermore, we show that in certain cases shift operators can be used to essentially invert the Abel transform by an invariant differential operator.

Establishing terminology, let G be a connected non-compact semisimple Lie group with finite center, maximal compact subgroup K and corresponding Cartan decomposition 𝔤 = 𝔨 ⊕ 𝔭 of the Lie algebra. The group G_{0} = K ⋉ 𝔭 acts on the flat symmetric space G_{0}/K ≃ 𝔭 as isometries for the Killing form. The group of isometries of 𝔭 thus obtained is known as the Cartan motion group.

Choose a maximal abelian subspace 𝔞 ⊂ 𝔭. Then there is a restriction isomorphism Res _{𝔞}^{𝔭} : C^{∞}(𝔭)^{K} ↦ C^{∞}(𝔞)^{W}, with analogues for compactly supported smooth invariant functions and for invariant polynomials. Let Σ be the restricted roots with multiplicities m_{α} (α ∈ Σ). We consider Σ to be a subset of 𝔞 by means of the Killing form ( . , .) and we let W be the Weyl group, acting on 𝔞.

The spherical function ψ_{λ} (λ ∈ 𝔞_{ℂ}) on the symmetric space 𝔭 satisfies the equations ∂(p)ψ_{λ} = p(λ)ψ_{λ} for all p in the algebra S(𝔭)^{K} of K-invariant polynomials on 𝔭, where as usual ∂(p) is the constant coefficient differential operator corresponding to p. Furthermore, ψ_{λ}(0) = 1. By [18, Corollary 2.3, p. 402], ψ_{λ} is the unique K-invariant function on 𝔭 with these two properties.

We let R ⊂ 𝔞 be the normalized root system of the finite reflection group W (so some roots in a component BC_{n} of Σ will coincide in R). Choose the multiplicity function k : R ↦ ℝ as k_{α} = 1/4 ∑_{β∈ℝα∩Σ} m_{β}. With these multiplicities, we define the corresponding operators T_{p}(k) (acting on functions on 𝔞) for the finite reflection group W. The crucial observation, with Rad denoting radial part, is the following:

(6.2)  Rad(∂(p))f = T_{Res _{𝔞}^{𝔭}p}(k)f (p ∈ S(𝔭)^{K}, f ∈ C^{∞}(𝔞)^{W} ) .

Before proving this relation, let us note that it shows how the algebra of radial parts can be obtained from Dunkl operators, as follows. One computes the operator T_{q}(k) for each polynomial q in a set of fundamental invariants, and then restricts it to the invariant functions on 𝔞. This restriction acts as a differential operator (a moment’s thought will make this clear), and the differential operators which are thus obtained generate the algebra of radial parts.

Turning to the proof of 6.2, we first recall [7] that

Δ_{k} = Δ_{0} + 2 ∑_{α∈R_{+}} k_{α}M_{(α^{∗})^{−1}} {∂_{α} − M_{(α^{∗})^{−1}}(1 − r_{α})} .

Let Δ_{𝔭} denote the ordinary Laplacian on 𝔭. By the choice of k, the expression for Rad Δ_{𝔭} in [18, Proposition 3.13, p. 270], and the above expression for Δ_{k}, we have Rad Δ_{𝔭}f = Δ_{k}f for f ∈ C^{∞}(𝔞)^{W}.

Now let p ∈ S(𝔭)^{K} be homogeneous of degree n and suppose f ∈ C^{∞}(𝔞)^{W}. We apply (6.1) first on 𝔭 (with k = 0) and then on 𝔞 (for our particular choice of k) to prove (6.2):

Rad(∂(p))f = \frac{1}{n!}Rad ((ad \frac{Δ_{𝔭}}{2} )^{n} M_{p}) f

= \frac{1}{n!} \{(ad \frac{Rad Δ_{𝔭}}{2} )^{n} (Rad M_{p})\} f

= \frac{1}{n!} \{(ad \frac{Δ_{k}}{2} )^{n} (M_{Res _{𝔞}^{𝔭}p})\} f

= T_{Res _{𝔞}^{𝔭}p}(k)f.

After having identified the radial parts in terms of Dunkl operators, it is now easy to show how the Bessel kernel J_{W}, i.e., the W -invariant component of Exp_{W} as in (4.8), is related to the restrictions of the spherical functions to 𝔞. Indeed, if λ ∈ 𝔞_{ℂ}, then we note that J_{W} (λ, k, .) has an extension J_{W}^{ext}(λ, k, .) to a smooth Kinvariant function on 𝔭. This extension is evidently equal to 1 in 0, and satisfies the equations ∂(p)J_{W}^{ext}(λ, k, .) = p(λ)J_{W}^{ext}(λ, k, .) for all p ∈ S(𝔭)^{K}, as a consequence of (6.2) and the very definition of radial parts. By the uniqueness mentioned above, we conclude that J_{W}^{ext}(λ, k, x) = ψ_{λ}(x) for λ ∈ 𝔞_{ℂ} and x ∈ 𝔞.

In this context of the Cartan motion group, we will now proceed to identify the restrictions to the invariants C^{∞}(𝔞)^{W} of the linear automorphisms W_{k}, V_{k} : C^{∞}(𝔞) ↦ C^{∞}(𝔞) from Theorem 5.1. Recall [18, p. 467] that

(6.3)  ∫_{𝔭} f(x) dx = (2π)^{\frac{dim 𝔭}{2}} c_{k}^{−1} ∫_{𝔞} Res _{𝔞}^{𝔭}f(x)w_{k} dx (f ∈ C_{c}^{∞}(𝔭)^{K}).

The constant is determined by considering the Gaussian. Since we have identified the spherical functions as Bessel functions, we see from this integral formula that the normalized spherical transform F, defined by

Ff(λ) = (2π)^{− \frac{dim 𝔭}{2}} ∫_{𝔭} f(x)ψ_{−iλ}(x) dx (λ ∈ 𝔞, f ∈ C_{c}^{∞}(𝔭)^{K}),

factors as F = D_{k}Res _{𝔞}^{𝔭}. From this relation one sees that the inversion, Plancherel and Paley–Wiener theorems for the spherical transform follow from the corresponding theorems for D_{k}, when specialized to the invariants (the first two admittedly being almost trivial for the Cartan motion group).

Let 𝔮 be the orthoplement of 𝔞 in 𝔭 and define the (flat analogue of) the Abel transform:

Af(x) = ∫_{𝔮} f(x + q) dq (f ∈ C_{c}^{∞}(𝔭)^{K}, x ∈ 𝔞).

Since the spherical transform can also be written as

Ff(λ) = (2π)^{− \frac{dim 𝔭}{2}} ∫_{𝔭} f(x)e^{−i(λ,x)} dx (λ ∈ 𝔞, f ∈ C_{c}^{∞}(𝔭)^{K}),

Fubini’s theorem yields an additional factorization of F, namely F = (2π)^{− \frac{dim 𝔮}{2}} D_{0}A. The relation (2π)^{− \frac{dim 𝔮}{2}} D_{0}Af = D_{k}Res _{𝔞}^{𝔭}f for f ∈ C_{c}^{∞}(𝔭)^{K} and the Paley–Wiener theorems for D_{0} and D_{k} together then show that the Abel transform establishes a linear isomorphism between C_{c}^{∞}(𝔭)^{K} and C_{c}^{∞}(𝔞)^{W}. It is easily verified that A : C_{c}^{∞}(𝔭)^{K} ↦ C_{c}^{∞}(𝔞)^{W} is in fact a linear homeomorphism, so that the isomorphisms A^{t} : D′(𝔞)^{W} ↦ D′(𝔭)^{K} and (A^{−1})^{t} : D′(𝔭)^{K} ↦ D′(𝔞)^{W} are defined.

Recall the symmetry S from Section 4, defined by Sf(x) = Sf(−x), and note that, as maps from C_{c}^{∞}(𝔞)^{W} into itself, we have

D_{k}E_{0} = D_{k}SD_{0} = (2π)^{\frac{dim 𝔮}{2}} D_{k}SFA^{−1} = (2π)^{\frac{dim 𝔮}{2}} D_{k}SD_{k}Res _{𝔞}^{𝔭}A^{−1}
= (2π)^{\frac{dim 𝔮}{2}} D_{k}E_{k}Res _{𝔞}^{𝔭}A^{−1} = (2π)^{\frac{dim 𝔮}{2}} Res _{𝔞}^{𝔭}A^{−1}.

Let Ext _{𝔞}^{𝔭} : C^{∞}(𝔞)^{W} ↦ C^{∞}(𝔭)^{K} be the inverse of Res _{𝔞}^{𝔭}. Using (6.3) we find, for f ∈ C^{∞}(𝔞)^{W} and ψ ∈ C_{c}^{∞}(𝔞)^{W}:

⟨W_{k}f, ψ⟩_{𝔞} = (2π)^{\frac{dim 𝔞}{2}} c_{k}^{−1}⟨(D_{k}E_{0})^{t}(fw_{k}), ψ⟩_{𝔞}
= (2π)^{\frac{dim 𝔞}{2}} c_{k}^{−1}⟨fw_{k}, D_{k}E_{0}ψ⟩_{𝔞}
= (2π)^{\frac{dim 𝔭}{2}} c_{k}^{−1}⟨fw_{k}, Res _{𝔞}^{𝔭}A^{−1}ψ⟩_{𝔞}
= ⟨Ext _{𝔞}^{𝔭}f, A^{−1}ψ⟩_{𝔭}
= ⟨(A^{−1})^{t}Ext _{𝔞}^{𝔭}f, ψ⟩_{𝔞}.

We conclude that W_{k} = (A^{−1})^{t}Ext _{𝔞}^{𝔭}, as maps with domain C^{∞}(𝔞)^{W}. For the right hand side, we know a priori only that it maps C^{∞}(𝔞)^{W} into D′(𝔞)^{W}, but this equality and Theorem 5.1 show that it is actually a linear automorphism of C^{∞}(𝔞)^{W}. It follows from this that (A^{−1})^{t}, which a priori is only known to map C^{∞}(𝔭)^{K} ⊂ D′(𝔭)^{K} into D′(𝔞)^{W}, establishes in fact a linear isomorphism between C^{∞}(𝔭)^{K} and C^{∞}(𝔞)^{W}. This enables us to identify the restriction of Dunkl’s intertwiner to C^{∞}(𝔞)^{W} as V_{k} = Res _{𝔞}^{𝔭}A^{t}, where the fact that A^{t} : C^{∞}(𝔞)^{W} ↦ C^{∞}(𝔭)^{K} is a linear isomorphism is then implicitly used in the interpretation of the right hand side.

Remark 6.1. For certain multiplicities of the restricted roots, the results on shift operators can be used to essentially invert the Abel transform, as follows. Assume that k_{α} = 1/4 ∑_{β∈ℝα∩Σ} m_{β} is an integer for all α ∈ R. Then (4.10) implies that there exists a W -invariant differential operator D̃ of order 1/4 ∑_{α∈Σ} m_{α}, with poles along the singular hyperplanes only, such that E_{k}g = D̃E_{0}g for all rapidly decreasing g ∈ C^{∞}(𝔞)^{W}. Therefore, if f ∈ C_{c}^{∞}(𝔞)^{W}, we find that Res _{𝔞}^{𝔭}A^{−1}f = (2π)^{− \frac{dim 𝔮}{2}} D_{k}SD_{0}f = (2π)^{− \frac{dim 𝔮}{2}} E_{k}D_{0}f = (2π)^{− \frac{dim 𝔮}{2}} D̃E_{0}D_{0}f = (2π)^{− \frac{dim 𝔮}{2}} D̃f. This shows that the Abel transform is for such multiplicities essentially inverted by the W -invariant differential operator (2π)^{− \frac{dim 𝔮}{2}} D̃.

Remark 6.2. In this section it has become clear that the theory of spherical functions for the Cartan motion group is intimately connected with the invariant part of the theory of Dunkl operators. It can, however, not be excluded that the general theory of these operators is also relevant in this context, for conceptual reasons which have gone unnoticed sofar. The author at least finds it hard to believe that, e.g., the description of the algebra of radial parts should not be the reflection of some more direct connection. There are, in effect, two additional indications that such a connection may exist.

The first of these is given by the estimates on the spherical functions ψ_{λ}. One has the integral representation

ψ_{λ}(x) = ∫_{K} e^{(λ,kx)} dk,

for λ ∈ 𝔞_{ℂ}and x ∈ 𝔞. The Kostant convexity theorem [18, Theorem 10.2, p. 473] then yields the estimate |ψ_{λ}(x)| ≤ exp(max_{w∈W} Re (wλ, x)). But alternatively, this also follows from the estimate for Exp_{W} in Section 2 and the identification of the spherical functions as generalized Bessel functions, as explained in this section. This natural occurrence of Exp_{W} points to relevance of the original Dunkl operators in the Cartan motion group case.

The second indication consists of Torossian’s application of Dunkl operators in the inversion of the Chevalley restriction isomorphism between the K-invariant polynomials on 𝔭 and the W -invariant polynomials on 𝔞 [35]. Here the original Dunkl operators make an appearance, not just the operators corresponding to invariant polynomials.

Remark 6.3. When the results in this section are combined with similar results for the Cherednik operators in Section 4.3 and with Theorem 4.13, one obtains a limit transition from the spherical functions for a Riemannian symmetric space of the non-compact type to the spherical functions for the corresponding Cartan motion group. This has previously been proved in a direct fashion by Ben Saïd and Ørsted in [2], where it is also shown how explicit formulas for the spherical functions for the Cartan motion group can be obtained through this limit transition.

References

[1] M. Abramowitz and I.A. Stegun, eds., Handbook of mathematical functions. Dover Publications, New York, 1970.

[2] S. Ben Saïd and B. Ørsted, Generalized Bessel functions I, Preprint (2004).

[3] S. Ben Saïd and B. Ørsted, Generalized Bessel functions II, Preprint (2004).

[4] I. Cherednik, A unification of Knizhnik–Zamolodchikov equations and Dunkl operators via affine Hecke algebras, Invent. Math. 106 (1991), 411-432.

[5] J.F. van Diejen and L. Vinet, Calogero–Moser–Sutherland Models. CRM Series in Mathematical Physics, Springer, 2000.

[6] C.F. Dunkl, Reflection groups and orthogonal polynomials on the sphere, Math. Z. 197 (1988), 33-60.

[7] C.F. Dunkl, Differential-difference operators associated to reflection groups, Trans. Amer. Math. Soc. 311, (1989), 167-183.

[8] C.F. Dunkl, Operators commuting with Coxeter group actions on polynomials, in “Invariant theory and tableaux,” Minneapolis, 1988, IMA Vol. Math. Appl. 19, 107-177.

[9] C.F. Dunkl, Integral kernels with reflection group invariance, Canad. J. Math. 43 (1991), 1213-1227.

[10] C.F. Dunkl, Hankel transforms associated to finite reflection groups, in “Proceedings of the special session on hypergeometric functions on domains of positivity, Jack polynomials and applications,” Tampa, 1991, Contemp. Math. 138, 123-138.

[11] C.F. Dunkl, M.F.E. de Jeu and E.M. Opdam, Singular polynomials for finite reflection groups, Trans. Amer. Math. Soc. 346 (1994), 237-256.

[12] C.F. Dunkl, Intertwining operators associated to the group S3, Trans. Amer. Math. Soc. 347 (1995), 3347-3374.

[13] C.F. Dunkl and Y. Xu, Orthogonal polynomials of several variables. Cambridge Univ. Press, 2001.

[14] C.F. Dunkl and E.M. Opdam, Dunkl operators for complex reflection groups, Proc. London Math. Soc. (3) 86 (2003), 70-108.

[15] C.F. Dunkl, Singular polynomials for the symmetric groups, Preprint (2004). ArXiv: math.RT/0403277.

[16] G.J. Heckman, A remark on the Dunkl differential-difference operators, in “Harmonic analysis on reductive groups,” Brunswick, 1989, Progr. Math. 101, 181-191.

[17] G.J. Heckman, Dunkl operators, Séminaire Bourbaki 828, 1996-97, Astérisque 245 (1997), 223-246.

[18] S. Helgason, Groups and Geometric Analysis. Academic Press, New York, 1984.

[19] S. Helgason, Geometric Analysis on Symmetric Spaces. American Mathematical Society, Providence, 1994.

[20] M.F.E. de Jeu, The Dunkl transform, Invent. math. 113 (1993), 147-162.

[21] M.F.E. de Jeu, Dunkl operators. Thesis, Leiden University, 1994.

[22] M.F.E. de Jeu, Subspaces with equal closure, Constr. Approx. 20 (2004), 93-157.

[23] G. Lusztig, Affine Hecke algebras and their graded version, J. Amer. Math. Soc. 2 (1989), 599-685.

[24] E.M. Opdam, Some applications of hypergeometric shift operators, Invent. Math. 98 (1989), 1-18.

[25] E.M. Opdam, Dunkl operators, Bessel functions and the discriminant of a finite Coxeter group, Comp. Math. 85 (1993), 333-373.

[26] E.M. Opdam, Harmonic analysis for certain representations of graded Hecke algebras, Acta Math. 175 (1995), 75-121.

[27] E.M. Opdam, Lecture notes on Dunkl operators for real and complex reflection groups, MSJ Memoirs 8, Math. Soc. of Japan, Tokyo, 2000.

[28] M. Rösler, Positivity of Dunkl’s intertwining operator, Duke Math. J. 98 (1999), 445-463.

[29] M. Rösler, Dunkl Operators: Theory and Applications, in “Orthogonal Polynomials and Special Functions,” Leuven, 2002, Lecture Notes in Math. 1817, 93-135.

[30] M. Rösler and M.F.E. de Jeu, Asymptotic analysis for the Dunkl kernel, J. Approx. Theory 119 (2002), 110-126.

[31] M. Rösler and M. Voit, Markov processes related with Dunkl operators, Adv. Appl. Math. 21 (1998), 575-643.

[32] W. Rudin, Functional analysis. Tata McGraw-Hill, New Delhi, 1973.

[33] L. Schwartz, Théorie des distributions. Hermann, Paris, 1978.

[34] E.C. Titchmarsh, The theory of functions. Oxford Univer. Press, 1986.

[35] C. Torossian, Une application des opérateurs de Dunkl au théorème de restriction de Chevalley, C.R. Acad. Sci. Paris 318 (1994), 895-898.

[36] F. Treves, Topological vector spaces, distributions and kernels. Academic Press, New York, 1967.

[37] K. Trimèche, The Dunkl intertwining operator on spaces of functions and distributions and integral representation of its dual, Integral Transform. Spec. Funct. 12 (2001), 349-374.

[38] K. Trimèche, Paley–Wiener theorems for the Dunkl transform and Dunkl translation operators, Integral Transform. Spec. Funct. 13 (2002), 17-38.

[39] Y. Xu, Integration of the intertwining operator for h-harmonic polynomials associated to reflection groups, Proc. Amer. Math. Soc. 125 (1997), 2963–2973.

M.F.E. de Jeu, Mathematical Institute, Leiden University, P.O. Box 9512, 2300 RA Leiden, The Netherlands

E-mail address: mdejeu@math.leidenuniv.nl