Mary,
I have noticed this myself, but only on paper surfaces. There is less pastel adhesion on paper that on a gritty or sanded surface. I have not used a Wallis paper, but use the Golden Acrylic Ground for Pastel on various hard surfaces. The pastel adheres to that beautifully.
From my experience , pastel on paper has to be done with fewer layers and less reworking, so since I have a heavy hand I like the ruggedness of that surface.
There is a wonderful paper something like the Wallis. It comes in two sizes, one quite large and in many gorgeous colors.
www.nycentralart.com will send you out a sample packet for $5.00. It is made by Diane Townsend who also has her own famous line of pastels. It is a pigmented surface and will not fade like the Canson. They also have the best paper catalogue in the world and that is free. I have seen samples of the Art Spectrum and it does not look like it has enough tooth to me, so I have never used it.
I think sometimes we are afraid to use ground that are a bit rough, because we feel that would give a less than a desirably smooth finish. However, by the time I plaster on my pastels the tooth is s SO filled up, it could have been done on cement! Alicia Blue was done on a really rugged surface. I know because where I prepare my surfaces is an unheated spray room, and when it is cold I knock 'em out as fast as I can! BRRR!
I personally have had more luck doing pastel studies on paper rather than full bore pastels.
What fixative are you using? I like the Rowney Perfix. That is the one Dakota pastels mainly recommends. I find it does not shift the color markedly if at all, but then again I use several light layers and don't blast it.
Interestingly enough, Daniel Greene, in his out of print book "Pastel", uses pastel fixative to darken the under layers so the top layers really glow. This book is really worth searching out for a second hand copy.
I hope this helps. Good luck with your project!