Anywhere and everywhere or nowhere at all. You never really need things for ceremonies, but some basics like candles and incense are helpful. They don’t need to be any special kind.
Most ritual supplies can be found in unexpected places like garage sales. I’ve found beautiful goblets, candle holders, and other decorative items at such neighborhood sales. A coffee table can be used as an altar. Candles are easily gotten anywhere, including the supermarket. Incense can be easily found in gift shops, craft stores, drug stores, etc. I prefer stick incense because of their ease of use and the wide variety available (and you can use the leftover stick-ends to light the candles!). I also use loose incense on charcoal occasionally, but it's rather messy.
Crystals and other stones can be found in many places these days, particularly in science and nature themed shops. There are several sources for getting crystals, but I'd suggest looking in your phone book for "Lapidary supplies". A lapidary shop will usually have a wide selection of good crystals and won't charge you "new age" prices.
Things like wands or staffs are best found while walking in the woods. Athame's are trickier to find, but patience will usually draw the right blade to you. Some people prefer to make all of their own tools, but an Athame can be a big project to tackle. I made my first one out of a swiss army knife blade and a wooden handle made from driftwood. It was difficult and time-consuming to make, but the result was uniquely mine. I bought my current Athame in a cheap import store.
As you can see, your tools and supplies can come from anywhere you find them... and often, it seems, they will find you.
~Flame RavenHawk