Sounds like you're not getting any sound from the digital connection.
On many receivers you must map the digital input(s) to whatever source you want. Sound like you did that all ready - might want to double check.
Does the Denon auto-select between Dolby Digital and PCM (most do, buy my 1st gen DD processor ($1500) does not).
You should have some sound whether the source is DD or PCM. If you have nothing...
Then, finally, I'd try a different cable (different brand). Ive seen some optical cables spring load the tips, others do not.
Also, be careful with optical cables. Don't get the ends dirty. Watch the bend radius. If you bend or kink them (at any time), perhaps while pulling them through, you can break the internal optical line.
It's possible you have a dead optical component (on either side). Take the 501 (and an extra cable) to a friends and retry the optical connection. I think if you have something prerecorded you don't need a sat connection.