Everyone has their own opinions on what a vacation is. I would agree if don't like the heat and humidity the Summer is not the time to be there, the sun is very intense.
I have yet to meet anyone that didn't like Key west and all it's oddities, at least glad they went there once, and avoiding it would be avoiding one of the most visited vacation spots aside from theme parks for a reason. Southern most point you mentioned is there.
Duval street with all it's rich history (example Sloppy Joes) and the famous pier sunset street performers is memorable. If Dual Street is too much for you there is plenty else in Key West.
Staying in the area is expensive.
It is not remotely a tourist trap, unless a place that is a tourist destination is your definition of one.
Edited by tampa8, 05 May 2014 - 04:54 PM.