I've done my share of both, and have had a good time at both with a reasonable degree of success.
One design in Lightning class, I was invited to go up against the glass boats with my wooden oldie. They assured me there were a couple of other wood boats in the fleet (so I wouldn't feel alone at the back of the pack???). I upgraded sails (to a 1 yr old set), added barber haulers and an adjustable backstay, cleaned the bottom - then went out to take a first, by about 3'. I was helped greatly by two gung-ho crew in their mid-teens. Anyway, the fleet learned the lesson - if you're not competitive, it's probably the gear - upgrade and go faster.
One design in DN class - yeah, ice boats - but same story. The equipment of the class evolves (within the rules) and you have to keep up to be competitive and sail fast.
PHRF - two masthead sloops, one cb and one fin keel. Biggest difference here is that the boats are bigger and it's more expensive to keep the sails and running gear in competitive order. As to being different design, PHRF covers that pretty well - no complaints with PHRF. After a short while, I soon realised who my biggest competitors were (usually not boats of the same design), and knew I had to keep an eye on as to tactics/strategy. As to boats of the same design - they're probably not you biggest competition, but they're the boats most likely to hit you.
Biggest issues with both types of racing are the same - keeping your eye on the wind, the weather, the current, and your competition. Fly the right sails and trim them properly. Stay with a fleet heading the wrong way and sail into a hole, or take a flyer? Skipper's choice. Nothing to do with 1 design or PHRF.
My favorite sailing story concerns Stu Barrett sailing an International 14 - rounded the weather mark first and headed off for the reaching mark. About half way down the leg, he realised he was heading for the wrong mark. If he fell off for the proper mark, he'd lose about 1/3 the fleet. He reached down and uncleated his main halyard and let the main drop, cursing about bum luck as the fleet went by. When they'd gone, he raised his main and proceeded to the proper mark (and a win).