I think part of it is that Adams' hyper-realistic style doesn't mesh well with the types of stories in these books. It works for Batman, Green Arrow, and even Deadman (who didn't engage in a lot of superheroics), but could look silly in some instances. There's a World's Finest story he drew about Superman and Batman presenting "challenges" to each other on their birthday, one of which casually uses an atomic bomb. Not a good fit.

Artists like Swan and Anderson had more of an idealized realism, so Anderson could get away with things like the Spectre and a demon swinging comets by the tail at each other.