Considering their relative lack of fame before the first Hangover film, it's no surprise that Bradley Cooper, Ed Helms, and Zach Galifianakis were not the first choices. In The Hollywood Reporter's oral history of the franchise, director Todd Phillips opened up about who could've been. Besides Paul Rudd and Jack Black both turning it down, Phillips had a hard time with the brother-in-law character:
"We were writing the brother-in-law as a younger brother they had to take along with them -- like a Jonah Hill character instead of Zach [Jake Gyllenhaal also was considered]. Then we thought it'd be so much more awkward if it was an older brother who's still at home. [Thomas Haden Church was strongly considered.]
They probably made the right choice in terms of comedy, but imagine the handsomeness contest Gyllenhaal could've had with Cooper. It would've been legendary.