You can't really blame Bronn for not wanting to go up against the Mountain — have you seen that guy? He's huge!
Of course, it depends which guy you're talking about. In season one of Game of Thrones, Ser Gregor Clegane was played by Conan Stevens — he was the one who shocked us at the King's Hand tournament when he shoved a wooden lance through the throat of Ser Hugh one day, and then sliced off the head of his own mare and nearly killed Ser Loras the next. Only through the intervention of his little brother, Sandor Clegane — the Hound — was he stopped. And it's amazing that Sandor even stepped up, considering his burned face came at the hand of his giant brother. Littlefinger related their "lovely little tale of brotherly love" to Sansa thusly: "The Hound was just a pup, 6 years old, maybe. Gregor, a few years older, already a big lad. One evening, Gregor found his little brother playing with a toy by the fire. Gregor's toy. Gregor never said a word. He just grabbed his brother by the scruff of his neck and shoved his face into the burning coals, held him there while the boy screamed while his face melted." ("The worse thing was is that it was my brother who did it," Sandor says when it's finally his turn to tell the story.)
This, of course, was just the beginning of the Mountain's penchant for violence, especially towards those smaller than him. Growing up in the employ of the Lannisters, the Cleganes followed their orders, Gregor as a knight, and Sandor as the sworn shield of Joffrey. Both were asked to do horrible things, especially to children. Remember that the Hound killed Arya's friend, Micah, the butcher's boy. That's nothing compared to what the Mountain did to Elia Martell's kids. "Do you know what happened to the Targaryen children when the Mad King fell?" Catelyn Stark once asked Robb. "They were butchered in their sleep," he answered. And that was the kinder version of the story. Some 20 years ago, Tywin Lannister sent this most vicious brute to wipe out the Targaryens, so the Mountain bashed Elia's son's head against a wall, killed her daughter, and with their blood still on his hands, raped and killed Elia. The fact that he was knighted in the first place by Elia's husband Rhaegar Targaryen didn't seem to matter. (This is why Oberyn Martell shares an enthusiasm for "dead Lannisters," because his family never had any justice for this.)
In season one, Ned Stark heard about the Mountain running wild in the Riverlands — at Tywin Lannister's behest. A peasant came to court to tell of their homes being burned, women being raped, and children being set on fire, by a mysterious man "taller than any man I'd ever met." The court deduces that this would be "Tywin's mad dog," so Ned tried to administer some justice. He denounced Gregor Clegane, stripped him of ranks, titles, lands, and holdings, and sentenced him to death, sending Ser Berric Dondarrion to enforce this. This sentence, however, didn't come to pass — the Mountain isn't so easily killed.
But actors can be easily recast, especially when they're more talked about than seen onscreen. Ian Whyte (who has also played White Walkers and a giant on the show) took the part during season two, selecting prisoners for torture at Harrenhal, and now, in season four, Icelandic strongman Hafþór Júlíus Björnsson plays the part. He is six foot nine, and he most recently placed third in the World's Strongest Man competition, in 2013. (A ranking he also won in 2012.) "You're going to fight that?" Ellaria says in disbelief in the coming attractions, when she sees her paramour's combatant for the trial by combat. Even the Red Viper should take pause — this will be a battle for the ages.