When you see the Colossus get its attention by firing arrows at its 4 legs, this will cause him to climb up the side of the Colosseum.
Once here, using the windows or the large openings fire 2 more arrows, one at the top leg, the other at the bottom. This will cause him to fall and land on his back.
Make your way down quickly, and there will be 2 weak spots on his belly, climb him and attack these spots.
If he rolls over, repeat the process until he is defeated.