Once you see the Colossus, use your sword and R1 to beam the light onto his head and lower back, this will show his weak spots.
This Colossus will do 2 different attacks, a stab with his sword if you are close, or a longer charged attack which leaves his sword in the ground. To make this easier, get him to hit the big metal circle in the middle of the arena, this will cause him to lose the armour on his wrist so you can easily climb up (otherwise you need to jump onto his hips, then back onto his arm which is awkward.)
When his sword is stuck you can jump on and run up to his unprotected arm. Climb up his hair towards his head, using body hair and any ledges you can get to. Once you're at his head, you can attack him with your sword.
His last weak spot is his lower back, to save time you can just jump from his head and land on his 'shorts'. Once you see the weak spot stab it with the sword and he is down!