To fight Olaf you must first complete the other 3 fistfights on The Skellige Isles.
The first one is in Kaer Trolde Harbor
The second is in Arinbjorn
The third is in Harviken
The fight against Olaf then takes place on Urialla Harbor
Go to the arena to start the fight.
Dodge Olaf's attacks and throw 2-3 heavy punches at him before dodging. Rinse and repeat.
The trophy will unlock when you finish the dialogue after the fight.