To fight Olaf you must first complete the other 3 fistfights in Skellige.
The first one is in Kaer Trolde Harbor, in the town on the north west side of the main isle.
The second is in Arinbjorn, on the far west side of the main island.
And the third is in Harviken, on the small island in the south east corner of the map.
The fight against Olaf takes place on a small island in the northern part of the map in Urialla Harbor. Just go to the arena and beat Olaf there and the trophy unlocks. It should also be noted that for these fistfight quests, you don't necessarily need to be the suggested level. You should be fine as long as you know what you're doing with blocking, counter attacking, etc.