Worked after completing the game and liberating the area. There is a swordsman at Cloud Ridge Shrine gate (East of Toyotama). He spawns there on fast travel.
Use a smoke bomb to stun him and kick him off with Wind Stance upgrade. Just repeat if several kicks are needed to send him off the cliff.