Firstly, damaging the ice swan or the ice that they drop does not work. If ice is dropped you need to remain some distance away from it, as it breaks upon hitting the ground and those fragments can damage you.
Use speed gear for this and you can run past all of them before they even drop the ice with the exception of the last group of three swans. I was able to outrun the first two but I had to wait for the third swan to drop their ice, hang back and wait for it to break before proceeding on to the boss.