To successfully execute an alien, you have to stun them by weakening them enough so they sit there on their knees gazed for a few moments before they get back up again, walk up to them and press the
button to execute, Duke will kick or punch their head off. The ways I got to stun the aliens most of the time was to melee them up close with
until they go down, shoot at their torso a certain amount or throw something at them.
If you do not have this trophy by the mission
"The Shrunk Machine", then in this mission some workers will give you a
"Freeze ray". If you use this freeze way to freeze an enemy, you can walk up to them and do an easy execution, getting you towards the trophy for killing frozen aliens too.
Good luck!