Capturing a monster requires the Capture skill. Capture is a darkness skill, which means you'll need a character with a darkness skill slot to use it. You may only capture a monster when its HP count is 5% of its maximum or lower.
The easiest way to obtain "Capture" is by getting an early title in guild.
Merciful Strike is a good way to ensure that you don't kill the enemy you're trying to capture when lowering its HP. Merciful Strike does the normal damage of an 8 AP physical attack, but leaves the enemy with 1 HP if it otherwise would have killed it.