Unreal 4 uses the .FBX format for bringing in animation. Below is a link to the documentation about best practices and process for using .FBX in Unreal.
FBX Animation Pipeline
FBX Best Practices
Rigging in Unreal
Technically you can use any rig that you want or create but there are certain best practices and standards you can use that will make it more compatible with our packs and with the game engines retargeting features.
Unreal provides a free maya rigging tool you can check out here.