This doesn't fully stop betrayers, but it limits them to what they can do and leaves you with a lower chance of getting betrayed.
1. Do not invite anyone you have not talked to before, do not recruit just anyone who asks to join. Only invite people you've known for a while.
2. Make use of /f owner [name] giving people ownership of certain claimed chunks. Make sure the /f home is only owned by the faction admin to prevent someone pouring lava over the sethome. And anything private to certain members give only them ownership.
3. Only moderate people you 100% trust.
4. If someone is acting the least bit suspicious kick them then question them, even people you know and trust can turn on you.