Villagers attack you when you hit them many times. After you hit a villager once, your reputation goes down a level. The only way to get it back up is to trade with them.