One reason a pastor needs margin is because his family should be a very high priority. You know this. It’s hard to practice though when pastoral care and the work of the church pull you in every direction.
Fight the guilt of disappointing church people and love and spend time with your family.
Use any margin you can create to spend more time with your family. You won’t feel guilty about that.