Moderation is a tricky business. Firstly because doing it well requires excellent judgment and social intuition; secondly because being a moderator appeals hugely to people who lack either of those traits. Pick a bad forum anywhere on the Internet and somewhere along the line you're likely to find moderators who are either apathetic, incompetent, bullies or all of the above.
It might seem like the relatively small number of moderators in a given community would limit their influence over the overall culture, but this isn't the case. While the quality of your user base is notoriously difficult to effect in the short term, over the long term it will be almost entirely determined by the kind of users that continue posting. That factor in turn will be decided by what your rules are and how they are enforced. Smart, socially capable people have better things to do than hang around a community that's full of trolls they know will never be turfed out. Unless you're dealing with a community small enough that you can moderate it entirely on your own, these decisions are going to largely be made by your moderating team.
The problem, of course, is who to pick. Moderating is frequently a thankless task. Volunteering to be The Man puts a user into a position where they will be frequently attacked or insulted by the trolls in the community and can quickly turn a fun hobby into an unpaid customer service position. The upsides to the position are the ability to take a hand in shaping the community that they care about. An uglier way of expressing that upside is the ability to exert power in that same space. The moderators you want are the ones who are interested in the first upside. The trick is separating them from those that are interested in the second.
The best moderators I've known were often the ones that needed the most persuading to take the job. Conversely, I have a standing policy that any user who openly expresses a desire to be a moderator should be blacklisted from it. In truth, a more nuanced view is probably wiser. There are a few standard qualities that you should look for. It goes without saying that you should look for active members of the community, as an absentee moderator is at best ineffective and at worst will cause harm as users become frustrated as their questions go unanswered.
However, if you have a few outlier users who post far more than other users I would caution against putting them in moderating positions. Users who are spending all day on your forums are likely to lack the kind of social skills required of a moderator and are also likely to quickly burn out and leave the forum. Another commonly looked for trait that can be misleading is metamodding (also known as backseat modding). It's tempting to look at a user who is showing a keen knowledge of the rules and a willingness to contribute and think they'd make a great mod. In my experience, this keenness can quickly become the kind of overzealousness that bad moderators are made of.
So, after all the warnings are dealt with, what should you look for in a moderator? Obviously this will vary depending on the specific needs of your community, but here's a checklist of good questions to ask yourself when looking at candidates: