Bad intent is a disposition, not a feeling
It’s common to think that someone else is arguing in bad faith. In a recent blog post, Nate Soares claims that this intuition is both wrong and harmful:
I believe that the ability to expect that conversation partners are well-intentioned by default is a public good. An extremely valuable public good. When criticism turns to attacking the intentions of others, I perceive that to be burning the commons. Communities often have to deal with actors that in fact have ill intentions, and in that case it's often worth the damage to prevent an even greater exploitation by malicious actors. But damage is damage in either case, and I suspect that young communities are prone to destroying this particular commons based on false premises.
To be clear, I am not claiming that well-intentioned actions tend to have good consequences. The road to hell is paved with good intentions. Whether or not someone's actions have good consequences is an entirely separate issue. I am only claiming that, in the particular case of small high-trust communities, I believe almost everyone is almost always attempting to do good by their own lights. I believe that propagating doubt about that fact is nearly always a bad idea.
It would be surprising, if bad intent were so rare in the relevant sense, that people would be so quick to jump to the conclusion that it is present. Why would that be adaptive?
What reason do we have to believe that we’re systematically overestimating this? If we’re systematically overestimating it, why should we believe that it’s adaptive to suppress this?
There are plenty of reasons why we might make systematic errors on things that are too infrequent or too inconsequential to yield a lot of relevant-feeling training data or matter much for reproductive fitness, but *social intuitions are a central case of the sort of things I would expect humans to get right *by default. I think the burden of proof is on the side disagreeing with the intuitions behind this extremely common defensive response, to explain what bad actors are, why we are on such a hair-trigger against them, and why we should relax this.
Nate continues:
My models of human psychology allow for people to possess good intentions while executing adaptations that increase their status, influence, or popularity. My models also don’t deem people poor allies merely on account of their having instinctual motivations to achieve status, power, or prestige, any more than I deem people poor allies if they care about things like money, art, or good food. […]
One more clarification: some of my friends have insinuated (but not said outright as far as I know) that the execution of actions with bad consequences is just as bad as having ill intentions, and we should treat the two similarly. I think this is very wrong: eroding trust in the judgement or discernment of an individual is very different from eroding trust in whether or not they are pursuing the common good.
Nate's argument is almost entirely about mens rea - about subjective intent to make something bad happen. But mens rea is not really a thing. He contrasts this with actions that have bad consequences, which are common. But there’s something in the middle: following an incentive gradient that rewards distortions. For instance, if you rigorously A/B test your marketing until it generates the presentation that attracts the most customers, and don’t bother to inspect why they respond positively to the result, then you’re simply saying whatever words get you the most customers, regardless of whether they’re true. In such cases, whether or not you ever formed a conscious intent to mislead, your strategy is to tell whichever lie is most convenient; there was nothing in your optimization target that forced your words to be true ones, and most possible claims are false, so you ended up making false claims.
More generally, if you try to control others’ actions, and don’t limit yourself to doing that by honestly informing them, then you’ll end up with a strategy that distorts the truth, whether or not you meant to. The default state for any given constraint is that it has not been applied to someone's behavior. To say that someone has the honest intent to inform is a positive claim about their intent. It's clear to me that we should expect this to sometimes be the case - sometimes people perceive a convergent incentive to inform one another, rather than a divergent incentive to grab control. But, if you do not defend yourself and your community against divergent strategies unless there is unambiguous evidence, then you make yourself vulnerable to those strategies, and should expect to get more of them.
I’ve been criticizing EA organizations a lot for deceptive or otherwise distortionary practices (see here and here), and one response I often get is, in effect, “How can you say that? After all, I've personally assured you that my organization never had a secret meeting in which we overtly resolved to lie to people!”
Aside from the obvious problems with assuring someone that you're telling the truth, this is generally something of a nonsequitur. Your public communication strategy can be publicly observed. If it tends to create distortions, then I can reasonable infer that you’re following some sort of incentive gradient that rewards some kinds of distortions. I don’t need to know about your subjective experiences to draw this conclusion. I don’t need to know your inner narrative. I can just look, as a member of the public, and report what I see.
Acting in bad faith doesn’t make you intrinsically a bad person, because there’s no such thing. And besides, it wouldn't be so common if it required an exceptionally bad character, anyway. But it has to be OK to point out when people are not just mistaken, but following patterns of behavior that are systematically distorting the discourse - and to point this out publicly so that we can learn to do better, together.
(Cross-posted on LessWrong.)
[EDITED 1 May 2017 - changed wording of title from "behavior" to "disposition"]
18 Comments
This is not to say humans are not well calibrated to social situations, only that for a fixed level of accuracy we should expect more false positives to false negatives for most behaviors that ancestrally could have resulted in (genetic) death since for any fixed level of accuracy all you can do is shift between making type I and type II errors.
I don't think this actually breaks the overall argument, though: it's just not evidence supporting the supposition of where the burden of proof lies.
- Purpose: a person is considered to have acted purposefully when the harm done was the conscious goal of the actor. When a person takes an action with the explicit goal of causing harm, their mental state is considered purposeful. - Knowledge: a person is considered to have acted knowingly when they were aware that their actions would almost certainly cause harm, but such harm was not their conscious goal. When a person takes an action that they know will cause harm, their mental state is considered knowledgeable. - Recklessness: a person is considered to have acted recklessly when they acted in conscious and unreasonable disregard of a known risk. A person is reckless where they are aware that their actions have a substantial risk of causing harm, and such risk is unreasonable under the circumstances. - Negligence: a person is considered to have acted negligently when they took an unreasonable risk that they should have known about, but were not consciously aware of. Where a person is unaware that their actions pose an unreasonable risk of causing harm, but they should have known the risk, their mental state is negligent.
I like this formulation. It's not about comprehending the wrongness of one's act, but it is about comprehending the likelihood of harm. A purposeful state of mind is the strongest mental state, and the one people usually think of when they hear "bad intent," but it's only one of four culpable mental states.
I think examining someone's mens rea is a critically important step in conflict resolution. There's a big difference between someone who didn't realize their actions would cause harm and a person who was well aware that their actions would likely cause harm, but proceeded anyway. And in terms of what Soarez is talking about, there's a big difference between talking to someone who *wants* to harm me, someone who harms me through recklessness, and someone who didn't know their words would cause harm.
I got stuck at the title here. Bad intent is a feeling not a behaviour by definition, and bad faith is typically defined to include bad intent.
Are you saying that actual bad feeling is so rare that we needn't have language to describe it (you say mens rea is 'not really a thing') so can repurpose the language to describe your middle ground scenario?
[After Ben said he should have used 'disposition' not 'behaviour' in the title]
That sounds more defensible, but still not very defensible. There's a well-understood meaning of bad intent (or at least, I thought there was?), and the example you have given doesn't meet that definition.
In any case even if you do expand the scope I want to have language to describe actual deliberate intent to deceive; the partner who cheats and then chooses to cover it up, the insurance salesman who sells you something they know you don't need, politicians who say they will do x knowing full well x is impossible. These examples aren't rare, they are extremely common. Mens rea is very much a thing.
I want to clarify why I feel strongly about this. Expanding definitions of words with strong positive or negative connotations is a pretty basic example of rhetorical trickery; you associate the connotations with whatever it is you intend to then praise or criticise. I don't think Ben intended a trick, but in a post ostensibly about honest communication that's not great.
I somewhat regret accepting Nate's framing of a single axis from good to bad faith, and intend to write about how to talk about this more granularly.
I've gotten a fair amount of pushback on the "mens rea" thing, and it's helped me clarify my position. Mens Rea, in law, refers to a variety of tests applied to behavior, in order to selectively punish behavior that one ought to have been able to know to avoid, since such behavior is more deterrable. Setting up these standards necessarily involves some amount of modeling the possible mental states of people doing various kinds of harms, but the law generally doesn't directly try to measure someone's mental state, and there's no such one thing as having a guilty mind, that can be assessed directly in all such cases.
Likewise, in discourse, we can try to model what someone ought to have been able to figure out and acknowledge, and use this to define standards of conduct. But the actual standards we apply publicly have to be standards of conduct, not standards of subjective states, especially since self-deception is common.
I think it's more natural to talk about different degrees and kinds of bad faith, then to make a stark distinction between bad and good faith. Most of us aren't perfect, but we can try to do better, and create public standards that help.
Likewise, in discourse, we can presume that people intend to cause the natural and probable results of their statements. We can not only model what they ought to have known, but also what they actually did know and expect. We can't do this with certainty, but we can estimate the degree to which our conversation partner was (a) genuinely trying to reach a common understanding vs. (b) focused on winning an argument. It's an important distinction, and not one I feel we can gloss over by focusing only on conduct. The degree to which a discussion is going to be productive is directly related to the intentions of the people having the discussion. Intentions can be inferred from conduct, but at the root, it's the intentions, not the conduct, that are going to determine whether the discussion is going to be productive.
For instance, there's a big difference between someone committing the sunk cost fallacy because they don't realize that's what they're doing vs. someone intentionally making an emotional appeal to sunk costs despite knowledge that it's a fallacy.
This is, I think, why there's so much focus on "bad faith." Rationalists tend to like argument and debate, if it's done with a person who is genuinely trying to reach an understanding. Even if they're not arguing fairly, as long as they are trying, we tend to forgive easily. But that goes out the window if people are arguing unfairly on purpose. I think that's why this is even an issue.
We have a model of what a reasonable person acting in good faith would do in response to various circumstances. This involves modeling their hypothetical state of mind, as they genuinely try to understand the consequences of their actions, etc. We then use this as the standard against which we measure the behavior of actual people (at least if they're not deeply incapacitated). If we imagine the good-faith hypothetical person might have made the same mistake in the same situation, then the behavior is excused. If not, then we can say that there's mens rea, in cases where that's relevant. This is a lot easier to show, than specific claims about the mental state of the person accused.
I think this is roughly what should happen in discourse as well, though of course there will be lots of difficult to decide cases. This implies that in cases where norms have not yet been worked out, it's helpful to have a neutral arbiter, and refocus the conversation on what amount of interpretive labor we would expect a generically reasonable person in the same situation to do and why, with an eye towards setting good precedent, rather than the mental state of either party.
I don't think I'm quite understanding your proposed discourse norm. IIRC, this discussion started with people telling Nate Soarez "I think the people I've been talking with aren't arguing in good faith" or something to that effect. Are you suggesting that, if I'm in such a situation, I should judge against an objective standard? Or I should appeal to a neutral arbiter? I think it would be more effective just to say "I'm starting to feel like you are not arguing in good faith. Here is why. I'm willing to give you the benefit of the doubt, but please explain why you're doing [suspicious behavior]."
I think my issue is that I just don't believe there's an objective standard that will effectively separate the well-intentioned from the willfully obtuse. I think focusing on someone's actual intentions is the only way to determine whether it's worthwhile to keep the discussion going.
I'm continuing this discussion because it seems to me that we are both pursuing a shared goal of figuring out the best discourse norms to have in our communities. You're clearly responding in the spirit of collaboration and understanding, and I hope you feel the same way. At the same time, I almost never comment on most blogs because that sort of thing doesn't happen. You just get people defending their point, and not interested in actual engagement. I infer the intentions from behavior, but if someone shows me that they are interested in a genuine exchange of ideas (even if all they do is say so), I am usually willing to engage.
When you ask questions like this, you start to notice that there are degrees of bad faith. Often it's finite and can be overcome if you're willing to do extra work, but this willingness is exploitable and you might at some point want to refuse, and just say "it's not my job to educate you." But sometimes people really do have a mental block that amounts to infinite bad faith, in which case, you're making concessions to Suicide Rock, which is nearly always a bad idea.
I didn't use a phrase like "bad faith," in part because I worried it would put too much focus on the individual who said the thing, while my sense was that they were just saying what lots of people were implicitly thinking already. But it's important to be able to talk about behavior patterns that promote dishonesty, and if we have strong norms against accusations of bad faith, that means that the people trying to point out such problems have to watch their words carefully, while as far as I can tell, no corresponding burden is applied to people executing such patterns.
It would be much, much better if we'd just figure out how to chill out about accusations of bad faith. None of us are perfect! Most of us are dishonest at least a little bit, because we live in a culture saturated with dishonesty at all levels. When people argue (as Alexander did above) that something I'm doing seems dishonest, and I think they're pointing to something that's plausibly real, I'm thankful for the help noticing, which gives me the opportunity to try to do better next time.