Intuitions and Cross-cultural Variation, Part I: Methodology

In light of this exchange with Jonathan Weinberg, I felt the need to defend the honor of intuition-driven rationalism. This will be the first part of a two part entry critiquing Weinberg, Nichols, and Stitch's, "Normativity and epistemic intuitions" (Philosophical Topics, 29 (2001): 429-460)--hereafter, WNS. I want to start with methodological issues, because I believe this paper has some serious design problems (some of which are likely to recur in related work) which make it unreasonable to draw any substantive conclusions from the experiments.

First, those of us who believe that intuitions are a source of basic evidence have been careful to isolate a particular sort of intellectual episode which is distinct from a judgment that p, belief that p, compulsion to believe that p, and a host of other related cognitive states. The states in question, rather, are states in which a given proposition seems necessarily true. Consequently, any study which purports to show that there is cross-cultural variation in intuitions must establish that the subjects are indeed basing their answers on intuitions. This is a base requirement for getting into the game. So the first question that needs to be asked is how WSN establish this preliminary requirement? The answer appears to be this: by operationally defining intuition as "response to question." The assumption (I gather) is that if a person is given a philosophical vignette and given a forced-choice answer concerning that vignette, their answer will be based on an intuition (at least often enough and in a more or less randomly distributed way across subjects). That is, the methodology presupposes that every answer (positive or negative) involves an intuition.

But it is wholly mysterious why they think this. In fact, the most common situation in my experience is that when a given case fails to convince somebody, the most common reason is that they fail to have the intuition that I am trying elicit and not that they have the opposite intuition. For instance, physicalists typically don't claim to have the intuition that zombies are not possible (though, of course, they believe this); rather, they typically claim not to have the intuition that zombies are possible. But this important distinction is simply lost on the WSN approach. So a very rudimentary issue is what exactly the WSN results show. Granting everything else, do the WSN results show that there are cross-cultural differences in intuitions? Do they show that there is cross-cultural variation in the having of intuitions in response to certain examples? Or do they simply show that there is cross-cultural variation in (forced choice) judgments? Not all of these questions have equal bearing on the epistemological status of intuitions, and some of them are downright irrelevant.

Moreover, it is not adequate to respond (as WSN respond) that that while "the sorts of intuitions that [their] experiments collect are not the sorts of intuitions that some [sic!: any] IDR theorists would exploit," nevertheless their "findings raise serious questions about the suggestion that [philosophically relevant intuitions] are anything close to universal." Why? In response to a related concern they say, "It is extremely likely that [the observed differences] would be recapitulated—or even strengthened—in any reflective process." But why on earth should we accept this? Do WSN have a theory about the connection between intuitions in their sense (i.e., forced-choice responses to written vignettes) and intuitions in the philosophically relevant sense? If so, it would be nice to hear it. They also suggest that "if philosophers' intuitions on simple Gettier intuitions are [philosophically relevant] intuitions, then [their] data indicate that strong intuitions are far from universal" because they get cross-cultural variation in a Gettier-case. But this is to suppose, mistakenly, that reading and making a judgment about a Gettier case is sufficient for having a philosophically relevant intuition about that case. It is as if they think you can individuate intuitions by the cases they are about rather than the characteristics they have. Nobody, of course, believes this; they are battling a straw man.

This leads me to a second methodological point. In the WSN experiments, subjects are asked to read a vignette and give one of two responses: (A) the person "really knows" or (B) the person "only believes". This terminology is also reflected in the phrasing of the vignettes. This decision is bizarre from a design perspective. The most obvious probe is to ask whether or not: (A) the person knows or (B) the person doesn't know. This makes it pretty clear to the subject what is being asked of them. The WSN response options, by contrast, leave a lot of room for interpretation. In particular, it is well known that the addition of focal stress by means of locutions like "really" and "only" can significantly affect meaning and are frequently used to indicate that the term in question is not being used with its standard meaning. How exactly are the subjects to understand the response "only believes"? The intention, presumably, is for it to be read as "only believes, but doesn't know". Nevertheless, it is not unnatural to read it instead as "believes without justification". And clearly if one reads it this way, the best response in the Gettier vignettes is "really knows". And what about the other answer, "really knows"? Does this suggest to respondents (it seems to) that the experimenters are assuming that he appears to know?

Of course, WSN try to control for this issue. (The fact that they have to try to control for it in the first place is suggestive of the sorts of concerns one might reasonably have over the experimental design.) Here is what they do: They worry that subjects were interpreting "really know" as a question about subjective certainty. To rule this out, they gave subjects the following vignette:

Dave likes to play a game with flipping a coin. He sometimes gets a "special feeling" that the next flip will come out heads. When he gets this "special feeling", he is right about half the time , and wrong about half the time. Just before the next flip, Dave gets that "special feeling", and the feeling leads him to believe that the coin will land heads. He flips the coin, and it does land heads. Did Dave really know that the coin was going to land heads, or did he only believe it?

Frankly, I don't see how this rules out the subjective certainty reading. The case is not clearly one in which Dave is subjectively certain and, in fact, it is most natural for him not to be understood this way. But if that is so, then whether or not subjects are reading "really knows" as "subjectively certain" the best answer is the one the subjects gave, namely, "only believes". Moreover, this one control is not nearly enough to rule out all possible interpretations of the answers as indicated above.

So their response to this problem--a problem of their own making--is inadequate. Again, what exactly do the data show? Do they show cross-cultural variation in intuitions? Or do they merely show cross-cultural variation in how one resolves the open interpretive questions about the two choice options?

Third methodological concern. It is a commonplace observation that some people find some examples more convincing than other examples. Moreover, variation in people's intuitions about a given case can depend on such factors as how people are filling in various parts of the back story or their grasp of various concepts occurring in the story. (Think here, for instance, of how hard it is to get students who don't have a background in logic or phil language to grasp Gettier's first counterexample, which turns on understanding the way definite descriptions work.) In my experience, most of the work in convincing students of a given point comes simply in getting them to understand the concepts involved in the story and/or making explicit what background assumption are or are not being made. This is NOT (as WSN suggest) a matter of "coaching intuitions" but rather a matter of getting the students to understand example in the intended way. These considerations suggest that, when designing experiments of the sort run by WSN, a great deal of care and thought needs to go into phrasing the vignettes.

But WSN do a pretty poor job on this count on at least some of the vignettes. Here, to take one example, is their Gettier vignette:

Bob has a friend, Jill, who has driven a Buick for many years. Bob therefore thinks that Jill drives an American car. He is not aware, however, that her Buick has recently been stolen, and he is also not aware that Jill has replaced it with a Pontiac, which is a different kind of American car. Does Bob really know that Jill drives an American car, or does he only believe it?

Unfortunately, it is not even clear that this is a Gettier case. After all, in colloquial English to say that someone drives an x is to say that they own an x. (Notice, for instance, that if Jill were merely renting the Pontiac and someone asked here, "What kind of car do you drive?" it would be misleading, perhaps even false, for her to say "A Pontiac" w/out further clarification.) But, of course, the mere fact that Jill's car was stolen does not mean that she no longer owns it. So if the question is understood as one about ownership, this isn't a Gettier case. Moreover, it might be that some subjects are telling themselves a back story according to which Bill knows that Jill prefers American made cars (she is driving a Buick, after all). Who knows, maybe East Asians are more likely to tell some such back story than Westerners. And let me just register my own impression that this is for me far from an obvious Gettier case. The point is not that these possible explanations are the right ones. The point is rather that the vignette is just badly written to control for such things.

Let me put this concern more generally. IDR theorists typically think that philosophically relevant intuitions are a function of one's grasp of the example. Obviously, if two subjects understand a given scenario in even slightly different ways, they may respond to it differently (either by having different intuitions or, more likely, by one or the other simply failing to have an intuition about the case). So in this sort of experimental design it is essential that the vignettes be written in such a way as to effectively guarantee common understanding of the given case. Obviously, this is a tall order. But at the very least what one would hope for are examples that are cultural (and class) neutral to as great an extent as possible. Doing so will help to ensure that all subjects are actually responding to the same case, which is essential if the results are to have any validity whatever. As the preceding discussion suggests, it is not clear to me the WSN are very successful on this count.

In light of these methodological reservations, I don't think any substantive philosophical conclusions can be drawn from the WSN experiments as they stand. Moreover, some the concerns voiced here will be very, very hard to overcome in any sort of simple read-a-vignette-and-give-an-answer-style experimental setting.

[Update: Incidentally, I have been reading through "The instability of philosophical intuitions" and, despite the much touted tentativeness of the WSN conclusions, we nevertheless find comments such as this (p. 4): "Weinberg, Nichols and Stitch revealed that epistemological intuitions vary according to factors such as cultural and educational background."

One of my biggest concerns about WSN is that, even under the most charitable of readings, it is at best a pilot study. The sample sizes, for instance, are simply so small (24 East Asians; 24 Sub-continenters) that the external validity (i.e., generalizability) of the results is highly suspect. It is doubtful that a scientist with this data would be very comfortable generalizing to Rutgers undergrads, much less East Asian culture! Nonetheless, here we have a forthcoming PPR paper which boldly asserts the variability as experimentally established fact.]

Comments

[marc moffett]

Jonathan, Thanks for your comments on the other thread. I am going to leave you with the last word on that one.

There are a bunch of things that I want to say about the preceding, but I really want to focus on your characterization of snobbism. It seems to me that you are running together two issues which we ought to seperate for the sake of conceptual clarity.

The first concerns the question of how widespread one takes intuitions to be. The second concerns the question of how closely connected intuitions are ontologically/epistemically to other sorts of relatively common grounds for judgment.

You seem to believe that if you believe in some degree of categorial distinctness between intuitions and other things, then you must also believe that intuitions are not commonplace. Now maybe there is an argument for that, but I don't see up front why that is. I believe that intuitions are ontologically and epistemically special (actually, I believe this to a greater extent then, say, George or John) but I nevertheless believe that they are really quite commonplace. Nevertheless, I believe that hunches, etc., are present even when intuitions aren't; they outstrip the intuitions. Second, I believe that under various sorts of non-canonical circumstances we might have intuitions but they are contraindicated by these other factors--which tend to be more heavily influenced by top-down factors.

The upshot, is that there is no quick path from categorial distinctness to anti-populism. There might, however, be a limited range of snobbiness about how one handles non-canonical cases. By analogy, when comparing really good wines to really bad wines, the wine snob and I pretty much agree. But when the judgment requires a more delicate palate, I just defer.

[jonathan weinberg]

"You seem to believe that if you believe in some degree of categorial distinctness between intuitions and other things, then you must also believe that intuitions are not commonplace." That's not quite my view. A better statement of my view would be: if you believe in some degree of categorial distinctness between intuitions _and our ordinary sorts of categorizations_, then you must also believe that intuitions are not commonplace. Which looks almost like a tautology, I think.

Now, if you think that intuitions are rather easily gotten, then you have a version of populism (as I suggested would characterize Dan's earlier proposal). But that view would have no teeth to resist the WNS findings. If intuitions are pretty common, and just _occasionally_ crossed up with hunches and the like, then we should expect that our subjects were reasonably well tapped into their intuitions -- in the absence, that is, of any not-purely-speculative reason to suspect that our groups would be differently inclined to have hunches instead of intuitions or something like that. It is hard to see how to square this view that intuitions are fairly common, though, with the kind of line taken in the main post, which seems to take the priors to be very low that our subjects could be basing their answer on intuitions.

To sum up: One could have the view that intuitions are fairly quotidian, which is not a form of snobbism, but then one should acknowledge that our findings make likely that the different groups have differences in intuitions as well, and indeed may be directly manifesting those differences in their answers. One could instead have the view that intuitions are radically discontinuous from ordinary categorizations, and hence from anything our study was tracking, in which case one is an adherent of snobbism & thus needs to address the substantive methodological worries I raised. And one could have an intermediate view in which intuitions per se are not so common, and thus not to be expected to be found in what was directly driving our subjects' responses; yet the cognitive machinery underlying intuitions is (as a psychologist would expect) continuous in various ways with the machinery of ordinary categorizations. And so this last view is still stuck with our findings providing evidence (albeit more indirectly) of cultural variation of intuitions proper.

"But when the judgment requires a more delicate palette, I just defer." I take the claim here to be that philosophers should count as having a relevantly more delicate palate -- i.e., that philosophers should count as _expert_ intuiters of some sort -- to be an empirical claim in very bad need of substantiation. (Tim Williamson and Steven Hales have both made it in print, if you're looking for allies.) Philosophers are surely experts in many things, but I don't think we actually have reason to think that intuitions-about-funny-cases are one of them. But that, I think, would be the matter for another thread!

[marc moffett]

"if you believe in some degree of categorial distinctness between intuitions _and our ordinary sorts of categorizations_, then you must also believe that intuitions are not commonplace. Which looks almost like a tautology, I think." But again, this collapses distinctions I would prefer to make explicit. I think that intuitions are *one of* our ordinary bases for categorization/judgment, but that it is categorially distinct from other ordinary bases. (This is not equivalent to your third option above.) So this is a form of populism. But that does not imply that it has "no teeth to resist the WNS findings." The WSN findings, as I have been saying, gloss over the differences between these distinct bases. So, putting to one side the questionable size of your samples for EAs and SCs, I grant that the WSN findings show that *taken as a group* our ordinary bases for categorization/judgment exhibit cross-cultural variation. But to infer from this result that intuitions (being one of the ordinary bases) exhibit cross-cultural variation is simply to commit a fallacy of division. I think a number of those other faculties might vary considerably by culture and, consequently, might in and of themselves wholly account for the variation you found in some of the cases. Moreover, given that many of your results were fairly weak, any kind of confound of this sort would be worrisome.

"I take the claim here to be that philosophers should count as having a relevantly more delicate palate -- i.e., that philosophers should count as _expert_ intuiters of some sort -- to be an empirical claim in very bad need of substantiation." Well, that is a somewhat stronger claim than I intended. My claim is rather that someone who is self-consciously aware of the sort of cognitive episode that is wanted (an intuition) is less likely to base their judgment on things other than that sort of episode (or, at least, to know that they are doing this). And while I grant that this is an empirical claim, it is one which I do take to have quite high priors.

[jonathan weinberg]

I'm not denying the distinction, Marc -- I'm just denying that it can do any work for you here. Near as I can tell, we're just using the term "ordinary" in slightly & harmlessly divergent ways. Recasting in your terms, my argument goes: suppose that we know that a fully competent user of the language has made an ordinary categorial judgment under fairly ordinary conditions. The defender of intuition must answer the question: what is the likelihood that a person in doing so is having an intuition proper? If it is fairly likely, then our results are direct evidence of intuitional diversity. If it is not fairly likely, but one acknowledges the likely continuity between intuition-producing cognition and most of the rest of ordinary cognition, then our results are still good indirect evidence of intuitional diversity. If it is not fairly likely, and one insists on a radical discontinuity between intuition-producing cognition and the rest of ordinary cognition, then one runs into the problems of intuition snobbism, as I argued above. Whichever way you go, you've got big problems.

One might try to thread this needle by saying that the likelihood is _just low enough_ to duck our results, and yet _just high enough_ so that one doesn't run into snobbish difficulties. That would, of course, take some serious spelling out. And moreover, such an attempted defense would commit one to a far stronger empirical claim than the ones I've been making; and with even less empirical evidence than my own very-admittedly preliminary results. And that's all I need to score a victory: if the defenders of intuition need some very particular empirical results to come out just so in order for their arguments to work, then it turns out we have to get out of our armchairs after all.

[marc moffett]

But Jonathan, you are mischaractizing the view. Let me summarize:

(i) IDR theorists (at least those with whom I am concerned, we can agree to disagree about who falls into the camp), deny that every ordinary judgment is based on an intuition; they also deny that every intuition is of the same type. [Call this restrictivism]

(ii) IDR theorists maintain that many ordinary judgments are based on intuition. [Call this populism]

(iii) IDR theorists maintain that many judgments are based on multiple factors, possibly including intuitions.

(iv) IDR theorists deny that in multiple-factor cases intuitions *must* trump other considerations as the basis for people's decisions. [To see this, simply note that we are fallibilists.]

(v) IDR theorists maintain that intuitions and other bases for categorial judgment differ wrt to their reliability and fluidity among other things.

In virtue of (i) and (iii), the IDR theorist will deny that forced-choice judgments (even "ordinary" forced choice judgments) can, in general, be identified with rational intuitions. In virtue of (ii), IDR theorists don't count as snobs; that is, restictivism doesn't imply snobbism. In virtue of (iv) and (v) IDR theorists don't think that forced-choice responses must track intuitions, even weakly. Indeed, in cases where intuitions aren't in play at all, the whole talk of tracking is nonsense unless you take "no answer" as an answer (in which case intuitions will by definition fail to track forced choice judgments).

What this shows is that in order to make it plausible that intuitions exhibit cross cultural variability of any kind by way of forced-choice questionaires, it must be established that the cases are likely to produce intuitions and that those intuitions are not likely to be controverted by other factors in a way that might vary cross-culturally. [It must also be established that factors influencing the understanding of the cases must not vary cross-culturally, but this is a different point.]

Now if someone wants to concede that these conditions are met, then they should worry about other problems with the study. But, as I've said, I think that there are plenty of reasons to doubt that these conditions have been met in all cases. After all, here is a reason for thinking that in the Gettier case something else is going on since the EA subjects are not giving Gettier-responses. Now I am very nearly certain that intuition yields a verdict of "doesn't know" in the Gettier cases. And being a man of the people, I am inclined to say the most plausible explanation of your results is simply that your subject's responses weren't based on (or only on) intuitions or judgments which track intuitions. Of course, another response is that intuitions vary cross-culturally. But, and this is the point of an experimental confound, your study is utterly incapable of deciding between these two, very different interpretations of the data. And that was my point, that the study was confounded, not that the conclusion was wrong.