Matt Weiner (Opiniatrety: GREs--Why?) wonders why GRE scores are so heavily relied upon by many grad admissions committees. After all, don't they just give an advantage to "standardized test geeks" at the expense of measuring true philosophical potential? Matts question seems to presuppose that the other graduate admissions components do validly measure philosophical ability more or less in isolation from other arbitrary background skills or properties. I doubt that this is true.
Consider what might look like one of the more objective tools for fine-grained selection of candidates--the writing sample. I have no doubt that a strong writing sample is correlated (perhaps very weakly) with philosophical promise. But Matt's question concerning the GREs is comparitive, not absolute. That is, Matt is not wondering whether scoring well on the GREs has zero (or even negative) correlation with philosophical promise. Rather, he is wondering whether or not two students of equal philosophical ability might not have differential success in applying to grad school because the one and not the other has the property of being a standardized test geek. [Well, this actually misrepresents Matt's view. He does explcitly ask, "[W]hy should a good performance on the GRE predict a good performance as a philosopher?" I want to put aside the question of whether or not the GREs have zero (or negative) predictive value, and focus on Matt's dopelganger who believes that the GREs aren't predictively worthless but who questions whether or not using them selects for philosophically arbitrary characteristics. This is the interesting question. If the GRE has zero predictive value, then (if the considerations below are correct) it is just adds an additional arbitrary component to the selection process. If it has negative predictive value, then we should use high GRE scores to rule candidates out!]
The corresponding comparitive question for writing samples is whether or not reliance on writing samples gives an advantage to applicants with some philosophically arbitrary characteristic X? I guess I think that it pretty obviously does. Here is a highly simplified case. Consider two students, A & B, who ex hypothesi have equal philosophical potential. Indeed, let them be identical except for the following characteristic: A, but not B, is sufficiently confident in her philosophical abilities to submit a paper she has written but not gone over thoroughly with her professors. Such confidence is at least as likely as not to be a good thing in grad school, but it is the death knell for admissions. Given that we are holding everything else constant, B will be more likely to be admitted than A simply becuase B has a markedly better writing sample than A. But the reason that B has the better sample is just that B is less intellectually confident and this, I will suppose, is a philosophically arbitrary characteristic.
As we make the cases more realistic, little "packages" of philosophically arbitrary properties will emerge as favored and other packages will emerge as disfavored. Generalizing, call the disjunction of all those favored packages the property of being good at the application process. So-labelling the property is a bit misleading since any student with a significant amount of philosophical potential will presumably be reasonably good at the application process. So think of this disjunctive property as demarcating all those (packages of) skills involved in applying that are just show, i.e., that don't have interesting statistical correlations with philosophical ability but which make for more philosophically appealing application packages. I don't mean here to invoke things like purely aesthetic packaging skills (though those can't be ignored either); rather, I mean to invoke characteristics/skills that make one student appear to have greater philosophical potential than another even though she doesn't. The moral I want to draw is that the very fact that there is an application process is going to carry with it a fair amount of arbitrariness. Singling out the GRE as arbitrary seems to me to wrongly suggest that the rest of the process is pretty objective.
There are, I think, some conclusions to be drawn about how we few the "caste-differences" between elite and non-elite programs, but I won't go into that here.
Final related note: remember how frequently college and professional scouts make errors in judgment about athletic potential. Does anyone really think that grad admissions committees are more thorough or more accurate?