Measuring Higher Education's Commitment to Civic Engagement: The Importance of Attention to Quality, Not Just Quantity

It is natural for those scholars and practitioners who are interested in the effectiveness of civic engagement programs in colleges to think about results in quantitative, "head count" terms. We talk about how many people vote, or participate in political parties, or join in community improvement campaigns. There is, I find, less attention to the quality of those engagements. I think this is unfortunate.

There are at least two reasons for this focus on quantitative measures. One is that quantity is much easier to define and measure than quality. It's far easier to find out how many people attended a meeting than to assess the quality of the discussion that took place, or the quality of individual contributions to the discussion. It's easier to find out how many of a college's graduates lead civic organizations than to find out how good they are as leaders. This is a perfectly good reason to track these quantitative measures and indeed there is good reason why we should want to see numbers like these go up. High levels of democratic participation, whether in voting, in community affairs or in political activity are good for society and low levels of participation can put democracy in jeopardy. But of course the fact that quality matters, doesn't mean it is all that matters.

I suspect there is a second reason for the focus on the "countable", one of a very different kind. This is a sense that focusing on the quality of civic engagement is itself undemocratic and unacceptably elitist or judgmental. It's easy to feel this discomfort: imagine a discussion not of whether somebody voted, but of "how well" they voted — or claiming that college graduates are not only more likely than others to vote, but that they actually vote "better". John Stuart Mill in fact argued that persons with more education should get more votes than those with less — a proposition that to modern eyes appears (rightly) both self-serving and shockingly undemocratic.

Yet I think that if we limit our appraisal of civic engagement to head counts, we miss out on something important. After all, if college is, in important measure, about expanding students' capacities for critical thinking, analytical reasoning and effective expression, we should be concerned that those qualities show up in the public lives of our students, and not only in their work lives and private leisure. And when we think about designing programs in colleges that aim to promote civic engagement, I think we want those programs to encourage the application of these developed capacities to problems and issues of civic life.

If these more qualitative dimensions of civic engagement are to be pursued purposefully by colleges, there are two big problems to address. First, we have to find ways to appraise the quality of civic engagement that are, so to speak, "content neutral"—that don't simply define quality in terms of our own particular political values or prejudices. And second, we have to find constructive ways to "operationalize" these qualitative aspects — to actually measure how well colleges promote high-quality civic engagement.

Neither of these problems is easily solved, and all I aim to do here is to get them squarely on the table and perhaps to offer some starting points in thinking about them.

Let me turn first to this issue I am calling "content neutrality" or "impartiality", bracketing for now the question of whether we can observe or measure the qualities I am going to describe. Tempting as it may be, we can't simply say that voting for the "wrong" candidate is ipso facto low-quality civic engagement. To begin to get a purchase on quality, we have to somehow get behind the bare act of voting (or of showing up at a political rally) to learn more about the basis for the act undertaken. I think we can identify three elements that contribute to the quality of a civic or political action or decision (of which the third is controversial). In shorthand, these are information, reasons, and empathetic imagination.

Information is easiest. What is the informational basis of the voter's choice (or of the participant's contribution at a community meeting)? What did she know about the candidates' positions, past voting records, honesty? What was the informational basis of her intervention at the community meeting? Had she read the newspaper articles on the matters under discussion; were the claims she made in her remarks consistent with the known facts?

Yet facts alone are not enough. It's possible to have a great deal of information and still reason poorly. No doubt it is a fact that Arnold Schwarzenegger could bench press more weight than his opponent in the California governor's race. But it would be hard to conceive of an argument that would make that a reason to vote for him. And someone who voted for him for that reason would not exhibit a high quality of civic engagement. This is a trivial example, but there is plenty of evidence from campaign commercials that candidates (or their handlers) believe that people often base their voting decisions on poorly reasoned arguments. And, of course, when people join in community meetings or political discussions, the quality of their reasoning is more directly on display.

To be sure, judging the quality of a person's reasons for his or her position is less of a black-and-white matter than judging knowledge of relevant facts (which admittedly is not itself entirely free of dispute). We are less likely to encounter pure logical fallacies in people's political arguments than we are matters like overgeneralization, excessive weighting of particular pieces of evidence, reliance on claims that are poorly supported by evidence, and so on. Rather than simple mistakes, these are failures of judgment. Some might argue that we cannot fairly evaluate the quality of reasoning in cases where the evidence is not dispositive, but I don't think that is right. Indeed, as Derek Bok argues in Our Underachieving Colleges, teaching students to arrive at reasonable conclusions in ambiguous cases is one of the most important things for colleges to do, and one at which they often fail badly.

It is more controversial to suggest that "empathetic imagination" is a proper standard for appraising the quality of civic engagement.¹ Suppose someone votes in favor of a public project that will benefit him and his friends modestly at the price of great harm to others in the town whom he doesn't know personally or care about. His knowledge of the facts is impeccable and his instrumental reasoning from them is accurate. He is entirely willing to acknowledge that these other people will be hurt severely, and he will say that he just doesn't care. Is it justifiable to claim that his civic engagement is flawed? Some might worry that such a judgment is no more than a prejudice displayed by "bleeding heart liberals" against their "hard-headed" conservative counterparts.

I don't think that's the case. The argument sketched above, "this is good for me and I don't care about you," is not one that could be persuasive to those harmed by the proposed project if made in a democratic forum. When making arguments in the public forum or when acting as a citizen (as in voting), one should seek to argue or act in ways that are reasonable from everyone's point of view (or as Thomas Scanlon has put it, to act or reason in a way that no one who was motivated to find agreement could reasonably reject). To believe that the public sphere is simply one more arena in which to pursue one's private interest is a fundamental misunderstanding and displays a low quality of civic engagement. That this is the case is indicated by the fact that people almost never publicly make arguments of this kind ("vote for this because it is good for me" — they will at least gesture in the direction of claiming that the harm to others is less, or the gain to them is greater, than might appear). Now the capacity to grasp how a proposed policy or electoral outcome will affect others requires is the capacity to "put oneself in another's shoes", a capacity that requires not only sound reasoning, but "empathetic imagination". Valuing this quality in appraising civic engagement is not, I think, simply a political prejudice but a proper standard that is in the relevant sense impartial. Indeed thoughtful conservatives generally argue that policies that seem to neglect the interests of the disadvantaged really are in their long run interest (or at least serve social interests that outweigh the individual costs, when viewed impartially).

If these sketchy remarks make it at least plausible that we can define quality in civic engagement, and in some measure sort out its dimensions, what are the prospects for measuring it? Here I think my main point is that measurement should be understood in a much broader way than is commonly done. We often think of measurement as inherently numerical and, even more ambitiously, as being consistent with an interval scale like temperature, in which it is meaningful to compare the difference between a change from 0 to 10 degrees and one from 80 to 90 degrees. It's important not to be a prisoner of such a narrow understanding of measurement and to recognize that one can meaningfully speak of measurement whenever items can be ranked in terms of "better" and "worse". It is better to aim for measures of things we really care about than to confine ourselves to those items that can be measured with (an often specious) precision. As Amartya Sen observed in the context of the somewhat analogous problem of measuring the standard of living, "it is better to be vaguely right than to be precisely wrong".

That said, are there realistic prospects for gauging the quality of civic engagement generally and, more specifically, for relating measures of quality to the character of educational experiences? It's easiest to see the possibility of measurement for the first of the three categories I reviewed, information. There are, for example, surveys of the kinds of information people access regularly about newspaper reading and radio listening, for example. There have also been studies of civic knowledge, seeing what samples of citizens know about the legal structure, governmental institutions and such².

Good measures of the quality of political reasoning and especially of empathetic understanding are considerably harder to envision. It would be hard to get them from pencil and paper surveys or multiple choice questions, which can be useful in learning what information citizens know. The most plausible way to gather evidence about these dimensions of civic activity would be through extended interviews or possibly through observation of people participating in civic and political discussions. It is conceivable that such inquiry could be conducted on a scale that would shed some light on the distribution of quality of political reasoning in a broad sample of the American population, but it doesn't seem very likely. It is much more plausible that an individual college or universities, or a group of colleges, could engage with groups of current students or alumni to learn something about how their college experiences influenced the ways they came to political judgments and to decisions about action. My arguments earlier in this paper suggest that it is possible to bring qualitative judgments to bear in appraisal of these judgments and decisions. Although there are many difficulties in relating such evidence to particular features of the college experience, but such projects may be worth considering.

There are some very important pitfalls to be aware of in judging the quality of citizens' political reasoning. As Meira Levinson observed in a recent essay on deliberative democracy, judgments of the reasonableness of citizens' political views and arguments must be sensitive to the fact that basic background assumptions and factual beliefs differ widely across groups in a racially and economically divided society like that of the United States. African Americans, for example, for whom the history of the Tuskegee experiments is much more salient than it is for White Americans, are likely to be much more inclined to attribute bad motives or conspiratorial intent to American medical practices than are Whites. Widely differing experiences create quite different starting points for members of different groups. This reality certainly complicates but does not, I think, preclude responsible assessment of the quality of civic engagement across members of different social groups.

I have tried to argue here that judgments about the quality of civic engagement can in principle be made in defensible ways and to suggest that such judgments could and should play a role in the evaluation of college's efforts to promote civic engagement. In saying this, I don't mean to downplay the worth of quantitative measures of civic engagement. I feel sure that more is better. I only want to argue that better is better too. As Barbara Holland said in responding to my proposal for this paper, "The ultimate measure of success is rewarding lives for individuals, and healthy communities composed of "engaged" and interactive residents." We cannot achieve success, measured in this way, without attending to the quality of the engagement we foster.

Do you have something to say? Leave your remarks in the dicussion. Return to the Visioning Papers table of contents.