[Editor’s note:] The following discussion is based on an edited set of e-mails among Joseph Ciarrochi, John Mayer and John Michela. Some of these were posted on EMONET. (Link broken.) They are posted with the permission of the contributors.
John Michela is Associate Professor of Industrial/Organizational Psychology at the University of Waterloo, Canada.
Joseph Ciarrochi is Associate Professor at the University of Woolongong, Australia.
John Mayer is Professor of Psychology at the University of New Hampshire, United States.
This discussion is a continuation of an earlier e-mail exchange concerning ability measures and self-report measures on EMONET (not posted here). After reviewing that discussion, John Michela posted the opening question here...
John Michela: I agree that there is going to be some overlap of self-report with a valid measure of ability, but why use the imprecise self-report if something is available that measures the ability directly?
Joseph Ciarrochi: I think this is the important question....and it has been troubling many people (including myself) for awhile.
I would first suggest that if two measures do not correlate, this does not mean that one measure is less accurate than the other. Rather, the two measures may be reflecting different latent processes. That is, they could both be accurate but measuring different things.
John Mayer: Can you explain why are you speaking here about latent variables rather than just plain old variables?
Ciarrochi: The idea is that you have observed variables (the answers people provide on a measure) and latent variables (the underlying psychological process that is presumed to cause the observed variables). "Latent" is a great word in conveying the inferred nature what we measure. It helps to avoid two logical errors when evaluating measurement models (e.g., Kline, 2005):
The naming fallacy... just because a particular latent variable is assigned a particular label -- e.g., EI, social skill -- does not mean that the hypothetical construct is understood or correctly named. ... "verbal labels should be viewed as conveniences and not substitutes for critical thinking."
The reification fallacy... the belief that a hypothetical construct must correspond to a real thing.
Ciarrochi (continued): Selecting and choosing between self-reports and ability based measures is more complex than first seems. You first need to ask the question, "what is the purpose of the measure?" That is, why are we using them? There seem to be three major purposes of EI measures.
The first is prediction and selection... this one is real important to you organizational folks. For example, you often want to be able to select the best possible managers. Failing to do so can cost you much money and time. So when your purpose is prediction, you often ask the question, does EI predict variance in performance over and above personality, IQ, and other traditional organizational measures. I am not sure what the answer to this question is, but at present it looks like ability based measures of EI are fairly distinctive from traditional organizational measures (though the data is still coming in). David Rosete in my lab, and people in Jack Mayer and Peter Salovey's lab, and Susan David in Australia, and many others out there... , are finding that the MSCEIT predicts variance over and above personality and IQ. I am not so sure about self-report EI measures. They don't seem to be doing as well, though again, the full evidence is not in yet.
Mayer: In the case of emotional intelligence measures, because you are measuring an ability, I believe you can make an a priori judgment, a presumptive judgment, that ability measures will have far superior validity. This gets back to John Michela’s point -- “Why not just use an ability measure where one exists?”.
Writing recently in Psychological Review, Borsboom et al. (2004) offer an interesting perspective on validity, in which they have observed that most central way to assess test validity -- indeed, perhaps the only relevant criterion for test validity is:
... if (a) the [mental] attribute exists and (b) variations in the attribute causally produce variation in the measurement outcomes. (Borsboom, et al., 2004, p. 1061).
They elaborate that, central to determining this causal relationship between the psychological entity and the test score is a good understanding of the mental qualities or processes involved, and how they might bring about alterations in test scores in the instrument under consideration.
Central to our understanding of mental ability is that it “causes” correct answers on criterion-report/ability-report scales. Central to our understanding of alexithymia -- or any more direct measure of emotional intelligence measured specifically by self-report -- is that self-judgment measures an individual’s self-concept. A person’s self-concept, in the case of ability, is one step removed from the actual ability. Since actual ability can be directly assessed, it ought to be for the best results.
It is for these reasons that I believe one can make a presumptive judgment that a criterion-report/ability-report scale can measure emotional intelligence and a self-judgment scale cannot do so as well -- if at all. Of course, if other findings seriously undermined that position, well, that would be another matter. I haven’t seen that, though.
Ciarrochi: Okay... The second reason we employ such scales, though, is influence.
This goal is about intervening to help people engage in more emotionally intelligent behavior (defined below). When influence is your primary goal, EI-relevant measures are only useful if they help guide the intervention.
A measure may not predict unique variance in behavior, but nevertheless may be useful in guiding an intervention. For example, let's say the following model is "true."
”Self-reported difficulty identifying feelings (alexithymia) leads to negative affect, which, in turn, leads to counterproductive workplace behaviors.”
In this model, if you wanted to predict counterproductive behavior, your best predictor would be negative affect. Alexithmia would predict no unique variance. However, if you wanted to reduce counterproductive behavior, it is a different story. You could try to directly reduce negative affect (which I have argued elsewhere is often difficult). Or you could seek to increase the ability of the person to identify emotions, and this would lead to decreases in negative affect and decreases in conterproductive behavior. So the alexithymia measure could be used both to evaluate who might need your intervention, and to evaluate the extent the intervention is actually working.
A third purpose of EI-relevant measures is to assess "ability" or potential. There is a lot of controversy about whether self-report measures assess ability (do people really know their own ability), or are assessing typical functioning , or assessing self-concept not connected to anything in the "real" world. My guess is that they assess typical functioning, rather than what one is capable of... .
Mayer: My impression is that there is less controversy in that regard; I think people recognize that self-judgments are just that. A further comment -- I don’t believe the typical versus maximal (or optimal) performance distinction is a terribly clear one. First, ability tests do capture effort in test-taking, but I think “maximal” overstates most test-takers’ investment in taking the test. In fact, I suspect they measure something closer to typical ability.
Ciarrochi. This sounds like a great future study. You can look at whether people can be motivated to perform better on EI tests like the MSCEIT.
Mayer: Hofstee (2001, p. 43) tells us that Cronbach (1949) introduced the terms typical and maximal performance, and “did not even bother to define intelligence and personality apart from this distinction and simply subsumed them under the difference in method”. Since then those who use such terms associate “typical” with personality, and “maximal” with intelligence. Hofstee (2001, p. 46) has raised the point that it is also possible to measure maximal personality and typical intelligence, but I think it is clearer and more accurate to speak of self-judged versus ability measures.
Anyway, can you give us an example of where ability and self-judgment scales overlap and where there would be any controversy?
Ciarrochi: Here is an example. The MSCEIT measures "difficulty identifying emotions." The Taylor Alexithymia Scale (TAS) is a self-report measure that also assesses difficulty identifying emotions. The two scales barely correlate. I would say that this is because they index different valid constructs, rather than the self-report being inaccurate. How do I know? The TAS-20 scale has an incredible amount of validity evidence associated with it. For a review (e.g., Taylor, 2000). For example, alexithymia predicts who is likely to die young, who is likely to develop emotional disorders, who is likely to somatisize, get physically ill, etc. It predicts clinician ratings of alexithymia. Finally, alexithmyia predicts important variance over and above positive and negative affectivity, self-esteem, hope, and other constructs. (We are finding these results in 8th graders and uni students for example.)
So... One can not ignore the mass of evidence around self-reported difficulty identifying emotions... It is a reliable, useful, and distinctive measure. (That is what the evidence says now.)
Mayer: You seem to be implying that there could be two different sorts of measures -- uncorrelated -- that both assess EI. How could that be?
Ciarrochi: You are right!! They can't be assessing the same thing. Maybe we should not call everything "EI." I suggest a very particular way of using EI (when the assumption is that the measure is assessing ability), which I think is 100% compatible with what you are saying. For other variables, I suggest we talk about them as indexing processes that promote emotionally intelligent behavior (EIB). These processes may or may not be related to EI, as defined by you and I. (The term "processes" is commonly used in clinical and intervention research, so i think it is a good word.)
Ciarrochi: Now, on another note, we have conducted a study with managers, using the alexithymia scale and the MSCEIT. David Rosete and I find that it is the MSCEIT perceiving emotions scale that is the powerful predictor of managerial effectiveness. Indeed, we have observed perhaps the largest MSCEIT related effect in the literature (explaining over 20% of variance in behavior). So I am a fan of the MSCEIT. In contrast, The alexithymia scale does not predict performance. Thus, If we are talking about managerial performance, I would prefer the msceit to the alexithymia scale. (But the evidence is still coming in.)
Michela: The research described by Joseph suggests that the difference between self report measures in that research and the MSCEIT is that the self-report measures are, indeed, about the self and the MSCEIT is about others. Management is about dealing with other people. Vastly overstated, depression etc. are about dealing with yourself. MSCEIT thus may be more about dealing with other people. The self-report measures seem to be about the self. In providing self-report data, people may be rather unable to assess, judge, and rate their effects on other people; maybe they just don't know (typically, not in every case). But the MSCEIT items may assess the abilities that end up being consequential in their effects on other people and how they deal with other people. Ergo MSCEIT is seen to be associated with social performance (management), self-report with self-performance (self-management of well-being).
Ciarrochi: The fallacy I am trying to avoid is that just because two things have been given the same name does not mean that 1) they are the same or 2) they must relate. (I think we should avoid giving things the same name, but, hey, that is beyond my control, because I don't make tests and measures). If two things with the same name do not relate, it does not follow that one thing is "accurate" and one thing is not accurate.
Ciarrochi: My solution to this controversy is as follows (and I say this most humbly knowing it could be wrong). We need to distinguish between emotional intelligence (EI) proper (as an ability) and emotionally intelligent behavior (EIB). Simply put, emotionally unintelligent behavior occurs when emotions impede effective action, and emotionally intelligent behavior occurs when emotions do not impede effective action, or when emotions facilitate effective action. Emotional intelligence (as an ability) is one set of processes hypothesized to promote emotionally intelligent behavior. There are other potential processes, many of which may be assessed by self-report measures (I review these in my chapter for the second edition of EI in everyday life).
Mayer: I agree with the idea that we need descriptions of how people behave, but what about just “effective social or socio-emotional functioning?” My concern is, how do we identify intelligence functioning -- of any kind, cognitive or emotional? The whole point is that intelligent, including emotionally-intelligent, functioning can be clever, strategic, and plastic. So, it is difficult to create a benchmark for it. One must identify the individual’s goals, and know whether they are aiming for short or long-term solutions, and the like.
Ciarrochi: This is indeed difficult, but I think worthwhile. Already, there is substantial research on how to identify people’s personal goals and strivings. (See, for example, the work of Higgins and Sheldon and others.) This research has focused on why people do what they do (i.e., are they intrinsically or extrinsically oriented), on what goals they find to be important, and to what extent they have achieved the important goals. People are not always good at achieving important goals, and often see their emotions and emotionally charged thoughts as the barriers to these goals (see Hayes et al., 1999). For example, people might say they are “too anxious” to meet new people (a valued direction). Or they might say they want to be a more respectful boss, but they just get “too angry” to act effectively. In these instances, emotions act as barriers to effective action (and this could be classified as emotionally unintelligent behavior). In other instances, people might be able to have their anxiety and meet new people, have their anger and deal effectively with their employees (more emotionally intelligent behavior).
Research is needed to better understand emotionally intelligent behavior (and its value as a term). I think such research would be worthwhile. (By the way, clinicians have no trouble getting this term. They are constantly seeing clients who are telling them their sadness or anxiety or anger gets in the way of what they want to do. I think the same thing happens with nonclinical groups.)
Ciarrochi: There is a lot of really valuable work out there that I think is focusing on processes that lead to emotionally intelligent behavior. Offhand, I think of the work of Richard Boyatzis, Bill Ickes, Saarni, Elias, and others (I know there are many others). Other people, such as Ziedner, Mathews, and Roberts, have examined situational influences on emotionally intelligent behavior... Some great work...
So I think we should only use "emotional intelligence" when we are making the assumption that our test is taping into an ability. We should maybe say "processes that are hypothesized to promote EIB" if we don't want to assume that the "processes" reflect an ability...
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061-1071.
Cronbach, L. J. (1949). Essentials of psychological testing. New York: Harper & Row.
Hayes, S. C., Strosahl, K. D., et al. (1999). Acceptance and commitment therapy: An experiential approach to behavior change. New York: The Guilford Press. (1999). xvi, 304 pp.
Higgins, T. E. (1998). Promotion and Prevention: Regulatory Focus as a Motivational Principal. Advances in Experimental Social Psychology, 30.
Hofstee, W. K. B. (2001). Intelligence and personality: Do they mix? In J. M. Collis & S. Messick (Eds.), Intelligence and personality: Bridging the gap in theory and measurement (pp. 43-60). Lawrence Erlbaum Associates, Publishers.
Kline, R. B. (2005). Principles and practice of structural equation modeling (2 nd ed.). New York: Guilford Press.
Sheldon, K. M., & Kasser, T. (2001). "Getting older, getting better? Personal strivings and psychological maturity across the life span." Developmental Psychology, 37(4), 491-501.
Taylor, G. J. (2000). Recent developments in alexithymia theory and research. Canadian Journal of Psychiatry, 45, 134-142.