by Carl-Erik Mattlar

Prologue

Motto: Concordia Res Parvae Crescunt, Discordia Maximae Dilabuntur

When one tries to understand a situation at any specific point of time it is sometimes helpful, and sometimes necessary, to study how the situation developed over a number of years. For me, it is a natural way of looking at the emergence of what is currently being called R-PAS. My first love at the university was world history, and I earned a major in this subject before going into psychology. Therefore, after listening to Greg Meyer in Jyväskylä (2005), at the 10 year Anniversary of the Finnish Rorschach Association for the Comprehensive System, I started gathering information on what now seems to be a new System for understanding the information obtained from the Rorschach cards.

Introduction

Over the last six decades there has emerged a wealth of various and differently founded criticisms of the Rorschach application. These criticisms have usually been productive because they led to further development of the method. Currently, it is my opinion that the major criticisms of the Rorschach Comprehensive System which are coming from our colleagues, the developers of the R-Pass Method, have to date not contributed to further development of the Rorschach Comprehensive System, but rather have confused and bewildered many in the psychological community. The criticisms coming from our R-PAS colleagues seem to be founded in elegant statistical analysis, but appear to be unfortunately based on questionable assumptions that have squelched scientific debate; and therefore, appear to be more irrational than rational.

Our goal should be to continue Dr. Exner’s deliberate and methodical evolutionary process for a still better Comprehensive System. It should be an evolution based on meticulous research. Please keep in mind, that there are already on PsycINFO more than 9000 studies on the Rorschach.

There is, actually, quite a lot of evidence, although some claim that there is not enough research. During the years 1999?2004 I wrote a paper on the utility of the Comprehensive System. I was satisfied ? and the research volume grew during the years up to now (C-E. Mattlar, SARJ 2005, pp, 3-31).

Of interest is what Don Viglione (a developer of the R-PAS System) wrote ten years ago (Psychological Assessment, 1999, 11: [3]):

“Taking the time to read and to understand the empirical literature in refereed journals over the last 20 years leads to the conclusion that the Rorschach variables are useful for many purposes in clinical, forensic, and educational settings. This conclusion rests on a synthesis of the empirical literature emphasizing ecologically valid, behavioral, real-life criteria…. The assumption that the Rorschach is not useful … is mistaken and contrary to evidence” (p. 260).

John E. Exner, Jr. died a month before the 2006 Society for Personality Assessment Annual Meeting in San Diego. He had, during a period of nearly 50 years, implemented a steady evolutionary process for the Rorschach Comprehen­sive System. He had, from the very beginning, emphasized the importance of reliability and empirically substantiated validity, i.e., what is now called an “evidence based” method.

Dr. Exner writes in the Preface of the 4^th Edition of the Basic Foundations, 2003, some important statements:

- (p. xiii) “The breadth of data that had accumulated by 1976 was much more extensive than could easily be scattered through a series of articles and a decision to create Volume 2 was made.

- (p.xiv) “Among those there is a group who deserves special recognition. They have served on the staff of the Rorschach Research Foundation (Rorschach Workshops®) for varying intervals …….They have trained and/or supervised more than 800 examiners who have participated in more than 500 investigations.”

– “This new revision of Volume 1 is not a brief book, but the Rorschach is not a simple test. The term “Comprehensive” remains appropriate. The System does reflect the hard won wisdom of those who researched and developed the test long before this project was initiated, plus the new information that has unfolded through the work of many practitioners during the past three decades.”

In the preface of the Primer (2000) Exner gives an overview of the development of the Comprehensive System, stating, i.e.:

“The objective was to put forth an approach to the test that would have a solid empirical basis, could be learned easily, and could be applied in a standard way. ……. Clearly, we have not (in 1974) finished the task. .. We had adopted some variables which, on closer scrutiny, failed to achieve the objective for which they were intended, and we had made some decisions based on small samples of data that later proved to be flawed. …… Over the next several years a flurry of new research occurred, and additions and changes to the System began to be reported with regularity. …….. All three volumes were revised again during the 1990s, but even then the research could not be brought to closure. New variables have been discovered. New interpretive rules and strategies have evolved. And, as usual, new questions have surfaced. …. Rorschach interpretation requires common sense. …. The expanding research base underpinning the System has caused the volumes about the System, especially Volume 1, to increase in size, creating a potential of information overload for the beginner. Obviously, research findings are important and should not be neglected. ….. Interpretation of the Rorschach consists of dealing with clusters of data. Each cluster relates to a psychological feature of the person. This work focuses on each of the clusters separately, and then addresses the matter of organizing all of the findings. …- The material in this work reflects the up to-date findings regarding the System and its use. There are some new variables, recently added to the System, and those familiar with the System will note that some older variables have been deleted from the Structural Summary. This is a result of “fine tuning” the System in light of many new analyses.”

In the Preface of the Workbook 2001;Fifth Edition, 2005 Exner writes “The new System now appears to stand about as complete as possible….”.

Anne Andronikof has written in Rorschachiana (Exneriana-II) the following:

“John E. Exner, Jr. (1928–2006) was an outstanding personality in the field of clinical psychology and had a profound influence on the psychologists, practitioners, teachers, and researchers who had the privilege to encounter him either personally or through his writings and teachings. He was a charismatic person whose gentleness, generosity, and modesty have tended to overshadow his seminal contribution to psychological assessment. As is common with outstanding personalities, John Exner has also aroused jealousies, rivalries, and, at times, even resentment.” Part of this paper was presented at the XIXth International Congress of Rorschach and Projective Methods, July 22–26, 2008, Louvain, Belgium.

Rorschach developments after John E. Exner’s death – suggested changes, the new criticism, and the alternative measures

The first “New Program,” with the intention to change the Interpretive System and “streamline” the Rorschach began with these presentations:

Innovations for the Comprehensive System

- The Evidence Base and Recommended Changes for the Comprehensive System, presentation by G.J.Meyer and D.J. Viglione in Arlington, 7/3 2007.

- Draft for Rorschach Interpretation Principles & Sequence, an: “Evidence-Based Hierarchical Approach”, by D.Viglione and G.Meyer – 10/2006, presented by D.J.Viglione in Helsinki 16/3 2007.

First Presentation

In the first presentation, in Arlington, the Rorschach Comprehensive System seems still to be preserved. A number of changes were presented:

Taking the R, the Lambda and the EB into consideration when interpreting, and an alteration of the Form Quality evaluation where the conventional form code, Fo, (and then also F+), the unusual form code, Fu, and the distorted form code, F-, all are eliminated.

This “new” Form Quality variable seems to be quite close to the XA%. I did not participate in this Symposium, but the following was taken from the handout that stated:

“Given data on stability, scoring accuracy, R with Lambda, and reference samples, we should interpret cautiously, taking into consideration appropriate limitations, and mentally place wider bands of uncertainty around our scores, reference data, and clinical inferences”

It should be pointed out, however, that it was John Exner’s tradition to continue to collect data, explore issues, and solidify the empirical foundations for the Comprehensive System. Exner considered the CS as a “work in progress.”

Second Presentation

In the second presentation fundamental changes were proposed and consisted of two parts:

The first part:

1. An adjustment to international normative expectations is recommended for at least 20 variables.

2. A new Form Quality evaluation is proposed, (where, e.g., the measures of conventionality and idiosyncrasy are eliminated).

3. All variables should be considered dimensional, and the “if – then” statements in the Primer and the Basic Foundations should be eliminated,

4. Generate strengths or positive interpretations as appropriate.

The second part

It implies a total (revolutionary, not evolutionary) revision of interpretive strategies.

1. The key variable-, cluster- and step-wise interpretive approach is eliminated. It is replaced with an “Evidence-Based Hierarchical Approach.”

2. Rorschach adequacy data: The prerequisites are a scrutiny and interpretation of R, Lambda, Pure F%, Complexity.

-Positive versus Negative Predictive Power, overestimates versus underestimates. (One has to ask, “How can this be implemented?”)

- Implications pertaining to the findings regarding Expressive-Suppressive Dimension. (Again, there are questions of how to identify these concepts and how to implement the interpretations.)

- Sufficient Connections/Relationships within Response to Measure Thought Disorder?

- DQ+, Blends, Multiple/Secondary Contents, Z. (How can this be accomplished?)

3. Complexity, Engagement – Meyer’s First Factor

- Complexity – DQ+, S integration, Blends, Multiple/Secondary Contents, Z, FQ+

- Simplicity – Common D, Do, DQv, Pure F, A only [single A, (A),Ad, (Ad) content], within Card PSV

- Simplicity = Do Fo A

4. Emphasize scores that incorporate multiple response features and have replicable evidence for construct validity such as actuarial Indices, broadband data, overall severity of psychopathology, and a sound behavioural experiential foundation.

- Actuarial – PTI (which is deleted or changed as a consequence of the new Form Quality scoring), DEPI (which is to be excluded), CDI, S-CON, HVI, OBS.

- Broadband – EII-2, X-%, WSUM6, GHR:PHR, EA, es ….

- Consider non-CS scores that have been regularly validated in the research literature, including Rorschach Oral Dependency Scale (Masling, Bornstein), Mutuality of Autonomy (Urist), some of the Lerner scales, Rorschach Prognostic Rating Scale (Klopfer), Elizur Scales of Anxiety and Hostility. – Can this realistically be interpreted as “streamlining?”)

- De-emphasize variables that lack sound behavioral or experiential foundation and have not generated much supportive research, e.g., a:p ratio imbalance, S, S- , C’, D vs AdjD, FM, High Egocentricity w/o reflection, or that have consistently failed to be validated (DEPI) for diagnostic purposes.

If we do as our R-PAS colleagues suggest is it to be understood that these variables actually should not be used any more, then, for instance, should we not evaluate stress tolerance and situational stress any longer?

5. Read through Exner’s interpretive routine to generate additional hypotheses for behavioral/experiential valid scores that have not been subjected to robust validation research (So, does this mean that we should generate a structural summary to read the variables but not consider them valid?)

- Consider “these” interpretations to be more speculative of ancillary – e.g., Dd, DQv, T, FD, Blends FC:CF+C, An+Xy, Food, parenthesized human content, PER, CP, COP, Intellectualization Index, PSV.

- Carefully and sceptically evaluate these hypotheses by considering them in light of other sources of information or by exploring them in discussion with the client and his/her significant others (e.g., referral therapist, spouse)

6. Organize Information in Domain Approach – Each Domain influences subsequent domains: Thought, Affect/Impulse, Self, Interpersonal.

7. Sequence of Scores

8. Words and Behavior

An overview of the revisions and changes in the program presented at the XIX ISR Congress in Leuven, Belgium, 2008, by Greg, Don and Joni.

1. Revisions regarding Administration and Inquiry

- New Variable Engagement/Complexity (R w/ L) – this requires new, stratified norms.

- R-Optimized administration. This is to be implemented by “Prompt for 2 responses and Pull after 4 responses.

This implies a radical change in one of the leading principles in the Comprehensive System, that examination including the inquiry is non-directive. The examiner registers the perceptions, and interferes as little as possible with the subject’s work!

At least 14 responses should be considered adequate but more responses are not necessary since research has shown that not much additionalinformation is gained from an excessive number of responses.

These two changes, put into effect, imply that new norms have to be gathered. Then it is also necessary to decide if these norms should depict healthy people (like the norms gathered by Exner at Rorschach Workshops), or “ordinary people”, i.e. random samples.

The R-PAS task, to gather new norms, using new administration rules, and only those variables considered valid (including a number of new variables) world-wide is immense. The training of examiners from all over the world will take many years. – It is impossible to use the data published in the JPA Supplement 2007, since in these different examination procedures were followed.

Adequately trained examiners is a major issue. Matsumoto’s child study, (FIN Japan Children final version August 9, 2009), and also the paper by A. Lis et al [2007] showed that trained examiners obtain a mean Lambda of 0.69, compared to 2.07 for less trained examiners, and a mean T of 1.22 compared to 0.24).

Joni Mihura’s 2008 Validity Review

This is based on a “Rorschach” search in PsycINFO and MEDLINE, English language papers only, 1974 and after. Result: 2.134 citations.

Green: Good support

Yellow: Some support

Red: Absence of evidence, or Mixed findings, or Limited support

An evaluation of Mihura-G5

The exclusion criteria used by Joni Mihura, and as we can see, accepted by the G5 group, seems rather strange. First, a comparison with the evaluation of Rorschach research used by John Exner (Basic Foundations, 1986, p. 22, and 2003, p. 24). John Exner writes:

“Concurrent with the three surveys, another project was completed at the Foundation that involved a systematic review of all published Rorschach research. The purpose of the project was to categorize, cross reference, and evaluate studies as they related to the variables of administration, scoring, and interpretive postulates that had evolved in each of the systems. By 1970 the Rorschach literature consisted of nearly 4000 articles and 29 books, plus Rorschach’s monograph. ….. About half of the nearly 4000 articles were purportedly research works.” However, when scrutinized against contemporary standards, more than 600 were seriously flawed, and an additional group of 800 articles also contained flaws. This evaluation of the Rorschach literature yielded nearly 600 research works that were methodologically sound and included appropriate data analysis.”

It should be mentioned that none of these studies are considered valid by the G5.

A consequence of the G5 decision to consider only peer-reviewed English language studies published at or after 1974 valid, is that none of the Great Systematisizers (Klopfer, Beck, Hertz, Piotrowski, and Rapaport) had produced valid, reliable Rorschach methods. None of the European methods, were, according to this view of science, valid either, including Hermann Rorschach, Oberholzer, Bohm, Alcock, Ames, Anzieu, etc., and none of these individuals had produced anything valid. – As a matter of fact, Hermann Rorschach’s monograph was never peer reviewed. After its publication (Ellenberger 1954, p. 207), the Psychodiagnostics was met with indifference in Switzerland and overt hostility in Germany. Prof. William Stern, the advocate of “personalism“ launched a vehement attack: He stated that no test could ever seize and comprehend the human personality; he declared Rorschach’s methodology faulty, his evaluation artificial and arbitrary, and his statistics insufficient. Had the Psychodiagnostics been peer reviewed, it most certainly would never have been published.

Over time, according to John E. Exner, p xiv in the Preface of the 4^th Edition of the Basic Foundations, 2003, more than 500 investigations were conducted at the Rorschach Workshops. Most of these were never published. Further, Rorschach research has been conducted in France, Germany, Spain, Japan, Argentina, Brazil, Italy, Finland, and Israel, just to mention a few non-English speaking countries, and published domestically in a number of languages. All of these studies have been ignored by the G5.

At the ERA Congress, Prague, CZ, August 27, 2009, Gregory J. Meyer and Joni Mihura presented a refinement of the above model:

The revision of the First Program – August 27, 2009

Below, are the four broad conceptual categories by G5

(To the best of my knowledge there are no systematic guidelines regarding how to interpret a record, cf. also the SPA Annual Meeting in .Boston 2011.)

Particularly interesting, to me, is that emotions/affect, is not considered a “domain”, or “conceptual category.” It could be concluded that the academically successful theoretically oriented statisticians, members of the G5, do not consider the emotional life particularly important.

The R-PAS group has suggested that a total of 24 variables be eliminated from the structural summary. These variables are:

1. Affect cluster: CP (Color Projection)

2. Affect: SumC’:WSumC

3. Constellations: DEPI

4. Constellations: OBS,

5. Ideation: Sum6

6. Ideation: MQ None

7. Ideation: a:p ratio

8. Interpersonal: Isolation Index

9. Interper­sonal: (H)+(Hd)+(A)+(Ad)

10. Mediation: FQ+

11. Mediation: S-

12. Processing: W:M ratio

13. Processing: Zd

14. Processing: PSV

15 .Resources: Adjusted D score

16. Resources: FM+m

17. Resources: Sum Shading > [FM+m]

18. Resources: Adjusted es

19. Resources: EB Per

20. Self-Perception: Egocentricity Index

Further, 40 variables are found to not have sufficient validity (color yellow). Thus, if used, they should be used with extreme caution. These are the following:

63. Affect: WSumC (low)

64. Affect: Pure C

65. Affect: Affective Ratio

66. Affect: S

67. Affect: Col-Shading Blends

68. Constellations: CDI

69. Ideation: Ma:Mp Ratio

70. Ideation: Intellectualization Index

71. Ideation: M-

72. Ideation: DR

73. Ideation: FAB + INC Level 2

74. Ideation: ALOG

75. Ideation: DV

76. Ideation: DR

77. Interpersonal: a:p Ratio

78. Interpersonal: Pure H

79. Interpersonal: all H content

80. Interpersonal: AG

81. Interpersonal: COP:AG ratio

82. Interpersonal, COP + AG

83. Interpersonal, Food content

84. Interpersonal, PER

85. Mediation: XA%

86. Mediation: Xu%

87. New: 1^st Factor

88. New: 2^nd Factor

89. New: 3^rd Factor

90. Processing: Dd

91. Processing: DQv + DQv/+

92. Processing: W:D:Dd

93. Resources: SumC’

94. Resources: SumT

95. Resources: SumV

96. Resources: FM

97. Resources: es

98. Resources: Sum Shading

99. Self-Perception: FD

100. Self-Perception: An + Xy

101. Self-Perception: H:(H)+Hd+(Hd) Ratio

102. Self-Perception: SumV.

Ultimately, according to the G5, Summary of Variable Validity, August 29, 2009, only the following 37 variables, five of which are new, are found to be valid, and should thus be used in interpretation:

26. Affect: FC:CF+C

27. Affect: Blends

28. Constellations: PTI

29. Constellations: S-CON

30. Constellations: HVI

31. Ideation: Level 2 Special Scores

32. Ideation: WSum6

33. Ideation: Level 1 vs Level 2 Special Scores

34. Ideation: Morbid

35. Interpersonal: COP

36. Interpersonal: GHR

37. Interpersonal: PHR

38. Mediation: WDA%

39. Mediation: X-%

40. Mediation: Popular

41. Mediation: X+%

42. New: Trauma Content Index

43. New: ROD

44. New: EII-2

45. New: Complexity

46. New: Extended Aggression (AgC)

47. Processing: Zf

48. Processing: DQ+

49. Resources: R,

50. Resources: Lambda

51. Resources: Lambda with R

52. Resources: EB

53. Resources: EA

54. Resources es>EA

55. Resources: D score

56. Resources: EA with D Score

57. Resources, m + Y

58. Resources: m

59. Resources: SumY

60. Resources: M

61. Self-Perception: Morbid

62. Self-Perception: Fr+rF.

Regarding the decision not to use the “cluster and step approach” the following:

At this point we are not planning to use the CS cluster approach, which provided a logical way to progress through the data. Note also that the cluster and sequential steps were not empirically studied for accuracy or validity. They were a practical and logical approach. Our approach has some of the same elements. As of now our selection of variables to interpret is based on three considerations. The first is the evidence supporting each variable. Second, we have organized scores into four broad conceptual categories. These categories are logically determined and likely to be revised and renamed somewhat before we’re done. – Actually, the conceptual categories – latercalled “domains” are “logically determined,” or, maybe, “arbitrarily created”. Since there are no rules for the interpretation either, every user of the R-PAS can arbitrarily interpret the variables in any order he/she finds fit.

This subjectivity seems strange, considering the scientific rigor with which the G5 has criticized John E. Exner’s cluster and step-wise approach.

The statement regarding the Comprehensive System Cluster and Step approach is, however, not quite true. The Exner textbook, The Rorschach, A Comprehensive System, Volume 2: Interpretation, Second Edition, 1991, p. 142 states:

“Even the discriminant function, inter­correlational, and factorial studies that demonstrate that variables fall into clusters tend to convey the notion that each cluster of variables is, somehow, insular and should be interpreted as such. Unfortunately, such an interpretive routine serves only to create a fragmented portrait of the subject.”

Above, Exner, informs the alert reader that the cluster structure is not a leisure-chair product, but, that it is empirically implemented using statistical measures.

Additional specifications by the G5, September 8^th, 2009

The current G5 (Robert Erard, Phil Erdberg, Greg Meyer, Joni Mihura, and Don Viglione) ap­proached the Rorschach Community September 8, 2009, with an Announcement regarding their evidence-focused approach to the Rorschach. Their intention is, as far as I have understood it, to further clarify “The revision of the First Program – August 27, 2009, in Prague”.They emphasized the following changes, maybe not quite clearly formulated in the Prague presentation:

1. New R-Optimized administration guidelines,

2. Interpret normative data adjusted for protocol complexity (determined by R and

Lambda),

3. Use of composite international reference values for normative benchmarks.

4. Focus interpretations on variables with a robust evidence base,

5. Implement an international effort to increase form quality validity and coverage.

6. Use standardized scores (Z-scores, T-scores, for all variables?) to aid interpretation,

7. Improve inquiry guidelines.

Conclusion.

After all the changes presented by the G5, it is not feasible for them to use the term “Comprehensive System” for their “new method” involving changes in administration, inclusion of variables, etc. The System they have created is something fundamentally other than the Comprehensive System.

Naming the New System

Currently, it appears that the New System is being introduced as the “Rorschach Performance Assessment System – R-PAS”.

6. R-PAS Introduced at the SPA Annual Meeting in San Jose March 24-28 2010.

The G5 appeared to use the SPA Annual Meeting in San Jose March 2010 as a market place where they were able to apply considerable political influence in demonstrating the advantages of their new System. According to the SPA Program the R-PAS advocates (members of the G5) had 16 presentations. To me, this seems excessive. It was the core group (Meyer, Viglione, and Mihura), who primarily did the presenta­tions: Gregory Meyer 11, Donald Viglione 8, and Joni Mihura 6. Phil Erdberg 3, Bob Erard 2.

The title of Erard’s presentation actually revealed a skepticism (“Sounds Great, but Can I Take it to Court?”). Phil Erdberg conducted a case analysis: “Foundations of Interpretation Applied to a case.” While I have heard Phil Erdberg give many wonderful presentations on the Rorschach I found all of the above presentations I attended, including the one by Phil Erdberg, to be somewhat disorganized and confusing.

I was surprised that the members of the G5 were allowed to have so much time at the SPA Annual Convention. It may be related to the fact that Bob Erard was the President-elect of the SPA, and Gregory Meyer Editor of the Journal for Personality Assessment. I understand that, for the G5, it was of utmost importance to act as forceful as possible in presenting their approach.

Following my comments Virginia Brabender (SPA Board President) sent the following explanation:

“I just want to respond to one point that concerns SPA’s Annual Meeting. In the past few years, the number of submissions has not exceeded the number of slots in the scientific sessions. Therefore, the program committee was able to accommodate all submissions. If an area appeared to be represented more than other areas, it was merely because prospective presenters sent in many proposals on this topic. I assure you that it was not due to the operation of any bias.”

There are a couple more slightly odd phenomena. These renowned experts, in particular Meyer and Mihura, exclusively focused a critical eye on data Exner had published, in particular in Basic Foundations (2003). They deemed as invalid all research conducted before 1974. The rational was that the research was not published, not peer reviewed and not written in English.

It is interesting to note that Meyer and Mihura have conducted laudable meta-analyses, and produced elegant statistics but have not been interested in conducting any “recommended replication studies”. The reason seems rather enigmatic because such replications, tedious as they are, would be immensely important: Either they confirm the findings reported by Rorschach Workshops and John Exner or then they do not. Hopefully researchers around the world will accept the challenge and begin to consider doing replication studies for the Rorschach Comprehensive System.

The claim that only studies published in 1974 or later could be taken into account has an interesting consequence. John Exner writes in Basic Foundations, 2003, page 24 the following:

“By 1970 the Rorschach literature consisted of nearly 4.000 articles and 29 books, plus Rorschach’s monograph…The evaluation of Rorschach literature yielded nearly 600 research works that were methodologically sound and included appropriate data analysis.”

The Comprehensive System was built on these studies. Since the G5 consider the studies not valid, it would be interesting to know on what grounds they are rejected apart from the publication date.

To be included, the G5 required that all Rorschach Workshop studies should have been published in English language in Journals using peer review. There are many of these studies. On page 2 in Basic Foundations (2003) Exner reported that more than 800 examiners working at the Rorschach Workshops implemented more than 500 investigations . By 1982 over 400 studies were prepared (about 10 years from early 70′s to 1982).

These studies (over 400), if published in the JPA would have filled the 60 or more issues and nothing else, at that time, could have been published in the JPA. That would, of course, have been impossible. One should keep in mind that it is one matter to conduct an investigation and another to prepare a paper suitable for publication. The time is takes for investigators can be a significant barrier for publishing, so many forego the publication process.

There is, however, another, more troubling question – by which authority does the G5 establish for the Rorschach community what is acceptable and what is not acceptable as valid?

The implication: In negating this research is that the unpublished Rorschach Workshops studies are unreliable. There has been some insinuation that John Exner might have manufactured his data, which of course, would be defined “scientific fraud.”

Many of you have heard about Sir Cyril Burt who has remained notorious for the research fraud. “The most sensational charge of scientific fraud this century is being leveled against Sir Cyril Burt. Leading scientists are convinced that Burt published false data and invented crucial facts to support his controversial theory that intelligence is largely inherited.” (Grillie, 1976)

Cyril Burt received his knighthood as a reward for his work on factor analysis and statistics on the inheritance of IQ and his twin research. Following his death there emerged indications that he falsified the published data on the twin research. Such information suggests that there can be significant problems with even published and peer reviewed data.

Concerning Exner, many current colleagues maintained a long standing relationship with John. I have known John Exner since the IRS Congress in Sao Paulo 1987, and have been in close contact with him ever since. A relationship with John Exner probably goes back even further for Nakamura, Sendin, Andronikof, Ritzler and Sciara. It is difficult to know for sure whether John did or did not falsify his data. However, our personal knowledge of him makes it hard to believe that he did, both because of the quality of work be demanded from those around him and because his unpublished Rorschach Workshop studies were conducted by so many people working with him.

It is unlikely that none of the more than 800 young researchers working with him, involved in more than 500 investigations, the Preface of the Basic Foundations, 2003, (p. xiv) would not have told someone about the falsified data had it existed. Don Viglione, member of the Rorschach Research Council and one of the G5 group, was one of John’s graduate students and has thus first-hand knowledge of many of the unpublished Rorschach Workshop studies. If he thought these studies were unreliable, he certainly would have said so and provided evidence to this effect. If he has no basis for considering them unreliable, he should not be one of the people labeling them as an unacceptable valid reference. In 2005 Phil Erdberg co-authored a textbook with John Exner, The Rorschach: A Comprehensive System, Volume 2, Advanced Interpretation, 3^rd Edition. Certainly Phil would not have allowed false information published in a textbook he was co-authoring. None of our G5 colleagues raised these issues prior to John Exner’s death and were proud to serve on the Rorschach Research Council.

Details of the Introduction of R-PAS at the SPA Annual Meeting in San Jose March 24-28 2010

This information is based on the Handout distributed by Meyer et al at the R-PAS Workshop in San Jose, “R-PAS Rationale for Variable Inclusion and Exclusion.”

In their handout, the G5 seemed to consider the terms “Cluster” and “Domain” as interchangeable. However, later (pp 34, 35) they used only the term “Domain.” In a following handout (p 96) regarding the term “Domain” they referred to: 4 Logically Organized Domains”, i.e., they are not created using statistical nor scientific methods:

- Engagement and Cognitive Complexity

- Perception and Thinking

- Stress and Distress

- Self- and Other-Representation

Domain: Engagement and Cognitive Complexity
02. Complexity (NEW) (cf. p. 28, p.37).The coding and calculation is extremely

03. R

04. F+% (Replaces Lambda, which is eliminated, as the avoidant style).

05. Blends %.

06. Synthesis.

07. EA.

08. EA-es.

09. EA with EA-es Score. (The D-score is eliminated.

10.) M:C. The term EB is not used, nor styles introversive, extratensive, ambitent.

11. M.

12. FC:CF+C.

- – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -

41. Location Array (NEW).

42. W : D : Dd.

43. Dd.

44. Space Integration (NEW). (SI = Si & Sri) (All NEW) Sum Space is eliminated.

45. Vague.

46. Prompts (NEW).

47. Pulls (NEW):

48. < or > (Sideways responding.) (NEW)

49. Vista. (The interpretation is totally complex and cannot be done by hand). changed as compared to RCS, i.e. (NEW).)

50. FD, (Also here, the interpretation is totally different as compared to RCS, or: NEW).

51. R8,9,10%. (Former Afr., i.e. interpretation is totally different as compared to RCS. I.e. NEW.)

52. WSumC. (The interpretation is changed as compared to RCS. NEW.)

53. Pure C. (Also here the interpretation is changed as compared to RCS. NEW.)

54. Ma:Mp. (Also here the interpretation is changed as compared to RCS. NEW.

55. Intellectualization Index. (Here newcoding criteria are used, i.e., NEW.)

Domain: Perceptual and Thinking Problems

14, EII-2/3. (NEW) The calculating of this variable, p.37, is extremely complicated and it is not possible to do by hand.

15. Thought and Perception Composite. (TP-Comp., p 37) (NEW) The calculation of this variable is extremely complicated and it is not possible to do it by hand.

16.WSum6.

17. Level 2 Cog. Scores.

18. Level 1 vs, 2 Cog. Scores.

19.X-%,

20. X+%.

21. WDA%.

22. Popular.

- – - – - – - – - – - – - – - – - – - – - – - – - – - – - – -

57. Xu%. Coding is revised. (NEW) Inter-rater reliability could be better, as well as Interpretation validity. Still, it is included amongst the variables in R-PAS.

Domain: Stress and Distress

24. m. Weiner has (JPA 1997, 68(1), 5-19)shown that m is associated with helpless-ness, and Y with hopelessness. – The G5 does not make this distinction.) Interpretation slightly changed, or NEW.

25. SumY. (On the contrary, G5 links Y with helplessness., i.e., slightly changed interpretation, or NEW)

26. m+Y.

27. Morbid. An interesting observation is that MOR still is included in Suicide Composite as follows: +0.193*(ÖMOR). This is a NEW application.

28. Suicide Composite (S-Comp). “ One of the 17 variables was dropped from the index ” . The calculation is totally revised, and impossible to calculate by hand. NEW.

- – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - -

59. V. NEW, The interpretation is totally changed: ( “ -response processes suggest a strength interpretation as the generic interpretation of Vista. It is a cognitive feat to … ”) new interpretation.

60. Sum Shading. NEW. Negative interpretation should be made cautiously, since articulating shading, texture, vista,
and achromatic color is indicative of perceptual and conceptual sensitivity. Still, SumShading is also a measure of emotional distress. Interpretation is revised.

61. Color-Shading Blends. NEW. The supposed self-destructive behavior has not been studied in not-at-risk populations. … It seems, in the first place to be a measure of experiential nuances.- – - – - – - – - – - – - – - – - – - – - – - – - – - – - – - – -

62. C’. This variable has not been studied against criteria for internalized distress or suppressions of natural emotional reactions. At the present, the RCS interpretation guidelines should be avoided when C’ is simply scored for black or white attributes. NEWcoding rules.

63. FM. There is no clear support in the research literature for the inference that it assesses unmet need states. That inter- pretation also is not supported by the phenomenology of the animal movement response. This and traditional normative data may tie it with more immature ideational activity. (Using very old research!) NEW, revised interpretation.

64. Critical Contents-3. NEW. (An+Bl+Ex+Fi+Sx+Xy+AGM+MOR). Response process would suggest that these contents are related to the emergence of disrupting themes in consciousness. New content structure, new interpretation.

65. Dramatic Content Scale. (Bl+Ex+Sx+AGM+MOR). NEW, New variable, and thus new interpretation

66. Trauma Content Index. (=[An+Bl+Sx+AGM+MOR]/R.) NEW. This Index (Judith Armstrong) was not included in the RCS.

Domain: Self and Other Representation

30. ROD%. NEW. New variable.

31.MOAH : MOAP. NEW. (MOAH = Health MOAP = Pathology.)
[MOAH/(MOAH + MOAP)] New variable, not really been studied. new coding, new interpretation.

32. GHR:PHR. [GHR/ (GHR+PHR)]. NEW. Slightly revised calculation, as well as intrerpretation.

33. GHR. NEW theory, interpretation revised.

34. PHR. NEW theory, interpretation revised.

35. AGC (Aggressive Content) NEW variable, theory, and interpretation.

36. AGP (Aggressive Past) NEW variable, theory, and interpretation.

37. Morbid. Cf. Stress and Distress, 27.

38. Vigilance Composite (V-Comp, cf p.37.) NEW. Extremely complicated cal- culation. The interpretation rules not clear.

39. COP.

- – - – - – - – - – - – - – - – - – - – - – - – - – - – -

68. M-. Human Movement -. NEW to some extent. Interpretation slightly revised.,

69. All H Contents. G5: This score has not really been studied.

70. r. NEW: the rF and Fr discrepancy is eliminated. Card rotation should be considered. G5 ignores Hilsenroth’s research.

71. a:p. [a/(a+p)] G5: There has been little research on this variable.

72. AGM. G5: This is a rather narrowly defined score… it misses AGC .. is easily manipulated … and shows distribution problems.

73. T. NEW. FT and TF are eliminated. G5: The research support of T is not consistently strong.

74. Space Reversal. (SR =Sr&Sri) NEWThe literature provides negative findings linking all space responses with oppositionality, but there is older (!!) literature that supports ..

75.Pure H:NPH; [H/(H+(H)+Hd+(Hd))] NEW. Research has not supported … that this ratio of having perceptions of self and others based on real vs imaginary experiences.

76. Pure H. (NEW) This score has not really been studied.

77. PER. (NEW) This variable has some empirical support …but we need to further differentiate types.

78. FD. (NEW) A number of revisions: The idea that this score indicates capacity for objective self-evaluation is not supported … nor solely related to self-perception …Eventually the Coding criteria probably should be revised.

79. Vista. (NEW) Clear coding revision. The Vista provides more evidence for cognitive Resources than for …. negative or painful affect. .. Weak support for latter.

80. An+Xy. (NEW) There is little good research on this index.

No Longer Scored or Interpreted

The reason for eliminating these variables is mainly, according to the G5, insufficient research confirming the validity of these variables. (The exclusion criteria were: Studies published before 1974, studies published in other languages than English, and studies published in journals not using peer review.)

82. Color Projection. Low base rate, and no supporting research.

83. SumC’:WSumC. There is no research base.
84. CDI. No support for interpersonally based depressive subtypes, and the issue of limited coping resources is well handled with the complexity and engagement variable.
85. DEPI. As a diagnostic indicator, it is clear that this index does not work well.
86.OBS. This index might work well if FQ were adjusted to more realistic cut-offs.

87. a:p ratio Ideation. There is no research support for a:p as a measure of cognitive rigidity. (Cf. Basic Foundations, 2003, pp 436-438.)

88.MQ-none (?). A very low base rate score that has not been researched. (Cf. with WSum6. BF 2003, pp 427-430.)

89. Sum6. This variable is largely redundant

90. (H)+(Hd)+(A)+(Ad). There is no research base for this That the G5 do criticize this variable is slightly strange, since it is not included in the RCS (cf. BF 2003, and the Primer 2000.)
91. Fd content. Almost no direct research to support as a measure of dependency.
92. Isolation Index (Bt, Cl, Ge, Ls, Na). There are a couple of supportive studies, but the scoring is too broad.
93. FQ+. There is no research support for this low base-rate variable.
94. S-. This score works as a FQ variable but research does not tie it specifically angry or hostile affect.

95. XA%. There is fine support for this variable but it is essentially the complement of X-%.

96. PSV. Some research support but it blurs two distinct types of cognitive inflexibility and ignores other important perseverative phenomena.

97. W:M ratio. …this construct is theoretically interesting …. but the ratio has no research support.

98. Zd. The literature provides negative findings, and the logic of the difference score is questionable. (Cf., however, BF 2003, pp. 356-360!!)

99. Zf. …is related to complexity and has good support … measure of sophistication … (but) is a conceptually confounded variable….

100. es. … there is some research supporting …. (but) es is included in the R-PAS only in conjunction with EÁ rather than on its own.

101. Adjusted D Score. The logic of the score is problematic and virtually impossible to test in research. (Cf., however, BF 2003, pp.255-256, 265-269.)

102. es>EA. This is now computed as a dimensional difference score of EA minus es.

103. Adjusted es. Not needed because AdjD was dropped.

104. EB Per. EB style itself is now considered on a continuum (because the evidence does not support a pathological interpretation of ambitence…….)

105. FM+m. Research does not (yet) support combining FM with m.

106. Lambda. Although this variable has consistent research support, it has a problematic, highly skewed distribution.

107. R with Lambda. This is essentially equal with complexity. (Cf. P.37 in Handout re how to calculate Complexity!)

108. Sum Shading > (FM + m). The research support is very weak … The rationale to compare these shading variables to FM + m in a ratio is not clear.

109. Egocentricity Index. This is the most studied CS variable with almost no empirical support as a measure of self focus or self-esteem and related concepts. .. the phenomenological roots of seeing pairs remains obscure.

110. Form Dominance of Shading (FY, YF vs Y). New learners have reported that one of the most difficult coding distinctions . It often necessitates additional inquiry questions. The sum of these scores has not been interpreted.

111. Hh, Sc. These cores are not interpreted.

112. Hx. It is rare, complicated to score and thus somewhat unreliable.

Conclusion

According to the above list, over 31 variables are “No Longer Scored or Interpreted.” In addition, there are a number of new variables (Complexity, < or > sideways responding, Location Array, Space Integration, Prompts, Pulls, EII 2/3, TP-Composite, Critical Contents, Trauma Content Index, Dramatic Content Scale, AGC, AGP, MOAH, MOAP, ROD%, Vigilance Composite), or – at least – 18 totally new variables. Further, either the coding and/or the interpretation has been more or less revised (even totally changed) for at least 26 variables. Additionally, the FT, TF and T are collapsed into just “T”, the FV., VF and V into just “V”, the FC’, C’F, and C’ into just “C’”,the FY, YF and Y into just “Y”. And, finally, the Fr and rF into just “r”, or 14 variables are changed into just 5. This means that

89 variables are affected by the creation of the R-PAS.

R-PAS Presentation at SPA in Boston, 2011

In March, 2011, Greg Meyer, Don Viglione, Joni Mihura, and Robert Erard (Phil Erdberg did not participate) gave a whole day Workshop describing the status of the R-PAS. I made the following observations:

1. There appeared to be no further changes in their rules regarding examination and the coding.

2. For reference data they decided to use 1396 records from the International Data Pool. They used a Modeling Procedure for 640 of these1396 records, they did model these records according to the R-Optimized Procedure. They ignored the fact that the international protocols were gathered according to the Comprehensive System rules for administration, including Inquiry for a number of variables that are excluded from the RCS when creating the R-PAS.

3. Greg clarified that the G5, so far, had not collected any records of their own; however, in the Handout (page 66) they mentioned a target database of 123 US protocols collected using the final R-Optimized instruction by experienced examiners. Regarding examiners they stated: “largely advanced students at AIU or UT”. In my experience student examiners (as well as poorly trained psychologists) get a high number of Pure F (High Lambda) records. On the same page (66) the G5 refers to a target database of 123 US protocols collected using the final R-Optimized instructions by experienced examiners.

There is a study (Shaffer, Erdberg, Haroian, JPA 1999, 73 (2), 305-316, where 123 subjects were examined by student examiners – and the mean Lambda was 1,22. The important finding here is that, although the supervisors were extremely competent, the students gathered records with an extremely high Lambda. (The R-PAS did not exist at that time, so the G5 may be referring to another study with 123 subjects. This should be clarified.)

The G5 apparently intends to gather R-Optimized records internationally. Therefore, Greg has an agreement with the President of the European Rorschach Association for the Comprehensive System, Adriana Lis, that she will, implement a data gathering project .

The G5 then described the interpretation. The variables are presented in “4 Logically Organized Domains” (Engagement and Cognitive Complexity, Perception and Thinking, Stress and Distress, Self- and Other Representation). They apparently use raw scores and standard scores (presented on Page 1 and Page 2 of their handout) as percentile values: 60, 70, 80, 90 100, 110, 120, 130, and 140 and when interpreting, they simply look at the Page 1 and Page 2 Profiles, and start with the variable that first catches the eye, then the next, and so forth. It appears to be a totally subjective and unsystematic approach. This is interesting because it is an interpretive approach most psychologists, at least in Finland, used during the decades 1950-1980.

Complexities of the RCS compared to R-PAS

Complexity of variable coding and calculation

The calculation of some of the variables (e.g., Suicide Composite) is complex but gives a seemingly trustworthy impression. In the Comprehensive System it is possible to calculate all the indexes by hand. This is totally impossible for R-PAS.

Complexity of interpretation

John Exner wrote in the Preface (p. XIV) of the Basic Foundations, 2003;

“This new revision of Volume 1 is not a brief book, but the Rorschach is not a simple test. The term Comprehens­ive remains appropriate. The System does reflect the hard-won wisdom of those who researched and developed the test long before this project was initiated plus the new information that has unfolded through the work of many practitioners during the past three decades…”

This has certain implications. I have personally many times heard John Exner say that a minimum of 100 lectures is required for learning how to correctly gather a record and code it, and that an additional 100 lectures are required for learning how to interpret the record. I am inclined to agree. In Finland, we are using a program of 165 hours in class, plus homework (the participants gather records, code them – and, eventually, try to interpret) They also read quite a lot. – At the first lecture I usually give an overview of what is ahead, and, further, I simply state that for learning the Comprehensive System you have to be bright, dedicated, and prepared to work hard for about one and a half years. On the other side, the program is immensely rewarding – it is an adventure to learn how to find your way in the maze of Exnerian symbols! It is a process of pursuing excellence and high competence versus mediocrity.

Historically, before R-PAS, psychologists have claimed that the Comprehensive System was too difficult and that students were not able to learn it. My response is to say: “fine – use simpler, more commonplace methods.” There are self-rating scales, easy to administer, and easy to code (the computer mostly does this). These easier methods are what seem to appeal to the less ambitious whether or not it provides the best assessment information. Consider, however, how easy it is to fake good/bad on these instruments.

For some reason, in Norway, first Professor Geir Nielsen, and later on Ellen Hartmann, and also Cato Grønnerød created a “short form” for the Rorschach, called “the Norwegian Rosrchach” (NR). In a way, the G5 is now doing the same. Their system is described as very simple by their students. In San Jose, one of Mihura’s students rapturously declared: “Ýou just press the button and out comes the profile, just as with the WAIS”. That may be true, but not even Phil Erdberg was able to interpret the “profile that came out” in San Jose during his presentation at SPA.

There is, however, another major problem for interpretation for R-PAS: Since the G5 has changed the Comprehensive System so profoundly, neither the Basic Foundations, nor the Primer, can be used when interpreting “the profile.” The G5 will need to produce a number of validity studies for all the new variables (18), all the more or less revised variables (26), and the collapsed variables (14). Otherwise, their new System, R-PAS, actually is not “evidence based”.

The Issue of Excellence versus Quick and Easy

The Rorschach Comprehensive System most certainly is “comprehensive.” There are, however, other professions requiring quite a lot of study and preparation. If your intention is to be able to perform open heart surgery or brain surgery it simply is not enough to read an introductory book on medicine. It takes 7 years of training to become a psychoanalyst. Using Finland as exemplar, there are 3-4 years specialized programs in neuro-psychology, work and organizational psychology, developmental and educational psychology, health psycho9logy, and psychotherapy. All include a thesis and an exam, and lead to a Licentiate Degree in Psychology. ‘although the Finnish RCS Educational Program is outstanding in international comparison, it is less demanding than these programs.

A major “marketing” argument for R-PAS is that it “trims down” the Comprehensive System so that it will be easy to use even by non-psychologists if they have the computer calculation & interpretation program. Just as Mihura’s student rapturously declared: “You just press the button and out comes the profile, just as with the WAIS”

Expertise in the Rorschach Comprehensive System requires sufficient intellectual resources, dedication, and diligence. The alternative is to conduct a shallow, superficial assessment of the subject’s personality, primarily based on less reliable self-report measures.

For the Rorschach Comprehensive System there are “assistance” programs available: the RIAP (the Rorschach Interpretation Assistance Program), and the ROR-SCAN (Philip F. Caracena). Both produce the Sequence of Scores, the Constellations, and the Structural Summary, as well as a number of Interpretive Statements. Neither of them consider the background (the history of the subject) nor the wealth of information included in the referral. Therefore when I teach the long Finnish Educational Program, I tell the participants to use just the Structural Summary (and the Sequence of Scores, and the Constellations) when implementing the cluster and step-wise interpretation. The new learners are also encouraged to take into consideration the history (and the referral, if there is any).

Coding, inter-rater reliability, and reference data.

R-PAS collapses the shading variables and reflection variables into just 5 (14 in the RCS) which may make the coding easier. It is, however, still possible to code, for instance, T when Y would have been correct, and C’ when Y would have been correct – as well as Vista instead of FD. But consider, there are also the 18 totally new variables, as well as 26 variables for which coding rules and/or interpretation guidelines have been changed. Since the R-PAS administration rules have been changed the inter-rater reliability should be studied. In addition, as pointed out above, normative/expectancy values studies should be implemented world-wide. Otherwise the R-PAS actually is not “evidence based”.

Currently for R-PAS there is no software to purchase. The G5 has a scoring program that is web based with a fee structure for those who want to access the program. This might be a problem for the international community. For example the President of the Finnish Rorschach Association for the Comprehensive System, Heikki Toivakka, sent me a message stating that if the scoring system resides on a server within the US it would be a barrier to participation because Finnish legislation does not permit patient data to be sent to a server abroad.

While there is increasing globalization it may still be a problem for some countries to send patient data to a US server. It is interesting to consider that if Rorschachers around the world start to deposit their records on a server in the US, controlled by the G5, the G5 will be gathering and controlling an immense quantity of data that can be used for any number of purposes.

Further Development and Use of the Comprehensive System

We can accept as a fact that the RCS is a demanding method, that only bright and dedicated psychologists can master it. A reasonable conclusion would then be to (as in medicine, where we have general practitioners, and also highly skilled specialists) create a short form of the RCS, using, e.g., only a small number of important variables. The general practitioner users would rely on this method, and the advanced Comprehensive System users should use the full System. This proposition was actually made many years ago by the Norwegian Professor Ellen Hartman, and, before her, by the Norwegian Professor Geir Nielsen.

In the fall of 2009 Anne Andronikof and other colleagues formed an new organization now known as “The International Rorschach Organization for the Comprehensive System’ (IROCS) . The founding members of this worldwide group are: Anne Andronkof, Carl-Erik Mattlar, Noriko Nakamura, Concepcion Sendin, Anthony Sciara, and Barry Ritzler.

The IROCS goal is to provide leadership for the Rorschach Comprehensive System psychologists around the world who are interested in the further development and implementation of reliable and valid research with the RCS

If possible it would be helpful to identify most of the over 500 studies mentioned by Exner related to Rorschach Workshops. A number of them are mentioned primarily in different Editions of Volume 1. It would be especially valuable if researchers around the world would begin to replicate these studies. This presents significant opportunities for young researchers to explore the validity of variables of the RCS using the information from Exner and the information currently advocated by the G5.

The IROCS group will start to develop research designs for studies confirming the validity for variables where the empirically confirmed validity according to the G5 is insufficient. A number of studies will also be implemented demonstrating how specific CS variables relate to the personality characteristics we infer from them. This is a future plan.

Epilogue

There are many psychologists who have assured me that they are continuing to use the Comprehensive System and are unwilling to quickly adopt a new system that is yet to be clearly delineated, tested and proven. Both the supporters the R-PAS and the RCS have much work to do to improve on what I feel has been seminal work by John Exner.

As a member of the IROCS I will do my utmost to further the competent use of the Rorschach Comprehensive System, in which sense I also will try to bring together these enthusiasts – and I will do my best to forward the teaching of the Rorschach Comprehensive System.

In 1966 Weiner, Exner and Sciara published an article “Is the Rorschach Welcome in the Courtroom?” [1] The article reported on a survey of almost 8000 federal and state court cases in which psychologists presented Rorschach testimony. That survey revealed in only six cases was the appropriateness of the Rorschach challenged and in only one case was the testimony not admitted. The conclusion of the study was clearly that the Rorschach is welcome in the courtroom. That study is now 14 years old and a similar study should be initiated to explore if there are different rates of acceptance now.

Click here to read the entire article: THE RORSCHACH COMPREHENSIVE SYSTEM USE IN THE FORENSIC SETTING

The Rorschach Comprehensive System as developed by John E. Exner, Jr., Ph.D. has been the predominant system in use for administration, coding and interpretation of the Rorschach since shortly after its inception in 1974. Dr. Exner died in 2006 leaving Rorschach Workshops, the legal rights to all his writings, and all his files to his family. While Rorschach Workshops still continues to sell his books and forms, there is no psychologist in the family to continue research, writing, and publication of new information.

This article will look at the impact his death will have on the Comprehensive System (CS). For whatever reason, Dr. Exner did not appoint a successor. The lack of an identified successor to continue CS research has left a leadership void. There is a current initiative by some psychologists to develop a new system for use with the Rorschach. This plan for the development of a new system will be discussed along with how the CS will continue to grow and change in the future.

Click here to read the entire article: Rorschach Comprehensive System: Current issues

In December, 2007, a supplement to the Journal of Personality Assessment was published with the stated purpose of “…provid(ing) CS users with a compendium of country-specific or locale-specific norms…”  (Meyer, Erdberg, &amp; Shaffer, 2007, p S201).  Even though this statement indicates the motivation behind the Supplement was to  establish a set of country-specific or locale-specific norms, the editors, in their conclusions manuscript state that “…these projects also introduce the possibility of creating a composite set of international norms…”  (Meyer, Erdberg, &amp; Shaffer, 2007, p. S201)

It practically goes without saying that this is a legitimate project in that collecting norms for different countries and cultures is an important undertaking.  The development of a set of norms that are country or culture-specific is meaningful in that they may increase the strength of the Comprehensive System (CS) in multiple sites. We commend the amount of effort that went into these projects and the concerns the investigators demonstrated regarding important issues of reliability and correct procedures for using the CS.  A general result from these studies is well worth noting:  i.e., there is remarkable consistency across most of the international studies.  The editors of the Supplement emphasize this consistency and Ritzler (2004), the first author of these cautionary notes, has also suggested that this international consistency is an indication that the Rorschach method is relatively culture-free.  Nevertheless, there are some difficult-to-explain differences, especially those between studies from the same country reported in the Supplement.  Furthermore, we are concerned that some of the consistency across nations may be the result of methodological deficiencies that we will discuss in this manuscript.

Click here to read the entire article: Rorschach Comprehensive System International Norms:  Cautionary Notes