"Accreditation is a system for recognizing educational institutions and professional programs affiliated with those institutions for a level of performance, integrity, and quality which entitles them to the confidence of the educational community and the public they serve." (CORPA, 1996)

• foster excellence in postsecondary education through the development of uniform national criteria and guidelines for assessing educational effectiveness;
• encourage improvement through continuous self-study and review;
• assure the educational community, the general public, and other agencies or organizations that an institution or program has clearly defined and appropriate objectives, maintains conditions under which their achievement can reasonably be expected, is in fact accomplishing them substantially, and can be expected to continue to do so;
• provide counsel and assistance to established and developing institutions and programs; and
• endeavor to protect institutions against encroachments which might jeopardize their educational effectiveness or academic freedom.

At its June, 1996 Board meeting, the Board of Directors of the American Evaluation Association charged an Accreditation Task Force with the responsibility of developing draft standards for the accreditation of graduate-level specializations in evaluation that are associated with existing degree programs. The AEA Board decided that it was not feasible for AEA to accredit graduate-level degree programs in evaluation. There simply are not enough programs wholly dedicated to evaluation to warrant this approach. Instead, the Board felt that it was more promising to accredit specializations in evaluation in existing degree programs that address evaluation such as programs in applied social research, health care, education and business.

Developing a set of accreditation standards is one of the first steps to developing an accreditation process. To elicit an initial set of accreditation standards from the AEA members, the Task Force decided to undertake an Internet-based process using the EVALTALK listserve participants. This approach was taken for several reasons. First, it assured a relatively rapid turnaround time. The entire study took place over several months in the Summer and Fall of 1996. Second, the method was inexpensive and easy to implement. Funds were not available at this preliminary stage of accreditation development for conducting a member-wide survey, nor was such a survey deemed practicable. The purpose of this initial stage was to draft an accreditation program that could subsequently be taken to the membership for extensive review and revision. Third, an Internet-based process was thought to have substantial research value in itself. The Internet is a critical new technology and no process like this had previously been undertaken over that technology. It was not clear whether such a process could even work and, if it did, how well it would function. Thus, this project is in large part designed as an evaluation feasibility test.

The core methodology selected to elicit accreditation standards was concept mapping. Concept mapping was chosen for several reasons. First, it was convenient. The chair of the Task Force is the originator of this technology and could provide the computer and programming expertise and access required at no cost. Second, concept mapping is an excellent method for just this type of purpose, the eliciting and organizing of a set of ideas from a target population. Third, concept mapping has advantages of methodological rigor over alternative methods of accomplishing this task. At its core, concept mapping uses advanced multivariate statistical methodologies (i.e., multidimensional scaling and hierarchical cluster analysis) that enable the analysis and graphing of group results.

The concept mapping process needed to be adapted in several ways in this first-ever Internet-based version. A World Wide Web site was constructed to manage the project, provide continuous information to participants, and allow participants to interact online. Participants used the website to enter or brainstorm accreditation standards, download materials for the sorting and rating task, and view the resulting maps and pattern matches. While it was not feasible to interpret maps interactively with participants (this was done by the facilitator), the website does allow participants to comment on any of the results directly on the website.

The concept mapping process discussed here was first described by Trochim and Linton (1986). Trochim (1989a) delineates the process in detail and Trochim (1989b) presents a wide range of example projects. Concept mapping has received considerable use and appears to be growing in popularity. It has been used to address substantive issues in social services (Galvin, 1989; Mannes, 1989), mental health (Cook, 1992; Kane, 1992; Lassegard, 1993; Marquart, 1988; Marquart, 1992; Marquart et al, 1993; Penney, 1992; Ryan and Pursley, 1992; Shern, 1992; Trochim, 1989a; Trochim and Cook, 1992; Trochim et al, in press; Valentine, 1992), health care (Valentine, 1989), education (Grayson, 1993; Kohler, 1992; Kohler, 1993), educational administration (Gurowitz et al, 1988), training development (McLinden and Trochim, in press) and theory development (Linton, 1989, Witkin and Trochim, 1996). Considerable methodological work on the concept mapping process and its potential utility has also been accomplished (Bragg and Grayson, 1993; Caracelli, 1989; Cooksy, 1989; Davis, 1989; Dumont, 1989; Grayson, 1992; Keith, 1989; Lassegard, 1992; Marquart, 1989; Mead and Bowers, 1992; Mercer, 1992; SenGupta, 1993; Trochim, 1985 , 1989c, 1990, 1993).

Concept mapping combines a group process (brainstorming, unstructured sorting and rating of the brainstormed items) with several multivariate statistical analyses (multidimensional scaling and hierarchical cluster analysis) and concludes with a group interpretation of the conceptual maps that result. This paper illustrates the use of concept mapping for developing standards for accrediting graduate-level specializations in evaluation.

Subjects

Because anyone with access to the web could theoretically join in the brainstorming, there is no simple way of knowing who or how many people did so. For the sorting and rating phase, twenty-three people participated. Demographic data for this sample are given in Table 1.

The sample is clearly not representative of the AEA member population. Only half of the sample were current AEA members. Nearly half were students. Only two of the 23 were faculty members. There are good reasons for why the sample is so small and relatively unrepresentative. The sorting and rating task is a fairly demanding one. It requires about 1 1/2 to 2 hours to accomplish, a considerable demand on participant’s time especially during the late summer months when the data was being collected. In this study, it also required that the participant was able to navigate to the website, obtain the required materials, and figure out how to return them correctly.

Because the sample is small and relatively unrepresentative, the results of this study must be considered preliminary. The study may have more value for determining the feasibility of accomplishing evaluation tasks like this over the Internet than it does as a study of accreditation standards.

Procedure

Generation of Conceptual Domain. Participants generated statements on the website using a structured brainstorming process (Osborn, 1948) guided by a specific focus prompt that limits the types of statements that are acceptable. The focus statement or criterion for generating statements was operationalized in the form of the instruction to the participants:

Generate statements (short phrases or sentences) that constitute specific standards that you believe AEA should include in its Standards for Accreditation of Graduate Programs and Specializations in Evaluation.

One specific standard I believe AEA should include in its Standards for Accreditation of Graduate Programs and Specializations in Evaluation is that...

The original statement set was edited for spelling and grammar before being made available for sorting and rating. Structuring of Conceptual Domain. Structuring involved two distinct tasks, the sorting and rating of the brainstormed statements. Participants had two options for participating in this stage of the project. The instructions for these two options are given below:

Option 1: Using The Concept System Remote Program

This option will only work if you have a Windows-based PC. Here are the steps you need to take to accomplish this option:

• Contact me (Bill Trochim) by e-mail to tell me you want to use this option. I will create a dataset for you and e-mail this dataset to you. When you receive the dataset from me, continue with the next step.
• Click on the Concept System Remote Module Setup File to download the program to your machine (the program is 1,192KB, or just over 1MB in size, so it may take a few minutes to download on your machine). Your browser will ask you where on your disk you would like to place the setup file -- don't forget to write down the location.
• To run the setup, in the Program Manager, choose the File --> Run menu item (in Windows 95 choose Start --> Run) and then click on the Browse button. Navigate to the cssetup.exe file and click on it. Then click on the OK button. The setup program will start -- just follow the instructions for installing the program. (Note: the program automatically comes with an uninstall capability. To remove the program from your disk after you've installed it, just click on the uninstall icon).
• Start the Concept System Remote program by double-clicking on the icon in the program group that was set up for you. The first screen you will see has an 'Open Database' button. Click on it to get the standard Windows dialog box and navigate to the dataset file I e-mailed to you. When you click on OK you will be shown a logon box that asks for your Username and Password. For your Username, use the first 8 letters of your last name, with the first letter capitalized and all others lowercase. For the Password, use the word 'concept' all in lowercase. At that point, you will be logged into the project and there will be instructions you can follow. At any time you can hit the F1 key to get help with what you're doing.
• When you've finished entering your demographics, sorting and rating data, exit the Concept System Remote Program.
• E-mail the dataset back to me at wmt1@cornell.edu

I estimate it will take about 45 mins - to 1 hr 30 mins for you to do the data entry depending on how comfortable you are with computers.

Option 2: Downloading the Instructions and Entering Data Manually

In this option, you simply have to download a word processor document version of the instructions and materials, print the file, follow the enclosed instructions, and either mail or fax the results back to me (the address and FAX number is in the instructions). If you have trouble with the download, or would rather I e-mail you the file, send e-mail to Bill Trochim asking me to send you the manual materials -- and be sure to tell me which of the Word processor formats below you would like it in.

To download the file, just click on the format you would like below. Your browser will ask you where you want to put the file -- be sure to record where you store it. If you don't see a format for your word processor, try downloading another format for your machine-type -- odds are your word processor can read it.

• Microsoft Word for the Mac, Version 4
• Microsoft Word for the Mac, Version 5
• Microsoft Word for Windows, Version 2
• Microsoft Word for Windows, Version 6
• Word Perfect for DOS Version 5
• Word Perfect for Windows Version 5

I estimate that it will take about 1 hr 30 mins - to 2 hrs 30 mins to accomplish the data entry steps depending on how fast you are.

The outstanding strength of the pile sort task is the fact that it can accommodate a large number of items. We know of no other data collection method that will allow the collection of judged similarity data among over 100 items. This makes it the method of choice when large numbers are necessary. Other methods that might be used to collect similarity data, such as triads and paired comparison ratings, become impractical with a large number of items (p. 25).

For the remote and manual rating task, the brainstormed statements were listed in questionnaire form and each participant was asked to rate each statement on a 5-point Likert-type response scale in terms of how important the statement where 1=relatively unimportant (compared with the rest of the statements); 2=somewhat important; 3=moderately important; 4=very important, and, 5=extremely important. Because participants were unlikely to brainstorm statements that were totally unimportant with respect to accreditation, it was stressed that the rating should be considered a relative judgment of the importance of each item to all the other items brainstormed.

Data Analysis. Twenty-three participants had sorting and rating data that was used in the analysis. The concept mapping analysis begins with construction from the sort information of an NxN binary, symmetric matrix of similarities, X_ij. For any two items i and j, a 1 was placed in X_ij if the two items were placed in the same pile by the participant, otherwise a 0 was entered (Weller and Romney, 1988, p. 22). The total NxN similarity matrix, T_ij was obtained by summing across the individual X_ij matrices. Thus, any cell in this matrix could take integer values between 0 and {N, e.g., 11} (i.e., the number of people who sorted the statements); the value indicates the number of people who placed the i,j pair in the same pile.

The total similarity matrix T_ij was analyzed using nonmetric multidimensional scaling (MDS) analysis with a two-dimensional solution. The solution was limited to two dimensions because, as Kruskal and Wish (1978) point out:

Since it is generally easier to work with two-dimensional configurations than with those involving more dimensions, ease of use considerations are also important for decisions about dimensionality. For example, when an MDS configuration is desired primarily as the foundation on which to display clustering results, then a two-dimensional configuration is far more useful than one involving three or more dimensions (p. 58).

The usual statistic that is reported in MDS analyses to indicate the goodness of fit of the two-dimensional configuration to the original similarity matrix is called the Stress Value. A lower stress value indicates a better fit. In a study of the reliability of concept mapping, Trochim (1993) reported that the average Stress Value across 33 projects was .285 with a range from .155 to .352. The Stress Value in this analysis was .248.

The x,y configuration was the input for the hierarchical cluster analysis utilizing Ward's algorithm (Everitt, 1980) as the basis for defining a cluster. Using the MDS configuration as input to the cluster analysis in effect forces the cluster analysis to partition the MDS configuration into non-overlapping clusters in two-dimensional space. There is no simple mathematical criterion by which a final number of clusters can be selected. The procedure followed here was to examine an initial cluster solution that was the maximum thought desirable for interpretation in this context. Then, successively lower cluster solutions were examined, with a judgment made at each level about whether the merger seemed substantively reasonable. The pattern of judgments of the suitability of different cluster solutions was examined and resulted in acceptance of the eight cluster solution as the one that preserved the most detail and yielded substantively interpretable clusters of statements.

The MDS configuration of the statement points was graphed in two dimensions. This "point map" displays the location of all the brainstormed statements with statements closer to each other generally expected to be more similar in meaning. The point map is shown in Figure 1.

Figure 1. Point map of the accreditation statements.

The point map shows the brainstormed statements in relation to each other. Generally, statements that are closer to each other on the map were sorted together by more participants. For instance, you can see that statements 2, 4, and 18 are located on top of each other on the far left side of the map. These statements should be very similar in meaning. To see what these statements are, you can look them up in the listing of statements shown in Table 2.

A "cluster map" was also generated displaying the original statement points enclosed by polygon shaped boundaries for the eight clusters. This is shown in Figure 2.

Figure 2. Point and cluster map of accreditation statements.

The 1-to-5 rating data was averaged across persons for each item and each cluster. This rating information was depicted graphically in a "cluster rating map" that showed the cluster average rating using the third dimension as shown in Figure 3.

Figure 3. Cluster rating map showing the average importance rating for all clusters across all participants.

This map shows the labels that were selected for each of the eight clusters. In general, the labels were suggested by an analysis of the sort pile labels of all participants. This map no longer shows the statement points because all of the individual points tend to make the map harder to read. But the points are still there in each of the clusters. You can examine the statements by cluster listing in Table 3 to see which statements were grouped together by the analysis. One interesting feature of this map is the clear split between the three program-related clusters on the top and left and the other curriculum-related clusters on the right and bottom. This suggests that if we had to put all of the statements into two broad accreditation categories, it might be sensible to label them Program and Curriculum.

The map also shows the cluster layers indicating the average importance of each cluster for all of the participants. It was somewhat surprising that the cluster 'Field Experience/Practicum' was rated highest overall, and that the ' course-related clusters in the lower right were rated so low. One hypothesis is that this might be because there were so many students doing the importance ratings. This is discussed later in the pattern match of students versus non-students. The range of average ratings is only from 3.24 to 3.93 (on a 5-point scale). While this might not seem like much, it is important to remember that this range reflects averages of averages (a cluster average is the average of all of the statement averages in the cluster). Because of this, even though the standard errors of these distributions would be rather small, even slight differences could be significant (significance testing has not been done here because the primary interest was in exploring relative importance and in explicitly testing mean differences).

The ratings in Figure 3 are aggregated across all participants. However, it is reasonable to expect that different stakeholder subgroups would differ in their importance ratings. In concept mapping, this type of group comparison is examined using pattern matching. The pattern match is depicted graphically using a "ladder graph." The pattern match comparing students versus non-students is shown in Figure 4.

Figure 4. Pattern matching comparison of student versus non-student average importance ratings for the clusters.

The figure is called a ladder graph because, if there is a strong match (and the scales are the same) the lines would be nearly horizontal, looking a bit like a "ladder." This "ladder graph" shows the pattern match of importance ratings between students and non-students. There are several noteworthy agreements -- both groups see 'Field Experience/Practicum' as most important and 'Quantitative Approaches' as least important. There are also a few notable disconnections. Students see Program Philosophy as more important while non-students see 'Curriculum Philosophy' as more important. The correlation of .68 at the bottom of the graph shows the overall level of agreement between the two groups. It's worth noting that the hypothesis about the high importance rating for 'Field Experience/Practicum' being due to the large number of student raters is not corroborated in this figure.

The pattern match comparing those employed in an academic context with non-academics is given in Figure 5.

Figure 5. Pattern matching comparison of persons employed in an academic environment versus non-academics average importance ratings for the clusters.

Here we see the pattern match of importance ratings between persons whose primary job is in an academic context versus all others. Although there is a moderate level of consensus here, there are some notable disconnects. For instance, the academics place more importance of 'Faculty Qualifications' while the non-academics place greater emphasis on 'Program Context'.

Finally, the pattern match comparing those who have a Ph.D., with those who do not is shown in Figure 6.

Figure 6. Pattern matching comparison of persons with a Ph.D. versus non-Ph.D.’s average importance ratings for the clusters.

In this pattern match, there is a major disconnection on 'Faculty Qualifications' with Ph.D.s seeing them as very important while non-Ph.D.s don't. This is hardly surprising. The Ph.D.s would be more likely to know from experience how important qualified faculty can be to the overall quality of a program.

The maps also suggest a useful taxonomy for the AEA accreditation standards. The three clusters at the top of the map are related to program the evaluation training program context -- the faculty qualifications and program resources. The five clusters at the bottom are related to the curriculum and the student learning experience. This two-fold distinction and the sub-categories within it are a useful way to classify the standards that were generated.

The maps have already had a direct influence on the broader discussion within AEA regarding accreditation. The structure of the maps was used in organizing the standards section of the Draft AEA Accreditation Guidelines (Trochim and Riggins, 1996) presented to the AEA Board at their annual meeting in November, 1996. Furthermore, the individual standards statements were also included in that draft. It is expected that the Draft Guidelines will undergo a process of review and comment by the AEA membership during 1997. This review will help to revise these original standards, adding to them if needed, and should give a better indication of how generalizable the results are for AEA membership as a whole. As important as these substantive implications are, perhaps the more important conclusion is methodological in nature. This project was the first time that a facilitated concept mapping process was undertaken over the Internet. The experience pointed out some of the advantages and disadvantages of the Internet as a platform for conducting research. The most significant problem is associated with sampling. Clearly, it is difficult to assure a valid sample when eliciting volunteer participants over the Internet. Many people who are in the target population do not have access to this technology or, if they do, were not participating in the listserve and consequently never even heard about this project. Even assuming that people heard, they needed to have a fairly robust set of computing skills and equipment and a fairly high degree of motivation to be able to participate effectively.

In spite of this limitation for this study, the fact that the project was completed at all demonstrates that the Internet can be used as a research platform for studies of this kind. It is easily possible to envision settings where the major sampling and access problems would not be a liability. For instance, in organizations that are using intranets and have a fairly "captive" population, it is likely that a target sample could be achieved with high quality. This study indicates that a corporation or non-profit organization could use their internal intranet or the broader Internet to accomplish a concept mapping entirely through that technology. In such a context, it would be reasonable to expect that people might either be required to participate or would be more highly motivated. In addition, there would more likely be an organizationally-determined level of access to and experience in using the Internet that would assure that participants are capable of accomplishing the work. The technology could be used to enable participants to come together world-wide without the expense of traveling to a common site. And, participants are able to work on the project on their own time because this approach does not require that they all be "online" simultaneously.

This project provided a starting point for the formulation of standards for accrediting graduate-level specializations in evaluation. At almost no cost, it was possible to involve participants from all over the world to map a broad set of accreditation standards and explore consensus about their relative importance. The standards generated here are already being used as the foundation for accreditation guidelines that will be revised over the next year. And, the project demonstrates the feasibility of using the Internet as a platform for conducting a facilitated group-based concept mapping project on a world-wide basis.

Caracelli, V. (1989). Structured conceptualization: A framework for interpreting evaluation results. Evaluation and Program Planning. 12, 1, 45-52.

Commission on Recognition of Postsecondary Accreditation (1996). Directory of Recognized Agencies and Supporters of Accreditation. Washington, D.C.

Cook, J. (1992). Modeling staff perceptions of a mobile job support program for persons with severe mental illness. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Cooksy, L. (1989). In the eye of the beholder: Relational and hierarchical structures in conceptualization. Evaluation and Program Planning. 12, 1, 59-66.

Davis, J. (1989). Construct validity in measurement: A pattern matching approach. Evaluation and Program Planning. 12, 1, 31-36.

Davison, M.L. (1983). Multidimensional scaling. New York, John Wiley and Sons.

Dumont, J. (1989). Validity of multidimensional scaling in the context of structured conceptualization. Evaluation and Program Planning. 12, 1, 81-86.

Everitt, B. (1980). Cluster Analysis. 2nd Edition, New York, NY: Halsted Press, A Division of John Wiley and Sons.

Everitt, B. (1980). Cluster Analysis. 2nd Edition, New York, NY: Halsted Press, A Division of John Wiley and Sons.

Galvin, P.F. (1989). Concept mapping for planning and evaluation of a Big Brother/Big Sister program. Evaluation and Program Planning. 12, 1, 53-58.

Grayson, T.E. (1992). Practical issues in implementing and utilizing concept mapping. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Grayson, T.E. (1993). Empowering key stakeholders in the strategic planning and development of an alternative school program for youth at risk of school behavior. Paper presented at the Annual Conference of the American Evaluation Association, Dallas, TX.

Gurowitz, W.D., Trochim, W. and Kramer, H. (1988). A process for planning. The Journal of the National Association of Student Personnel Administrators, 25, 4, 226-235.

Kane, T.J. (1992). Using concept mapping to identify provider and consumer issues regarding housing for persons with severe mental illness. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Keith, D. (1989). Refining concept maps: Methodological issues and an example. Evaluation and Program Planning. 12, 1, 75-80.

Kohler, P.D. (1992). Services to students with disabilities in postsecondary education settings: Identifying program outcomes. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Kohler, P.D. (1993). Serving students with disabilities in postsecondary education settings: Using program outcomes for planning, evaluation and empowerment. Paper presented at the Annual Conference of the American Evaluation Association, Dallas, TX.

Kruskal, J.B. and Wish, M. (1978). Multidimensional Scaling. Beverly Hills, CA: Sage Publications.

Kruskal, J.B. and Wish, M. (1978). Multidimensional Scaling. Beverly Hills, CA: Sage Publications.

Lassegard, E. (1992). Assessing the reliability of the concept mapping process. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Lassegard, E. (1993). Conceptualization of consumer needs for mental health services. Paper presented at the Annual Conference of the American Evaluation Association, Dallas, TX.

Linton, R. (1989). Conceptualizing feminism: Clarifying social science concepts. Evaluation and Program Planning. 12, 1, 25-30.

Mannes, M. (1989). Using concept mapping for planning the implementation of a social technology. Evaluation and Program Planning. 12, 1, 67-74.

Marquart, J.M. (1988). A pattern matching approach to link program theory and evaluation data: The case of employer-sponsored child care. Unpublished doctoral dissertation, Cornell University, Ithaca, New York.

Marquart, J.M. (1989). A pattern matching approach to assess the construct validity of an evaluation instrument. Evaluation and Program Planning. 12, 1, 37-44.

Marquart, J.M. (1992). Developing quality in mental health services: Perspectives of administrators, clinicians, and consumers. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Marquart, J.M., Pollak, L. and Bickman, L. (1993). Quality in intake assessment and case management: Perspectives of administrators, clinicians and consumers. In R. Friedman et al. (Eds.), A system of care for children's mental health: Organizing the research base. Tampa: Florida Mental Health Institute, University of South Florida.

McLinden, D. J. & Trochim, W.M.K. (In Press). From Puzzles to Problems: Assessing the Impact of Education in a Business Context with Concept Mapping and Pattern Matching. In J. Phillips (Ed.), Return on investment in human resource development: Cases on the economic benefits of HRD - Volume 2. Alexandria, VA: American Society for Training and Development.

Mead, J.P. and Bowers, T.J. (1992). Using concept mapping in formative evaluations. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Mercer, M.L. (1992). Brainstorming issues in the concept mapping process. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Nunnally, J.C. (1978). Psychometric Theory. (2nd. Ed.). New York, McGraw Hill.

Osborn, A.F. (1948). Your Creative Power. New York, NY: Charles Scribner.

Penney, N.E. (1992). Mapping the conceptual domain of provider and consumer expectations of inpatient mental health treatment: New York Results. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Romney, A.K., Weller, S.C. and Batchelder, W.H. (1986). Culture as consensus: A theory of culture and informant accuracy. American Anthropologist, 88, 2, 313-338.

Rosenberg, S. and Kim, M.P. (1975). The method of sorting as a data gathering procedure in multivariate research. Multivariate Behavioral Research, 10, 489-502.

Rosenberg, S. and Kim, M.P. (1975). The method of sorting as a data gathering procedure in multivariate research. Multivariate Behavioral Research, 10, 489-502.

Ryan, L. and Pursley, L. (1992). Using concept mapping to compare organizational visions of multiple stakeholders. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

SenGupta, S. (1993). A mixed-method design for practical purposes: Combination of questionnaire(s), interviews, and concept mapping. Paper presented at the Annual Conference of the American Evaluation Association, Dallas, TX.

Shern, D.L. (1992). Documenting the adaptation of rehabilitation technology to a core urban, homeless population with psychiatric disabilities: A concept mapping approach. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Shern, D.L., Trochim, W. and LaComb, C.A. (1995). The use of concept mapping for assessing fidelity of model transfer: An example from psychiatric rehabilitation. Evaluation and Program Planning, 18, 2.

Trochim, W. (1985). Pattern matching, validity, and conceptualization in program evaluation. Evaluation Review, 9, 5, 575-604.

Trochim, W. (1989a). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1, 1-16.

Trochim, W. (1989b). Concept mapping: Soft science or hard art? Evaluation and Program Planning, 12, 1, 87-110.

Trochim, W. (1989c). Outcome pattern matching and program theory. Evaluation and Program Planning, 12, 4, 355-366.

Trochim, W. (1990). Pattern matching and program theory. In H.C. Chen (Ed.), Theory-Driven Evaluation. New Directions for Program Evaluation, San Francisco, CA: Jossey-Bass.

Trochim, W. (1993). Reliability of Concept Mapping. Paper presented at the Annual Conference of the American Evaluation Association, Dallas, Texas, November.

Trochim, W. and Cook, J. (1992). Pattern matching in theory-driven evaluation: A field example from psychiatric rehabilitation. in H. Chen and P.H. Rossi (Eds.) Using Theory to Improve Program and Policy Evaluations. Greenwood Press, New York, 49-69.

Trochim, W., Cook, J. and Setze, R. (1994). Using concept mapping to develop a conceptual framework of staff's views of a supported employment program for persons with severe mental illness. Consulting and Clinical Psychology, 62, 4, 766-775.

Trochim, W. and Linton, R. (1986). Conceptualization for evaluation and planning. Evaluation and Program Planning, 9, 289-308.

Trochim, W. and Riggins, L. (1996). AEA Accreditation Guidelines. Draft guidelines presented to the American Evaluation Association Board of Directors, Annual Meeting, November, 1996.

Valentine, K. (1989). Contributions to the theory of care. Evaluation and Program Planning. 12, 1, 17-24.

Valentine, K. (1992). Mapping the conceptual domain of provider and consumer expectations of inpatient mental health treatment: Wisconsin results. Paper presented at the Annual Conference of the American Evaluation Association, Seattle, WA.

Weller S.C. and Romney, A.K. (1988). Systematic Data Collection. Newbury Park, CA, Sage Publications.

Weller, S.C. and Romney, A.K. (1988). Systematic Data Collection. Sage Publications, Newbury Park, CA.

Witkin, B. and Trochim, W. (1996). A concept map analysis of the construct of listening. Paper presented at the annual conference of the International Listening Association.