Ecological Validity: Difference between revisions

Latest revision as of 13:54, 22 July 2026

Dr Peter Wyman, in interview with Tim McGowan, describes the key finding that Ecological Validity is critical to preventative mental health care interventions.

Ecological validity means an intervention has to work the way the real group actually works, not the way a tidy protocol assumes it should. A training exercise can be well designed in principle and still fail once it meets an intact team, because the team already has its own habits, hierarchy, and culture that the exercise never accounted for.

This is not just a design preference; it is a documented failure mode. Discussing an unrelated military skills programme, Wyman et al. (2020) note that "investigators of the Royal Air Force study noted that cognitive training may lack ecological validity without addressing unit functioning" (Introduction). In plain terms, teaching someone a skill in isolation does not help if the group around them still runs on the old norms; the skill has nowhere to land.

Wingman-Connect was built specifically to avoid that failure, and its own construction shows what ecological validity looks like in practice. Rather than importing an existing programme unchanged, the developers started from one that worked elsewhere and deliberately reshaped it: "Wingman-Connect was developed... by adapting a suicide prevention program used in public education settings (Sources of Strength)," and, critically, "group training for units was selected to align with Air Force culture, a departure from the Sources of Strength model of training selected opinion leaders" (Wyman et al., 2020, Methods). The original school-based model worked by training influential student leaders; the military version abandoned that structure entirely, because Air Force culture does not organise itself around informal student opinion leaders the way a school does. Ecological validity, here, meant changing the delivery mechanism itself to match the population, not just translating the language of the materials.

The trial's own conclusions treat this redesign as central to why it worked, not incidental to it. The authors state plainly that "group training that builds cohesive, healthy military units is promising for upstream suicide prevention and may be essential for ecological validity" (Wyman et al., 2020, Abstract), and in their discussion they go further: "an innovation of Wingman-Connect was training in natural organizational units," and this approach "may be critical for ecological validity of military suicide prevention programs... and for transfer of skills into the organizational culture" (Wyman et al., 2020, Discussion). The claim being made is specific: it is training delivered through the unit's own existing structure, rather than through an imported curriculum layered on top of it, that allows the skills learned in training to actually persist once training ends.

This case is instructive precisely because it is a single, well-documented example rather than a broad literature. Wingman-Connect demonstrates what ecological validity requires, genuine structural adaptation to a population's real social organisation, but the evidence for this principle again comes from one research group's work with military populations, and has not been tested as a general design rule across other settings.

Context: Ecological Validity in Schools

Schools are where the ecologically valid design that Wingman-Connect later adapted originated. Wyman (2014) frames schools as "normative social systems... settings for universal interventions" that "serve the broadest populations" (p. S252), precisely because a school already has its own dense peer structure that an intervention can work through rather than compete with. The Sources of Strength model, which Wingman-Connect departed from for military culture, was itself built around this fit: training "high school student peer leaders to prepare them to modify norms through their natural social networks" (Wyman, 2014, p. S254). The lesson Wingman-Connect's redesign makes explicit in hindsight is that this peer-leader structure was not incidental; it was the specific feature that made the school version ecologically valid, and any future adaptation of it should treat that structure as load-bearing, not decorative.

Context: Ecological Validity among Remote Workers in IT and Technology

No study has tested whether a network-cohesion intervention like Wingman-Connect retains ecological validity when delivered to a distributed software team, so this section can only flag the risk, not resolve it. The same caution Wyman et al. (2020) raise about training that "may lack ecological validity without addressing unit functioning" applies directly here, because distributed teams function differently from colocated ones in ways that are well documented. Herbsleb and Mockus (2003) found that developers separated across sites report markedly less "teamness," a gap tied to how "distance interfered with the usual stages by which individuals become coherent groups or teams" (Discussion). A cohesion-building programme designed for an in-person team and delivered unchanged to a distributed one would risk exactly this failure; it has not been tested either way.

References

Herbsleb, J. D., & Mockus, A. (2003). An empirical study of speed and communication in globally distributed software development. IEEE Transactions on Software Engineering, 29(6), 481–494. https://doi.org/10.1109/TSE.2003.1205177

Wyman, P. A. (2014). Developmental approach to prevent adolescent suicides: Research pathways to effective upstream preventive interventions. American Journal of Preventive Medicine, 47(3, Suppl. 2), S251–S256. https://doi.org/10.1016/j.amepre.2014.05.039

Wyman, P. A., Pisani, A. R., Brown, C. H., Yates, B., Morgan-DeVelder, L., Schmeelk-Cone, K., Gibbons, R. D., Caine, E. D., Petrova, M., Neal-Walden, T., Linkh, D. J., Matteson, A., Simonson, J., & Pflanz, S. E. (2020). Effect of the Wingman-Connect upstream suicide prevention program for Air Force personnel in training: A cluster randomized clinical trial. JAMA Network Open, 3(11), e2022532. https://doi.org/10.1001/jamanetworkopen.2020.22532

@@ Line 1: / Line 1: @@
-=== Summary ===
-=== Notes and References ===
+[[File:Ecological Validity.mp4|thumb|Dr Peter Wyman, in interview with Tim McGowan, describes the key finding that Ecological Validity is critical to preventative mental health care interventions.]]
+Ecological validity means an intervention has to work the way the real group actually works, not the way a tidy protocol assumes it should. A training exercise can be well designed in principle and still fail once it meets an intact team, because the team already has its own habits, hierarchy, and culture that the exercise never accounted for.
+This is not just a design preference; it is a documented failure mode. Discussing an unrelated military skills programme, Wyman et al. (2020) note that "investigators of the Royal Air Force study noted that cognitive training may lack ecological validity without addressing unit functioning" (Introduction). In plain terms, teaching someone a skill in isolation does not help if the group around them still runs on the old norms; the skill has nowhere to land.
-'''''Effectiveness Trial of Wingman-Connect Implemented Across Career Phases'''''
+Wingman-Connect was built specifically to avoid that failure, and its own construction shows what ecological validity looks like in practice. Rather than importing an existing programme unchanged, the developers started from one that worked elsewhere and deliberately reshaped it: "Wingman-Connect was developed... by adapting a suicide prevention program used in public education settings (Sources of Strength)," and, critically, "group training for units was selected to align with Air Force culture, a departure from the Sources of Strength model of training selected opinion leaders" (Wyman et al., 2020, Methods). The original school-based model worked by training influential student leaders; the military version abandoned that structure entirely, because Air Force culture does not organise itself around informal student opinion leaders the way a school does. Ecological validity, here, meant changing the delivery mechanism itself to match the population, not just translating the language of the materials.
-''Submitted to the National Institute of Mental Health, PAR21-130, 10/13/2022''
+The trial's own conclusions treat this redesign as central to why it worked, not incidental to it. The authors state plainly that "group training that builds cohesive, healthy military units is promising for upstream suicide prevention and may be essential for ecological validity" (Wyman et al., 2020, Abstract), and in their discussion they go further: "an innovation of Wingman-Connect was training in natural organizational units," and this approach "may be critical for ecological validity of military suicide prevention programs... and for transfer of skills into the organizational culture" (Wyman et al., 2020, Discussion). The claim being made is specific: it is training delivered through the unit's own existing structure, rather than through an imported curriculum layered on top of it, that allows the skills learned in training to actually persist once training ends.
-''Group training that builds cohesive, healthy military units is promising for upstream suicide prevention and may be essential for ecological validity''
+This case is instructive precisely because it is a single, well-documented example rather than a broad literature. Wingman-Connect demonstrates what ecological validity requires, genuine structural adaptation to a population's real social organisation, but the evidence for this principle again comes from one research group's work with military populations, and has not been tested as a general design rule across other settings.
-'''''Data-informed implementation preparation.'''''
+=== '''Context: Ecological Validity in Schools''' ===
+Schools are where the ecologically valid design that Wingman-Connect later adapted originated. Wyman (2014) frames schools as "normative social systems... settings for universal interventions" that "serve the broadest populations" (p. S252), precisely because a school already has its own dense peer structure that an intervention can work through rather than compete with. The Sources of Strength model, which Wingman-Connect departed from for military culture, was itself built around this fit: training "high school student peer leaders to prepare them to modify norms through their natural social networks" (Wyman, 2014, p. S254). The lesson Wingman-Connect's redesign makes explicit in hindsight is that this peer-leader structure was not incidental; it was the specific feature that made the school version ecologically valid, and any future adaptation of it should treat that structure as load-bearing, not decorative.
-''A “voltage drop” and poor sustainment are common as interventions transition from controlled efficacy trials to regular use.''
+=== '''Context: Ecological Validity among Remote Workers in IT and Technology''' ===
+No study has tested whether a network-cohesion intervention like Wingman-Connect retains ecological validity when delivered to a distributed software team, so this section can only flag the risk, not resolve it. The same caution Wyman et al. (2020) raise about training that "may lack ecological validity without addressing unit functioning" applies directly here, because distributed teams function differently from colocated ones in ways that are well documented. Herbsleb and Mockus (2003) found that developers separated across sites report markedly less "teamness," a gap tied to how "distance interfered with the usual stages by which individuals become coherent groups or teams" (Discussion). A cohesion-building programme designed for an in-person team and delivered unchanged to a distributed one would risk exactly this failure; it has not been tested either way.
-''Conversely, scaled out interventions are more impactful with careful preparation using implementation science frameworks.''
+=== '''References''' ===
+Herbsleb, J. D., & Mockus, A. (2003). An empirical study of speed and communication in globally distributed software development. ''IEEE Transactions on Software Engineering, 29''(6), 481–494. <nowiki>https://doi.org/10.1109/TSE.2003.1205177</nowiki>
-''“The concept of ecological validity has become familiar to psychologists. It reminds them that the artificial situation created for an experiment may differ from the everyday world in crucial ways. When this is so, the results may be irrelevant to the phenomena that one would really like to explain.”''
+Wyman, P. A. (2014). Developmental approach to prevent adolescent suicides: Research pathways to effective upstream preventive interventions. ''American Journal of Preventive Medicine, 47''(3, Suppl. 2), S251–S256. <nowiki>https://doi.org/10.1016/j.amepre.2014.05.039</nowiki>
-''To bridge the gap between lab and life, many researchers have called for experiments with more ‘ecological validity’ to ensure that experiments more closely resemble and generalize to the ‘real-world.’''
+Wyman, P. A., Pisani, A. R., Brown, C. H., Yates, B., Morgan-DeVelder, L., Schmeelk-Cone, K., Gibbons, R. D., Caine, E. D., Petrova, M., Neal-Walden, T., Linkh, D. J., Matteson, A., Simonson, J., & Pflanz, S. E. (2020). Effect of the Wingman-Connect upstream suicide prevention program for Air Force personnel in training: A cluster randomized clinical trial. ''JAMA Network Open, 3''(11), e2022532. <nowiki>https://doi.org/10.1001/jamanetworkopen.2020.22532</nowiki>
-''Holleman GA, Hooge ITC, Kemner C, Hessels RS. The 'Real-World Approach' and Its Problems: A Critique of the Term Ecological Validity. Front Psychol. 2020 Apr 30;11:721. doi: 10.3389/fpsyg.2020.00721. PMID: 32425850; PMCID: PMC7204431.''
-''The main intention for cultural adaptation frameworks (EVM) is to increase the cultural acceptability and effectiveness of the psychological treatment. This is accomplished by making changes that align with the culture of the beneficiary population, while maintaining the components of the evidence-based research that supports the treatment''
-''The EVM was selected because it is based on the view that individuals must be understood within their cultural, social, and political environment. The EVM framework serves to ‘culturally center’ an intervention through eight dimensions that must be incorporated for an intervention to have ecological validity and be embedded within the cultural context (Bernal, 2003). These dimensions include language, persons, metaphors, content, concepts, goals, methods, and context (Table 1 in online Supplementary material).''
-''Sangraula M, Kohrt BA, Ghimire R, Shrestha P, Luitel NP, Van't Hof E, Dawson K, Jordans MJD. Development of the mental health cultural adaptation and contextualization for implementation (mhCACI) procedure: a systematic framework to prepare evidence-based psychological interventions for scaling. Glob Ment Health (Camb). 2021 Feb 19;8:e6. doi: 10.1017/gmh.2021.5. PMID: 33996110; PMCID: PMC8082944.''
-'''''NIMH'''''
-'''''Data-informed implementation preparation.'''''
-''A “voltage drop” and poor sustainment are common as interventions transition from controlled efficacy trials to regular use. Conversely, scaled out interventions are more impactful with careful preparation using implementation science frameworks.''
-'''''JAMA Open Network, October, 2020'''''
-''group training design ecologically valid and appealing for participants. Wingman-Connect development benefited from strong organizational collaboration, resulting in an ecologically valid program.''
-'''''Evidence-based Methodology:''''' ''A commitment to follow the evidence of impact. Networks of Support’s structure and key elements strengthen relationship networks, cohesion, adaptive coping, and group norms. Group-based, interactive training uses research-informed strategies.''
-'''''JAMA Open Network, October, 2020'''''
-'''''Research Outcomes – Evidence Based Key Findings include:'''''
-* ''Increased class cohesion, morale, positive bonds, and acceptability of help seeking''
-* ''Reduction in suicide risks scores, depression, and reactivity to anger''
-* ''Nearly 50% reduction in occupational impairment (corrective training, negative counseling)''
-'''''Measurable Impact on both Protective Factors and Risk Factors'''''
-''Wingman-Connect was tested using the '''gold-standard research design with Airmen in Training''' from 2017-2019 (randomized controlled trial).  Training showed statistically significant increases in group and individual level protective factors, as well as reductions in negative outcomes including and depression and suicide risk.  ''
-''Class exercises create more cohesive units with skills extended into group culture.''' ''' Wingman Connect training is delivered to organizational units using interactive exercises to build key group and individual protective factors.''
-''Wingman-Connect impact promoting more cohesive, healthy training classes was a mechanism that reduced risk for suicidal thoughts and behaviors. More cohesive, healthy classes ‘lifted up’ individuals who were vulnerable to fitness and mental health problems. Study findings suggest that group training for organizational units may be critical for ‘transfer’ of training concepts and skills into the organization’s culture.''
-'''''Social Science & Medicine'''                     Social Science & Medicine 296 (2022) 114737''
-''The current attention on the detection and treatment of suicidal military members is necessary, but ultimately insufficient to meet the needs of many suicidal service members. Results from our study suggest that the status quo progression for many service members at elevated suicide risk is one of worsening disconnection, evidenced by the trajectory of declining valued connections found in the active control condition.''
-''Program achieved the objective of strengthening the network structure of unit relationships over time, including socially integrating those currently at elevated risk for suicide. This is an especially important hypothesis since W-CP training has no content encouraging Airmen to form connections with at-risk peers.''
-'''''NIMH'''''
-''Effective upstream suicide prevention programs that can be implemented in real-world settings are urgently needed to address the significant and growing problem of suicides.''
-''No RCT-validated universal programs shown to reduce vulnerability to suicide are in wide use. To fill this gap, we developed the Wingman-Connect Program–group-based prevention approach that strengthens protective relationship networks and skills for managing challenge.   This strategy of reducing suicide risk could be highly impactful as a complement to the predominant approach of trying to mitigate risk only after suicidal individuals are identified.''
-''Wyman has led research (NIH, CDC funded) testing that program, which disseminates skills for social health through youth peer networks.<sup> </sup> The high-energy, interactive training improves student connectedness and coping norms, and protective effects spread school-wide including adult help for suicidal youth.<sup> </sup> A study aggregating three RCTs (N=78 schools; 39,900 students) showed fewer suicide deaths in schools implementing this approach.'' <sup> </sup>
-=== Automatically generated stuff from ChatGPT to look over ===
-In the context of preventative mental health care, "ecological validity" refers to the extent to which the findings and interventions of mental health research or practices are applicable or relevant to real-world settings. This concept emphasizes the importance of ensuring that research studies, assessment tools, and interventions are reflective of the conditions, challenges, and dynamics that individuals encounter in their everyday lives. Ecological validity is crucial for the effective translation of theoretical knowledge and clinical insights into practical strategies that can be implemented in various environments, such as schools, workplaces, communities, and healthcare settings.
-For preventative mental health care, ecological validity ensures that:
-# Assessment Tools are relevant and applicable to the diverse situations and contexts in which individuals might experience mental health challenges. This means that tools should accurately reflect the range of symptoms and behaviors as they occur in natural settings, outside of the controlled conditions of a clinical or research environment.
-# Interventions are designed to be feasible and effective in the real world. For example, an intervention that promotes mental well-being through regular physical activity should consider factors like access to safe outdoor spaces, availability of time, and cultural attitudes towards exercise.
-# Research Findings can be generalized or applied to the target population in their everyday environments. Studies that have high ecological validity provide more useful and actionable insights for public health policies, school-based programs, workplace wellness initiatives, and community mental health services.
-# Cultural and Contextual Relevance is maintained, acknowledging that the effectiveness of preventative strategies may vary based on cultural norms, societal structures, and environmental factors. Interventions with high ecological validity respect and incorporate these differences to ensure relevance and effectiveness across diverse populations.
-Improving the ecological validity of preventative mental health care initiatives involves ongoing collaboration between researchers, practitioners, community members, and policy-makers. It also requires a commitment to participatory research methods, where interventions are co-developed with those who are meant to benefit from them, and continuous evaluation and adaptation based on real-world feedback and outcomes. By prioritizing ecological validity, preventative mental health care can offer more meaningful, accessible, and sustainable support to individuals and communities.