Psychometric properties of the University of California Los Angeles Loneliness Scale version 3: the European Portuguese version

Ansiedad y Estrés, (2023), 29(1), 18-26
https://doi.org/10.5093/anyes2023a3
https://www.ansiedadyestres.es
Bibliography reference

Psychometric properties of the University of California Los Angeles Loneliness Scale version 3: the European Portuguese version

INFO ARTICLE

---

Received 24 December 2021
Accepted 3 October 2022
Available online 2 November 2022

 

ABSTRACT

---

Currently, the golden standard measure to assess loneliness is the University of California, Los Angeles Loneliness scale version 3 (UCLA v3). Objective. The aim of this study was to analyze the psychometric properties of UCLA v3 for the European Portuguese population. Method. A sample of 282 participants was surveyed in Portugal. Results. Confirmatory factor analysis (CFA) supported a model portraying a global loneliness bifactor solution for positive and negative wording items, which achieved optimal fitness. Multi-group CFA indicates scalar and metric invariance across gender. Loneliness test scores (global score, positive items and negative items) correlated with well-established mental health indicators such as psychological stress, depressive and anxiety symptomatology, or psychological inflexibility. Internal consistency of the loneliness test scores was optimal for the global measure (a = .91; ? = .91) as well as for the positive (a = .87; ? = .87) and the negative factors (a = .86; ? = .88). Conclusions. Results support UCLA v3 as a reliable and valid measure of loneliness for future research studies interested in examining the prevalence of loneliness and impact in health in the context of Covid-19 in the Portuguese population and as a health indicator in health promotion and clinical interventions.

 

KEYWORDS

---

UCLA version 3
Loneliness
Portuguese population
Psychometric properties

Ansiedad y Estrés, (2023), 29(1), 18-26
https://doi.org/10.5093/anyes2023a3
https://www.ansiedadyestres.es
Bibliography reference

RESUMEN

---

Actualmente, la medida estándar mayormente utilizada para evaluar la soledad es la versión 3 de la escala de Soledad de la Universidad de California de Los Angeles (UCLA v3). Objetivo. Analizar las propiedades psicométricas de UCLA v3 para la población portuguesa europea. Método. Una muestra total de 282 personas fue encuestada en línea en Portugal. Resultados. El análisis factorial confirmatorio (AFC) respaldó un modelo que representa una solución global de soledad con dos factores relativos al contenido positivo y negativo de los ítems, y logrando un ajuste óptimo. El AFC multigrupo indicó invariancia métrica y escalar según género. Los puntajes de la prueba de soledad (puntaje global; ítems positivos e ítems negativos) correlacionaron con indicadores de salud bien establecidos como el estrés psicológico, la sintomatología depresiva y ansiosa o la inflexibilidad psicológica. La consistencia interna de los puntajes de la escala de soledad fue óptima tanto para la medida global (a = .91; ? = .91) como para las medidas positiva (a = .87; ? = .87) y negativa (a = .86; ? = .88). Conclusiones. Los resultados apoyan a la UCLA v3 como una medida confiable y válida de la soledad para futuros estudios de investigación interesados en examinar la prevalencia de la soledad y su impacto en la salud en el contexto de Covid-19 en la población portuguesa, y como un indicador de salud en programas de promoción de salud y de intervenciones clínicas.

 

PALABRAS CLAVE

---

UCLA versión 3
Población portuguesa
Propiedades psicométricas
 

Introduction

---

Loneliness refers to the perception of mismatch between the quantity and especially the quality of meaningful relationships an individual has and the relationships one desires. Loneliness or perceived social isolation is distinctive from being physically alone (Cacioppo & Williams, 2008). Moreover, the measures implemented to cope with Covid-19, such as mass home-confinements directives, including stay-at-home orders, quarantine, and isolation, should raise further concern about loneliness (Horesh et al., 2020; Beutel et al., 2017).

Although loneliness is often trivialized, evidence indicates that loneliness increases the risk for premature mortality up to 26%, an effect comparable with well-established risk factors for mortality such as tobacco, lack of physical activity or obesity (Holt-Lunstad et al., 2015). This adverse effect of loneliness on health is supposed to be mediated by the underlying stress response since, as social beings, loneliness involves the perception of lack of control to meet desired and meaningful social connections (Cacioppo & Cacioppo, 2012). Furthermore, recent studies suggest that loneliness may induce alterations in the stress response that may result in the downregulation of anti-inflammatory genes and upregulation of proinflammatory genes, increasing the risk of infectious and inflammatory diseases, similarly to other chronic exposures to adversity (Spithoven et al., 2019).

Processes involved in emotional regulation play an important role in loneliness (O'Day et al., 2019). A growing number of empirical studies focus on psychological (in)flexibility, a set of processes associated with being rigidly engaged with negative thoughts and unable to guide ones' behavior by direct contingencies or personal values (Wojnarowska et al., 2020). Moreover, the set of processes involved in psychological (in)flexibility are transdiagnostic in nature and are relevant across a wide range of mental health problems, including, but not limited, to loneliness (Wojnarowska et al., 2020; Levin et al., 2014).

Currently, the University of California, Los Angeles Loneliness Scale (UCLA) is the gold standard to measure loneliness worldwide (for a full overview see Russell, 1996). The original version (Russell et al., 1978) was found to be biased because its 20 items were worded in a negative format and included loneliness related words (Russell, 1996). Also, this scale showed poor validity evidence of the intended interpretation of test scores due to its high correlations with depression and self-esteem constructs (Russell, 1996). The second version fixed most of the previous problems and showed acceptable validity of the intended interpretation of test scores (Russell et al., 1980). However, some items were still worded in double-negative, and factor analysis for ulterior adaptations reported inconsistent internal structure (one, two, three, and four-factor solutions) (Zarei et al., 2016). The latest UCLA version 3 (Russell, 1996) offers a single bipolar loneliness factor solution along with two independent factors (one for positive items or non-loneliness items, and another for negative items or loneliness items), optimal validity of the intended interpretation of test scores, and good internal consistency reliability of test scores across different samples (Russell, 1996).

However, to our knowledge, the psychometric properties of the latest UCLA version 3 in the Portugal population have not been tested yet, which makes it more difficult to conduct reliable and valid epidemiological studies about loneliness prevalence, as well as measure the impact of Covid-19 in Portugal (Brooks et al., 2020) and compare these data with other countries.

The aim of this study was to analyze the psychometric properties of the UCLA version 3 for the European Portuguese language. First, validity of the intended interpretation of test scores was assessed examining internal structure (factor structure and measurement invariance across gender) and relation with other variables. It was expected that higher levels of loneliness would be associated with higher levels of perceived stress (Hackett et al., 2012), depression and anxiety symptoms (Cacioppo et al., 2006; Keller et al., 2022), and psychological inflexibility (Hernández-López et al., 2021). Second, internal consistency reliability of test scores and floor and ceiling effects were also assessed.

Methods

---

Participants

A total of 282 participants completed an online survey using the lime survey UC system. Inclusion criteria comprised age (> 18 years old) and being a European Portuguese native speaker.

The study was approved by the Ethical Committee of the Faculty of Psychology and Educational Sciences of the University of Coimbra and followed the Helsinki Declaration by the World Medical Association (Helsinki, 1964), Ethical Principles of the Chapter of Fundamental Rights of the EU, European Code of Conduct for Research Integrity, the General Data Protection Regulation (GDPR).

Measures

A European Portuguese version of the original the UCLA version 3 (Russell, 1996) was administered. This is a 20-item scale to assess loneliness through a 4-point Likert-type scale from 1 ("never") to 4 ("often"). Items 1, 5, 6, 9, 10, 15, 16, 19 and 20 are reversely scored. Scores range from 20 to 80. Higher scores indicate higher loneliness or perceived social isolation. To check the European Portuguese version, see Appendix I.

This European Portuguese version was developed by the authors as follows. First, permission was granted by the author to develop and adapt his original version of the UCLA version 3 into European Portuguese. The language translation from English to European Portuguese was carried out following the guidelines proposed by the International Test Commission for the translation and adaptation of psychological and educational tests (International Test Commission, 2017) and WHO (WHO, 2010): first, a forward translation was done by a bilingual professional, and then a back-translation was conducted independently by a panel of three member experts. Disagreements were solved upon throughout discussions to reach the final wording.

A small-scale preliminary pilot study was conducted to evaluate feasibility, duration and explore minor adjustments needed to the Portuguese version of the UCLA v3. A sample of 7 participants (4 women and 3 men), whose age ranged from 18 to 51 years old (Mdn = 20.7) with secondary (1) and professional (6) levels of education, provided informed consent to be part of the pilot study. Results evidenced that minor changes in the wording of the items were needed to improve compressibility across different ages, gender, and socioeconomic levels in Portugal, before conducting the final version described in this paper. Consequently, item 3 ("How often do you feel that there is no one you can turn to?") was reworded using a more familiar expression for "you can turn to" and item 12 ("How often do you feel that your relationships with others are not meaningful"?) was added a brief explanation of the meaning of the word "meaningful". This process followed the guidelines of the American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME) (AERA et al., 2014).

The Perceived Stress Scale (PSS-10) validated for the Portuguese population (Trigo et al., 2010) is a 10-item scale that measures perception of stress with a 5-point Likert-type scale ranging from 0 ("never") to 4 ("frequently"). Items 4, 5, 7 and 8 are reversely scored. Its scores range from 0 to 40. Higher scores indicate higher levels of psychological stress. For this study, the PSS-10 test scores showed Cronbach's a = .68.

The Patient Health Questionnaire of Depression and Anxiety (PHQ-4) is a 4-item questionnaire to measure depression and anxiety associated with functional impairment and disability days (Löwe et al., 2010). It utilizes a 4-point Likert-type scale ranging from 0 ("Not at all") to 4 ("Nearly every day"). Its scores ranged from 0 to 12. Higher scores indicate higher levels of depressive and anxiety symptomatology. For this study, the PHQ-4 test scores showed Cronbach's a = .87.

The Acceptance and Action Questionnaire-II (AAQ-II), validated for the Portuguese population (Pinto-Gouveia et al., 2012), is a 7-item questionnaire that measures psychological inflexibility with a 7-point Likert-type ranging from 1 ("never true") to 7 ("always true"). Scores rank between 7 and 49. Higher scores indicate the need for control and avoiding aversive thoughts and feelings. For this study, the AAQ-II test scores showed Cronbach's a = .88.

Design and procedure

A descriptive cross-sectional study was conducted, and standards of current psychometrics were followed (AERA et al., 2014; Hernandez et al., 2020).

Regarding data collection, participants were invited via email and social networks providing a link to the online 4-step survey, including sociodemographic and self-reported clinical information. All participants provided informed consent. Data were collected for 3 months (October-December 2019). Participation was confidential and anonymous.

Data analysis

The IBM Statistical Package for the Social Sciences (SPSS) software and the Mplus version 7.4 for Windows were used to conduct the statistical analyses.

To test the internal structure of the intended interpretations of the UCLA version 3 scale, we used a confirmatory factor analysis (CFA) approach. Different models were tested, considering previous works on the internal structure of the instrument (for a summary, see Russel et al., 1996). The following measurement models were tested: a unifactorial model (Model 1) wherein all items loaded on a single first-order factor; a two correlated factor model (Model 2), with some of the items loading on a positive measure (non-loneliness items i.e., measuring feeling connected) and the other items loading on a negative measure (loneliness items i.e., measuring feeling lonely); and a bifactor model (Model 3) that considered that items loaded simultaneously on a global measure and on one of the two additional measures (i.e., positive or negative), and that factors were orthogonal (i.e., not-correlated). The rationale behind testing these different models relies on the conceptualization of loneliness as a connectedness/social isolation continuum where lonely individuals find themselves at one edge unwittingly (Cacioppo & Cacioppo, 2012). Finally, the maximum likelihood robust estimator (MLR) was used given that the data was not multivariate normal according to Mardia's Multivariate Normality Test [?2(59.47) = 2795.16, p < .001 for skewness and kurtosis ?2(520.13) = 22.68, p < .001]. Goodness of fit was considered based on the two-index approach proposed by Hu and Bentler (1999): the fit of the model was considered acceptable if it achieved a Standardized Root Mean Square Residual (SRMR) = .08 combined with either a Comparative Fit Index (CFI) = .95 or with a Mean Square Error of Approximation (RMSEA) = .06.

A multi-group CFA was performed on the best fitting model, to assess the measurement invariance across gender. The measurement invariance tests are a series of hierarchically nested confirmatory factor analyses by increasing levels of cross-group equality constraints. Thus, first, configural invariance tests whether each common factor is associated with identical items across groups, fixing construct dimensionality to be invariant. Second, metric invariance, was tested by constraining item loadings only to be equivalent across groups. Third, scalar invariance was tested at the intercept level, whereby the intercepts are constrained to be equal across groups in addition to factor loadings being fixed to equality, to examine whether the comparisons of group means are meaningful. Given that the best fitting model was the bifactorial model (see the Results section), it was necessary to constrain factor variances (and not to the first item of each factor) to 1.0 so that the bifactorial solution could be identified when testing for both metric and scalar invariance. Invariance was established when the added restrictions did not lead to a worse model fit. Invariance across gender was considered when, after adding restrictions, non-significant result of ?2 difference (??2) test was found and the change value of comparative fit indices (?CFI) were = .01 (Byrne, 2010). Because we used the MLR estimator, we computed the Satorra-Bentler scaled chi-square difference test.

To examine gender differences of the global test scores an independent sample t-test was conducted considering p values of = 0.5 as significant different.

Validity of the intended test scores interpretation was tested using Pearson's correlations between the global loneliness test scores and test scores from different well-established health indicators: psychological stress, depression and anxiety symptomatology, and psychological inflexibility. The same analysis was conducted for the test scores of the items grouped into the positive measure and the test scores of the items grouped into the negative measure and the well-established health indicators.

Internal consistency reliability of the global test scores and the test scores of the items grouped into the positive and into the negative measure was assessed based on the calculation of Cronbach's a and McDonald's ?, considering values of 0.80 or above as acceptable for both cases (Nunnally & Berstein, 1994).

Finally, floor and ceiling effects were evaluated for the direct global test scores of individuals scoring at the lowest or highest possible UCLA version 3. Floor or ceiling effects were considered absent when the percentage of individuals scoring at the highest or lowest possible scores was between 1-15% (Terwee et al., 2007). If the percentage was > 15%, validity and reliability were considered compromised (Terwee et al., 2007).

Results

---

Sample description

Sociodemographic characteristics of the sample are detailed in Table 1. Age ranged from 18 to 70 years old (Mdn = 40; SD = 12.476).

Validity evidence based on internal structure: Factor structure and goodness of fit

A CFA approach was performed with the 20 items to examine the scale's structure and adequacy (n =282). Three models were tested; fit indicators of the three models under analyses are presented in Table 2. The bifactor model was the best fitting model and had an acceptable fit according to the two-index approach proposed by Hu and Bentler (199). The bifactor model (i.e., Model 3) was, therefore, chosen by the authors as the most adequate to represent the theoretical model and was in accordance with the aim of developing a measure that would allow the assessment of global feelings of loneliness. For local adjustment indices, standardized regression weights (SRW) varied between -0.18 (item 17) to 0.67 (item 20) for the global measure. Alike what was found by Russell et al (1996), all those SRW values were significant at p < .05; the same was found for SRW values of items loading on the Negative measure, which ranged from .28 (item 17) to .62 (item 11). Again, like findings by Russell et al. (1996), SRW for items loading on the Positive measure were not always significant (i.e., items 5, 6, 9 and 10) and some had negative SWR values (i.e., items 1, 6, 9 and 10). Squared Multiple Correlations (SMC) varied from 0.11 (item 17) to 0.81 (item 20). Therefore, the 20 items of the UCLA version 3 present both adequate global and local adjustments. Only item 17 evidenced low loading, as previously reported in the original scale (.21) (Russell, 1996) and other adaptations (Lasgaard, 2007). Based on those previous findings and bearing comparability across studies, item 17 was maintained in the scale.

Invariance across gender

Results indicated that men and women understood loneliness homogeneously. Table 3 presents a summary of goodness-of-fit indices for measurement invariance across gender groups. Before undertaking the measurement invariance tests, the confirmatory factor analyses were separately conducted for the male and female groups, which confirmed goodness-of-fit for both of those groups. The baseline unconstrained model tested the structure of the scale across both gender groups simultaneously. Results showed an acceptable model fit, further indicating that the factor structure model fitted the data well in both groups (configural invariance). Subsequently, a measurement weight model was tested with factor loadings constrained to be equal across gender groups. When compared to the baseline unconstrained model, no significant changes occurred (??2(40) = 48.13, p = .18; ?CFI = .003), indicating that the factor loadings were invariant across groups (metric invariance). Finally, a measurement intercept model was tested. When compared to the measurement weight model, no significant changes occurred (??2(17) = 14.18, p = .65; ?CFI = .001), indicating that the intercepts are similar across groups (scalar invariance). Taken together, these results suggest that this scale is fit to assess general feelings of loneliness, in both gender groups.

Loneliness scores across gender

Overall, independent sample t-test showed no gender differences between men test scores (M = 43.27; SD = 9.7) and women test scores (M = 42.30; SD = 8.57) on the global direct scores (t(280) = -.850; p < .396).

Validity evidence based on relations with other variables

The global loneliness test is positively correlated with psychological stress (PSS-10) (r = .396, p < .001); depression and anxiety symptomatology (PHQ-4) (r = .539, p < .001); and psychological inflexibility (AAQ-II) (r = .603, p < .001).

The test scores of the items grouped into the positive measure (after being reverse coded) positively correlated with well-established mental health indicators such as psychological stress (PSS-10) (r = .362, p < .001); depression and anxiety symptomatology (PHQ-4) (r = .501, p < .001); and psychological inflexibility (AAQ-II) (r = .590, p < .001). In the same way, the test scores of the items grouped into the negative measure were also positively correlated with PSS-10 (r = .341, p < .001); PHQ-4 (r = .458, p < .001); and AAQ-II (r = .478, p < .001).

In another words, in alignment with what was expected, people who feel lonely are more susceptible to feel stressed, to show anxiety and depression symptoms, and to have a strong need for control and avoidance of aversive thoughts and feelings.

Internal consistency

Both Cronbach's and McDonald's indicated good internal consistency for the global loneliness test scores (a = .91 and ? = .91) as well as for the items grouped into the positive measure (a = .87 and ? = .87) (positive or non-loneliness items) and items grouped into the negative measure (a = .86 and ? = .88) (negative or loneliness items) for the Portuguese version. The inter-item correlation analysis evidenced no negative correlations and the corrected item-normal correlation analysis evidenced values between .25 to .70. However, removing item 17 increases Cronbach's (a = .92) and McDonald's (? = .92).

Floor or ceiling effects

Discussion

---

This study provides the first version of the UCLA version 3 for the Portuguese language, reporting good internal consistency of the test scores, a global bipolar structure, and gender invariance; all of these findings are consistent with the original version (Russell, 1996). This result is important because UCLA version 3 is currently the gold standard to measure loneliness and a growing body of literature supports the impact of loneliness on health.

Regarding the validity based on internal structure, confirmatory factor analysis evidenced a bifactor model that considered items loading simultaneously on a global measure and on one of the two additional measures (i.e., positive, or negative) and that factors were orthogonal (i.e., not correlated). This finding is consistent with previous studies (Russell, 1996). Indeed, individuals move between two edges, feeling connected or feeling lonely (Cacioppo & Cacioppo, 2018).

Validity based on relations with other variables supports the interpretation of test scores of the Portuguese version of UCLA version 3. The association between the global loneliness measure and its positive and negative measures with a number of well-established health indicators denotes an optimal discriminative capacity of the scale. First, the association with psychological stress is consistent with theoretical models about loneliness as a social stressor (Coyle & Dugan, 2012). Indeed, failing to connect with significant others may result in the perception of the social environment as dangerous and threatening (Cacioppo & Hawkley, 2009). These results are also consistent with the social information processing (SPI) model of loneliness (Spithoven et al., 2017), where lonely people tend to code social experiences more negatively and expect negative social interactions, in both cases, leading to unaware hypervigilance towards stressful social interactions and confirmatory behaviours, which in turn enhance the perception of feeling alone (Spithoven et al., 2017). Second, in line with previous studies, the UCLA version 3 scores positively correlated with depressive or anxiety symptomatology scores, which may be mediated by poor emotional regulation (Vanhalst et al., 2012; Jones et al., 1990). Finally, the positive association between loneliness scores and psychological inflexibility test scores are consistent with previous studies, where lonely people tend to avoid social interactions due to expected - and sometimes received - rejection and failed attempts of connection (Cacioppo & Hawkley, 2009). Lonely people may even avoid intimacy which further leads to perceived social isolation.

The test scores of this version of the UCLA version 3 evidenced good reliability for its use among the Portuguese population, based on McDonald's and Cronbach's indexes. Results of this study are consistent with previous studies that reported an a = .92 (Lasgaard, 2007) or a from .89 to .94 across different samples in the original study (Russell, 1996). Interestingly, item 17 ("how often do you feel shy") showed low loading into the negative measure, which might be due to its reference to the personality trait shyness. Both loneliness and shyness, deal with how we interact or relate to the social context by withdrawing from social environments. However, shyness is proposed as a stable personality trait that does not necessarily involve loneliness, as the perception of social isolation (Jones et al., 1990). Lonely people may withdraw from social interactions to avoid being rejected, and shy people to avoid feeling awkward or negatively evaluated/rejected by others (Zimbardo, 1982).

The European Portuguese version of the UCLA version 3 expands the possibilities for loneliness research studies interested in examining its prevalence and impact in health (e.g., context of Covid-19). Moreover, as loneliness is considered a public health matter and has shown strong correlation with well-established health indicators (perceived stress, depression and anxiety symptomatology, and psychological inflexibility), the present version of the scale can be taken as a health indicator of health promotion programs and clinical interventions. Findings from the current work may lead to insights on how to better accept and defuse from negative internal experiences associated with loneliness (Masi et al., 2011).

In sum, this study provides a new version of the UCLA version 3, a reliable and valid measure of loneliness for the Portuguese population for both research and clinical purposes. The UCLA version 3 is the gold standard measure of loneliness, so international comparisons with other countries require this version. It is important to note that the items grouped into the positive measure are less coherent; therefore, this subscale should not be used by itself. Results of this study should be considered with caution since data are based on a convenience sample with a relatively high educational level, unavoidable self-reported measures, and a cross-sectional design.

Funding

---

This work was partially supported by the predoctoral grant Fundación Caja Navarra (FBCAN) and the Public University of Navarre (Resolution 589/2021). Additional Funding was provided by the European Union–NextGenerationEU through the Grant for the Requalification of the Spanish University System for 2021-2023 at the Public University of Navarre (Resolution 1402/2021). _AG-Á is supported by the Portuguese Foundation for Science and Technology [Individual Call to Scientific Employment Stimulus – 3rd Edition 2019-2020.02059-CEECIND]. The Center for Research in Neuropsychology and Cognitive and Behavioral Intervention (CINEICC) of the Faculty of Psychology and Educational Sciences of the University of Coimbra is supported by the Portuguese Foundation for Science and Technology and the Portuguese Ministry of Education and Science through national funds and co-financed by FEDER through COMPETE2020 under the PT2020 Partnership Agreement [UID/PSI/01662/2013].

Conflicts of interest

---

The authors declare they have no conflict of interest.

Acknowledgement

---

We thank Professor Víctor López Guerra, Ph.D. (Associate Professor at the Universidad Técnica Particular de Loja, Ecuador) for his technical support and Professor José Luis Padilla, Ph.D. (Full Professor at the Universidad de Granada, Spain) for his valuable and constructive suggestions.

References

---

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. United States of America: American Educational Research Association.

Brooks, S. K., Webster, R. K., Smith, L. E., Woodland, L., Wessely, S., Greenberg, N., & Rubin, G. J. (2020). The psychological impact of quarantine and how to reduce it: rapid review of the evidence. The Lancet, 395(10227), 912-920. https://doi.org/10.1016/S0140-6736(20)30460-8

Byrne, B. M. (2010). Structural equation modeling with AMOS: basic concepts, applications, and programming (multivariate applications series). New York: Taylor & Francis Group.

Byrne, B. M. (2013). Structural equation modeling with Mplus: Basic Concepts, Applications, and Programming. Routledge.

Cacioppo, S., & Cacioppo, J. T. (2012). Decoding the invisible forces of social connections. Frontiers in Integrative Neuroscience, 6(1), 51. https://doi.org/10.3389/fnint.2012.00051

Cacioppo, J. T., & Cacioppo, S. (2018). Loneliness in the modern age: an evolutionary theory of loneliness (ETL). In Advances in Experimental Social Psychology (Vol. 58, pp. 127-197). Academic Press. https://doi.org/10.1016/bs.aesp.2018.03.003

Cacioppo, J. T., & Hawkley, L. C. (2009). Perceived social isolation and cognition. Trends in Cognitive Sciences, 13(10), 447-454. https://doi.org/10.1016/j.tics.2009.06.005

Cacioppo, J. T., Hughes, M. E., Waite, L. J., Hawkley, L. C., & Thisted, R. A. (2006). Loneliness as a specific risk factor for depressive symptoms: cross-sectional and longitudinal analyses. Psychology and Aging, 21(1), 140–151. https://doi.org/10.1037/0882-7974.21.1.140

Coyle, C. E., & Dugan, E. (2012). Social isolation, loneliness and health among older adults. Journal of Aging and Health, 24(8), 1346-1363. https://doi.org/10.1177/0898264312460275

Hackett, R. A., Hamer, M., Endrighi, R., Brydon, L., & Steptoe, A. (2012). Loneliness and stress-related inflammatory and neuroendocrine responses in older men and women. Psychoneuroendocrinology, 37(11), 1801-1809. https://doi.org/10.1016/j.psyneuen.2012.03.016

Hernandez, A., Hidalgo, M. D., Hambleton, R. K., & Gomez-Benito, J. (2020). International test commission guidelines for test adaptation: A criterion checklist. Psicothema, 32(3), 390-398.

Hernández-López, M., Cepeda-Benito, A., Díaz-Pavón, P., & Rodríguez-Valverde, M. (2021). Psychological inflexibility and mental health symptoms during the COVID-19 lockdown in Spain: A longitudinal study. Journal of Contextual Behavioral Science, 19(1), 42–49. https://doi.org/10.1016/j.jcbs.2020.12.002

Holt-Lunstad, J., Smith, T. B., Baker, M., Harris, T., & Stephenson, D. (2015). Loneliness and social isolation as risk factors for mortality: a meta-analytic review. Perspectives on Psychological Science, 10(2), 227-237. https://doi.org/10.1177/1745691614568352

Horesh, D., Kapel Lev-Ari, R., & Hasson-Ohayon, I. (2020). Risk factors for psychological distress during the COVID-19 pandemic in Israel: Loneliness, age, gender, and health status play an important role. British Journal of Health Psychology, 25(4), 925-933. https://doi.org/10.1111/bjhp.12455

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: a Multidisciplinary Journal, 6(1), 1-55. https://doi.org/10.1080/10705519909540118

International Test Commission. (2017). The ITC Guidelines for Translating and Adapting Tests (Second edition). [www.InTestCom.org].

Jones, W. H., Rose, J., & Russell, D. (1990). Loneliness and social anxiety. In Handbook of Social and Evaluation Anxiety (pp. 247-266). Springer, Boston, MA.

Keller, F. M., Derksen, C., Kötting, L., Dahmen, A., & Lippke, S. (2022). Distress, loneliness, and mental health during the COVID-19 pandemic: Test of the extension of the Evolutionary Theory of Loneliness. Applied Psychology. Health and Well-being, 10.1111/aphw.12352. Advance Online Publication. https://doi.org/10.1111/aphw.12352

Lasgaard, M. (2007). Reliability and validity of the Danish version of the UCLA Loneliness Scale. Personality and individual differences, 42(7), 1359-1366. https://doi.org/10.1016/j.paid.2006.10.013

Levin, M. E., MacLane, C., Daflos, S., Seeley, J. R., Hayes, S. C., Biglan, A., & Pistorello, J. (2014). Examining psychological inflexibility as a transdiagnostic process across psychological disorders. Journal of Contextual Behavioral Science, 3(3), 155-163. https://doi.org/10.1016/j.jcbs.2014.06.003

Löwe, B., Wahl, I., Rose, M., Spitzer, C., Glaesmer, H., Wingenfeld, K., Schneider, A., & Brähler, E. (2010). A 4-item measure of depression and anxiety: validation and standardisation of the Patient Health Questionnaire-4 (PHQ-4) in the general population. Journal of Affective Disorders, 122(1-2), 86-95. https://doi.org/10.1016/j.jad.2009.06.019

Masi, C. M., Chen, H. Y., Hawkley, L. C., & Cacioppo, J. T. (2011). A meta-analysis of interventions to reduce loneliness. Personality and Social Psychology Review, 15(3), 219-266. https://doi.org/10.1177/1088868310377394

Nunnally, J. C., & Bernstein, I. H. (1994). The Assessment of Reliability. Psychometric Theory, 3(1), 248-292.

O'Day, E. B., Morrison, A. S., Goldin, P. R., Gross, J. J., & Heimberg, R. G. (2019). Social anxiety, loneliness, and the moderating role of emotion regulation. Journal of Social and Clinical Psychology, 38(9), 751-773. https://doi.org/10.1521/jscp.2019.38.9.751

Pinto-Gouveia, J., Gregório, S., Dinis, A., & Xavier, A. (2012). Experiential avoidance in clinical and non-clinical samples: AAQ-II Portuguese version. International Journal of Psychology and Psychological Therapy, 12(2), 139-156. https://www.redalyc.org/articulo.oa?id=56023336002

Rubin, K. H., Coplan, R. J., Bowker, J. C., & Menzer, M. (2014). Social withdrawal and shyness. In P. K. Smith & C. H. Hart (Eds.), Wiley Blackwell Handbooks of Developmental Psychology. The Wiley Blackwell Handbook of Childhood Social Development (p. 434–452). Wiley Blackwell.

Russell, D. W. (1996). UCLA Loneliness Scale (Version 3): Reliability, validity, and factor structure. Journal of Personality Assessment, 66(1), 20-40. https://doi.org/10.1207/s15327752jpa6601_2

Russell, D., Peplau, L. A., & Cutrona, C. E. (1980). The revised UCLA Loneliness Scale: concurrent and discriminant validity evidence. Journal of Personality and Social Psychology, 39(3), 472. https://doi.org/10.1037/0022-3514.39.3.472

Russell, D., Peplau, L. A., & Ferguson, M. L. (1978). Developing a measure of loneliness. Journal of Personality Assessment, 42(3), 290-294. https://doi.org/10.1207/s15327752jpa4203_11

Spithoven, A. W., Bijttebier, P., & Goossens, L. (2017). It is all in their mind: A review on information processing bias in lonely individuals. Clinical Psychology Review, 58(1), 97-114. https://doi.org/10.1016/j.cpr.2017.10.003

Spithoven, A. W. M., Cacioppo, S., Goossens, L., & Cacioppo, J. T. (2019). Genetic contributions to loneliness and their relevance to the evolutionary theory of loneliness. Perspectives on Psychological Science, 14(3), 376-396. https://doi.org/10.1177/1745691618812684

Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., Bouter, L. M., & de Vet, H. C. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60(1), 34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012

Trigo, M., Canudo, N., Branco, F., & Silva, D. (2010). Estudo das propriedades psicométricas da Perceived Stress Scale (PSS) na população portuguesa. Psychologica, (53), 353-378. https://doi.org/10.14195/1647-8606_53_17

Vanhalst, J., Luyckx, K., Raes, F., & Goossens, L. (2012). Loneliness and Depressive Symptoms: The Mediating and Moderating Role of Uncontrollable Ruminative Thoughts. The Journal of Psychology, 146(1), 259 - 276. https://doi.org/10.1080/00223980.2011.555433

Wojnarowska, A., Kobylinska, D., & Lewczuk, K. (2020). Acceptance as an Emotion Regulation Strategy in Experimental Psychological Research: What We Know and How We Can Improve That Knowledge. Frontiers in Psychology, 11(1), 242. https://doi.org/10.3389/fpsyg.2020.00242

World Health Organization. (2020). Process of translation and adaptation of instruments. Alcohol, Drugs and Addictive Behaviours Unit. Retrieved May 31, 2020, from https://www.who.int/substance_abuse/research_tools/translation/en/

Zarei, S., Memari, A. H., Moshayedi, P., & Shayestehfar, M. (2016). Validity and reliability of the UCLA loneliness scale version 3 in Farsi. Educational Gerontology, 42(1), 49-57. https://doi.org/10.1080/03601277.2015.1065688

Zimbardo, P. G. (1982). Shyness and the stresses of the human connection. In L. Goldberger and S. Breznitz (Eds.), Handbook of stress: Theoretical and Clinical Aspects (pp. 466–481). New York: Free Press.