community health care research articles

Journal of Community Health

The Publication for Health Promotion and Disease Prevention

The Journal of Community Health, a peer-reviewed publication, offers original articles on the practice, teaching, and research of community health. Coverage includes preventive medicine, new forms of health manpower, analysis of environmental factors, delivery of health care services, and the study of health maintenance and health insurance programs. Serving as a forum for the exchange of ideas and clarification, the journal features articles on projects that make a significant impact on the education of health personnel.

• Pascal James Imperato

Latest issue

Volume 49, Issue 3

Latest articles

The influence of the covid 19 pandemic on food insecurity among cancer survivors across new york state.

• Marlene Camacho-Rivera
• Katherine Haile
• Erica Phillips

YouTube as a Source of Information on Air Pollution: Significance for Community Health

• Corey H. Basch
• Helen Yousaf
• Charles E. Basch

Improving Care for People Aging with HIV: A Collaborative Quality Improvement Approach

• Daniel Belanger
• John Wikiera
• Eugenia L. Siegler

The Social Determinants of Suicide among Female Service Members and Veterans

• Rachel M. Larson
• Wanda A. Wright

A Ten Year Experience of Men’s Health Events in a Socioeconomically Diverse City in the United States – Lessons Learned

• Shane Tinsley
• Navid Mahabadi
• Elisabeth I. Heath

Journal updates

Call for papers, journal information.

• CAB Abstracts
• Current Contents/Social & Behavioral Sciences
• Google Scholar
• Japanese Science and Technology Agency (JST)
• OCLC WorldCat Discovery Service
• Semantic Scholar
• Social Science Citation Index
• TD Net Discovery Service
• UGC-CARE List (India)

Rights and permissions

Editorial policies

© Springer Science+Business Media, LLC, part of Springer Nature

• Find a journal
• Publish with us
• Track your research

• Research article
• Open access
• Published: 13 April 2018

Achieving successful community engagement: a rapid realist review

• E. De Weger 1 , 2 ,
• N. Van Vooren 1 ,
• K. G. Luijkx 2 ,
• C. A. Baan 1 , 2 &
• H. W. Drewes 1  

BMC Health Services Research volume  18 , Article number:  285 ( 2018 ) Cite this article

51k Accesses

92 Citations

35 Altmetric

Metrics details

Community engagement is increasingly seen as crucial to achieving high quality, efficient and collaborative care. However, organisations are still searching for the best and most effective ways to engage citizens in the shaping of health and care services. This review highlights the barriers and enablers for engaging communities in the planning, designing, governing, and/or delivering of health and care services on the macro or meso level. It provides policymakers and professionals with evidence-based guiding principles to implement their own effective community engagement (CE) strategies.

A Rapid Realist Review was conducted to investigate how interventions interact with contexts and mechanisms to influence the effectiveness of CE. A local reference panel, consisting of health and care professionals and experts, assisted in the development of the research questions and search strategy. The panel’s input helped to refine the review’s findings. A systematic search of the peer-reviewed literature was conducted.

Eight action-oriented guiding principles were identified:

Ensure staff provide supportive and facilitative leadership to citizens based on transparency;

foster a safe and trusting environment enabling citizens to provide input;

ensure citizens’ early involvement;

share decision-making and governance control with citizens;

acknowledge and address citizens’ experiences of power imbalances between citizens and professionals;

invest in citizens who feel they lack the skills and confidence to engage;

create quick and tangible wins;

take into account both citizens’ and organisations’ motivations.

Conclusions

An especially important thread throughout the CE literature is the influence of power imbalances and organisations’ willingness, or not, to address such imbalances. The literature suggests that ‘meaningful participation’ of citizens can only be achieved if organisational processes are adapted to ensure that they are inclusive, accessible and supportive of citizens.

Peer Review reports

Ageing populations with increasingly complex health and care needs, growing health inequalities, and the challenging financial climates in OECD countries, have long emphasised the need for the provision of better and more efficient care [ 1 ]. In an effort to tackle such problems, a diverse range of organisations, including healthcare providers, insurance companies, municipalities and patient representatives are collaborating to implement new models of care [ 2 , 3 , 4 ]). Community engagement (CE) is increasingly seen as a key component of such new collaborative models of care. Communities often have a more holistic view of health and wellbeing, thus enabling organisations to look beyond their own interests and traditional remits [ 5 ]. The assumption is that involving communities can act as a lever for change to bring a wider range of services together even including, schools and local businesses, which would then be more tailored to the needs of the communities themselves. Many suggest that such tailored and integrated services would ultimately lead to improved community health [ 6 , 7 ]. Others also believe that as citizens become more engaged and empowered to shape their local services, the management of their own health and wellbeing would also improve [ 8 ]. Many health and care organisations in the Netherlands have started implementing new CE interventions; however, there is limited insight regarding the best ways to implement successful CE initiatives.

Previous studies have evaluated different types of CE interventions that have been implemented with the aim of improving local health and care services or neighbourhoods’ healthy living infrastructure [ 9 , 10 , 11 , 12 ]. Earlier literature reviews have focused on how CE interventions affect populations’ health and social outcomes [ 8 , 13 ] or organisational structures and processes [ 14 , 15 ]. Each of these studies has shed some light on the problems that prevent CE interventions from reaching ‘meaningful’ citizen participation. For example, earlier studies have shown how power imbalances and the inaccessibility of organisational structures and processes experienced by citizens can prevent CE interventions from producing the intended outcomes and can instead lead to mistrust between citizens and professionals [ 9 , 16 , 17 , 18 ]). However, while these earlier studies have been insightful, they do not provide professionals with the information they need to successfully implement CE interventions in their own contexts. This is partly because previous studies have provided limited insight into the ways in which the different contextual factors (e.g. existing service fragmentation) and underlying mechanisms (e.g. staff’s support and facilitation making citizens feel valued) influence CE intervention outcomes (e.g. levels of community trust).

To start providing such information, this rapid realist review (RRR) sets out eight guiding principles for ‘meaningful’ participation. The principles are based on a review of the peer-reviewed literature and are underpinned by an investigation of which CE interventions work, for whom, how, to what extent and in which contexts. The principles, along with the contextual factors and the mechanisms that influence the outcomes of CE interventions are useful for policymakers and professionals to explore when struggling with the implementation of their own CE intervention. The review specifically investigated the application of CE in health and social care, focusing on the macro and meso levels of CE, e.g. developing policies, designing, implementing and delivering health and care services, setting service and policy priorities. The review addressed the following research questions:

What are the action-oriented guiding principles by which community engagement interventions can be implemented successfully?

What are the mechanisms by which these principles operate? What are the contextual factors influencing the principles?

What impact do the interactions between contextual factors and mechanisms have on CE intervention outcomes?

This review applied the rapid realist review (RRR) methodology. The realist methodology aims to highlight the impact that interactions between the contextual factors and the mechanisms have on intervention outcomes [ 19 ]. RRRs aim to provide a similar knowledge synthesis as traditional systematic realist reviews, but within a considerably shorter timeframe to ensure the relevance and applicability of results for the stakeholders [ 20 , 21 , 22 , 23 ].

The review was undertaken in consultation with a local reference panel. As this RRR represents the first stage of a four-year mixed methods multiple case study evaluating six community engagement interventions in the Netherlands, the local reference panel consisted of the six CE interventions’ stakeholders, including professionals, citizens and citizen representatives who will be further developing and implementing the interventions. The panel also included experts in the fields of health inequalities, citizen participation, and public health, to ensure the review addressed relevant gaps in the literature. The review followed five iterative stages, which have been applied and described by others previously [ 21 , 22 , 23 ]:

Developing and refining research questions

Searching and retrieving information

Screening and appraising information

Synthesising information

Interpreting information

Because there are such wide-ranging definitions and interpretations of CE, an important first step was to find one clear definition that the authors could then apply throughout each stage of the review. Based on a preliminary search of the literature and early consultations with the panel, the authors chose the following definition of community engagement:

‘Involving communities in decision-making and in the planning, design, governance and/or delivery of services. Community engagement activities can take many forms including service-user networks, healthcare forums, volunteering or interventions delivered by trained peers’ ([ 24 ], p. xiii).

The authors engaged with the stakeholders of the six interventions at the start of the review to ensure their key areas of interest were covered in the review, and also consulted with the other experts in the local reference panel to confirm that the review addressed relevant gaps in the literature.

In consultation with the library scientist at the National Institute for Public Health and the Environment (RIVM), and based on the chosen definition and the preliminary search of the literature, the review search terms and search strings were agreed (See Additional file 1 ) and applied in the electronic databases, Embase and Scopus. These two databases were chosen as they were deemed by the library scientist to be the most relevant to the review’s subject area. Furthermore, Embase and Scopus are two of the largest international databases with a focus on health and social research and include trade journals as well. Upon reviewing the results of these two databases, the authors felt that enough rich data had been obtained and so did not search any other databases in order to speed up the process to ensure the stakeholders received the relevant information on time and in line with their CE intervention implementation schedules.

The draft inclusion and exclusion criteria were developed based on the preliminary search and were tested by two reviewers (EdW and NvV). Based on this test, the reviewers decided to expand the original criteria to ensure closer alignment with the review’s scope and the chosen CE definition. The reviewers screened the papers in two stages. During the first stage, papers’ titles and abstracts were screened, for example, based on whether the CE interventions described involved citizens or communities in the decision-making, planning, design, governance or delivery of health and care services or policies. EdW and NvV applied these criteria to the titles and abstracts and rated papers: (a) ‘red’, if papers did not follow the agreed definition of CE and/or if topics fell clearly outside the scope; (b) ‘amber’ if this was unclear; or (c) ‘green’, if the papers clearly applied the same definition and discussed topics within the scope. Initially, EdW and NvV both screened the same 100 papers to ensure standardisation of the screening process. After this, the reviewers each reviewed a different stack of papers to speed up the screening process. EdW and NvV crosschecked and discussed the papers rated ‘amber’ or ‘green’ to ensure consistency in their approach. Additionally, HD sampled 40 papers—20 papers which NvV and EdW had both screened, 10 papers which EdW had screened and 10 papers which NvV had screened—to ensure EdW’s and NvV’s screening was rigorous, consistent and standardised. Papers rated ‘red’ did not continue to the second, full-text, screening stage (Table 1 ).

During the second screening stage, EdW and NvV assessed the full text of those papers that had been rated ‘green’ or ‘amber’ for methodological rigour using the Mixed Methods Appraisal Tool (MMAT) [ 25 ] and for relevance. Relevance was assessed by asking questions like whether CE was the paper’s main subject area and whether the CE interventions described operated on Rowe & Frewer’s [ 26 ] ‘Public Participation’ level. In line with the O’Mara-Eves et al. [ 24 ] definition, the authors used Rowe & Frewer’s [ 26 ] classification of public participation to assess whether the interventions described in the literature operated on the ‘public participation level’ whereby citizens are not merely receiving information from organisations (public communication level), or merely providing information to organisations (public consultation), but are actively engaged in dialogue with organisations (Table 2 ) [ 26 ].

Data extractions were conducted on the final set of selected papers using an extraction template (available upon request) . The template was used to extract data regarding the interventions’ strategies, activities and resources, and the context, mechanisms and outcomes directly stipulated in the papers. To aid the reviewers during the extraction process and to ensure consistency and transparency, the authors specified CE-oriented definitions of important realist concepts. The realist methodology is still developing and as such, realist evaluators continue to unpack and operationalise terms like ‘context’, ‘mechanisms’, and ‘interventions’ and how these interrelate [ 27 ]. The following CE-oriented definitions of the realist concepts were applied:

Intervention : refers to interventions’ implemented activities, strategies and resources [ 27 ] e.g., citizen advisory panel meetings, neighbourhood clean-up activities, or citizen learning opportunities.

Mechanism : the concept of ‘mechanism’ does not refer to the intentional resources offered or strategies implemented within an intervention. Rather, it refers to what ‘triggers’ participants to want to participate, or not, in an intervention. Mechanisms usually pertain to cognitive, emotional or behavioural responses to intervention resources and strategies [ 28 ], e.g., citizens feeling more empowered due to learning opportunities.

Context : pertains to the backdrop of an intervention. Context includes the pre-existing organisational structures, the cultural norms and history of the community, the nature and scope of pre-existing networks, and geographic location effects [ 28 , 29 ], e.g., pre-existing levels of trust between communities and organisations or previous experience of CE interventions.

Outcome: refers to intended or unexpected intervention outcomes [ 28 ] e.g. sustainability, quality integration of services (macro); citizens’ level of involvement in health and care services (meso); citizens’ health and wellbeing outcomes (micro).

Using completed extractions, EdW and NvV created Context-Mechanism-Outcome configurations (CMOs) in order to understand and explain why CE interventions work, or not, and to generate the action-oriented guiding principles. For this review, the authors only created CMOs if those contexts, mechanisms and outcomes were explicitly correlated in the papers themselves to avoid conjecture. After drafting the configurations, the mechanisms of the CMOs were first clustered per type of CE intervention in order to ensure that the eventual principles were underpinned by mechanisms found across the range of CE interventions and thus across different contexts—i.e. (a) citizens involved in health and care organisations or neighbourhood panels, forums, boards, steering groups, planning and decision-making committees; (b) community-wide volunteering and community group activities in health and care related subjects; and (c) peer delivery. After this initial round of clustering, NvV and EdW searched for keywords in those mechanisms and then thematically clustered the mechanisms according to those keywords—independent of the types of intervention—thus generating the guiding principles. As papers were able to contribute to multiple principles, EdW and NvV also checked that each principle was based on mechanisms from several different papers to ensure the principles were transferable across different interventions and contexts. The final draft of the clustered mechanisms was shared with the other authors to confirm the mechanism themes and to refine the principles. The mechanisms of the CMOs were chosen as the basis for generating the principles, because the question of what makes citizens want to participate or not, are central to the CE literature and to the local reference panel. This question is inherently related to the concept of ‘mechanisms’—what ‘turns on in the minds of program participants and stakeholders that make them want to participate or invest in programs’ [ 28 ].

Finally, the authors held a workshop in order to present the review’s findings, including the final draft of the principles, to the local reference panel. During the workshop, the panel discussed the applicability of the principles within their local contexts and whether they are experiencing similar issues in the development of their own CE interventions. Confirming that the final draft of principles and their corresponding mechanisms had face validity, the workshop provided rich anecdotal evidence, thus further refining and finalising the principles discussed below.

After the removal of duplicates, the search resulted in 2249 potentially relevant papers (see Fig.  1 ). After the first title and abstract screening stage, 205 papers were selected to continue to the second full text screening stage. After applying the full-text inclusion and exclusion criteria and removing a further four papers as they contained no information on contexts or mechanisms and excluding six literature reviews to ensure this review’s findings were based on primary data, a total of 20 papers were used for data extraction.

Flowchart of document inclusion and exclusion process

The majority of papers focused on CE interventions involving citizens in healthcare organisations’ or neighbourhood panels, forums, boards, steering groups, or planning and decision-making committees [ 9 , 17 , 18 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 ]. For example, [ 32 ] study described the Australian District Aboriginal Health Action Groups (DAHAGs), which included both Aboriginal community members and healthcare professionals who together, identified local solutions to important Aboriginal health problems [ 32 ]. Five papers investigated CE interventions involving community group activities or community wide volunteering ([ 11 , 12 , 31 , 41 ]; Schoch-Spana). For example, Hamamoto et al. [ 11 ] described how community volunteering and actions groups, together with the local Community Centre, developed and implemented a new healthy living infrastructure in the local neighbourhood. Only two papers described peer delivery interventions [ 10 , 16 ]. For example, De Freitas & Martin’s [ 16 ] study showed how Cape Verdean migrant mental health service-users were empowered and actively engaged in supporting and recruiting other service-users (Table 3 ).

A total of eight guiding principles was identified through the literature and enriched and triangulated by the panel’s input. Table  3 summarises the enabling contexts and mechanisms underpinning the principles that organisations can build on to ensure CE interventions are successful. It is worth noting that constraining contexts and mechanisms are largely two sides of the same coin—e.g. lack of previous positive relationships between organisations and communities (context) and lack of quick wins worsened residents’ feelings of hopelessness and powerlessness (mechanism) . The following section first describes each principle using the evidence from the literature review, including examples of individual CMO configurations underpinning the principle (full list of individual CMO configurations available upon request) . After each principle, the panel’s reflections will be summarised separately—the panel’s input did not change the wording of the principles and instead triangulated and enriched the literature findings (Table 4 ).

Guiding principle 1: Ensure staff provide supportive and facilitative leadership to citizens based on transparency

Supportive and facilitative leadership refers to organisational leadership that supports citizens in their roles and tasks without being too directive or restrictive. Such support should be based on transparency allowing both citizens and professionals to easily share information with each other. This helps to ensure that all those involved in CE interventions are clear on the expected outcomes [ 30 , 32 , 37 , 40 ]. One of the examples from the literature involves a hospital setting up a Citizen’s Advisory Panel not just to address the hospital’s significant deficit, but also to create community support for the required service changes and to foster closer relationships with the community. From the start, the Board was transparent about the difficult financial situation and stipulated that the Panel’s role was to make far-reaching recommendations regarding the Hospital’s operations and processes in order to make the hospital more efficient [ 30 ]. The Board supported the Panel, for example, by giving and receiving presentations and by enlisting the help of an external consultancy who facilitated the Panel in developing their recommendations. While the Panel felt anxious about the magnitude and complexity of their task and their own recommendations on how best to address the hospital’s significant deficit (context) , the supportive yet not directive facilitation and transparency of the Board ensured that the Panel remained positive and motivated throughout the process ( mechanism ). Ultimately, the Panel members stated they would engage in such interventions again (outcome) . Furthermore, the Board approved the majority of the Panel’s recommendations ( outcome ), which resulted in a balanced budget ( outcome ). While some in the wider community were angry about the service cuts, the overall response of the community was positive ( outcome ) [ 30 ].

As evidenced in Table 3 , CE interventions operate within a wide range of contextual factors relating to leadership. Enabling contextual factors include previous positive experiences of CE [ 30 ] and organisational structures providing separate points of connections between communities and local services (e.g. quarterly meetings, whole-area forums) [ 32 ]. Constraining contextual factors include engaging communities with pre-existing low-levels of community readiness to mobilise around a health or neighbourhood issue or citizens with deteriorating health [ 40 ]. If contextual constraints are not acknowledged, interventions will likely be met with resistance. For example, unsupportive leadership that is unable to release control to citizens living in low-income neighbourhoods, leads to those citizens feeling frustrated and disempowered [ 40 ]. However, the literature shows that CE interventions operating within constraining contextual factors do not automatically fail as long as such constraints are acknowledged and addressed within the intervention by supportive and facilitative leadership [ 30 , 32 , 40 ].

Local reference panel reflections

The panel acknowledged the importance of fostering supportive leadership and offering specific points of connection between communities and their local services. The panel proposed having one consistent professional in a leadership position whom citizens can contact if they need further information or support.

Guiding principle 2: Foster a safe and trusting environment enabling citizens to provide input

Creating forums where citizens and professionals alike feel comfortable enough to put forward ideas is critical to CE interventions’ success. Engagement processes and activities should, therefore, be adjusted to suit citizens’ needs and organisations should take steps to reduce practical as well as cultural barriers to enable their full participation [ 16 , 32 , 33 , 34 , 35 , 39 , 41 , 42 ]. Examples from the literature include holding meetings and activities when convenient to citizens, taking into account citizens’ language needs (e.g. less jargon), and ensuring activities aimed at ethnic minorities are culturally sensitive [ 16 , 34 , 42 ]. In Schoch-Spana et al.’s [ 42 ] study the enabling organisational infrastructure (context) helped management to create a safe environment for the community to ask questions during deliberative meetings, which helped to build trust and cohesion (mechanism) . Management’s efforts, in turn, enabled staff to dedicate time to building trusting relationships with the community (outcome) which then meant that citizens were more likely to come forward and volunteer their own time (outcome) . Luluquisen & Pettis’s [ 34 ] study highlighted that it is important for organisations to consider citizens’ potentially differing needs and cater to different groups so that safe spaces can be created for those different groups (e.g. a youth only steering group, separate from adult boards). Creating a safe and trusting environment is especially important in contexts of marginalisation and racism. Often, such communities are mistrustful of local services [ 42 ], especially if past engagement efforts have failed to bring any improvements [ 32 ]. Failing to accommodate citizens’ needs would result in citizens feeling intimidated by, e.g. professional meetings [ 41 ]. Before implementing any plans, organisations will need to invest time and resources into addressing these contextual factors [ 16 , 32 , 35 , 42 ].

The panel emphasised that local neighbourhoods do not consist of one homogeneous group of citizens with the same interests and needs. This means that local citizens sometimes have opposing views and priorities and that organisations should play a mediating role by, for example, setting up safe forums where such tensions can be openly discussed.

Guiding principle 3: Ensure citizens’ early involvement

Citizens should be involved as early as possible, though the point of citizen involvement should be discussed with citizens. Where possible, organisations should engage citizens in the identification and prioritisation of their own healthcare needs. In doing so, organisations ensure that their priorities and definitions of health are aligned with those of the citizens they serve [ 9 , 12 , 31 , 37 , 39 ]). Veronesi & Keasey’s [ 39 ] study showed how the early involvement of citizens was critical in overcoming initial staff resistance to the proposed reorganisation of an acute hospital. While staff was resistant to any change, the local community and patient representatives welcomed the chance to improve the failing local hospital and thus became active drivers for change. [ 30 ] study is particularly interesting, as the conflict of interest did not centre on the differences of opinions between ‘the organisation’ as a whole and ‘the community’. Instead, the organisation’s upper management seemed at odds with its staff, thereby creating a unique opportunity to leverage the community’s input to make the required changes. The literature includes several examples of how failing to include citizens early on negatively affected the outcome of CE interventions [ 9 , 12 , 37 ]). For example, Carlisle (2010) evaluated a Panel consisting of professionals and local community members tasked with tackling the social exclusion and health inequalities experienced by deprived local communities (context) . However, the Panel had already been operational for over a year before any community members were able to join (context ). Because the professionals in the Panel had already allocated funds and resources, the community members felt ‘like tokens’ on the Panel (mechanism) and were keen to ‘present a united front’ against the professionals (outcome) . This ultimately led to a tense and uncollaborative relationship (outcome) [ 9 ]. Organisations will struggle to involve citizens early on, if contextual power imbalances between professionals and citizens are not addressed and organisations maintain overall control of interventions’ projects and plans. Ultimately, citizens who are shut out of strategic and decision-making stages end up feeling disempowered and demotivated to continue their engagement [ 12 ]. Instead, the early involvement of citizens can build momentum and motivate others to join CE interventions [ 12 , 31 ].

While the panel recognised that early involvement of citizens is important, in the panel’s experience, citizens often struggle to participate if organisations have not yet worked out any concrete goals or plans, as they prefer having something tangible to discuss. They suggested organisations support citizens to turn their own ideas into workable plans and strategies.

Guiding principle 4: Share decision-making and governance control with citizens

Organisations should encourage citizens to take on governance and decision-making roles within CE interventions [ 9 , 12 , 31 , 32 , 33 , 34 ]). The literature includes many examples of how organisations maintain control of the management, governance and planning of CE interventions [ 9 , 12 , 31 , 33 ]). For example, in Carlisle’s study (2010), once the citizens had joined the Panel, the professionals continued to maintain control by monopolising the meetings by ‘wading through large quantities of complex paperwork’ and the tenant council of one of the interventions evaluated by Yoo et al. [ 40 ] maintained control by, for example, cancelling meetings at the last minute. The literature also highlights examples of how organisations can share control by amending interventions’ governance and management structures and processes [ 12 , 32 , 33 , 34 ]. For example, Durey et al.’s [ 32 ] study showed that in the context of the community’s marginalisation and mistrust of culturally inappropriate mainstream health services (context) , enabling Aboriginal community members to choose their own representatives on the DAHAGs was valued by community members (mechanism) and led to more authority being placed in the hands of the community (outcome) . Renedo & Marston [ 18 ] took a broader view and recommended that organisations examine the way in which professionals view and discuss citizen participation and to enable citizens to shape their own roles and identities instead. This way citizens will not have to adapt to organisations’ ‘elite systems’ and are valued for their own unique input. Such sharing of control is harder to achieve in contexts of marginalised communities with lower levels of readiness and hierarchical organisational structures, and when interventions have been developed ‘top down’. In such contexts, engaged citizens quickly feel as if professionals dismiss their views [ 34 ]. Ultimately, as Lang et al. [ 12 ] highlighted, citizens’ willingness to participate in interventions significantly depends on the extent to which organisations are willing and able to share control.

The citizens and citizen representatives on the panel echoed the review’s findings and suggested organisational processes should be more tailored to citizens as they find it difficult to navigate organisational processes and structures. In their experience, citizens are often unaware of which organisations or professionals to approach with their ideas or what processes they are expected to follow.

Guiding principle 5: Acknowledge and address citizens’ experiences of power imbalances between citizens and professionals

Addressing power imbalances between citizens and professionals is crucial to CE interventions’ success. However, there are several factors, which contribute to citizens’ relative powerlessness [ 9 , 17 , 18 , 33 , 34 ]). Firstly, as we have seen, organisational structures and hierarchies are tipped towards professionals rather than citizens leading interventions’ most influential aspects as professionals continue to hold key decision-making and governance positions [ 32 , 33 , 34 ]. Secondly, studies like Renedo & Marston [ 18 ] and Lewis [ 17 ] highlights that the way in which professionals view and discuss citizens contains contradictions that maintain the institutional status quo. For example, Lewis [ 17 ] discussed how professionals dismissed and undermined engaged service-users of a mental health service, because of their having a mental illness. The professionals called into question the validity of service-users’ contributions by suggesting their mental health issues ‘made them unreasonable’. Renedo & Marston [ 18 ] explained that professionals’ contradictory discourse and expectations of engaged citizens limits citizens in the type and scope of contributions they are able to make—e.g. having professional-level skills, while at the same time being a ‘genuine’ citizen in the local area. Such discourses maintain a clear division between a ‘powerful us’ (professionals) and disempowered ‘others’ (engaged citizens). Constraining contextual factors, which may make it harder to address power imbalances, include disadvantaged communities used to being at the lower end of the privilege spectrum, and organisations remaining symbolic institutions of power and hierarchy [ 9 , 17 , 32 , 33 , 34 ]). For example, Lewis [ 17 ] showed that a lack of institutional status (context) can lead to citizens feeling out of place and unsure of how to contribute to organisationally run meetings (mechanism) . This led to some citizens feeling silenced (outcome) , while others felt angry and shouted out their views (outcome) , which in turn was dismissed by professionals (outcome) . Ultimately, CE initiatives will only be successful if organisations address power imbalances, share control with citizens and professionals and citizens view each other as legitimate and equal partners in the interventions [ 17 , 18 , 33 ].

The panel recognised the importance of the equal status between organisations and citizens. They felt that achieving such a balance would require open and honest discussions between organisations and citizens about their respective roles in a more equal CE structure.

Guiding principle 6: Invest in citizens who feel they lack the skills and confidence to engage

Organisations should offer learning opportunities to citizens who feel they lack the skills and confidence to engage. Without being offered the opportunity to learn the required skills and capabilities, many, more vulnerable, citizens will likely feel unable to effectively engage [ 10 , 12 , 16 , 18 , 32 ]. For example, Crondahl & Eklund Karlsson [ 10 ] evaluated a CE learning intervention, which aimed to empower socially excluded and discriminated against Roma citizens to become health promotion coordinators in their own Roma communities (context) . The training programme helped the Roma coordinators to develop a greater sense of control and empowerment (mechanism) , which led to increased self-acceptance and to a sense of positive Roma culture (outcome) [ 10 ]. De Freitas & Martin’s study [ 16 ] of a mental health provider’s advocacy project supporting migrants with mental health issues showed how culturally sensitive training programmes empowered disadvantaged service-users. The organisation recognised that the participants did not feel as if they had the required skills and delivered training aimed at raising their awareness about the causes of their disadvantage and the tools they could use to alleviate the causes. Additionally, the already engaged service-users were trained to deliver peer-support groups to other service-users. The direct peer recruitment enabled the marginalised service-users to recognise their own entitlement to participation and enable them to successfully recruit other migrants with mental health issues into the service. The peer supporters themselves increased their social interactions, improved their communication skills and adhered more to their own treatment plans [ 16 ]. Constraining contextual factors include organisations that maintain power imbalances, have unclear remits for citizens, and have tense relationships with communities. Organisations operating within such contexts will struggle to provide the right learning opportunities to citizens who do not already feel empowered; again highlighting the importance of first addressing such constraining contextual factors.

The panel agreed that in their experience it was difficult to engage citizens who are not already empowered. Most of their citizen-participants not only live in the area, but also work in the local healthcare sector. The stakeholders are still searching for the best ways of engaging more disadvantaged citizens, but suggested ‘buddying up’ vulnerable citizens with the already engaged citizens.

Guiding principle 7: Create quick and tangible wins

Quick wins are important for CE interventions to build and maintain momentum among citizens [ 11 , 32 , 34 , 40 , 41 ]. Hamamoto et al. [ 11 ] described how a local community health centre engaged thousands of citizens in tangible projects promoting active living in the area, e.g. a mother’s walking group, bicycle repair and distribution programme. Though the local community had pressing and visible socio-economic needs including a deteriorating infrastructure not easily lending itself to physical activity (context) , the early successes in the initial stages of the intervention provided momentum and energy for citizens to come together towards other common and achievable goals (mechanism) . This led to thousands of citizens to volunteer for health-related activities and youth programmes (outcome) . However, the community centre struggled to engage citizens in broader policy development (outcome) , partly because the Centre did not have enough supportive resources or clearly defined policy issues to mobilize the community around. Kegler et al. [ 41 ] evaluated communities’ participation in California Healthy Cities and Communities programmes and found that sites, which focused on tangible mobilisation efforts such as neighbourhood clean-ups typically, generated more spin-off activities, and had more citizens participating in projects’ implementation phases. Studies like Durey et al. [ 32 ] and Yoo et al. [ 40 ] suggest that quick wins are especially important for interventions where communities’ previous experiences of CE, or health and care services more generally, have been negative and failed to show any benefits to citizens , this is especially true for CE interventions with marginalised and low-income communities. In such contexts, a lack of quick, concrete improvements can worsen citizens’ feelings of powerlessness and will likely result in citizens being less likely to participate in future interventions [ 40 ]. While, quick wins which result in changes that improve services, help to create communities’ trust in the engagement processes and can trigger citizens’ dedication and ability to push through difficulties and obstacles [ 32 , 34 ].

The panel echoed Hamamoto et al.’s [ 11 ] findings and highlighted the difficulties in maintaining citizens’ engagement in interventions that had achieved the quick wins and were running smoothly. The panel mentioned that regularly relating to citizens how their input is being used and how it contributes to successful outcomes can be helpful in maintaining citizens’ interest. They felt that such transparency might also force organisations to actually use citizens’ input.

Guiding principle 8: Take into account both citizens’ and organisations’ motivations

Organisations should enable citizens to participate in activities and projects that truly interest and motivate them, instead of channelling their participation to other projects [ 11 , 12 , 16 , 17 , 36 , 38 , 39 , 42 ]. The community centre evaluated by Hamamoto et al. [ 11 ], for example, enabled citizens to solely provide input into those projects, which truly interested them, which meant that citizens remained engaged for the entire length of their specific project. As Van Eijk & Steen [ 38 ] argued, citizens cannot pay attention to every topic and are often engaged in an ad hoc manner, contingent on specific problems. For example, their study of a mental healthcare provider’s Client Council showed that citizens’ motivations for joining the Council were mostly personal—e.g. because they were a service-user or the carer of a service-user and wanted more social interactions with others in a similar situation. The implication was that as soon as that personal connection disappeared, their commitment to the Council decreased. In Lewis’ [ 17 ] study concerning a policy and planning committee operating within a hierarchical organisation (context) , service-users’ own negative mental health service-usage experiences motivated them to take part in the committee with the aim of improving mental health services and to find solidarity with others (mechanism). However, because the committee was ineffective in addressing the poor quality standards which were the cause of service-users’ negative experiences, the citizens felt they would be better off forming their own forms of active citizenship relating to mental health services (mechanism) . Eventually the service-users did split off from the committee and set up, for example, a mental health charity and a mental health social firm (outcome) [ 17 ]. Such examples show that organisations should be transparent about the problems the organisation is facing and about their own motivations, especially if it is their intention to make cost-savings, and listen openly to citizens’ negative experiences. Aligning motivations can enhance personal citizens’ personal connections with services and can enable longer-term collaboration between citizens and organisations.

The panel recognised the importance of aligning CE interventions with citizens’ own interests and motivations. In their experience, for example, citizens are less interested in CE initiatives focusing on an entire municipality; while initiatives centred on their local neighbourhood, attract more input from citizens.

As far as the authors are aware, this is the first review to develop guiding principles for the successful implementation of community engagement interventions. Using the realist methodology, the rapid review identified eight guiding principles and highlighted the different enabling and constraining contextual factors and mechanisms, which influence the effectiveness of CE interventions. The literature findings, which resonated with the Dutch local reference panel, provide policymakers and practice leaders with an understanding of the key principles, which promote the engagement of citizens in the health and care setting. The aim of this information is to enable professionals to implement their own effective CE interventions.

While this review has not examined the interactions between the eight guiding principles, they appear interrelated. For example, those in leadership positions play an important role in ensuring CE interventions are enacted in a safe and trusting environment for citizens, which in turn seems tied into the power imbalances between citizens and organisations. Future studies could examine the nature and extent of the principles’ interactions and how these can be used to reach more ‘meaningful participation’—for example by investigating ‘ripple effect mechanisms’ [ 43 ].

Even without the examination of principles’ interrelatedness, it is clear that the existence of power imbalances and organisations’ willingness, or not, to address such imbalances, is an especially important thread throughout the principles. The literature suggests that ‘meaningful participation’ of citizens can only be achieved if organisational processes are adapted to ensure they are inclusive, accessible and supportive of citizens, for example by placing citizens in decision-making and leadership positions and providing relevant learning opportunities [ 16 , 32 , 33 , 34 ]. This holds especially true for interventions seeking to engage communities with lower levels of capacity and higher levels of deprivation. In such cases, organisations should first invest significant time and resources in developing positive and trusting relationships with communities [ 10 , 16 , 40 , 42 ]. However, the literature contains more examples of how failing to build more equal organisational structures results in worsening relationships and the deterioration of citizens’ empowerment. Studies like Carlisle (2010), Lewis [ 17 ] and Renedo & Marston [ 18 ] have shown that even though organisations implement CE interventions, ostensibly with the aim of involving citizens more deeply in their organisation, professionals continue to maintain their ‘business as usual’ approach.

Future studies will be needed to continue broadening our understanding of CE. Firstly, it remains unclear why professionals and organisations implement CE interventions, but then ‘maintain their business as usual’ approach. Renedo & Marston [ 18 ] suggest part of the problem lies in the professional discourse around citizen engagement, but new evaluations could question wider aspects of this problem by investigating other underlying mechanisms and contextual factors, which prevent organisations from fully adapting their processes and structures. As CE is expected to bring a wider range of services together specifically around citizens’ views and needs, new studies could investigate, for example, how service fragmentation and funding competition hamper professionals’ willingness to truly take on board citizens’ more holistic and potentially remit-transcending views. Some of the studies included in this review indicate that service fragmentation and a lack of funding aggravate uncollaborative citizen-professional relationships, especially if professionals place an emphasis on the self-sustainability of (marginalised) communities [ 31 , 9 , 33 ]. Secondly, while there have been some studies highlighting how CE interventions which address power imbalances can tailor specific health and care services or local neighbourhoods to citizens’ needs [ 32 , 33 , 34 , 39 ]; little is known about whether CE actually enables the implementation of new collaborative models of care centred on citizens’ preferences. Finally, too few evaluations have investigated interventions involving low-income or ethnically diverse communities. There are even fewer studies focusing on other vulnerable or disadvantaged groups like the frail elderly, LGBTQ citizens, or less abled citizens [ 13 , 44 , 45 , 46 ]. This could partly be because not many CE interventions with such target groups have been implemented, in which case new studies could examine why that is the case—perhaps the reason lies partly in the context of the disadvantage and marginalisation such groups face. Because contexts of power imbalances, marginalisation and discrimination are hugely influential, such studies will be key to ensuring our understanding of CE is more inclusive and complete and can be tailored more closely to different citizens’ needs. As the local reference panel pointed out, citizens are not one homogenous group with the same needs, priorities and preferences.

Studies using a wider-range of quantitative methodologies and those reporting on the more negative results or aspects of the studied CE interventions would help close such gaps. To date, most CE evaluations have been qualitative and based on case studies and have not explicitly discussed the studies’ negative results. These case studies have provided rich anecdotal evidence, but to further develop our understanding of which CE interventions work or not, for whom, how, in which contexts and to what extent, new studies should use mixed-methods in order to quantify findings, thus providing a richer evidence-base. The authors will attempt to address such remaining gaps in the multiple-case study going forward, using the principles and the underlying CMOs as the initial programme theories.

On a separate note, though the focus of this paper was not the application of the realist methodology, important questions arose during the review’s data analysis stage. The first issue relates to theory development using CMO configurations. While most realist papers clearly highlight that CMO configurations were key to the development of the theories under discussion, most do not actually describe how the CMO configurations then led to those theories [ 47 ]. Within the papers that do describe this analytical process, there seems to be no consistency as to whether the theories are centred on the contexts, mechanisms, or outcomes of the configurations. For example, previous evaluators have put interventions as related to outcomes central [ 21 , 23 ], others have placed only outcomes in the limelight (e.g. [ 48 ]) or outcomes and contexts [ 49 ], and similarly to Kane et al. [ 50 ], we saw mechanisms as critical for our guiding principles. The methodology’s inherent flexibility brings many, creative, benefits, however, it also raises important questions regarding the generation of results. For example, it is currently not clear whether we would have drawn the same conclusions if we had chosen context or outcomes as the core of this review’s analysis. For example, due to the review’s focus on mechanisms—i.e. what makes citizens or communities want to participate or not—and our aim of providing policymakers and professionals with the evidence to implement their own effective CE strategies—the outcomes within our individual CMO configurations often relate to citizens’ or communities’ behavioural changes and the impact on organisational processes, rather than say the impact on citizens’ health and wellbeing. However, the local reference panel’s valuable input indicates that our results have face validity.

Relatedly, the methodology’s flexibility and dynamic nature is again one of its key strengths as it provides rich and detailed information, partly because of its recognition that interventions and their contexts are complex and varying. However, there is a tension between the recognition that all contexts are in a way unique, and the generalisability of the results. This tension is only partly addressed by searching for the same mechanisms and outcomes in different contexts. Ultimately, if the methodology is to continue to evolve and improve, realist evaluators should not only be transparent about how they constructed CMOs and generated theories, but also why they choose that specific approach and endeavour to show that the results are indeed generalisable across different contexts and care settings.

Study limitations

This study has two main limitations. Firstly, though this rapid review’s literature search was systematic, it was by no means exhaustive in a conscious effort to speed up the process and to share the findings as quickly as possible with stakeholders. This limitation has been mitigated by collaborating with the local reference panel to confirm and supplement the findings. Secondly, while the realist methodology is helpful in uncovering multifaceted and complex issues like power imbalances in CE, the methodology is still developing, which means that key concepts are not always understood or applied in the same manner. Other researchers may therefore find it difficult to build on this review’s findings, however, the authors have attempted to address this limitation by clearly stipulating the applied understandings of key concepts and describing, in detail, how and why the CMOs and principles were developed.

By highlighting the contextual factors and mechanisms, which can influence the outcome of CE interventions, the eight guiding principles can hopefully guide professionals to develop their own successful interventions. While the principles are based on a wide range of contextual factors, professionals are encouraged to interpret and adapt the findings to the contexts of their own local settings and explore which activities and mechanisms would lead to the most inclusive and diverse CE interventions. Organisations should pay specific attention to sources of contextual power imbalances and find the most appropriate ways to empower, motivate and upskill citizens so they may take shared control of initiatives.

Abbreviations

• Community engagement

Context-mechanism-outcome configurations

Action-oriented guiding principles

Organisation for Economic Co-operation and Development. Fiscal Sustainability of Health Systems: bridging health and finance perspectives. http://www.oecd.org/publications/fiscal-sustainability-of-health-systems-9789264233386-en.htm . Accessed 16 Feb 2017.

Drewes HW, Heijmnk R, Struijs JN, et al. Samen werken aan duurzame zorg: Landelijke monitor proeftuinen:. http://www.rivm.nl/Documenten_en_publicaties/Wetenschappelijk/Rapporten/2015/juli/Samen_werken_aan_duurzame_zorg_Landelijke_monitor_proeftuinen. . Accessed 16 Feb 2017.

Iacobucci G. NHS plan calls for new models of care and greater emphasis on prevention. BMJ Open. 2015; www.bmj.com/content/349/bmj.g6430

Shortell SM, Wu FM, Lewis VA, et al. A taxonomy of accountable care organizations for policy and practice. Health Serv Res. 2014;49(6):1883–99.

PubMed   PubMed Central   Google Scholar  

Drewes HW, Struijs JN, Baan C.A. How the Netherlands is Integrating Health and Community Services. https://catalyst.nejm.org/netherlandsintegrating-health-community-services/ . Accessed 16 Feb 2017.

Iroz-Elardo N. Health impact assessment as community participation. Community Development Journal. 2015;50(2):280–95.

Article   Google Scholar  

O’Neill K, Williams KJ, Reznik V. Engaging Latino residents to build a healthier community in mid-City san Diego. Am J Prev Med. 2008;34(3):S36–41.

Article   PubMed   Google Scholar  

Milton B, Attree P, French B, et al. The impact of community engagement on health and social outcomes: a systematic review. Community Development Journal. 2011;47(3):316–34.

Carlisle S. Tackling health inequalities and social exclusion through partnership and community engagement? A reality check for policy and practice aspirations from a social inclusion Partnership in Scotland. Critical Public Health. 2008;20(1):117–27.

Crondahl K, Eklund KL. Roma empowerment and social inclusion through work-integrated learning. Sage Open. 2015:1–10. http://journals.sagepub.com/doi/pdf/10.1177/2158244015572275 .

Hamamoto MH, Deraug DD, Yoshimura BS. Building the base: two active living projects that inspired community participation. Am J Prev Med. 2009;37(6S2):S345–51.

Lang R, Roessl D, Weismeier-Sammer D. Co-operative Governance of Public–Citizen Partnerships: Two Diametrical Participation Modes. In: Gnan L, Hinna A, Monteduro F, editors. Conceptualizing and Researching Governance in Public and Non-Profit Organizations (Studies in Public and Non-Profit Governance, Volume 1). Emerald Group Publishing Limited. p. 227-46. https://www.emeraldinsight.com/doi/abs/10.1108/S2051-6630%282013%290000001013 .

Cyril S, Smith BJ, Possamai-Inesedy A. Exploring the role of community engagement in improving the health of disadvantaged populations: a systematic review. Glob Health Action. 2015;8(1):29842.

Dalton J, Chambers D, Harden M. Service user engagement and health service reconfiguration: a rapid evidence synthesis. Health Serv Delivery Res. 2015;21(3):195–205.

Kenny A, Hyett N, Sawtell J, et al. Community participation in rural health: a scoping review. BMC Health Serv Res. 2013;13:64.

Article   PubMed   PubMed Central   Google Scholar  

De Freitas C, Martin G. Inclusive public participation in health: policy, practice and theoretical contributions to promote the involvement of marginalised groups in healthcare. Soc Sci Med. 2015;135:31–9.

Lewis L. User involvement in mental health services: a case of power over discourse. Sociol Res Online. 2014;19:1.

Renedo A, Marston C. Healthcare professionals’ representations of ‘patient and public involvement’ and creation of ‘public participant’ identities: implications for the development of inclusive and bottom-up community participation initiatives. J Community Applied Psychol. 2011;21(3):268–80.

Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997.

Google Scholar  

Khangura S, Konnyu K, Cushman R, et al. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10.

Saul JE, Willis CD, Bitz J, et al. A time-responsive tool for informing policy making: rapid realist review. Implement Sci. 2013;8:103.

Stolee P, Elliott J, McNeil H. Choosing healthcare options by involving Canada’s elderly: a protocol for the CHOICE realist synthesis project on engaging older persons in healthcare decision making. BMJ Open. 2015;5:11.

Willis CD, Saul J, Bevan H, et al. Sustaining organizational culture change in health systems. J Health Organ Manag. 2014;30(1):2–30.

O’Mara-Eves A, Brunton G, McDaid D. Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis. Public Health Res. 2013;1:4.

Pluye P., Robert E., Cargo M., et al. Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. http://mixedmethodsappraisaltoolpublic.pbworks.com . Accessed 24 Jan 2017, 2015.

Rowe G, Frewer LJ. A typology of public engagement mechanisms. Sci Technol Hum Values. 2005;30(2):251–90.

Lacouture A, Brenton E, Guichard A, et al. The concept of mechanisms from a realist approach: a scoping review to facilitate its operationalization in public health program evaluation. Implement Sci. 2015;10:153.

Jagosh J, Pluye P, Macaulay C, et al. Assessing the outcomes of participatory research: protocol for identifying, selecting, appraising and synthesizing the literature for realist review. Implement Sci. 2011;6:24.

MacFarlane F, Greenhalgh T, Hughes HC, et al. A new workforce in the making? A case study of strategic human resource management in a whole-system change effort in healthcare. J Health Organ Manag. 2011;25(1):55–72.

Chan YE, Benecki LA. Evaluating the success of a Hospital’s community engagement process. Health Care Management Forum. 2013;26(1):20–5.

Clark N, Lachance L, Doctor LJ, et al. Policy and system change and community coalitions: outcomes from allies against asthma. Am J Public Health. 2010;100(5):904–12.

Durey A, McEnvoy S, Swift-Otero V. Improving healthcare for Aboriginal Australians through effective engagement between community and health services. BMC Health Serv Res. 2016;16:224.

Kelaher M, Sabanovic H, La Brooy C, et al. Does more equitable governance lead to more equitable health care? A case study based on the implementation of health reform in aboriginal health Australia. Soc Sci Med. 2014;123:278–86.

Luluquisen M, Pettis L. Community engagement for policy and systems change. Community Development. 2014;45(3):252–62.

Montesanti SR, Abelson J, Lavis JN, et al. The value of frameworks as knowledge translation mechanisms to guide community participation practice in Ontario CHCs. Soc Sci Med. 2015;142:223–31.

Pennel CL, McLeroy KR, Burdine JN, et al. A mixed-methods approach to understanding community participation in community health needs assessments. J Public Health Manag Pract. 2015;23(2):112.

Tenbensel T, Cummings J, Ashton T, et al. Where there’s a will, is there a way? Is New Zealand’s publicly funded health sector able to steer towards population health? Soc Sci Med. 2008;67(7):1143–52.

Van Eijk C, Steen T. Why engage in co-production of public services? Mixing theory and empirical evidence. Int Rev Adm Sci. 2016;82(1):28–46.

Veronesi G, Keasey K. Patient and public participation in the English NHS: an assessment of experimental implementation processes. Public Manag Rev. 2015;17:4.

Yoo S, Butler J, Elias TI. The 6-step model for community empowerment: revisited in public housing communities for low-income senior citizens. Health Promot Pract. 2008;10(2):262–75.

Kegler MC, Ellenberg PJ, Twiss JM. Evaluation findings on community participation in the California healthy cities and communities program. Health Promot Int. 2009;24(4):300–10.

Schoch-Spana M, Kirk ST, Morhard R. Local health department capacity for community engagement and its implications for disaster resilience. Biosecur Bioterror. 2013;1(2):118–29.

Jagosh J, Bush PL, Salsberg J. A realist evaluation of community-based participatory research: partnership synergy, trust building and related ripple effects. BMC Public Health. 2015;15:725.

Holroyd-Leduc J, Resin J, Ashley L, et al. Giving voice to older adults living with frailty and their family caregivers: engagement of older adults living with frailty in research, health care decision making, and in health policy. Res Involv Engagem. 2016;2:223.

Nieder TO, Strauss BM. Transgender health care in Germany: participatory approaches and the development of a guideline. Int Rev Psychiatry. 2015;27(5):416–26.

Rifkin SB. Examining the links between community participation and health outcomes: a review of the literature. Health Policy Plan. 2014;29(2):98–106.

Marchal B, van Belle S, Hoerée T, et al. Is realist evaluation keeping its promise? A review of the published empirical studies in the field of health systems research. Evaluation. 2012;18(2):192–212.

Jagosh J, Bush PL, Salberg J, et al. A realist evaluation of community-based participatory research: partnership, synergy, trust building and ripple effects. BMC Public Health. 2012;15:725.

MacDonald M, Pauly B, Wong G, et al. Supporting successful implementation of public health interventions: protocol for a realist synthesis. Syst Rev. 2016;5:54.

Kane SS, Gerretsen B, Scherpbier R, et al. A realist synthesis of randomised controlled trials involving use of community health workers for delivering child health interventions in low and middle income countries. BMC Health Serv Res. 2010;10:286.

Download references

Acknowledgements

The authors wish to acknowledge and thank the local reference panel for their valuable time, insight and guidance.

Availability of data and materials

All data generated and analysed during this study are included in the published article and supplementary information files. Templates used for data extraction and analysis are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Department of Quality of Care and Health Economics, Centre for Nutrition, Prevent and Health Services, National Institute for Health and the Environment (RIVM), P.O. Box 1, 3720 BA, Bilthoven, The Netherlands

E. De Weger, N. Van Vooren, C. A. Baan & H. W. Drewes

Tilburg University, Tranzo, Tilburg School of Social and Behavioural Sciences, PO Box 90153, 5000 LE, Tilburg, The Netherlands

E. De Weger, K. G. Luijkx & C. A. Baan

You can also search for this author in PubMed   Google Scholar

Contributions

Research design was developed by EdW and reviewed and approved by HD, KgL and CB EdW and NvV acquired, analysed and interpreted data. EdW wrote the paper and HD, KgL and CB critically reviewed all drafts and the final copy. All authors made substantial contributions to conception and design and read and approved the final manuscript

Corresponding author

Correspondence to E. De Weger .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was not required as data was retrieved through a literature review. The local reference panel was verbally informed of the fact that anonymous meeting notes (e.g. without any person-identifiable information, including initials) would be taken with the sole purpose of refining the literature review data. Each member of the local reference panel provided verbal consent. This is in compliance with Dutch national guidelines: https://www.tilburguniversity.edu/upload/ddc3ce11-1e82-4bf7-ac6d-e813999e5037_CODE%20OF%20ETHICS%20FOR%20RESEARCH%20IN%20THE%20SOCIAL%20AND%20BEHAVIOURAL%20SCIENCES%20DSW%20J%20%20%20.pdf and http://ec.europa.eu/research/participants/data/ref/fp7/89867/social-sciences-humanities_en.pdf .

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Search strings. (DOCX 32 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

De Weger, E., Van Vooren, N., Luijkx, K.G. et al. Achieving successful community engagement: a rapid realist review. BMC Health Serv Res 18 , 285 (2018). https://doi.org/10.1186/s12913-018-3090-1

Download citation

Received : 13 November 2017

Accepted : 03 April 2018

Published : 13 April 2018

DOI : https://doi.org/10.1186/s12913-018-3090-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Citizen engagement
• Community participation
• Rapid realist review
• Realist evaluation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Search by keyword
• Search by citation

Page 1 of 501

Living in urban or rural environments affect the sleep quality of the elderly in Bushehr (Southern Iran): emphasizing the active and inactive of the elderly

Sleep disorders have a significant negative impact on mental and physical health, especially among the elderly. Various factors can affect the sleep quality of elderly people. The aim of this research to inves...

• View Full Text

Early-life tobacco smoke exposure and stroke risk: a prospective study of 341,783 and 352,737 UK Biobank participants

Stroke is a life-threatening condition that causes a major medical burden globally. The currently used methods for the prevention or prediction of stroke have certain limitations. Exposure to tobacco in early ...

“That was one of my most difficult and biggest challenges”: experiences, preconditions and preventive measures of health-oriented leadership in virtual teams – A qualitative study with virtual leaders

Health-oriented leadership (HoL) has a positive impact on health- and work-related outcomes of employees in face-to-face settings. Increased digitization during the COVID-19 pandemic has led to many changes an...

Qualitative multi-stakeholder evaluation of the adoption, implementation and sustainment of the school-based dietary intervention “Jump-in”

Comprehensive school-based programs applying the WHO Health Promoting School Model have the potential to initiate and sustain behavior change and impact health. However, since they often include intervention e...

How Ohio public library systems respond to opioid-related substance use: a descriptive analysis of survey results

Public libraries in the United States have experienced increases in opioid-related substance use in their communities and on their premises. This includes fatal and non-fatal overdose events. Some libraries ha...

Association of longitudinal trajectories of fasting plasma glucose with all-cause and cardiovascular mortality among a Chinese older population: a retrospective cohort study

The association between fasting plasma glucose (FPG), an important indicator of overall glycemic status, and the risk of cardiovascular mortality has been well investigated. The longitudinal study can repeated...

Assessing family function: older adults vs. care nurses: a cross-sectional comparative study

This study aimed to assess family function in home care for older adults. Understanding family dynamics is essential for providing quality care to older adults choosing to age in place.

The effects of meteorological factors and air pollutants on the incidence of tuberculosis in people living with HIV/AIDS in subtropical Guangxi, China

Previous studies have shown the association between tuberculosis (TB) and meteorological factors/air pollutants. However, little information is available for people living with HIV/AIDS (PLWHA), who are highly...

The impact of healthy nutrition education based on traffic light labels on food selection, preference, and consumption in patients with acute coronary syndrome: a randomized clinical trial

Acute Coronary Syndrome is the most common heart disease and the most significant cause of death and disability-adjusted life years worldwide. Teaching a healthy eating style is one preventive measure to preve...

Development and psychometric properties of an instrument to measure perception of aphrodisiac use among undergraduates in a southwestern Nigerian university

A tool to measure perception of aphrodisiac use by undergraduates students of University of Ibadan, Oyo State, Nigeria was developed and validated. The study was hinged on several theories that could explain p...

The epidemiology of major depression among adults in Norway: an observational study on the concurrence between population surveys and registry data – a NCDNOR project

Mental health problems, and major depression in particular, are important public health issues. Following trends in the prevalence of major depression is difficult because of the costs and complications of dia...

Factors associated with the co-utilization of oral rehydration solution and zinc for treating diarrhea among under-five children in 35 sub-saharan Africa countries: a generalized linear mixed effect modeling with robust error variance

Even though childhood diarrhea is treated with a simple treatment solution, it continues to be one of the leading causes of under-five child mortality and malnutrition globally. In resource-limited settings su...

Heated tobacco products- well known or well understood? A national cross-sectional study on knowledge, attitudes and usage in Pakistan

Heated tobacco products (HTPs) are reshaping the tobacco industry and just recently, a plan was sought to regularize HTPs in Pakistan. Pakistan provides an intriguing case study in this context, as tobacco use...

Process evaluation of a pragmatic feasibility trial on smokeless tobacco cessation intervention delivered in dental hospitals

Article 14 of the WHO ‘Framework Convention on Tobacco Control’ recommends, that all oral healthcare providers provide support for tobacco cessation, to all patients. Despite evidence on the effectiveness of t...

Regional and rural-urban patterns in the prevalence of diagnosed hypertension among older U.S. adults with diabetes, 2005–2017

Hypertension prevalence among the overall US adult population has been relatively stable during the last two decades. However, whether this stabilization has occurred across rural-urban communities and across ...

Discontinuation of HIV oral pre-exposure prophylaxis: findings from programmatic surveillance within two general population HIV programs in Nigeria

As oral PrEP scales up in Nigeria, information about uptake, use pattern and client preference in a real-world, implementation setting is invaluable to guide refining service provision and incorporation of ora...

Characteristics of children readmitted with severe pneumonia in Kenyan hospitals

Pneumonia is a leading cause of childhood morbidity and mortality. Hospital re-admission may signify missed opportunities for care or undiagnosed comorbidities.

Association between sleep patterns and galectin-3 in a Chinese community population

Irregular sleep patterns have been associated with inflammation. Galectin-3, a novel biomarker, plays an important role in inflammation. We investigated the relationship between sleep patterns and galectin-3 i...

Temporal change in prevalence of BMI categories in India: patterns across States and Union territories of India, 1999–2021

The problem of overweight/obesity often coexists with the burden of undernutrition in most low- and middle-income countries. BMI change in India incorporating the most recent trends has been under-researched.

Unlocking the potential: exploring the impact of dolutegravir treatment on body mass index improvement in underweight adults with HIV in Malawi

The introduction of dolutegravir (DTG) in treating HIV has shown enhanced efficacy and tolerability. This study examined changes in weight gain and body mass index (BMI) at 6- and 12-months after post-initiati...

Sociodemographic determinants of vaccination and willingness to pay for COVID-19 vaccines in Hungary, results of a cross-sectional online questionnaire

Several different coronavirus disease (COVID-19) vaccines were authorized and distributed all over the world, including Hungary, but vaccination rates and acceptance of the different vaccines varied through 20...

Exploring the relationship between dietary patterns and obesity among Nigerian adults: a cross-sectional study

No previous study has investigated the association between dietary pattern and both general and abdominal obesity risk among adults in Nigeria. This study aimed to evaluate the associations between dietary pat...

Correction: Prognostic factors of time to first abortion after sexual debut among fragile state Congolese women: a survival analysis

The original article was published in BMC Public Health 2021 21 :525

The effects of trade liberalization on inequality in nutrition intake: empirical evidence from Indian districts

Despite the positive impact of trade liberalization on food availability in India, severe inequality in nutrition consumption at the district level persists. Empirical evidence on the relationship between trad...

Financial inclusion and improved water usage among households in Ghana

In Ghana, about 76% of households are at risk of drinking water polluted with faecal matter, hence, poor sanitation and unsafe water are responsible for 80% of all diseases in the country. Given this, some stu...

How do labour market conditions explain the development of mental health over the life-course? a conceptual integration of the ecological model with life-course epidemiology in an integrative review of results from the Northern Swedish Cohort

The aim of this study was to contribute to the theoretical development within the field of labour market effects on mental health during life by integrating Bronfenbrenner’s ecological model with mainly earlie...

Gender differences in attitudes towards psychological help-seeking among chinese medical students: a comparative analysis

Medical students are known to be at a greater risk of psychological disorders compared to the general population. However, their rate of help-seeking behavior is low. The purpose of this study was to explore t...

Cultural and co-designed principles for developing a Māori kaumātua housing village to address health and social wellbeing

The current study is a case study of a Māori (Indigenous people of New Zealand) organisation and their developmental processes in creating a kaumātua (older people) housing village for health and social wellbe...

Characteristics and spending patterns of high-cost child patients: findings from Fujian in China

The health condition during childhood has been shown to influence an individual’s health and socioeconomic status in adulthood. Understanding the concentration and persistence patterns in children’s healthcare...

Correction to: Burden of colorectal cancer and its risk factors in the North Africa and Middle East (NAME) region, 1990–2019: a systematic analysis of the global burden of disease study

The original article was published in BMC Public Health 2024 24 :557

Social media influence on COVID-19 vaccine perceptions among University students: a Malawi case study

The global fight against the COVID-19 pandemic relies significantly on vaccination. The collective international effort has been massive, but the pace of vaccination finds hindrance due to supply and vaccine h...

Knowledge and attitude towards home quarantine instructions and associations with history of Covid-19 infection in Malaysia

Although COVID-19 has entered the endemic phase, individuals infected with COVID-19 are required to adhere to home quarantine measures. By exploring the public’s knowledge and attitude towards recommended home...

Knowledge, attitude, and practice toward allergic rhinitis among parents in Ningbo, China

This study examined the knowledge, attitude, and practice (KAP) toward allergic rhinitis (AR) among parents.

The usage and costs of national drug price-negotiated anticancer medicines in a first-tier city in Northeast China: a study based on health insurance data

The National Drug Price Negotiation (NDPN) policy has entered a normalisation stage, aiming to alleviate, to some extent, the disease-related and economic burdens experienced by cancer patients. This study ana...

Exploring predictors and prevalence of postpartum depression among mothers: Multinational study

Postpartum depression (PPD) affects around 10% of women, or 1 in 7 women, after giving birth. Undiagnosed PPD was observed among 50% of mothers. PPD has an unfavorable relationship with women’s functioning, ma...

New sexually transmitted HIV infections from 2016 to 2050 in Guangdong Province, China: a study based on a dynamic compartmental model

In Guangdong Province, China, there is lack of information on the HIV epidemic among high-risk groups and the general population, particularly in relation to sexual transmission, which is a predominant route. ...

Impacts of an abbreviated personal agency training with refugee women and their male partners on economic empowerment, gender-based violence, and mental health: a randomized controlled trial in Rwanda

We assessed the impact of a personal agency-based training for refugee women and their male partners on their economic and social empowerment, rates of intimate partner violence (IPV), and non-partner violence...

Association between social capital and utilization of essential public health services among elderly migrants: a multilevel logistic study based on the 2017 China migrant dynamic survey (CMDS)

As the number of elderly migrants in China continues to grow, it is necessary to pay closer attention to their health and health services. Some studies have confirmed that social capital plays a significant ro...

Trends of physical activity and recreational screen time among Chinese children and adolescents: a national study from 2017 to 2019

The prevalence of physical inactivity and sedentary behavior among children and adolescents is a growing public health concern. This study aims to examine the trends in Physical Activity (PA) and Recreational ...

A community based intervention to modify preventive behaviors of cutaneous leishmaniasis in children: a randomized controlled trial based on PRECEDE PROCEED model

Iran ranks among the top six countries globally with a significant incidence of Cutaneous Leishmaniasis (CL). Using planning models is one community-based intervention to promote preventive behaviors. The purp...

Unused, expired pharmaceuticals and their disposal practices among the general public in Burdur-Türkiye: a cross-sectional study

Unused pharmaceuticals are currently a public health problem. This study aimed to identify unused pharmaceuticals, research practices about the disposal methods, classify the medicines according to Anatomical ...

Prevalence trends of anemia impairment in adolescents and young adults with HIV/AIDS

Anemia is a common complication of HIV/AIDS, particularly in adolescents and young adults across various countries and regions. However, little is known about the changing prevalence trends of anemia impairmen...

Associations of obesity with chronic inflammatory airway diseases and mortality in adults: a population-based investigation

The association between obesity and respiratory diseases has been confirmed. However, few studies have reported the relationship between obesity and the risk and mortality of chronic inflammatory airway diseas...

How did the use of the social marketing approach in Egyptian communities succeed in improving breastfeeding practices and infants’ growth?

Improving breastfeeding practices does not always link to interventions relying only on improving nutrition awareness and education but needs cultural and behavioral insights .

Navigating health challenges: the interplay between occupation-imposed movement restrictions, healthcare access, and community resilience

Transportation plays a significant role in health, community resilience, and access to basic needs such as healthcare, social services, education, and job opportunities. Health and community resilience are, ho...

Socioeconomic determinants of myalgic encephalomyelitis/chronic fatigue syndrome in Norway: a registry study

Previous research has shown that socioeconomic status (SES) is a strong predictor of chronic disease. However, to the best of our knowledge, there has been no studies of how SES affects the risk of Myalgic enc...

Analysis of the current situation and factors influencing bullying in junior high schools in backward areas of Western, China & A case study of Qingyang City in Gasu

Qingyang is located in the northwest of China. By analyzing the current situation and risk factors of bullying in junior high schools in Qingyang City, and identify relevant data for formulating prevention and...

Predictive effects of organizational justice on job satisfaction in bus drivers: the moderating effects of role overload and proactive personality

There have been few longitudinal studies on Chinese bus drivers and the individual differences in the relationships between organizational justice and job satisfaction. This study examined the organizational j...

Nonpharmacological intervention therapies for dementia: potential break-even intervention price and savings for selected risk factors in the European healthcare system

New effective treatments for dementia are lacking, and early prevention focusing on risk factors of dementia is important. Non-pharmacological intervention therapies aimed at these factors may provide a valuab...

An investigation into the acceptability, adoption, appropriateness, feasibility, and fidelity of implementation strategies for birth companionship in Tehran: a qualitative inquiry on mitigating mistreatment of women during childbirth

A birth companion is a powerful mechanism for preventing mistreatment during childbirth and is a key component of respectful maternity care (RMC). Despite a growing body of evidence supporting the benefits of ...

Important information

Editorial board

For authors

For editorial board members

For reviewers

• Manuscript editing services

Annual Journal Metrics

2022 Citation Impact 4.5 - 2-year Impact Factor 4.7 - 5-year Impact Factor 1.661 - SNIP (Source Normalized Impact per Paper) 1.307 - SJR (SCImago Journal Rank)

2023 Speed 32 days submission to first editorial decision for all manuscripts (Median) 173 days submission to accept (Median)

2023 Usage  24,332,405 downloads 24,308 Altmetric mentions 

• More about our metrics

Peer-review Terminology

The following summary describes the peer review process for this journal:

Identity transparency: Single anonymized

Reviewer interacts with: Editor

Review information published: Review reports. Reviewer Identities reviewer opt in. Author/reviewer communication

More information is available here

• Follow us on Twitter

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 13 May 2024

What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review

• Olivia R. Phillips 1 , 2   na1 ,
• Cerian Harries 2 , 3   na1 ,
• Jo Leonardi-Bee 1 , 2 , 4   na1 ,
• Holly Knight 1 , 2 ,
• Lauren B. Sherar 2 , 3 ,
• Veronica Varela-Mato 2 , 3 &
• Joanne R. Morling 1 , 2 , 5  

Research Involvement and Engagement volume  10 , Article number:  48 ( 2024 ) Cite this article

103 Accesses

2 Altmetric

Metrics details

There is increasing interest in using patient and public involvement (PPI) in research to improve the quality of healthcare. Ordinarily, traditional methods have been used such as interviews or focus groups. However, these methods tend to engage a similar demographic of people. Thus, creative methods are being developed to involve patients for whom traditional methods are inaccessible or non-engaging.

To determine the strengths and limitations to using creative PPI methods in health and social care research.

Electronic searches were conducted over five databases on 14th April 2023 (Web of Science, PubMed, ASSIA, CINAHL, Cochrane Library). Studies that involved traditional, non-creative PPI methods were excluded. Creative PPI methods were used to engage with people as research advisors, rather than study participants. Only primary data published in English from 2009 were accepted. Title, abstract and full text screening was undertaken by two independent reviewers before inductive thematic analysis was used to generate themes.

Twelve papers met the inclusion criteria. The creative methods used included songs, poems, drawings, photograph elicitation, drama performance, visualisations, social media, photography, prototype development, cultural animation, card sorting and persona development. Analysis identified four limitations and five strengths to the creative approaches. Limitations included the time and resource intensive nature of creative PPI, the lack of generalisation to wider populations and ethical issues. External factors, such as the lack of infrastructure to support creative PPI, also affected their implementation. Strengths included the disruption of power hierarchies and the creation of a safe space for people to express mundane or “taboo” topics. Creative methods are also engaging, inclusive of people who struggle to participate in traditional PPI and can also be cost and time efficient.

‘Creative PPI’ is an umbrella term encapsulating many different methods of engagement and there are strengths and limitations to each. The choice of which should be determined by the aims and requirements of the research, as well as the characteristics of the PPI group and practical limitations. Creative PPI can be advantageous over more traditional methods, however a hybrid approach could be considered to reap the benefits of both. Creative PPI methods are not widely used; however, this could change over time as PPI becomes embedded even more into research.

Plain English Summary

It is important that patients and public are included in the research process from initial brainstorming, through design to delivery. This is known as public and patient involvement (PPI). Their input means that research closely aligns with their wants and needs. Traditionally to get this input, interviews and group discussions are held, but this can exclude people who find these activities non-engaging or inaccessible, for example those with language challenges, learning disabilities or memory issues. Creative methods of PPI can overcome this. This is a broad term describing different (non-traditional) ways of engaging patients and public in research, such as through the use or art, animation or performance. This review investigated the reasons why creative approaches to PPI could be difficult (limitations) or helpful (strengths) in health and social care research. After searching 5 online databases, 12 studies were included in the review. PPI groups included adults, children and people with language and memory impairments. Creative methods included songs, poems, drawings, the use of photos and drama, visualisations, Facebook, creating prototypes, personas and card sorting. Limitations included the time, cost and effort associated with creative methods, the lack of application to other populations, ethical issues and buy-in from the wider research community. Strengths included the feeling of equality between academics and the public, creation of a safe space for people to express themselves, inclusivity, and that creative PPI can be cost and time efficient. Overall, this review suggests that creative PPI is worthwhile, however each method has its own strengths and limitations and the choice of which will depend on the research project, PPI group characteristics and other practical limitations, such as time and financial constraints.

Peer Review reports

Introduction

Patient and public involvement (PPI) is the term used to describe the partnership between patients (including caregivers, potential patients, healthcare users etc.) or the public (a community member with no known interest in the topic) with researchers. It describes research that is done “‘with’ or ‘by’ the public, rather than ‘to,’ ‘about’ or ‘for’ them” [ 1 ]. In 2009, it became a legislative requirement for certain health and social care organisations to include patients, families, carers and communities in not only the planning of health and social care services, but the commissioning, delivery and evaluation of them too [ 2 ]. For example, funding applications for the National Institute of Health and Care Research (NIHR), a UK funding body, mandates a demonstration of how researchers plan to include patients/service users, the public and carers at each stage of the project [ 3 ]. However, this should not simply be a tokenistic, tick-box exercise. PPI should help formulate initial ideas and should be an instrumental, continuous part of the research process. Input from PPI can provide unique insights not yet considered and can ensure that research and health services are closely aligned to the needs and requirements of service users PPI also generally makes research more relevant with clearer outcomes and impacts [ 4 ]. Although this review refers to both patients and the public using the umbrella term ‘PPI’, it is important to acknowledge that these are two different groups with different motivations, needs and interests when it comes to health research and service delivery [ 5 ].

Despite continuing recognition of the need of PPI to improve quality of healthcare, researchers have also recognised that there is no ‘one size fits all’ method for involving patients [ 4 ]. Traditionally, PPI methods invite people to take part in interviews or focus groups to facilitate discussion, or surveys and questionnaires. However, these can sometimes be inaccessible or non-engaging for certain populations. For example, someone with communication difficulties may find it difficult to engage in focus groups or interviews. If individuals lack the appropriate skills to interact in these types of scenarios, they cannot take advantage of the participation opportunities it can provide [ 6 ]. Creative methods, however, aim to resolve these issues. These are a relatively new concept whereby researchers use creative methods (e.g., artwork, animations, Lego), to make PPI more accessible and engaging for those whose voices would otherwise go unheard. They ensure that all populations can engage in research, regardless of their background or skills. Seminal work has previously been conducted in this area, which brought to light the use of creative methodologies in research. Leavy (2008) [ 7 ] discussed how traditional interviews had limits on what could be expressed due to their sterile, jargon-filled and formulaic structure, read by only a few specialised academics. It was this that called for more creative approaches, which included narrative enquiry, fiction-based research, poetry, music, dance, art, theatre, film and visual art. These practices, which can be used in any stage of the research cycle, supported greater empathy, self-reflection and longer-lasting learning experiences compared to interviews [ 7 ]. They also pushed traditional academic boundaries, which made the research accessible not only to researchers, but the public too. Leavy explains that there are similarities between arts-based approaches and scientific approaches: both attempts to investigate what it means to be human through exploration, and used together, these complimentary approaches can progress our understanding of the human experience [ 7 ]. Further, it is important to acknowledge the parallels and nuances between creative and inclusive methods of PPI. Although creative methods aim to be inclusive (this should underlie any PPI activity, whether creative or not), they do not incorporate all types of accessible, inclusive methodologies e.g., using sign language for people with hearing impairments or audio recordings for people who cannot read. Given that there was not enough scope to include an evaluation of all possible inclusive methodologies, this review will focus on creative methods of PPI only.

We aimed to conduct a qualitative systematic review to highlight the strengths of creative PPI in health and social care research, as well as the limitations, which might act as a barrier to their implementation. A qualitative systematic review “brings together research on a topic, systematically searching for research evidence from primary qualitative studies and drawing the findings together” [ 8 ]. This review can then advise researchers of the best practices when designing PPI.

Public involvement

The PHIRST-LIGHT Public Advisory Group (PAG) consists of a team of experienced public contributors with a diverse range of characteristics from across the UK. The PAG was involved in the initial question setting and study design for this review.

Search strategy

For the purpose of this review, the JBI approach for conducting qualitative systematic reviews was followed [ 9 ]. The search terms were (“creativ*” OR “innovat*” OR “authentic” OR “original” OR “inclu*”) AND (“public and patient involvement” OR “patient and public involvement” OR “public and patient involvement and engagement” OR “patient and public involvement and engagement” OR “PPI” OR “PPIE” OR “co-produc*” OR “co-creat*” OR “co-design*” OR “cooperat*” OR “co-operat*”). This search string was modified according to the requirements of each database. Papers were filtered by title, abstract and keywords (see Additional file 1 for search strings). The databases searched included Web of Science (WoS), PubMed, ASSIA and CINAHL. The Cochrane Library was also searched to identify relevant reviews which could lead to the identification of primary research. The search was conducted on 14/04/23. As our aim was to report on the use of creative PPI in research, rather than more generic public engagement, we used electronic databases of scholarly peer-reviewed literature, which represent a wide range of recognised databases. These identified studies published in general international journals (WoS, PubMed), those in social sciences journals (ASSIA), those in nursing and allied health journals (CINAHL), and trials of interventions (Cochrane Library).

Inclusion criteria

Only full-text, English language, primary research papers from 2009 to 2023 were included. This was the chosen timeframe as in 2009 the Health and Social Reform Act made it mandatory for certain Health and Social Care organisations to involve the public and patients in planning, delivering, and evaluating services [ 2 ]. Only creative methods of PPI were accepted, rather than traditional methods, such as interviews or focus groups. For the purposes of this paper, creative PPI included creative art or arts-based approaches (e.g., e.g. stories, songs, drama, drawing, painting, poetry, photography) to enhance engagement. Titles were related to health and social care and the creative PPI was used to engage with people as research advisors, not as study participants. Meta-analyses, conference abstracts, book chapters, commentaries and reviews were excluded. There were no limits concerning study location or the demographic characteristics of the PPI groups. Only qualitative data were accepted.

Quality appraisal

Quality appraisal using the Critical Appraisal Skills Programme (CASP) checklist [ 10 ] was conducted by the primary authors (ORP and CH). This was done independently, and discrepancies were discussed and resolved. If a consensus could not be reached, a third independent reviewer was consulted (JRM). The full list of quality appraisal questions can be found in Additional file 2 .

Data extraction

ORP extracted the study characteristics and a subset of these were checked by CH. Discrepancies were discussed and amendments made. Extracted data included author, title, location, year of publication, year study was carried out, research question/aim, creative methods used, number of participants, mean age, gender, ethnicity of participants, setting, limitations and strengths of creative PPI and main findings.

Data analysis

The included studies were analysed using inductive thematic analysis [ 11 ], where themes were determined by the data. The familiarisation stage took place during full-text reading of the included articles. Anything identified as a strength or limitation to creative PPI methods was extracted verbatim as an initial code and inputted into the data extraction Excel sheet. Similar codes were sorted into broader themes, either under ‘strengths’ or ‘limitations’ and reviewed. Themes were then assigned a name according to the codes.

The search yielded 9978 titles across the 5 databases: Web of Science (1480 results), PubMed (94 results), ASSIA (2454 results), CINAHL (5948 results) and Cochrane Library (2 results), resulting in 8553 different studies after deduplication. ORP and CH independently screened their titles and abstracts, excluding those that did not meet the criteria. After assessment, 12 studies were included (see Fig.  1 ).

PRISMA flowchart of the study selection process

Study characteristics

The included studies were published between 2018 and 2022. Seven were conducted in the UK [ 12 , 14 , 15 , 17 , 18 , 19 , 23 ], two in Canada [ 21 , 22 ], one in Australia [ 13 ], one in Norway [ 16 ] and one in Ireland [ 20 ]. The PPI activities occurred across various settings, including a school [ 12 ], social club [ 12 ], hospital [ 17 ], university [ 22 ], theatre [ 19 ], hotel [ 20 ], or online [ 15 , 21 ], however this information was omitted in 5 studies [ 13 , 14 , 16 , 18 , 23 ]. The number of people attending the PPI sessions varied, ranging from 6 to 289, however the majority (ten studies) had less than 70 participants [ 13 , 14 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. Seven studies did not provide information on the age or gender of the PPI groups. Of those that did, ages ranged from 8 to 76 and were mostly female. The ethnicities of the PPI group members were also rarely recorded (see Additional file 3 for data extraction table).

Types of creative methods

The type of creative methods used to engage the PPI groups were varied. These included songs, poems, drawings, photograph elicitation, drama performance, visualisations, Facebook, photography, prototype development, cultural animation, card sorting and creating personas (see Table  1 ). These were sometimes accompanied by traditional methods of PPI such as interviews and focus group discussions.

The 12 included studies were all deemed to be of good methodological quality, with scores ranging from 6/10 to 10/10 with the CASP critical appraisal tool [ 10 ] (Table  2 ).

Thematic analysis

Analysis identified four limitations and five strengths to creative PPI (see Fig.  2 ). Limitations included the time and resource intensity of creative PPI methods, its lack of generalisation, ethical issues and external factors. Strengths included the disruption of power hierarchies, the engaging and inclusive nature of the methods and their long-term cost and time efficiency. Creative PPI methods also allowed mundane and “taboo” topics to be discussed within a safe space.

Theme map of strengths and limitations

Limitations of creative PPI

Creative ppi methods are time and resource intensive.

The time and resource intensive nature of creative PPI methods is a limitation, most notably for the persona-scenario methodology. Valaitis et al. [ 22 ] used 14 persona-scenario workshops with 70 participants to co-design a healthcare intervention, which aimed to promote optimal aging in Canada. Using the persona method, pairs composed of patients, healthcare providers, community service providers and volunteers developed a fictional character which they believed represented an ‘end-user’ of the healthcare intervention. Due to the depth and richness of the data produced the authors reported that it was time consuming to analyse. Further, they commented that the amount of information was difficult to disseminate to scientific leads and present at team meetings. Additionally, to ensure the production of high-quality data, to probe for details and lead group discussion there was a need for highly skilled facilitators. The resource intensive nature of the creative co-production was also noted in a study using the persona scenario and creative worksheets to develop a prototype decision support tool for individuals with malignant pleural effusion [ 17 ]. With approximately 50 people, this was also likely to yield a high volume of data to consider.

To prepare materials for populations who cannot engage in traditional methods of PPI was also timely. Kearns et al. [ 18 ] developed a feedback questionnaire for people with aphasia to evaluate ICT-delivered rehabilitation. To ensure people could participate effectively, the resources used during the workshops, such as PowerPoints, online images and photographs, had to be aphasia-accessible, which was labour and time intensive. The author warned that this time commitment should not be underestimated.

There are further practical limitations to implementing creative PPI, such as the costs of materials for activities as well as hiring a space for workshops. For example, the included studies in this review utilised pens, paper, worksheets, laptops, arts and craft supplies and magazines and took place in venues such as universities, a social club, and a hotel. Further, although not limited to creative PPI methods exclusively but rather most studies involving the public, a financial incentive was often offered for participation, as well as food, parking, transport and accommodation [ 21 , 22 ].

Creative PPI lacks generalisation

Another barrier to the use of creative PPI methods in health and social care research was the individual nature of its output. Those who participate, usually small in number, produce unique creative outputs specific to their own experiences, opinions and location. Craven et al. [ 13 ], used arts-based visualisations to develop a toolbox for adults with mental health difficulties. They commented, “such an approach might still not be worthwhile”, as the visualisations were individualised and highly personal. This indicates that the output may fail to meet the needs of its end-users. Further, these creative PPI groups were based in certain geographical regions such as Stoke-on-Trent [ 19 ] Sheffield [ 23 ], South Wales [ 12 ] or Ireland [ 20 ], which limits the extent the findings can be applied to wider populations, even within the same area due to individual nuances. Further, the study by Galler et al. [ 16 ], is specific to the Norwegian context and even then, maybe only a sub-group of the Norwegian population as the sample used was of higher socioeconomic status.

However, Grindell et al. [ 17 ], who used persona scenarios, creative worksheets and prototype development, pointed out that the purpose of this type of research is to improve a certain place, rather than apply findings across other populations and locations. Individualised output may, therefore, only be a limitation to research wanting to conduct PPI on a large scale.

If, however, greater generalisation within PPI is deemed necessary, then social media may offer a resolution. Fedorowicz et al. [ 15 ], used Facebook to gain feedback from the public on the use of video-recording methodology for an upcoming project. This had the benefit of including a more diverse range of people (289 people joined the closed group), who were spread geographically around the UK, as well as seven people from overseas.

Creative PPI has ethical issues

As with other research, ethical issues must be taken into consideration. Due to the nature of creative approaches, as well as the personal effort put into them, people often want to be recognised for their work. However, this compromises principles so heavily instilled in research such as anonymity and confidentiality. With the aim of exploring issues related to health and well-being in a town in South Wales, Byrne et al. [ 12 ], asked year 4/5 and year 10 pupils to create poems, songs, drawings and photographs. Community members also created a performance, mainly of monologues, to explore how poverty and inequalities are dealt with. Byrne noted the risks of these arts-based approaches, that being the possibility of over-disclosure and consequent emotional distress, as well as people’s desire to be named for their work. On one hand, the anonymity reduces the sense of ownership of the output as it does not portray a particular individual’s lived experience anymore. On the other hand, however, it could promote a more honest account of lived experience. Supporting this, Webber et al. [ 23 ], who used the persona method to co-design a back pain educational resource prototype, claimed that the anonymity provided by this creative technique allowed individuals to externalise and anonymise their own personal experience, thus creating a more authentic and genuine resource for future users. This implies that anonymity can be both a limitation and strength here.

The use of creative PPI methods is impeded by external factors

Despite the above limitations influencing the implementation of creative PPI techniques, perhaps the most influential is that creative methodologies are simply not mainstream [ 19 ]. This could be linked to the issues above, like time and resource intensity, generalisation and ethical issues but it is also likely to involve more systemic factors within the research community. Micsinszki et al. [ 21 ], who co-designed a hub for the health and well-being of vulnerable populations, commented that there is insufficient infrastructure to conduct meaningful co-design as well as a dominant medical model. Through a more holistic lens, there are “sociopolitical environments that privilege individualism over collectivism, self-sufficiency over collaboration, and scientific expertise over other ways of knowing based on lived experience” [ 21 ]. This, it could be suggested, renders creative co-design methodologies, which are based on the foundations of collectivism, collaboration and imagination an invalid technique in the research field, which is heavily dominated by more scientific methods offering reproducibility, objectivity and reliability.

Although we acknowledge that creative PPI techniques are not always appropriate, it may be that their main limitation is the lack of awareness of these methods or lack of willingness to use them. Further, there is always the risk that PPI, despite being a mandatory part of research, is used in a tokenistic or tick-box fashion [ 20 ], without considering the contribution that meaningful PPI could make to enhancing the research. It may be that PPI, let alone creative PPI, is not at the forefront of researchers’ minds when planning research.

Strengths of creative PPI

Creative ppi disrupts power hierarchies.

One of the main strengths of creative PPI techniques, cited most frequently in the included literature, was that they disrupt traditional power hierarchies [ 12 , 13 , 17 , 19 , 23 ]. For example, the use of theatre performance blurred the lines between professional and lay roles between the community and policy makers [ 12 ]. Individuals created a monologue to portray how poverty and inequality impact daily life and presented this to representatives of the National Assembly of Wales, Welsh Government, the Local Authority, Arts Council and Westminster. Byrne et al. [ 12 ], states how this medium allowed the community to engage with the people who make decisions about their lives in an environment of respect and understanding, where the hierarchies are not as visible as in other settings, e.g., political surgeries. Creative PPI methods have also removed traditional power hierarchies between researchers and adolescents. Cook et al. [ 13 ], used arts-based approaches to explore adolescents’ ideas about the “perfect” condom. They utilised the “Life Happens” resource, where adolescents drew and then decorated a person with their thoughts about sexual relationships, not too dissimilar from the persona-scenario method. This was then combined with hypothetical scenarios about sexuality. A condom-mapping exercise was then implemented, where groups shared the characteristics that make a condom “perfect” on large pieces of paper. Cook et al. [ 13 ], noted that usually power imbalances make it difficult to elicit information from adolescents, however these power imbalances were reduced due to the use of creative co-design techniques.

The same reduction in power hierarchies was noted by Grindell et al. [ 17 ], who used the person-scenario method and creative worksheets with individuals with malignant pleural effusion. This was with the aim of developing a prototype of a decision support tool for patients to help with treatment options. Although this process involved a variety of stakeholders, such as patients, carers and healthcare professionals, creative co-design was cited as a mechanism that worked to reduce power imbalances – a limitation of more traditional methods of research. Creative co-design blurred boundaries between end-users and clinical staff and enabled the sharing of ideas from multiple, valuable perspectives, meaning the prototype was able to suit user needs whilst addressing clinical problems.

Similarly, a specific creative method named cultural animation was also cited to dissolve hierarchies and encourage equal contributions from participants. Within this arts-based approach, Keleman et al. [ 19 ], explored the concept of “good health” with individuals from Stoke-on Trent. Members of the group created art installations using ribbons, buttons, cardboard and straws to depict their idea of a “healthy community”, which was accompanied by a poem. They also created a 3D Facebook page and produced another poem or song addressing the government to communicate their version of a “picture of health”. Public participants said that they found the process empowering, honest, democratic, valuable and practical.

This dissolving of hierarchies and levelling of power is beneficial as it increases the sense of ownership experienced by the creators/producers of the output [ 12 , 17 , 23 ]. This is advantageous as it has been suggested to improve its quality [ 23 ].

Creative PPI allows the unsayable to be said

Creative PPI fosters a safe space for mundane or taboo topics to be shared, which may be difficult to communicate using traditional methods of PPI. For example, the hypothetical nature of condom mapping and persona-scenarios meant that adolescents could discuss a personal topic without fear of discrimination, judgement or personal disclosure [ 13 ]. The safe space allowed a greater volume of ideas to be generated amongst peers where they might not have otherwise. Similarly, Webber et al. [ 23 ], , who used the persona method to co-design the prototype back pain educational resource, also noted how this method creates anonymity whilst allowing people the opportunity to externalise personal experiences, thoughts and feelings. Other creative methods were also used, such as drawing, collaging, role play and creating mood boards. A cardboard cube (labelled a “magic box”) was used to symbolise a physical representation of their final prototype. These creative methods levelled the playing field and made personal experiences accessible in a safe, open environment that fostered trust, as well as understanding from the researchers.

It is not only sensitive subjects that were made easier to articulate through creative PPI. The communication of mundane everyday experiences were also facilitated, which were deemed typically ‘unsayable’. This was specifically given in the context of describing intangible aspects of everyday health and wellbeing [ 11 ]. Graphic designers can also be used to visually represent the outputs of creative PPI. These captured the movement and fluidity of people and well as the relationships between them - things that cannot be spoken but can be depicted [ 21 ].

Creative PPI methods are inclusive

Another strength of creative PPI was that it is inclusive and accessible [ 17 , 19 , 21 ]. The safe space it fosters, as well as the dismantling of hierarchies, welcomed people from a diverse range of backgrounds and provided equal opportunities [ 21 ], especially for those with communication and memory difficulties who might be otherwise excluded from PPI. Kelemen et al. [ 19 ], who used creative methods to explore health and well-being in Stoke-on-Trent, discussed how people from different backgrounds came together and connected, discussed and reached a consensus over a topic which evoked strong emotions, that they all have in common. Individuals said that the techniques used “sets people to open up as they are not overwhelmed by words”. Similarly, creative activities, such as the persona method, have been stated to allow people to express themselves in an inclusive environment using a common language. Kearns et al. [ 18 ], who used aphasia-accessible material to develop a questionnaire with aphasic individuals, described how they felt comfortable in contributing to workshops (although this material was time-consuming to make, see ‘Limitations of creative PPI’ ).

Despite the general inclusivity of creative PPI, it can also be exclusive, particularly if online mediums are used. Fedorowicz et al. [ 15 ], used Facebook to create a PPI group, and although this may rectify previous drawbacks about lack of generalisation of creative methods (as Facebook can reach a greater number of people, globally), it excluded those who are not digitally active or have limited internet access or knowledge of technology. Online methods have other issues too. Maintaining the online group was cited as challenging and the volume of responses required researchers to interact outside of their working hours. Despite this, online methods like Facebook are very accessible for people who are physically disabled.

Creative PPI methods are engaging

The process of creative PPI is typically more engaging and produces more colourful data than traditional methods [ 13 ]. Individuals are permitted and encouraged to explore a creative self [ 19 ], which can lead to the exploration of new ideas and an overall increased enjoyment of the process. This increased engagement is particularly beneficial for younger PPI groups. For example, to involve children in the development of health food products, Galler et al. [ 16 ] asked 9-12-year-olds to take photos of their food and present it to other children in a “show and tell” fashion. They then created a newspaper article describing a new healthy snack. In this creative focus group, children were given lab coats to further their identity as inventors. Galler et al. [ 16 ], notes that the methods were highly engaging and facilitated teamwork and group learning. This collaborative nature of problem-solving was also observed in adults who used personas and creative worksheets to develop the resource for lower back pain [ 23 ]. Dementia patients too have been reported to enjoy the creative and informal approach to idea generation [ 20 ].

The use of cultural animation allowed people to connect with each other in a way that traditional methods do not [ 19 , 21 ]. These connections were held in place by boundary objects, such as ribbons, buttons, fabric and picture frames, which symbolised a shared meaning between people and an exchange of knowledge and emotion. Asking groups to create an art installation using these objects further fostered teamwork and collaboration, both at an individual and collective level. The exploration of a creative self increased energy levels and encouraged productive discussions and problem-solving [ 19 ]. Objects also encouraged a solution-focused approach and permitted people to think beyond their usual everyday scope [ 17 ]. They also allowed facilitators to probe deeper about the greater meanings carried by the object, which acted as a metaphor [ 21 ].

From the researcher’s point of view, co-creative methods gave rise to ideas they might not have initially considered. Valaitis et al. [ 22 ], found that over 40% of the creative outputs were novel ideas brought to light by patients, healthcare providers/community care providers, community service providers and volunteers. One researcher commented, “It [the creative methods] took me on a journey, in a way that when we do other pieces of research it can feel disconnected” [ 23 ]. Another researcher also stated they could not return to the way they used to do research, as they have learnt so much about their own health and community and how they are perceived [ 19 ]. This demonstrates that creative processes not only benefit the project outcomes and the PPI group, but also facilitators and researchers. However, although engaging, creative methods have been criticised for not demonstrating academic rigour [ 17 ]. Moreover, creative PPI may also be exclusive to people who do not like or enjoy creative activities.

Creative PPI methods are cost and time efficient

Creative PPI workshops can often produce output that is visible and tangible. This can save time and money in the long run as the output is either ready to be implemented in a healthcare setting or a first iteration has already been developed. This may also offset the time and costs it takes to implement creative PPI. For example, the prototype of the decision support tool for people with malignant pleural effusion was developed using personas and creative worksheets. The end result was two tangible prototypes to drive the initial idea forward as something to be used in practice [ 17 ]. The use of creative co-design in this case saved clinician time as well as the time it would take to develop this product without the help of its end-users. In the development of this particular prototype, analysis was iterative and informed the next stage of development, which again saved time. The same applies for the feedback questionnaire for the assessment of ICT delivered aphasia rehabilitation. The co-created questionnaire, designed with people with aphasia, was ready to be used in practice [ 18 ]. This suggests that to overcome time and resource barriers to creative PPI, researchers should aim for it to be engaging whilst also producing output.

That useable products are generated during creative workshops signals to participating patients and public members that they have been listened to and their thoughts and opinions acted upon [ 23 ]. For example, the development of the back pain resource based on patient experiences implies that their suggestions were valid and valuable. Further, those who participated in the cultural animation workshop reported that the process visualises change, and that it already feels as though the process of change has started [ 19 ].

The most cost and time efficient method of creative PPI in this review is most likely the use of Facebook to gather feedback on project methodology [ 15 ]. Although there were drawbacks to this, researchers could involve more people from a range of geographical areas at little to no cost. Feedback was instantaneous and no training was required. From the perspective of the PPI group, they could interact however much or little they wish with no time commitment.

This systematic review identified four limitations and five strengths to the use of creative PPI in health and social care research. Creative PPI is time and resource intensive, can raise ethical issues and lacks generalisability. It is also not accepted by the mainstream. These factors may act as barriers to the implementation of creative PPI. However, creative PPI disrupts traditional power hierarchies and creates a safe space for taboo or mundane topics. It is also engaging, inclusive and can be time and cost efficient in the long term.

Something that became apparent during data analysis was that these are not blanket strengths and limitations of creative PPI as a whole. The umbrella term ‘creative PPI’ is broad and encapsulates a wide range of activities, ranging from music and poems to prototype development and persona-scenarios, to more simplistic things like the use of sticky notes and ordering cards. Many different activities can be deemed ‘creative’ and the strengths and limitations of one does not necessarily apply to another. For example, cultural animation takes greater effort to prepare than the use of sticky notes and sorting cards, and the use of Facebook is cheaper and wider reaching than persona development. Researchers should use their discretion and weigh up the benefits and drawbacks of each method to decide on a technique which suits the project. What might be a limitation to creative PPI in one project may not be in another. In some cases, creative PPI may not be suitable at all.

Furthermore, the choice of creative PPI method also depends on the needs and characteristics of the PPI group. Children, adults and people living with dementia or language difficulties all have different engagement needs and capabilities. This indicates that creative PPI is not one size fits all and that the most appropriate method will change depending on the composition of the group. The choice of method will also be determined by the constraints of the research project, namely time, money and the research aim. For example, if there are time constraints, then a method which yields a lot of data and requires a lot of preparation may not be appropriate. If generalisation is important, then an online method is more suitable. Together this indicates that the choice of creative PPI method is highly individualised and dependent on multiple factors.

Although the limitations discussed in this review apply to creative PPI, they are not exclusive to creative PPI. Ethical issues are a consideration within general PPI research, especially when working with more vulnerable populations, such as children or adults living with a disability. It can also be the case that traditional PPI methods lack generalisability, as people who volunteer to be part of such a group are more likely be older, middle class and retired [ 24 ]. Most research is vulnerable to this type of bias, however, it is worth noting that generalisation is not always a goal and research remains valid and meaningful in its absence. Although online methods may somewhat combat issues related to generalisability, these methods still exclude people who do not have access to the internet/technology or who choose not to use it, implying that online PPI methods may not be wholly representative of the general population. Saying this, however, the accessibility of creative PPI techniques differs from person to person, and for some, online mediums may be more accessible (for example for those with a physical disability), and for others, this might be face-to-face. To combat this, a range of methods should be implemented. Planning multiple focus group and interviews for traditional PPI is also time and resource intensive, however the extra resources required to make this creative may be even greater. Although, the rich data provided may be worth the preparation and analysis time, which is also likely to depend on the number of participants and workshop sessions required. PPI, not just creative PPI, often requires the provision of a financial incentive, refreshments, parking and accommodation, which increase costs. These, however, are imperative and non-negotiable, as they increase the accessibility of research, especially to minority and lower-income groups less likely to participate. Adequate funding is also important for co-design studies where repeated engagement is required. One barrier to implementation, which appears to be exclusive to creative methods, however, is that creative methods are not mainstream. This cannot be said for traditional PPI as this is often a mandatory part of research applications.

Regarding the strengths of creative PPI, it could be argued that most appear to be exclusive to creative methodologies. These are inclusive by nature as multiple approaches can be taken to evoke ideas from different populations - approaches that do not necessarily rely on verbal or written communication like interviews and focus groups do. Given the anonymity provided by some creative methods, such as personas, people may be more likely to discuss their personal experiences under the guise of a general end-user, which might be more difficult to maintain when an interviewer is asking an individual questions directly. Additionally, creative methods are by nature more engaging and interactive than traditional methods, although this is a blanket statement and there may be people who find the question-and-answer/group discussion format more engaging. Creative methods have also been cited to eliminate power imbalances which exist in traditional research [ 12 , 13 , 17 , 19 , 23 ]. These imbalances exist between researchers and policy makers and adolescents, adults and the community. Lastly, although this may occur to a greater extent in creative methods like prototype development, it could be suggested that PPI in general – regardless of whether it is creative - is more time and cost efficient in the long-term than not using any PPI to guide or refine the research process. It must be noted that these are observations based on the literature. To be certain these differences exist between creative and traditional methods of PPI, direct empirical evaluation of both should be conducted.

To the best of our knowledge, this is the first review to identify the strengths and limitations to creative PPI, however, similar literature has identified barriers and facilitators to PPI in general. In the context of clinical trials, recruitment difficulties were cited as a barrier, as well as finding public contributors who were free during work/school hours. Trial managers reported finding group dynamics difficult to manage and the academic environment also made some public contributors feel nervous and lacking confidence to speak. Facilitators, however, included the shared ownership of the research – something that has been identified in the current review too. In addition, planning and the provision of knowledge, information and communication were also identified as facilitators [ 25 ]. Other research on the barriers to meaningful PPI in trial oversight committees included trialist confusion or scepticism over the PPI role and the difficulties in finding PPI members who had a basic understanding of research [ 26 ]. However, it could be argued that this is not representative of the average patient or public member. The formality of oversight meetings and the technical language used also acted as a barrier, which may imply that the informal nature of creative methods and its lack of dependency on literacy skills could overcome this. Further, a review of 42 reviews on PPI in health and social care identified financial compensation, resources, training and general support as necessary to conduct PPI, much like in the current review where the resource intensiveness of creative PPI was identified as a limitation. However, others were identified too, such as recruitment and representativeness of public contributors [ 27 ]. Like in the current review, power imbalances were also noted, however this was included as both a barrier and facilitator. Collaboration seemed to diminish hierarchies but not always, as sometimes these imbalances remained between public contributors and healthcare staff, described as a ‘them and us’ culture [ 27 ]. Although these studies compliment the findings of the current review, a direct comparison cannot be made as they do not concern creative methods. However, it does suggest that some strengths and weaknesses are shared between creative and traditional methods of PPI.

Strengths and limitations of this review

Although a general definition of creative PPI exists, it was up to our discretion to decide exactly which activities were deemed as such for this review. For example, we included sorting cards, the use of interactive whiteboards and sticky notes. Other researchers may have a more or less stringent criteria. However, two reviewers were involved in this decision which aids the reliability of the included articles. Further, it may be that some of the strengths and limitations cannot fully be attributed to the creative nature of the PPI process, but rather their co-created nature, however this is hard to disentangle as the included papers involved both these aspects.

During screening, it was difficult to decide whether the article was utilising creative qualitative methodology or creative PPI , as it was often not explicitly labelled as such. Regardless, both approaches involved the public/patients refining a healthcare product/service. This implies that if this review were to be replicated, others may do it differently. This may call for greater standardisation in the reporting of the public’s involvement in research. For example, the NIHR outlines different approaches to PPI, namely “consultation”, “collaboration”, “co-production” and “user-controlled”, which each signify an increased level of public power and influence [ 28 ]. Papers with elements of PPI could use these labels to clarify the extent of public involvement, or even explicitly state that there was no PPI. Further, given our decision to include only scholarly peer-reviewed literature, it is possible that data were missed within the grey literature. Similarly, the literature search will not have identified all papers relating to different types of accessible inclusion. However, the intent of the review was to focus solely on those within the definition of creative.

This review fills a gap in the literature and helps circulate and promote the concept of creative PPI. Each stage of this review, namely screening and quality appraisal, was conducted by two independent reviewers. However, four full texts could not be accessed during the full text reading stage, meaning there are missing data that could have altered or contributed to the findings of this review.

Research recommendations

Given that creative PPI can require effort to prepare, perform and analyse, sufficient time and funding should be allocated in the research protocol to enable meaningful and continuous PPI. This is worthwhile as PPI can significantly change the research output so that it aligns closely with the needs of the group it is to benefit. Researchers should also consider prototype development as a creative PPI activity as this might reduce future time/resource constraints. Shifting from a top-down approach within research to a bottom-up can be advantageous to all stakeholders and can help move creative PPI towards the mainstream. This, however, is the collective responsibility of funding bodies, universities and researchers, as well as committees who approve research bids.

A few of the included studies used creative techniques alongside traditional methods, such as interviews, which could also be used as a hybrid method of PPI, perhaps by researchers who are unfamiliar with creative techniques or to those who wish to reap the benefits of both. Often the characteristics of the PPI group were not included, including age, gender and ethnicity. It would be useful to include such information to assess how representative the PPI group is of the population of interest.

Creative PPI is a relatively novel approach of engaging the public and patients in research and it has both advantages and disadvantages compared to more traditional methods. There are many approaches to implementing creative PPI and the choice of technique will be unique to each piece of research and is reliant on several factors. These include the age and ability of the PPI group as well as the resource limitations of the project. Each method has benefits and drawbacks, which should be considered at the protocol-writing stage. However, given adequate funding, time and planning, creative PPI is a worthwhile and engaging method of generating ideas with end-users of research – ideas which may not be otherwise generated using traditional methods.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Critical Appraisal Skills Programme

The Joanna Briggs Institute

National Institute of Health and Care Research

Public Advisory Group

Public and Patient Involvement

Web of Science

National Institute for Health and Care Research. What Is Patient and Public Involvement and Public Engagement? https://www.spcr.nihr.ac.uk/PPI/what-is-patient-and-public-involvement-and-engagement Accessed 01 Sept 2023.

Department of Health. Personal and Public Involvement (PPI) https://www.health-ni.gov.uk/topics/safety-and-quality-standards/personal-and-public-involvement-ppi#:~:text=The Health and Social Care Reform Act (NI) 2009 placed,delivery and evaluation of services . Accessed 01 Sept 2023.

National Institute for Health and Care Research. Policy Research Programme – Guidance for Stage 1 Applications https://www.nihr.ac.uk/documents/policy-research-programme-guidance-for-stage-1-applications-updated/26398 Accessed 01 Sept 2023.

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, Chant A. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019. https://doi.org/10.1111/hex.12888

Article   PubMed   PubMed Central   Google Scholar  

Street JM, Stafinski T, Lopes E, Menon D. Defining the role of the public in health technology assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. 2020. https://doi.org/10.1017/S0266462320000094

Article   PubMed   Google Scholar  

Morrison C, Dearden A. Beyond tokenistic participation: using representational artefacts to enable meaningful public participation in health service design. Health Policy. 2013. https://doi.org/10.1016/j.healthpol.2013.05.008

Leavy P. Method meets art: arts-Based Research Practice. New York: Guilford; 2020.

Google Scholar  

Seers K. Qualitative systematic reviews: their importance for our understanding of research relevant to pain. Br J Pain. 2015. https://doi.org/10.1177/2049463714549777

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis JBI. 2020. https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03

CASP. CASP Checklists https://casp-uk.net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/CASP-Qualitative-Checklist-2018_fillable_form.pdf (2022).

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006. https://doi.org/10.1191/1478088706qp063oa

Article   Google Scholar  

Byrne E, Elliott E, Saltus R, Angharad J. The creative turn in evidence for public health: community and arts-based methodologies. J Public Health. 2018. https://doi.org/10.1093/pubmed/fdx151

Cook S, Grozdanovski L, Renda G, Santoso D, Gorkin R, Senior K. Can you design the perfect condom? Engaging young people to inform safe sexual health practice and innovation. Sex Educ. 2022. https://doi.org/10.1080/14681811.2021.1891040

Craven MP, Goodwin R, Rawsthorne M, Butler D, Waddingham P, Brown S, Jamieson M. Try to see it my way: exploring the co-design of visual presentations of wellbeing through a workshop process. Perspect Public Health. 2019. https://doi.org/10.1177/1757913919835231

Fedorowicz S, Riley V, Cowap L, Ellis NJ, Chambers R, Grogan S, Crone D, Cottrell E, Clark-Carter D, Roberts L, Gidlow CJ. Using social media for patient and public involvement and engagement in health research: the process and impact of a closed Facebook group. Health Expect. 2022. https://doi.org/10.1111/hex.13515

Galler M, Myhrer K, Ares G, Varela P. Listening to children voices in early stages of new product development through co-creation – creative focus group and online platform. Food Res Int. 2022. https://doi.org/10.1016/j.foodres.2022.111000

Grindell C, Tod A, Bec R, Wolstenholme D, Bhatnagar R, Sivakumar P, Morley A, Holme J, Lyons J, Ahmed M, Jackson S, Wallace D, Noorzad F, Kamalanathan M, Ahmed L, Evison M. Using creative co-design to develop a decision support tool for people with malignant pleural effusion. BMC Med Inf Decis Mak. 2020. https://doi.org/10.1186/s12911-020-01200-3

Kearns Á, Kelly H, Pitt I. Rating experience of ICT-delivered aphasia rehabilitation: co-design of a feedback questionnaire. Aphasiology. 2020. https://doi.org/10.1080/02687038.2019.1649913

Kelemen M, Surman E, Dikomitis L. Cultural animation in health research: an innovative methodology for patient and public involvement and engagement. Health Expect. 2018. https://doi.org/10.1111/hex.12677

Keogh F, Carney P, O’Shea E. Innovative methods for involving people with dementia and carers in the policymaking process. Health Expect. 2021. https://doi.org/10.1111/hex.13213

Micsinszki SK, Buettgen A, Mulvale G, Moll S, Wyndham-West M, Bruce E, Rogerson K, Murray-Leung L, Fleisig R, Park S, Phoenix M. Creative processes in co-designing a co-design hub: towards system change in health and social services in collaboration with structurally vulnerable populations. Evid Policy. 2022. https://doi.org/10.1332/174426421X16366319768599

Valaitis R, Longaphy J, Ploeg J, Agarwal G, Oliver D, Nair K, Kastner M, Avilla E, Dolovich L. Health TAPESTRY: co-designing interprofessional primary care programs for older adults using the persona-scenario method. BMC Fam Pract. 2019. https://doi.org/10.1186/s12875-019-1013-9

Webber R, Partridge R, Grindell C. The creative co-design of low back pain education resources. Evid Policy. 2022. https://doi.org/10.1332/174426421X16437342906266

National Institute for Health and Care Research. A Researcher’s Guide to Patient and Public Involvement. https://oxfordbrc.nihr.ac.uk/wp-content/uploads/2017/03/A-Researchers-Guide-to-PPI.pdf Accessed 01 Nov 2023.

Selman L, Clement C, Douglas M, Douglas K, Taylor J, Metcalfe C, Lane J, Horwood J. Patient and public involvement in randomised clinical trials: a mixed-methods study of a clinical trials unit to identify good practice, barriers and facilitators. Trials. 2021 https://doi.org/10.1186/s13063-021-05701-y

Coulman K, Nicholson A, Shaw A, Daykin A, Selman L, Macefield R, Shorter G, Cramer H, Sydes M, Gamble C, Pick M, Taylor G, Lane J. Understanding and optimising patient and public involvement in trial oversight: an ethnographic study of eight clinical trials. Trials. 2020. https://doi.org/10.1186/s13063-020-04495-9

Ocloo J, Garfield S, Franklin B, Dawson S. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Sys. 2021. https://doi.org/10.1186/s12961-020-00644-3

National Institute for Health and Care Research. Briefing notes for researchers - public involvement in NHS, health and social care research. https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371 Accessed 01 Nov 2023.

Download references

Acknowledgements

With thanks to the PHIRST-LIGHT public advisory group and consortium for their thoughts and contributions to the design of this work.

The research team is supported by a National Institute for Health and Care Research grant (PHIRST-LIGHT Reference NIHR 135190).

Author information

Olivia R. Phillips and Cerian Harries share joint first authorship.

Authors and Affiliations

Nottingham Centre for Public Health and Epidemiology, Lifespan and Population Health, School of Medicine, University of Nottingham, Clinical Sciences Building, City Hospital Campus, Hucknall Road, Nottingham, NG5 1PB, UK

Olivia R. Phillips, Jo Leonardi-Bee, Holly Knight & Joanne R. Morling

National Institute for Health and Care Research (NIHR) PHIRST-LIGHT, Nottingham, UK

Olivia R. Phillips, Cerian Harries, Jo Leonardi-Bee, Holly Knight, Lauren B. Sherar, Veronica Varela-Mato & Joanne R. Morling

School of Sport, Exercise and Health Sciences, Loughborough University, Epinal Way, Loughborough, Leicestershire, LE11 3TU, UK

Cerian Harries, Lauren B. Sherar & Veronica Varela-Mato

Nottingham Centre for Evidence Based Healthcare, School of Medicine, University of Nottingham, Nottingham, UK

Jo Leonardi-Bee

NIHR Nottingham Biomedical Research Centre (BRC), Nottingham University Hospitals NHS Trust, University of Nottingham, Nottingham, NG7 2UH, UK

Joanne R. Morling

You can also search for this author in PubMed   Google Scholar

Contributions

Author contributions: study design: ORP, CH, JRM, JLB, HK, LBS, VVM, literature searching and screening: ORP, CH, JRM, data curation: ORP, CH, analysis: ORP, CH, JRM, manuscript draft: ORP, CH, JRM, Plain English Summary: ORP, manuscript critical review and editing: ORP, CH, JRM, JLB, HK, LBS, VVM.

Corresponding author

Correspondence to Olivia R. Phillips .

Ethics declarations

Ethics approval and consent to participate.

The Ethics Committee of the Faculty of Medicine and Health Sciences, University of Nottingham advised that approval from the ethics committee and consent to participate was not required for systematic review studies.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

40900_2024_580_MOESM1_ESM.docx

Additional file 1: Search strings: Description of data: the search strings and filters used in each of the 5 databases in this review

Additional file 2: Quality appraisal questions: Description of data: CASP quality appraisal questions

40900_2024_580_moesm3_esm.docx.

Additional file 3: Table 1: Description of data: elements of the data extraction table that are not in the main manuscript

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Phillips, O.R., Harries, C., Leonardi-Bee, J. et al. What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review. Res Involv Engagem 10 , 48 (2024). https://doi.org/10.1186/s40900-024-00580-4

Download citation

Received : 28 November 2023

Accepted : 25 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s40900-024-00580-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public and patient involvement
• Creative PPI
• Qualitative systematic review

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

Community-Engaged Research: Common Themes and Needs Identified by Investigators and Research Teams at an Emerging Academic Learning Health System

Megan b. irby.

1 Maya Angelou Center for Health Equity, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@ybrim

Keena R. Moore

2 Program in Community-Engaged Research, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@eroomrk (K.R.M.); ude.htlaehekaw@lladnari (I.R.)

Lilli Mann-Jackson

3 Department of Social Sciences and Health Policy and Program in Community-Engaged Research, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@nnaml

DeWanna Hamlin

4 Formerly of the Program in Community-Engaged Research, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; moc.oohay@nilmahtld

Isaiah Randall

Phillip summers.

5 Department of Radiology, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@sremmusp

Joseph A. Skelton

6 Department of Pediatrics and Program in Community-Engaged Research, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@notleksj

Stephanie S. Daniel

7 Department of Family and Community Medicine and Program in Community-Engaged Research, Wake Forest School of Medicine, Winston-Salem, NC 27157, USA; ude.htlaehekaw@leinads

Scott D. Rhodes

Associated data.

Data are available from the corresponding author.

Community-engaged research (CEnR) has emerged within public health and medicine as an approach to research designed to increase health equity, reduce health disparities, and improve community and population health. We sought to understand how CEnR has been conducted and to identify needs to support CEnR within an emerging academic learning health system (aLHS). We conducted individual semi-structured interviews with investigators experienced in CEnR at an emerging aLHS in the southeastern United States. Eighteen investigators (16 faculty and 2 research associates) were identified, provided consent, and completed interviews. Half of participants were women; 61% were full professors of varied academic backgrounds and departments. Interviews were audio-recorded, transcribed, coded, and analyzed using constant comparison, an approach to grounded theory. Twenty themes emerged that were categorized into six domains: Conceptualization and Purpose, Value and Investment, Community-Academic Partnerships, Sustainability, Facilitators, and Challenges. Results also identified eight emerging needs necessary to enhance CEnR within aLHSs. The results provide insights into how CEnR approaches can be harnessed within aLHSs to build and nurture community-academic partnerships, inform research and institutional priorities, and improve community and population health. Findings can be used to guide the incorporation of CEnR within aLHSs.

1. Introduction

1.1. community-engaged research.

Community-engaged research (CEnR) has emerged within public health and medicine as an approach to research designed to increase health equity, reduce health disparities, and improve community and population health. CEnR involves the affected community in research, and defines that community as any group of people affiliated by geographic proximity, special interest, health condition, or similar categories of shared identity. Rather than investigators and research teams from universities, government, or other types of research organizations approaching and entering a community with a preconceived notion of a community’s best interests, in projects that apply CEnR approaches, community members and representatives from community organizations collaborate and share research roles with academic investigators and research teams. Community members become not merely “targets” of research but also research partners. CEnR emphasizes relationship-building and trust; open communication; co-learning; reciprocal transfer of expertise; shared power, resources, and decision-making; and mutual ownership of the processes and products of research [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ].

Growing evidence suggests that including community members and representatives from community organizations in the design, implementation, and evaluation of research can lead to deeper, more informed, and nuanced understandings of health-related phenomena and identify actions (e.g., interventions, programs, and policy and system changes) that are more relevant, culturally congruent, and likely to be effective, sustained, and scalable, if warranted, to improve community and population health [ 6 , 7 , 9 , 10 , 11 ]. However, the conduct of CEnR is shaped by institutional (e.g., allocation of resources and time), community (e.g., local history and receptivity), and personal (e.g., background, experiences, and world views) factors. Thus, there is no standardized way to engage and partner with communities to conduct CEnR, and there is great variability across models of engagement and in the degree to which communities are included in the research process [ 8 , 9 ].

Community engagement often is viewed as a continuum that spans from outreach (less engaged), consultation, involvement, and collaboration, to shared leadership (more engaged) [ 3 , 4 , 7 , 12 , 13 ]. As such, the community engagement literature is extensive and features a wide range of theories, approaches, strategies, and methods, reflecting tremendous diversity in how community engagement is defined, implemented, and evaluated [ 1 , 2 , 6 , 7 , 13 ]. Despite this heterogeneity, CEnR includes several common critical elements. These include: (1) Collaboration with groups of people affiliated by geographic proximity, special interest, health condition, or other categories of shared identity; (2) Groups of community members, organizational representatives, and academic researchers adhering to common principles and norms to nurture trust and promote authentic partnership; (3) Focus on identifying and addressing the needs and priorities and harnessing the assets that affect health and well-being; and (4) Research as an approach to systematically uncover and understand health-related phenomena and improve community and population health [ 2 , 3 , 7 ].

1.2. Academic Learning Health Systems

Learning health systems (LHSs) are organizations or networks that pursue a cycle of perpetual learning processes embedded in daily practice. These processes inform evidence-based improvements in health care to yield better patient outcomes. This cycle combines quality improvement methods and data analytics from patient care, which are fed back into the LHS to inform institutional policies and structures and improve care delivery [ 14 ]. An academic learning health system (aLHS) prioritizes gathering new, generalizable knowledge to improve community and population health, in addition to promoting continuous improvement and innovation within its own health care delivery system. An aLHS draws on its embedded academic expertise in public health and medical education, health system sciences, translation, and population and community health; shares knowledge and imparts skills necessary to advance the system internally; and disseminates knowledge broadly to contribute to knowledge generation and advance science [ 15 , 16 ].

Although engaging patients to provide perspectives and feedback into various aspects of health systems has long been valued, to date, there has been no exploration of CEnR within aLHSs. This is due in part to the recent emergence of the aLHS as a concept and entity [ 15 , 16 , 17 ]. Understanding how CEnR has been conducted and identifying needs to support and enhance CEnR within an emerging aLHS could provide insights critical to advance our knowledge of effective CEnR methods and designs and strengthen the conduct of future research designed to increase health equity, reduce health disparities, and improve community and population health [ 18 ].

Wake Forest School of Medicine/Wake Forest Baptist Health (WFSM/WFBH), an emerging aLHS, has a long and successful history of CEnR [ 19 , 20 , 21 , 22 , 23 ]. A number of its investigators routinely incorporate community engagement in their research. We sought to explore how CEnR has been harnessed at WFSM/WFBH; how investigators at this emerging aLHS have incorporated principles of community engagement within their research; the degree to which community members and/or representatives from community organizations have been involved in the planning, implementation, evaluation, and/or dissemination phases of research; the nature of CEnR as it is implemented within the context of the local community; and what institutional supports are needed to facilitate and integrate CEnR within an aLHS.

2. Materials and Methods

2.1. setting.

WFSM/WFBH is an emerging aLHS serving more than 50 counties in North Carolina, Tennessee, Virginia, and West Virginia. It is comprised of WFSM and five hospitals in the Piedmont region of northwest North Carolina. In addition to training medical, physician assistant, and nursing students, WFSM has a broad array of biomedical graduate and postdoctoral training programs. It also has a large Division of Public Health Sciences with Departments in Biostatistics and Data Science, Epidemiology and Prevention, Implementation Science, and Social Sciences and Health Policy. A Clinical and Translational Science Award (CTSA) from the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), supports the WFSM/WFBH Clinical and Translational Science Institute (CTSI), which includes the Program in Community-Engaged Research (UL1TR001420). The Program aims to improve community and population health through promoting and facilitating community-relevant and engaged research, working in partnership with communities to identify health needs, priorities, and assets and to seek solutions to health challenges together [ 23 ].

2.2. Participants

We conducted individual semi-structured interviews with investigators experienced in community engagement and CEnR. We identified potential interviewees with assistance from the WFSM/WFBH CTSI and Institutional Review Board (IRB). A keyword search was performed within the institutional grants management software (InfoEd) and IRB databases to identify WFSM/WFBH investigators who had submitted research applications related to community engagement or CEnR within the past ten years. Search terms included “community”, “engagement”, “CBPR”, “participatory research”, and “partnership”. Abstracts from identified applications were reviewed to further determine eligibility. Investigators were then contacted by electronic mail and invited to participate in an interview. Interviews with consenting participants were then scheduled at a time and location convenient to the participant. Interviews were conducted by two trained study staff and digitally recorded, with study staff also taking clarifying notes. During interviews, participants were asked for the names of other investigators within the institution who also conducted CEnR to identify additional potential participants. Interviews were transcribed verbatim and subsequently verified.

2.3. Individual Semi-Structured Interview Guide

We chose a qualitative research approach to gain a broad spectrum of perspectives about CEnR within the context of an aLHS. This qualitative approach can yield perspectives and insights that are not previously known by researchers and thus would not emerge in a more close-ended quantitative approach [ 24 , 25 ]. A semi-structured interview guide was designed to collect both descriptive demographic data from each participant, including race/ethnicity, age, gender, position, academic degrees, background, current academic department, years at WFSM/WFBH, and CEnR training. The guide also captured socio-contextual and detailed descriptions of perceptions, experiences, and strategies specific to the use of community engagement and CEnR. Areas explored included: how each participant conceptualizes and defines CEnR; their preparation for CEnR, including formal training; perceived benefits of CEnR; challenges associated with CEnR; how communities are engaged in research; what roles community members and representatives from community organizations play in CEnR; how the region (i.e., southern United States) and the institution (i.e., an emerging aLHS) affect CEnR; how sustainability and dissemination of CEnR are incorporated into CEnR; and what CEnR lessons have been learned by each participant. The guide is summarized in Table 1 .

Abbreviated items from the individual semi-structured interview guide.

The semi-structured interview guide was drafted, reviewed, revised, and finalized by community members with experience in community-engaged research and experts in community engagement and aLHSs. All items were open-ended. The guide was pilot tested for comprehension and timing with 3 investigators; slight revisions to wording were made based on the pilot.

2.4. Analysis

Each interview transcript was coded by two analysts. Themes were identified through constant comparison, an approach to developing grounded theory, combining inductive coding with simultaneous comparison [ 25 ]. Using standard procedures [ 25 ], analysts first coded text and convened to compare their codes. They then identified and resolved any discrepancies through discussion. Matrices were used to identify similarities and differences within and across participants. Analysts identified, refined, and interpreted themes iteratively through discussion and by examining codes and rereading the transcripts. Findings and themes were presented to interview participants in a presentation for the WFSM/WFBH CTSI Program in Community-Engaged Research Affinity Group ( n = 27). The Affinity Group is a group of investigators, research team members, and others at WFSM/WFBH who are interested in CEnR. Members of the Community Stakeholder Advisory Committee (CSAC) ( n = 8) of the CTSI’s Program in Community-Engaged Research also attended. CSAC is comprised of representatives from community organizations who provide feedback to the CTSI regarding research infrastructure and policies, and to investigators and research teams regarding WFSM/WFBH research initiatives [ 26 ] Presentation attendees contributed to the refinement of themes and their interpretation through facilitated group discussion.

Human subject approval and oversight for this study were provided by the WFSM/WFBH IRB.

The keyword search in InfoEd and IRB databases yielded the names and research projects of 51 investigators. Sixteen investigators were confirmed as eligible based on a review of project abstracts; 14 of these responded to email invitations and agreed to participate. Four additional participants were referred by initial participants. In all, 18 investigators (16 faculty and 2 senior research associates) provided consent and completed interviews. Interviews averaged 45 min.

The participant sample ( Table 2 ) was 50% female, had a mean age of 55 years, was mostly White, and had varied academic backgrounds. Most participants held doctoral degrees (i.e., PhD, DrPH, EdD, MD, and MD/PhD). On average, participants had worked at WFSM/WFBH for nearly 14 years, and represented six academic departments. Nearly all participants indicated they had never received formal education or training in CEnR, though 100% reported “on-the-job” training and experience. All had been principal investigators on at least one federally funded research project.

Demographics of participants ( N = 18).

* Count and percent or mean and standard deviation.

3.1. Domains and Themes Related to Community Engagement and CEnR

Twenty themes emerged across six domains related to community engagement and CEnR within an aLHS ( Table 3 ): Conceptualization and Purpose, Value and Investment, Community-Academic Partnerships, Sustainability, Facilitators, and Challenges.

Domains and themes related to conducting community-engaged research (CEnR) at an academic learning health system (aLHS).

3.1.1. Conceptualization and Purpose of CEnR

Participants agreed that CEnR is a collaborative approach to research designed to improve health and well-being through participatory and better-informed inquiry, always with an eye on how knowledge generated can be translated and applied within the local affected community. However, participants noted that this local application of knowledge does not preclude its generalizability and transferability to other contexts, thus aligning with the broader research goals of an aLHS. Participants also identified CEnR as an approach relevant within many aspects of an aLHS, including education and training, quality improvement, clinical care, and clinical trials.

Participants highlighted the difficulties in defining “community” and emphasized that communities are heterogeneous, which can be challenging for investigators and research teams who may want “simple answers” (e.g., perspectives and insights) from community partners; as participants reported, working with one community does not yield one voice. CEnR was identified as requiring careful consideration of the various perspectives and insights of all partners. Finally, participants also identified additional goals of CEnR, including strengthening connections within the community, building resilience and capacity, and reducing the effects of marginalization.

3.1.2. Value and Investment in CEnR: Institutional, Professional, and Personal

Participants described CEnR as an under-appreciated and frequently misunderstood approach to research within many institutions, including federal funding agencies such as the NIH, the U.S. Centers for Disease Control and Prevention (CDC), and the Health Resources and Services Administration (HRSA). A participant reported that this misunderstanding among federal partners persists even when a project is required by the federal partner and funder and initially designed to be conducted using CEnR approaches. A participant provided an example of an intervention study that resulted in null findings; the participant attributed the study’s null findings to funder-required changes to the study design, recruitment and retention strategies, and an intervention that did not align with community partner perspectives and were contrary to project-specific community steering committee guidance.

Participants also noted that their professional and academic investment in CEnR stemmed from their own personal values, including community health, health equity, and social justice. Participants agreed that this commitment was not limited to traditional nine-to-five “work hours,” but that successful community engagement must be woven into daily life and interactions to be successful.

3.1.3. Community-Academic Partnerships

Participants expressed that the type and degree of engagement, and the ease with which partnerships are formed, depend on the extent to which communities feel they have been marginalized and how they perceive research and research institutions. Moreover, partnerships, how they function, and the roles of partnership members in CEnR vary profoundly across investigators and across projects. Participants identified many areas as essential to the development of strong and productive partnerships: fostering trust and mutual respect, balancing expertise across community and aLHS partners, investing time, facilitating open communication, embracing conflict as a strategy for resolution, and overcoming barriers (including addressing community mistrust based on previous community experiences with both research and health care). Participants noted the role of the Tuskegee Syphilis Study [ 27 ], the Guatemalan Syphilis Experiments [ 28 ]), and rampant anti-immigration rhetoric [ 29 ], racism, homophobia, and transphobia in the United States as contributing to community mistrust.

3.1.4. Sustainability of CEnR

Participants described sustaining CEnR as complex, citing both the need and the difficulty of maintaining community-aLHS partnerships beyond the time period of an individual project. They highlighted the difficulty in sustaining partnerships without funding for continued community involvement and effort from academic investigators and research teams. Additionally, participants noted that sustainability is influenced by the strength and quality of engagement throughout a project, and how well community partners were incorporated into the research process—from conception, study design and conduct, data analysis and interpretation, to the dissemination of findings.

3.1.5. Facilitators of CEnR

Participants emphasized the need for institutional support (e.g., funding, protected time, and respect) and research resources (e.g., CTSA and institutional research centers) that prioritize CEnR as integral to the academic mission of an aLHS. Participants also noted the immense value of having an IRB that is willing to learn the nuances of CEnR in order to approve and oversee CEnR.

3.1.6. Challenges of CEnR

Participants described the potential for burn out and strain resulting from the need to incorporate community engagement into daily life and the great time commitment accompanying CEnR. Other challenges identified by participants related to the discordance between community and academic goals within a given research project, and differences in overarching community and academic priorities that challenged collaboration. Participants also described the challenges of history and how an institution’s complicity in oppressive practices of the past influenced the development and maintenance of community research partnerships. For example, some participants cited WFSM/WFBH’s past involvement in the North Carolina Eugenics Program [ 30 ] as reducing community trust and engagement.

3.2. Enhancing Community-Engaged Research within an aLHS

We also identified eight emergent needs that could enhance CEnR within aLHSs ( Table 4 ). First, participants described the need for increased understanding among academic investigators, research teams, and healthcare providers of community contexts and assets, social determinants of health (also known as “social drivers of health”), and historical factors that influence community and population health. Participants reported that this increased understanding may be particularly critical because many investigators, research teams, and providers may be from other regions of the country or other parts of the world. Thus, although well intentioned, they may not sufficiently understand the local community or the contexts of the populations of interest and focus.

Emergent needs to support and enhance community-engaged research (CEnR) within an academic learning health system (aLHS).

Similarly, participants noted a profound need for increased understanding of CEnR within the aLHS and its value as an approach within community and population health, public health, and medicine. Participants expressed frustration that CEnR is often conflated with community outreach; formative or qualitative research; or behavioral and social sciences. Participants emphasized that members of communities have critical perspectives regarding research into locally identified needs and priorities. Such collaborations can harness the assets of both the community and the aLHS.

Third, participants also highlighted the need for training community members, academic investigators, and research teams to increase their understanding of and skills in partnering with communities and conducting CEnR. Participants noted that establishing authentic and productive partnerships to conduct CEnR is difficult, and, despite good intentions, many investigators and research teams at the emerging aLHS do not understand how to work effectively with community members and representatives from community organizations. At the same time, participants shared that community partners may not sufficiently understand the research process, how evidence and knowledge are generated, and the various components of and objectives inherent within an aLHS.

Fourth, participants noted that although the theories and principles underlying CEnR are well documented, effective frameworks and methods aligned with CEnR are needed. They noted the need for methodologic innovations. Participants noted methods such as photovoice [ 31 ], empowerment-based community forums [ 32 ], evidence academies [ 33 ], and citizen science [ 23 ] are well developed, further research approaches to and methods aligned with CEnR are needed.

Participants also noted that policy changes can positively affect health in multiple ways, whether within an aLHS, the local community, or nationally. They identified a need for evidence-based, practical guidance to increase the translation of CEnR findings into policies designed to improve community and population health.

Sixth, participants acknowledged the need for guidance on balancing the perspectives of community and aLHS partners. They reported that it can be difficult for academic investigators and research teams to know how to elicit community partner perspectives, to usefully share their own perspectives (based on theory, their own prior research, and the existing literature), build on and negotiate with community partners, and negotiate and compromise in ways that ensure sound science and maximize the success of a research project. Participants noted that CEnR requires weighing scientific rigor and what is realistic and “doable”; far too often, participants reported, investigators and research teams may choose rigorous research approaches that simply cannot be successfully implemented. Participants asserted not including community perspectives through CEnR could result in study designs that are inauthentic to how communities convene, interact, and take action; enrollment and retention plans that are not acceptable or realistic; and/or measurement that does not make sense to members of the community. In such situations, data collection may be sacrificed, analysis and interpretation of findings may be less accurate, and sustainability and meaningful dissemination of findings may not be possible.

Participants cited a need for a model to incorporate principles of CEnR into research mission, vision, and priorities of an aLHS. They suggested that the linkages between CEnR and aLHS need further exploration and articulation. Finally, participants reported the need for more institutional support for community engagement and CEnR. This support included pilot funding and protected time of investigators and research teams to establish partnerships, develop innovative methods, and explore integration of community perspectives into the priorities and processes of an aLHS.

4. Discussion

Our results provide insights critical to understanding how CEnR approaches function within an emerging aLHS and ways to further build and nurture community-academic partnerships and inform research and institutional priorities to increase health equity, reduce health disparities, and improve community and population health. In this study, we identified six primary domains of 20 themes related to the purpose of CEnR, its value within an aLHS, characteristics of effective community-aLHS partnerships, issues related to sustainability of CEnR, facilitators of CEnR within an aLHS, and challenges facing CEnR within an aLHS. Many of these domains have been explored in the broader CEnR literature; however, this is the first exploration of CEnR within an aLHS. We also uncovered eight needs that, if addressed, could support and enhance community engagement and CEnR within an aLHS. Several findings deserve highlighting.

First, participants expressed the need for academic investigators, research teams, and healthcare providers to better understand community contexts, social determinants of health, and historical factors influencing community and population health and participation in research. This view is consistent with previous research suggesting that health disparities will persist without better understanding of health and health-related phenomena within communities and the ongoing reluctance of community members to participate in research [ 11 ].

Participants also described the importance of the aLHS’s reputation in the community as critical to influencing trust and engagement, particularly among historically marginalized populations. Participants specifically noted WFSM/WFBH’s past involvement in the North Carolina Eugenics Program and other discriminatory practices committed against minority and vulnerable populations [ 30 ]; they also noted that many communities across the United States share similar historical narratives. Thus, although much work must done; the inclusion of CEnR within CTSAs and emerging aLHS is a step in the right direction.

Trust is built by and exists among individuals; community members may or may not trust an institution, but the ongoing commitment of investigators and research teams from an aLHS to partner with and listen to community members can overcome mistrust [ 21 ]. Thus, thorough training for academic investigators and research teams is essential. This training should include how to work collaboratively; how to encourage, elicit, and listen to diverse voices; and how to help partners organize for community and population health. While training in designing studies, reducing bias, and increasing validity are critical for investigators and research teams, skills in relationship building and maintenance and in negotiation and compromise are similarly critical [ 2 , 11 , 34 , 35 ]. For example, when investigators and research teams from an aLHS attend community fairs, church gatherings, community forums, or parties and celebrations, these informal settings help build and nurture trust among partners. These opportunities show commitment and allow attendees to further understand one another. Volunteering with a community organization or serving on local health coalitions are other ways to advance trust and develops genuine and mutually respectful relationships between researchers and communities. In addition, this involvement can open other doors by helping to identify others in the community who may be committed to working together [ 21 ].

Nearly all participants in this study highlighted the need for institutions to explicitly show their value of CEnR. This finding aligns with previous work that identified barriers attributable to institutional culture that shape research agendas and support for CEnR, and challenges in gaining support from institution leadership and top decision-makers [ 8 , 11 ]. Overcoming these challenges and cultural aspects specific to aLHSs likely requires leadership to further explore institutional readiness to serve as a partner to communities, the existence and appropriateness of structures to support CEnR scholarship, investigator and study staff training in CEnR, the extent to which aLHS goals align with principles of CEnR, and whether there is sufficient CEnR expertise within the institution to establish a commitment to CEnR [ 11 , 34 , 35 ]. Participants echoed the need for CEnR training opportunities (for investigators, research teams, and community members); education on effective models of engagement across diverse populations; resources to conduct CEnR authentically; and guidance for interpreting findings and disseminating information back to communities.

This study was conducted at a single aLHS; thus, findings may not be applicable to other aLHSs. Regardless, the study’s design and analysis may help other aLHSs and medical centers, as well as those seeking to incorporate CEnR approaches within public health and medicine, to engage and form authentic and long-lasting partnerships with communities. The sample size achieved saturation across interviews and provides valuable information regarding the number of investigators and research teams conducting CEnR. In this study, we did not collect data from community research partners, which was beyond the scope of our work. Future research is warranted to better understand perspectives of community members and community organizations with or without previous experience partnering with investigators and research teams within aLHSs.

5. Conclusions

While some findings from this study reflect the broader CEnR literature, these findings are important for informing CEnR approaches and can be used to guide the incorporation of CEnR within aLHSs. LHSs and aLHSs are becoming more established and numerous, and many institutions could benefit from our findings. In educating the next generation of academic investigators, research teams, and healthcare providers, an aLHS can incorporate CEnR as it strives to understand and increase health equity, reduce health disparities, and improve community and population health.

Author Contributions

Conceptualization, S.D.R.; methodology, S.D.R., M.B.I., K.R.M. and P.S.; software, S.D.R.; validation, M.B.I., D.H., I.R. and P.S.; formal analysis, S.D.R. and M.B.I.; investigation, M.B.I., S.D.R. and K.R.M.; resources, S.D.R.; data curation, S.D.R. and M.B.I.; writing—original draft preparation, S.D.R. and M.B.I.; writing—review and editing, S.D.R., M.B.I., K.R.M., L.M.-J., D.H., I.R., P.S., J.A.S. and S.S.D.; supervision, S.D.R.; project administration, K.R.M.; funding acquisition, S.D.R. All authors have read and agreed to the published version of the manuscript.

This research was funded by National Center for Advancing Translational Sciences, National Institutions of Health, grant number UL1TR001420.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board (or Ethics Committee) of Wake Forest School of Medicine (protocol code IRB00049597; date of initial approval: 04/10/2018, renewed annually).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Open access
• Published: 13 May 2024

The TransFORmation of IndiGEnous PrimAry HEAlthcare Delivery (FORGE AHEAD): economic analysis

• Aleksandra Stanimirovic   ORCID: orcid.org/0009-0000-8066-8876 1 , 2 , 3 , 4 , 5 ,
• Troy Francis 1 , 2 , 3 , 5 ,
• Susan Webster-Bogaert 6 ,
• Stewart Harris 6 ,
• Valeria Rac 1 , 2 , 3 , 4 , 5 on behalf of

the LIONESS study group

Health Research Policy and Systems volume  22 , Article number:  57 ( 2024 ) Cite this article

80 Accesses

1 Altmetric

Metrics details

Indigenous populations have increased risk of developing diabetes and experience poorer treatment outcomes than the general population. The FORGE AHEAD program partnered with First Nations communities across Canada to improve access to resources by developing community-driven primary healthcare models.

This was an economic assessment of FORGE AHEAD using a payer perspective. Costs of diabetes management and complications during the 18-month intervention were compared to the costs prior to intervention implementation. Cost-effectiveness of the program assessed incremental differences in cost and number of resources utilization events (pre and post). Primary outcome was all-cause hospitalizations. Secondary outcomes were specialist visits, clinic visits and community resource use. Data were obtained from a diabetes registry and published literature. Costs are expressed in 2023 Can$.

Study population was ~ 60.5 years old; 57.2% female; median duration of diabetes of 8 years; 87.5% residing in non-isolated communities; 75% residing in communities < 5000 members. Total cost of implementation was $1,221,413.60 and cost/person $27.89. There was increase in the number and cost of hospitalizations visits from 8/$68,765.85 (pre period) to 243/$2,735,612.37. Specialist visits, clinic visits and community resource use followed this trend.

Considering the low cost of intervention and increased care access, FORGE AHEAD represents a successful community-driven partnership resulting in improved access to resources.

Peer Review reports

Chronic diseases, such as diabetes, cancer, cardiovascular disease and chronic respiratory diseases, are major contributors to a reduced quality of life, increased risk of hospitalization and premature death [ 1 ]. These major chronic diseases, along with mood and anxiety disorders, account for about one-third of direct healthcare expenditures in Canada [ 1 , 2 ].

The chronic disease burden is heightened in Indigenous populations, including First Nations communities, who experience significantly worse health outcomes, with diabetes prevalence rates that are 3–5 times higher than the general population [ 3 , 4 , 5 ]. In addition, the rates of complications from diabetes are also 2 to 5 times higher in First Nations communities than the general population [ 6 , 7 ]. Chronic diseases have also been shown to impact the life trajectory and life expectancy of First Nations population. A Canadian study found that the burden of chronic diseases among 19 First Nations communities was higher at a relatively younger age in comparison to the general population [ 6 , 7 ]. In some cases, Indigenous populations have been shown to have a life expectancy 10–15 years less than non-indigenous populations [ 8 ].

Distrust between Indigenous populations in Canada and the healthcare system remains a significant driver of health inequities [ 9 ]. Significant gaps in care and higher rates of adverse health outcomes have also been identified within Indigenous populations due to a myriad of factors, including structural determinants of health, negative stereotyping and stigmatization [ 10 ]. A history of colonization that has undermined Indigenous culture and spiritual practices has created a lack of trust within Indigenous populations towards the healthcare system and has given rise to a structural barrier in utilizing the health system [ 6 ].

Given the dramatic rise of chronic diseases and gaps in care in Indigenous populations in Canada, a shift from the dominant episodic and responsive healthcare model most common in First Nations communities to one that places emphasis on proactive culturally appropriate prevention and chronic disease management is urgently needed [ 11 ].

The progressive nature of these diseases has substantial health and cost implications due to lost productivity and hospitalizations and poses a significant burden on patients, their families, society, and the healthcare system [ 2 , 5 ]. There is a growing need to modify healthcare systems to improve the implementation, appropriateness, and effectiveness of healthcare delivery [ 12 , 13 ].

The TransFORmation of IndiGEnous PrimAry HEAlthcare Delivery (FORGE AHEAD) research program

The TransFORmation of IndiGEnous PrimAry HEAlthcare Delivery (FORGE AHEAD) Research Program, co-designed with First Nations communities across Canada, aimed to improve chronic disease care and access to available resources by developing and evaluating community-driven, culturally-relevant primary healthcare models using quality improvement theory and processes [ 11 ]. FORGE AHEAD, initiated in 2013 and described in detail elsewhere, [ 11 ] was directed by a strong multidisciplinary and cross-jurisdictional research team that included First Nations community representatives, Indigenous and non-Indigenous healthcare providers, clinician scientists, academic researchers, policy decision-makers, knowledge-users, and collaborators. Many were nationally recognized leaders in their fields and all stakeholders had experience in working collaboratively with First Nations communities. FORGE AHEAD was guided by the principles of OCAP ® [OCAP ®  is a registered trademark of the First Nations Information Governance Centre (FNIGC)] [ 14 ] and community-based participatory research (CBPR) [ 15 ]. OCAP ® , standing for ownership, control, access, and possession, asserts that First Nations alone have control over data collection processes in their communities, and that they own and control how this information can be stored, interpreted, used or shared. CBPR ensures culturally appropriate processes by involving communities as equal partners in all phases of the research process from planning through to knowledge exchange. The Model for Improvement [ 16 ] was the process used to guide quality improvement (QI). The Model for Improvement is a modest yet powerful method for accelerating improvements by identifying priorities and testing different ideas for change through Plan-Do-Study-Act (PDSA) cycles. The Diabetes Registry and Surveillance System [ 7 , 17 ] was used to facilitate the development of community diabetes registries and clinical data.

The FORGE AHEAD objectives [ 11 ] were achieved through a series of 10 activities, two of which were implementing community and clinical quality improvement initiatives with evaluation including costing.

Including an economic assessment into the evaluation of quality improvement initiatives, such as FORGE AHEAD, is necessary when we seek to determine the sustainability of these programs and to lobby for the potential scaling up of culturally relevant healthcare models to improve trust in healthcare systems and chronic disease management within underserved First Nations communities. Additionally, the costs and cost-effectiveness of this program and others like it have not yet been reported.

Objectives of this economic assessment

The overarching goal of this assessment is to perform an economic evaluation to determine the cost of the FORGE AHEAD QI intervention and its implementation, and whether the QI intervention was cost-effective compared with standard diabetes care using a pre-post design and measured by the cost/resource utilization avoided. More specifically, the objectives of this study are to:

Identify the relevant resource categories (e.g., costs associated with organizing and operating the initiative); (descriptive analysis of relevant resource categories)

Assess resource use including hospital care, specialists’ visits and community care; (descriptive costing analysis)

Compare resource utilization and costs with the standard diabetes care provided in the same communities in the pre-intervention period (cost-effectiveness analysis).

Health economic analysis plan

This analysis is a study-based pre-post economic evaluation with two components: a descriptive cost analysis; and a cost-effectiveness study looking at resource utilization and attached costs. The analysis was conducted in accordance with the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) [ 18 ] statement while following Canadian Agency for Drugs and Technologies in Health (CADTH) guidelines [ 19 ] for economic evaluation in Canada.

Setting and location

The intervention was implemented in First Nations communities across Canada. The communities were located in Ontario, British Columbia, Alberta, Manitoba, Quebec and Newfoundland and Labrador [ 11 ].

Communities were recruited into the program through self-expressed interest in response to personal communication or regional distribution of program information with participation confirmed by a signed research and financial agreement. FORGE AHEAD partnered with 11 communities across six provinces and various isolation levels (Table  1 ). The program was completed by 9 communities, with 8 communities that collected clinical data using the Diabetes Registry and Surveillance System [ 7 , 17 ].

Study population

The study population consisted of all adults (≥ 18 years) diagnosed with type 2 diabetes (T2DM) from FORGE AHEAD community partners who were registered and had clinical data in the Diabetes Registry and Surveillance System.

Specific settings and locations were considered as FORGE AHEAD team partnered with 11 First Nations communities across six provinces (British Columbia, Alberta, Manitoba, Ontario, Quebec, Newfoundland and Labrador) and three isolation levels (isolated, non-isolated, and remote-isolated/semi-isolated).

Intervention description

The FORGE AHEAD quality improvement program was implemented in two 18-month waves: 5 communities in wave 1 and 6 communities in wave 2 [ 11 ]. Each community assembled both a clinical team and a community team and each team was led by a FORGE AHEAD-trained Community Facilitator. The teams completed readiness questionnaires and attended a series of three quality improvement workshops. The workshops provided quality improvement training and fostered a culture of change via plenaries with expert presenters, teams sharing community strengths and challenges, and breakout sessions where teams planned QI initiatives. The workshops were interspersed with 3-month action periods when the teams implemented their quality improvement initiatives with coaching support. FORGE AHEAD-trained Community Data Keepers managed the Diabetes Registry and Surveillance System, collecting baseline and follow-up patient-level clinical information and entered data on specialist visits, hospitalizations and the use of other community resources [ 11 ].

Study perspective and comparators

The study was undertaken using a payer perspective, using the Ontario Ministry of Health (OMH) as the payer. The comparator was the diabetes care received prior to the intervention implementation [ 20 ].

Time horizon and discount rate

We considered the lifetime of the QI intervention period (18 months). In line with CADTH guidelines [ 19 ], we did not consider a 1.5% discount rate as there were no costs and consequences that went beyond 1 year.

Selection, measurement and valuation of outcomes

Outcomes for the descriptive cost analysis were the total cost of intervention’s implementation and the cost per patient. In the second part of the economic evaluation, pre-post downstream resource utilization, the primary outcome was all-cause hospitalizations (with Length of Stay). Secondary outcomes were: all-cause specialist visits; clinic visits; and community resource use. Clinic visits included: interactions with Community Health Representatives; visits with a Nurse Practitioner; physician visits; visits with a Registered Dietitian; and visits with a Registered Nurse. Community resource use included: counselling for addiction; dialysis treatment; foot care procedures; mental health programs; nutrition programs; physical activity and healthy weight initiatives; psychosocial counselling; smoking cessation programs; substance abuse awareness programs; and wound care clinics.

Measurement and valuation of resources and costs

Data source included study-derived data and relevant published literature. For the descriptive cost analysis, costing data came directly from the FORGE AHEAD program, which provided detailed information on all costs. In the second part of the economic evaluation—pre-post downstream resource utilization—Data sources were derived from relevant published sources. More specifically, costs related to hospitalizations were estimated using Canadian Institute of Health Information (CIHI) resources. Costs for specialist visits were obtained using the 2021 Ontario Health Insurance Plan (OHIP) billing data. While FORGE AHEAD was implemented across Canada, in this assessment we took on the perspective of the OMH and therefore used OHIP billing data. Please note that as podiatrist services are not covered by OHIP, we used the average cost (Can$90.00) of a podiatrist’s visit as covered by private insurance in Ontario.

Data collection

For the descriptive cost analysis, we collected the costs in broad categories: staff (e.g., salaries, fee for services); equipment (e.g., equipment and depreciation costs, software costs, installation costs, and maintenance costs); communication (e.g., data transmission costs, modem costs, networking costs); administration (e.g., administration costs, supplies); and healthcare resource utilization.

For the pre-post downstream resource utilization analysis, we used a bottom-up approach to tally healthcare resource utilization costs over the study duration. Healthcare resource utilization costs consisted of hospitalizations and other relevant services as decided with the study stakeholders.

Currency, price date, and conversion

All costs are reported in 2023 Canadian dollars (2023Can$) using the Bank of Canada Consumer Price Index for inflation to a given base year [ 21 ].

Analytical methods

A study based economic evaluation has two parts. The first is a descriptive cost analysis of the aggregate and direct medical costs associated with the FORGE AHEAD QI initiative (including all QI activities). The second is a cost-effectiveness analysis to estimate whether the adoption of the FORGE AHEAD program represents a beneficial use of OMH resources. We assessed the cost-effectiveness of FORGE AHEAD as incremental differences in cost and number of resources utilization events (before and after program implementation). We did not consider a traditional cost-effectiveness approach where we only perceive lower costs and increased effectiveness in a positive sense.

In this study, we considered the intervention effective if it improved access and quality of patient care among participating communities, reflecting that increased use of resources results from improved equity and increased access to care resources in First Nations communities. The pre-intervention period was defined as May 2015 to April 2016 and the post-intervention period as May 2016 to May 2017.

Patient demographics

In Table  1 , we present characteristics of the 8 communities who used the Diabetes Registry and Surveillance System and the individuals ( N  = 2810) entered into the System who had information for both baseline and follow-up visits. Most communities (87.5%) were not isolated, had less than 5000 members (75%), and used electronic medical records (62.5%). Only 37.5% of communities had an onsite primary care physician. Individuals were on average 60.5 years old; 57.2% female; and had a median duration of diabetes of 8 years.

Descriptive analysis of relevant resource categories

Descriptive costs included program implementation costs incurred during the in-person and video conference workshops in participating First Nations communities. The total cost of program implementation over 18 months was $1,221,413.60. When we took into account the total population residing in the 11 communities ( N  = 43,793) where the FORGE AHEAD program was implemented, the cost per person was $27.89 (Table  2 ). Labour and staff salaries were the main driver of cost.

The cost per person rose to $434.67 (Table  2 ) when the calculation was limited to the 8 communities with 2810 individuals listed in the Diabetes Registry and Surveillance System with baseline data.

Descriptive analysis of resource use: specialists’ visits vs community care

Primary outcome—all-cause hospitalizations (and length of stay).

As noted in Table  3 there were 8 hospital admissions in the pre-intervention period and 243 hospital admissions in the post-intervention period. On average, the cost of hospital admission increased from $8,240.64 in the pre-intervention period to $10,639.26. The average patient length of stay in the hospital was 9.7 days in the pre-intervention period, with an average patient’s length of stay of 6.9 days in the post-intervention period. There was a minimal change to median hospital length of stay (from 4 to 3.5 days respectively) (Table  3 ).

In Table  4 , we provide information on the top reasons for hospital admissions. The majority of hospitalizations in both pre- and post-periods were non-diabetes related. We note that diabetes-related complications ( N  = 44), including diabetic ketoacidosis foot ulcers, cellulitis, gangrene, foot infections, necrotic toes, septic shock secondary to necrotizing fasciitis and osteomyelitis diabetic foot, remained the top reasons for hospital admissions.

In Table  5 , we present the findings related to the cost of hospitalizations, where we note that majority of hospitalizations in this analysis occurred in the post-intervention period. The total cost of the program implementation and hospital admissions in the post-intervention period was $3,957,025.97, or a cost per person of $90.35.

Secondary outcomes—specialist visits, clinic visits and community resource use

Table 6 displays trends in the use of other care resources: specialist visits; clinic visits; and community resource use. There was an increase in use health resources from the pre-intervention period to post-intervention period. Specialist visits, clinic visits and community resource use rose from 222 (pre) to 301 (post), 1853 (pre) to 2091 (post) and from 358 (pre) to 578 (post) respectively. Related to specialist visits, the main drivers of cost in the pre-intervention period were ophthalmologist visits, cardiologist visits and surgical consults. The main drivers of cost in the post-intervention period were cardiologist visits, endocrinologist visits and ophthalmologist visits. In reference to clinic visits, the main drivers of cost in the pre-intervention period were consultations with Nurse Practitioners and Community Health Representatives. In the post-intervention period, the main drivers of cost were physicians’ visits and consultations with Nurse Practitioners. The main drivers of cost in community resource use in the pre-intervention period included foot and wound care appointments. Foot care and wound care remained the main drivers of cost in the post-intervention period.

Cost-effectiveness analysis (pre-post resource utilization)

As illustrated in Table  5 , there was an increase in both the number and cost (N,$) of hospital admission visits from 8/$68,765.86 (pre-intervention period) to 243/$2,735,612.37 (post-intervention period).

There was an increase in use and costs (N,$) of specialist visits; clinic visits; and community resource use from the pre-intervention period to post-intervention period. More specifically, specialist visits, clinic visits and community resource use rose from 222/$27,378.42 (pre) to 301/$34,080.12 (post), 1853/$46,425.28 (pre) to 2091/$70,415.00 (post) and from 358/$26,906.50 (pre) to 578/$39,712.17 (post) respectively (Table  6 ).

This analysis is the first study to assess the cost of implementation of a community-driven, culturally relevant QI program and its impact on resource utilization in First Nation communities residing across Canada. The total cost of FORGE AHEAD implementation was $1,221,413.60 and cost/person $27.89 where labour and staff salaries were considered the main driver of cost. We note an increase in both the number and cost (N,$) of hospital admission visits from 8/$68,765.85 (pre period) to 243/$2,735,612.37 (post period). Results illustrate that specialist visits, clinic visits and community resource use followed this trend. Notably, there was an increase in resource utilization in the post-intervention period. Post-intervention impact on the increased use of healthcare resources should not be interpreted in the traditional cost-effectiveness context, where we assume decrease in costs and increase in effectiveness in a positive sense. Rather, they should reflect that increased use of resources resulted in improved equity and access to care resources in the participating First Nation communities.

FORGE AHEAD was the first Canadian study to demonstrate that a partnership with local clinical and community teams implementing a QI intervention can lead to improvements in diabetes management. The authors found that individuals were more likely to receive ≥ 75% of clinical practice guideline (CPG) recommended services compared to baseline (OR: 1.51; 95%CI: 1.27, 1.80) [ 22 ].

Findings from the FORGE AHEAD study align with other parts of the world that used a QI approach to target chronic disease care in Indigenous settings [ 23 ]. The ABCD study (Australia) demonstrated improvements in delivery of diabetes services across several health centres through three annual cycles. The initial ABCD report (2007) summarized results from 12 health centres indicated an improvement in HbA1c testing from 41 to 72% and an increase in the proportion of people at target HbA1c (< 7.0%) from 19 to 28% after two annual cycles. A subsequent ABCD report (2011) indicated that delivery of overall preventative services increased across 36 health centres through three annual cycles of continuous QI from 31 to 44% and delivery of diabetes services increased from 57 to 63% with improvements sustained in the fourth annual cycle. Contrary to ABCD study, in FORGE AHEAD we did not observe significant change in HbA1c. Similar to the ABCD study, the Indian Health Services (IHS) national T2DM audit and feedback program in the United States has demonstrated a decrease in mean HbA1c from 9% in 1996 to 8.1% in 2014 [ 24 ], representing yet another example how a community-led intervention can improve diabetes management. The IHS Division of Diabetes, Area Diabetes Consultants and the Tribal Leaders Diabetes Committee work with the Special Diabetes Programs for Indians (SDPI) grants to provide diabetes treatment and prevention services to IHS, Tribal and Urban Indian health programs [ 25 ]. Contrary to FORGE AHEAD which has been conceptualized as community-driven national research program that partners with Indigenous communities in Canada to improve chronic disease care and access to available resources whereas IHS is designed as a process to assess care and health outcomes for American Indians with diagnosed diabetes. Inherently FORGE AHEAD AND IHS may be serving a different purpose, resulting in different outcomes.

The increasing demand on the healthcare system to deliver evidence-based practice with scarce resources has led to a need to evaluate the cost-effectiveness of healthcare improvement and knowledge translation strategies [ 26 ]. This economic assessment found that the FORGE AHEAD program improved access to quality care and reduced underutilization of health services among First Nations communities. Underutilization of health services among chronic non-communicable disease sufferers, such as diabetes mellitus (DM), is recognized as a significant contributing factor to increased morbidity and mortality. Underutilization of health services can be perceived as failure to adopt an affordable health service that is highly possible to improve the quality or quantity of life [ 27 ]. It manifests behaviorally when individuals do not seek medical care when feeling ill or suspecting they should go [ 28 , 29 ]. Additionally, the underuse of health services lowers process-related healthcare costs in the long term because the necessary care is not provided, conversely, increasing the use of health services increases short-term costs [ 29 ]. Improvements in the quality of care provided will require an increase in costs initially; however, improvements could lower long-term costs by slowing disease progression, which may reduce complications and hospital readmissions [ 30 , 31 ].

Current economic frameworks lack many elements important to the worldview of Indigenous communities. An example of a framework that does include these elements is the First Nations Mental Wellness Continuum (FNMWC) (Fig.  1 ) [ 32 ]. FNMWC is a national framework that addresses mental wellness through culturally safe delivery of services among First Nations communities in Canada. While the framework is focused on mental health, it clearly depicts the tiers, complexities, and intricacies of Indigenous community involvement as interventions and programs are implemented, based on: Culture as a Foundation; Community Development, Ownership and Capacity Building; Quality Care Systems and Competent Service Delivery; Collaboration with Partners and Enhanced Flexible Funding. It supports culturally safe delivery of services. One of the key elements emphasized through FNMWC is the degree/level of Community Involvement to promote change. The economic evaluation of the FORGE AHEAD initiative, similarly, incorporated the community perspectives and how the intervention influences resources utilization at the Community level. FORGE AHEAD was co-designed with leaders from First Nation communities. In Indigenous settings, following the principles of OCAP ® is critical to facilitate the participation and development of the community to result in effective and culturally relevant clinical strategies/programs to improve chronic disease outcomes. Yet there remains a gap in how we evaluate Spiritual health using an economic lens. Future work should consider inclusion of cultural sensitivity.

First Nations Mental Wellness Continuum (FNMWC)

We propose a de novo Decision Determinants (DD) framework, Health Technology Assessment (HTA) framework for the evaluation of programs/interventions implemented in Indigenous populations [ 33 ]. The DD framework is similar in several respects to Health Technology Assessment (HTA) frameworks but is based on an explicit set of social values. HTA (evidence based medicine, economics, and bioethics/social science) is used to aggregate decision attributes and is rooted in a theoretical framework of optimal decision making rather than one related to broad social goals, such as health or welfare maximization. The purpose of the DD framework is to embrace a more holistic approach while considering the Spiritual lens of Indigenous populations. The framework will integrate the community aspect in evaluation, resulting in achievement of common goals, promoting interaction among members to address member concerns with sensitivity meanwhile celebrating heritage and tradition. Finally, for successful intervention implementation and adoption in Indigenous populations, it is fundamental to co-design and co-evaluate programs and interventions in partnership with an Indigenous evaluator.

Indigenous populations living in Canada are among the highest-risk populations for diabetes and related complications. Yet there are significant inequities in access to diabetes healthcare and outcomes for these Indigenous populations. Poor success of many diabetes management strategies highlights the limitations of health services when they are socially, culturally, and contextually irrelevant. Future studies need to include a more holistic approach and community involvement, with the DD framework being culturally sensitive to Indigenous populations by considering the value in combining elements of Western/Indigenous medicine.

FORGE AHEAD is the first Canadian study to demonstrate that a community-led QI intervention can improve diabetes management and healthcare access. Considering the intervention’s low cost and its potential to improve equity and access to care, FORGE AHEAD epitomizes a successful community-driven and culturally based partnership resulting in improved equity and access to resources in participating communities.

Availability of data and materials

The datasets generated during and/or analyzed during the current study are not publicly available to respect the wishes of our partnering First Nations communities and individual participants.

Betancourt M, Roberts K, Bennett T, Discoll R, Jayaraman G, Pelletier L. Monitoring chronic diseases in Canada: the Chronic Disease Indicator Framework. Ottawa: Public Health Agency of Canada; 2014. Contract No.: 1.

The Public Health Agency of Canada. The Economic Burden of Illness in Canada, 2010. Ottawa: Public Health Agency of Canada; 2017.

Dyck RF, Hayward MN, Harris SB, Group CS. Prevalence, determinants and co-morbidities of chronic kidney disease among First Nations adults with diabetes: results from the CIRCLE study. BMC Nephrol. 2012;13:57.

Article   PubMed   PubMed Central   Google Scholar  

Golden SH, Brown A, Cauley JA, Chin MH, Gary-Webb TL, Kim C, et al. Health disparities in endocrine disorders: biological, clinical, and nonclinical factors–an Endocrine Society scientific statement. J Clin Endocrinol Metab. 2012;97(9):E1579–639.

Tompkins JW, Mequanint S, Barre DE, Fournie M, Green ME, Hanley AJ, et al. National Survey of Indigenous primary healthcare capacity and delivery models in Canada: the TransFORmation of IndiGEnous PrimAry HEAlthcare delivery (FORGE AHEAD) community profile survey. BMC Health Serv Res. 2018;18(1):828.

Hayward MN, Mequanint S, Paquette-Warren J, Bailie R, Chirila A, Dyck R, et al. The FORGE AHEAD clinical readiness consultation tool: a validated tool to assess clinical readiness for chronic disease care mobilization in Canada’s First Nations. BMC Health Serv Res. 2017;17(1):233.

Harris SB, Naqshbandi M, Bhattacharyya O, Hanley AJ, Esler JG, Zinman B. Major gaps in diabetes clinical care among Canada’s First Nations: results of the CIRCLE study. Diabetes Res Clin Pract. 2011;92(2):272–9.

Article   PubMed   Google Scholar  

Public Health Agency of Canada. Health Status of Canadians 2016: Report of the Chief Public Health Officer—How Healthy Are We?. Government of Canada, 2017.

Vogel L. Broken trust drives native health disparities. Can Med Assoc J. 2015;187(1):E9.

Article   Google Scholar  

Gracey M, King M. Indigenous health part 1: determinants and disease patterns. Lancet. 2009;374(9683):65–75.

Naqshbandi Hayward M, Paquette-Warren J, Harris SB, On behalf of the FAPT. Developing community-driven quality improvement initiatives to enhance chronic disease care in Indigenous communities in Canada: the FORGE AHEAD program protocol. Health Res Policy Syst. 2016;14(1):55.

Hutchison B, Levesque J-F, Strumpf E, Coyle N. Primary health care in Canada: systems in motion. Milbank Q. 2011;89(2):256–88.

Friedberg MW, Hussey PS, Schneider EC. Primary care: a critical review of the evidence on quality and costs of health care. Health Aff (Millwood). 2010;29(5):766–72.

First Nations Information Governance Centre. The First Nations Principles of OCAP® Akwesasne, ON: First Nations Information Governance Centre; 2024. Available from: https://fnigc.ca/ocap-training/ .

Jagosh J, Bush PL, Salsberg J, Macaulay AC, Greenhalgh T, Wong G, et al. A realist evaluation of community-based participatory research: partnership synergy, trust building and related ripple effects. BMC Public Health. 2015;15(1):725.

Langley GL, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP. The improvement guide: a practical approach to enhancing organizational performance. San Francisco: Jossey-Bass Publishers; 2009.

Google Scholar  

Pace R, Hiwi BT, Zaran H, Harris S. 24—implementation and evaluation of a diabetes registry in first nations communities in Canada: data from the FORGE AHEAD study. Can J Diabetes. 2020;44(7):S11.

Husereau D, Drummond M, Augustovski F, de Bekker-Grob E, Briggs AH, Carswell C, et al. Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. BMJ. 2022;376: e067975.

CADTH. Guidelines for the economic evaluation of health technologies. Ottawa, Canada; 2017.

Tikkanen R, Osborn R, Mossialos E, Djordjevic A, Wharton G. International Health Care System Profiles2020. Available from: https://www.commonwealthfund.org/international-health-policy-center/countries/canada .

Bank of Canada. Inflation Calculator Ottawa: Bank of Canada; 2021. Available from: https://www.bankofcanada.ca/about/ .

Hayward MN, Pace R, Zaran H, Dyck R, Hanley AJ, Green ME, et al. Closing the indigenous health gap in Canada: results from the TransFORmation of IndiGEnous PrimAry HEAlthcare delivery (FORGE AHEAD) program. Diabetes Res Clin Pract. 2020;162: 108066.

Bailie R, Si D, Connors C, Weeramanthri T, Clark L, Dowden M, et al. Study protocol: Audit and Best Practice for Chronic Disease Extension (ABCDE) Project. BMC Health Serv Res. 2008;8:184.

Indian Health Service. Changing the Course of Diabetes: Turning Hope into Reality. Rockville, Maryland: Indian Health Service; 2014.

Indian Health Service. The Federal Health Program for American Indians and Alaska Natives Rockville, Maryland2022 [cited 2022 June 30]. Available from: https://www.ihs.gov/diabetes/about-us/ .

de la Perrelle L, Radisic G, Cations M, Kaambwa B, Barbery G, Laver K. Costs and economic evaluations of Quality Improvement Collaboratives in healthcare: a systematic review. BMC Health Serv Res. 2020;20(1):155.

Hu H, Jian W, Fu H, Zhang H, Pan J, Yip W. Health service underutilization and its associated factors for chronic diseases patients in poverty-stricken areas in China: a multilevel analysis. BMC Health Serv Res. 2021;21(1):707.

McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348(26):2635–45.

Nuckols TK, Escarce JJ, Asch SM. The effects of quality of care on costs: a conceptual framework. Milbank Q. 2013;91(2):316–53.

Anderson GF, Chalkidou K. Spending on medical care: more is better? JAMA. 2008;299(20):2444–5.

Article   CAS   PubMed   Google Scholar  

Hussey PS, Wertheimer S, Mehrotra A. The association between health care quality and cost: a systematic review. Ann Intern Med. 2013;158(1):27–34.

Health Canada. First Nations Mental Wellness Continuum Framework Summary Report. Ottawa: Health Canada; 2015.

Krahn M, Miller F, Bayoumi A, Brooker AS, Wagner F, Winsor S, et al. Development of the Ontario decision framework: a values based framework for health technology assessment. Int J Technol Assess Health Care. 2018;34(3):290–9.

Download references

We acknowledge the funding received from Canadian Institutes of Health Research, Canada (funding reference numbers #MCO 117675, RFN #297910, and #PME-133824), Grant from AstraZeneca Canada Inc. Canada (Dr. Stewart Harris—investigator initiated grant) and support from The Lawson Foundation, Canada. No funding partners were involved in the design of the study, data collection, analysis, interpretation of the data, writing the manuscript, or in the decision to submit this paper for publication.

Author information

Authors and affiliations.

Program for Health System and Technology Evaluation, Toronto General Hospital Research Institute, University Health Network, 10th Floor, Eaton North, 200 Elizabeth Street, Toronto, ON, M5G 2C4, Canada

Aleksandra Stanimirovic, Troy Francis & Valeria Rac

Ted Rogers Centre for Heart Research at Peter Munk Cardiac Centre, Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada

Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

Diabetes Action Canada, CIHR SPOR Network, Toronto, ON, Canada

Aleksandra Stanimirovic & Valeria Rac

Toronto Health Economics and Technology Assessment Collaborative, Toronto General Hospital Research Institute, University Health Network, Toronto, ON, Canada

Centre for Studies in Family Medicine, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada

Susan Webster-Bogaert & Stewart Harris

You can also search for this author in PubMed   Google Scholar

Contributions

AS, TF and VR have conceived of study design, data analysis, interpretation of findings and the broader discussion. SWB and SH have been responsbile for data collection; interpretation of findings and the broader discussion. All authors contributed to manuscript write up.

Corresponding author

Correspondence to Aleksandra Stanimirovic .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval for the FORGE AHEAD Program was received from the Western University Health Sciences Research Ethics Board ((#103895, approved June 17, 2013), the Health Research Ethics Board of Alberta (CHC-14-0054, approved December 1, 2014), the Cree Board of Health and Social Services of James Bay (#2014-DSP-03, approved October 2, 2014), Mi’kmaw Ethics Watch, Unama’ki College, Cape Breton University (approved January 29, 2014), and Mi’kmaq Confederacy Ethics Review Committee, Prince Edward Island (approved March 14, 2014).

Competing interests

AS, TF and VR declare no competing interests. SWB was funded from the grant supporting this study. SH has received consulting/advisory fees from AstraZeneca as well as research funds from AstraZeneca.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Stanimirovic, A., Francis, T., Webster-Bogaert, S. et al. The TransFORmation of IndiGEnous PrimAry HEAlthcare Delivery (FORGE AHEAD): economic analysis. Health Res Policy Sys 22 , 57 (2024). https://doi.org/10.1186/s12961-024-01135-5

Download citation

Received : 23 September 2023

Accepted : 30 March 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s12961-024-01135-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Indigenous populations
• Diabetes and its complications
• Community care
• Economic analysis

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

There’s a powerful story behind every headline at Ohio State Health & Discovery. As one of the largest academic health centers and health sciences campuses in the nation, we are uniquely positioned with renowned experts covering all aspects of health, wellness, science, research and education. Ohio State Health & Discovery brings this expertise together to deliver today’s most important health news and the deeper story behind the most powerful topics that affect the health of people, animals, society and the world.  Like the science and discovery news you find here? You can support more innovations fueling advances across medicine, science, health and wellness by giving today.

BROUGHT TO YOU BY

• The Ohio State University
• College of Dentistry
• College of Medicine
• College of Nursing
• College of Optometry
• College of Pharmacy
• College of Public Health
• College of Veterinary Medicine
• Ohio State Wexner Medical Center
• Ohio State's Comprehensive Cancer Center – James Cancer Hospital and Solove Research Institute

Subscribe. The latest from Ohio State Health & Discovery delivered right to your inbox.

New Healthy Community Center transforms historic library into beacon of wellness

Senior Staff Writer, Photographer Ohio State Wexner Medical Center

• Share on Facebook
• Share on Linkedin
• Share via Email
• Share this page

On a bright, sunny afternoon in early May , nearly 300 members of Columbus’ Near East Side gathered to celebrate their new community center. Among the first to arrive was Ann B. Walker, a centenarian deeply woven into the fabric of the neighborhood.

Seated in the front row at the ribbon-cutting ceremony of The Ohio State University Wexner Medical Center Healthy Community Center, Walker looked thoughtfully at the transformation of the building located on Long Street in the historically Black neighborhood. Once bustling with readers as the Martin Luther King Jr. branch of the Columbus Metropolitan Library, the structure has been turned into a hub of health and wellness, thanks to a $5.1 million investment from The Ohio State University Wexner Medical Center , which will operate the community center.

Walker, who will turn 101 this year, had good reason to reminisce. As an award-winning TV journalist, she once interviewed Rev. Martin Luther King Jr.

“I was just thinking about how much things have changed,” Walker shares, her gaze lingering on an image of King Jr. installed in the building. “It’s great that it can now be used as a health center. There are people in the area who need services and don’t have access to them.”

The Healthy Community Center is more than just a building: it’s a hub for the neighborhood, focusing on promoting healthier lifestyles. With plans to host dietary and exercise workshops, health screenings  and educational health programming, it’s a gathering space that’s geared to meet the real needs of the community.

From books to health: A legacy reimagined

“The history of this place begins with a longing to learn,” says Joshua Joseph, MD , the center’s medical director, at the opening ceremony.

He shared the center’s journey from its origin as the city’s East Side library in the 1960s. He recalled the words of Martin Luther King Sr., who spoke at the library’s dedication in 1969:

“You have a beautiful space conducive to reading, but books are worth nothing unless you read them.”

Dr. Joseph says those words are reflected in the building’s new purpose: “It’s a beautiful building, but it means nothing if it’s not filled every day with programming and people improving health and wellness."

Read about the events and programming taking place at the Healthy Community Center

Programming geared for a healthy community

The transformation was a true community effort, says the center’s director, Javonte McDonald. It required a commitment to understanding and meeting community needs, which influenced every aspect of the center, from the state-of-the-art teaching kitchen for nutrition classes to the café that supports local entrepreneurs to an art gallery that spotlights area artists.

“It will help address pressing needs in creative and collaborative ways to improve the health of the communities that we serve,” says John J. Warner, MD, chief executive officer of the Ohio State Wexner Medical Center and executive vice president at Ohio State.

Local artists contribute artwork displayed in the Healthy Community Center.

The Healthy Community Center helps fill a gap in providing access to healthy foods to area residents.

A demonstration kitchen will be used to teach healthy cooking and nutrition skills to community members.

Members of the community and leaders from the Ohio State Wexner Medical Center cut the ribbon for the grand opening of the Healthy Community Center.

The Healthy Community Center honors the legacy of the building's former use as the Martin Luther King Jr. branch of the Columbus Metropolitan Library.

Community members at the grand opening of the Healthy Community Center.

Dr. Joseph says his team is working closely with community partners to match services with community needs. “Ohio State has a vested interest in supporting healthy lifestyle changes so everyone has the opportunity to lead longer, healthier lives,” he says.

The Healthy Community Center will have a café with indoor and outdoor seating. The space will serve as a business incubator for an independent restaurant owner. The center’s partners include the Mid-Ohio Food Collective, Partners Achieving Community Transformation and the Growing and Growth Collective.

Longtime resident Pamela Shields, who founded the Urban Aging Residents Coalition, emphasized the center’s role in bridging health disparities .

“Our focus is health, wealth and wellness. We aim to get African Americans, people of color and marginalized groups healthy, attending their medical appointments and involved in research studies,” she says.

A neighborhood cornerstone

As the community looks ahead, the Healthy Community Center is set to become a bedrock of the Near East Side.

“My greatest hope is that the entire community sees this center as a resource,” says Julialynne Walker, with the Bronzeville Growers Market. She’s also Ann B. Walker’s daughter. “If everyone welcomes, embraces and uses it fully, we will see a change in our community’s health over time.”

Support initiatives that change lives like the Healthy Community Center

Your funding of the Ohio State Wexner Medical Center directly impacts our community.

• Community Health Care ,
• Community Outreach ,
• Community Resources ,
• Community Transformation ,
• Health and Wellness Center ,
• Healthy Communities ,
• Healthy Community Center ,
• Community Health
• Health and Society

Related websites

Articles on community health.

Study finds pressure to be perfect leads to parental burnout

Pressure to be perfect parents leads to burnout, mental health concerns.

Making health equity more than a dream

By Alayna DeMartini

Obstacles along his life’s path made Chyke Doubeni, MD, MPH, dedicate his career to helping removing barriers so all people, regardless of their race or income, can be healthy.

Parenting’s new crisis: the loneliness epidemic

Family’s passion fuels healthy transitions for adults with autism.

By Rebecca Walters

How Ohio State celebrates organ transplants as ‘Buckeyes for Life’

By Susannah Kistler

Get articles and stories about health, wellness, medicine, science and education delivered right to your inbox from the experts at Ohio State.

Required fields

Tell us more about yourself

By clicking "Subscribe" you agree to our Terms of Use . Learn more about how we use your information by reading our Privacy Policy .

• Português Br
• Journalist Pass

Welcoming the community into Alzheimer’s disease research studies

Lynda De Widt

Share this:

Sochenda Stephens is the program manager for the Alzheimer's Disease Research Center (ADRC) at Mayo Clinic in Florida . Her team is reaching out to communities in Jacksonville that are underrepresented in healthcare to attract more people to join research studies. Here in her own words, she talks about the work of the team.

What types of studies are underway at the Alzheimer's Disease Research Center?

We have multiple active observational studies that focus on brain health and aging and the diagnosis of Alzheimer's disease through analyzing fluid and neuroimaging biomarkers. We also have several active clinical trials that focus on developing medications that may slow or stop the progression of dementia or prevent the onset of cognitive symptoms in healthy people who are at high risk of developing symptoms of Alzheimer's disease.

How have you and the ADRC been able to increase access to research participation in underrepresented communities?

One key strategy has been expanding our Clinical Research Coordinators (CRCs) team, focusing specifically on increasing engagement within underrepresented communities. By tripling the number of CRCs focused on community outreach and diversifying our team to better reflect our community, we have enhanced our ability to engage with underrepresented populations, build relationships, educate and inform them about the importance of research participation.

Notably, we have six coordinators and three clinician-researchers who speak Spanish. Also, we have established partnerships with faith- and community-based organizations and leaders within underrepresented communities to foster trust and facilitate access to research opportunities.

What is the purpose of the new curriculum for clinical research coordinators?

Clinical research on the Florida campus has grown rapidly in the past several years. Our newly hired research coordinators complete a robust six-week onboarding and new-hire training curriculum that provides the resources and knowledge to equip them for success in their roles.

What do these clinical research coordinators do in the community?

Our CRCs play a crucial role in community outreach. They actively engage with underrepresented communities to better understand their research and health needs, provide education, raise awareness about research opportunities, address concerns or misconceptions about research participation and facilitate enrollment in research studies. They participate in community events and establish and maintain ongoing relationships with community members to ensure continued engagement and participation.

How else is the Alzheimer's Disease Research Center supporting research in underrepresented communities?

In addition to expanding our CRC team, the ADRC is involved in various initiatives to support research in underrepresented communities. This includes providing culturally sensitive education and outreach materials, offering transportation assistance for participants, collaborating with local healthcare providers to enhance awareness of research opportunities and involving community members in the research process through advisory boards and focus groups.

We are working with community organizations on give-back events, such as a book bag and school supplies giveaway, to involve the entire family. We also prioritize sharing individual findings with participants when appropriate and with Institutional Review Board approval. They can take this back to their primary care physicians to discuss next steps. We share community-based research in newsletters or at events.

What are some of the community events you support to inform people about Mayo Clinic research?

In 2023, we organized or participated in more than 40 community events, reaching more than 5,000 community members. We provided education on brain health and dementia risk factors, raised awareness about research opportunities at Mayo Clinic in Florida, and fostered dialogue with community members.

These events included health fairs, informational seminars, workshops, memory screenings and community forums. Also, we actively participated in cultural festivals, neighborhood gatherings, and outreach events hosted by community organizations to engage with diverse audiences and share information about our research initiatives. Our goal is to make research accessible, relevant and engaging for all community members.

How do you build trust with underrepresented populations before educating them about research studies?

Building trust in underrepresented populations is fundamental to our outreach efforts. We prioritize a "give first" approach by actively listening to the concerns and perspectives of our community members and facilitating a transparent, open, and culturally sensitive conversation. Our team takes the time to educate and address any misconceptions about research, establish meaningful relationships based on mutual respect and understanding, and enhance overall brain health whenever possible.

Why is this work important to you?

Our work addresses health disparities and ensures that all people, regardless of background or socioeconomic status, have equitable access to cutting-edge research opportunities. Black and Hispanic people have two times and 1.5 times higher risk of Alzheimer's disease and related dementia (ADRD) respectively, compared to non-Hispanic white people, yet they are underrepresented in ADRD studies, including clinical trials. By actively engaging with underserved communities, we can advance key scientific knowledge for the prevention, diagnosis, and treatment of ADRD, promote health equity, and improve outcomes for all members of our community.

• Mayo Clinic Health System celebrates grand opening of Mankato hospital expansion and modernization  Mayo Clinic and Mercy reach first major milestone in data collaboration

Related Articles

• Skip to main content
• Keyboard shortcuts for audio player

• Your Health
• Treatments & Tests
• Health Inc.
• Public Health

California's $12 billion Medicaid experiment stretches the definition of health care

Angela Hart

Alondra Mercado, a community health worker with the Central California Asthma Collaborative, helps provide services through an ambitious California Medicaid initiative. On a recent morning in March, she visited a family in Turlock to teach a mother how to control in-home asthma triggers that cause flare-ups in her young son. Angela Hart/KFF Health News hide caption

Alondra Mercado, a community health worker with the Central California Asthma Collaborative, helps provide services through an ambitious California Medicaid initiative. On a recent morning in March, she visited a family in Turlock to teach a mother how to control in-home asthma triggers that cause flare-ups in her young son.

TURLOCK, Calif. — For much of his young life, Jorge Sanchez regularly gasped for air, at times coughing so violently that he'd almost throw up. His mother whisked him to the emergency room late at night and slept with him to make sure he didn't stop breathing.

"He's had these problems since he was born, and I couldn't figure out what was triggering his asthma," Fabiola Sandoval said of her son, Jorge, now 4. "It's so hard when your child is hurting. I was willing to try anything."

In January, community health workers visited Sandoval's home in Turlock, a city in California's Central Valley where dust from fruit and nut orchards billows through the air. They scoured Sandoval's home for hazards and explained that harsh cleaning products, air fresheners, and airborne dust and pesticides can trigger an asthma attack.

The team also provided Sandoval with air purifiers, a special vacuum cleaner that can suck dust out of the air, hypoallergenic mattress covers, and a humidity sensor — goods that retail for hundreds of dollars. Within a few months, Jorge was breathing easier and was able to run and play outside.

The in-home consultation and supplies were paid for by Medi-Cal, California's Medicaid health insurance program for low-income residents.

Gov. Gavin Newsom is spearheading an ambitious $12 billion experiment to transform Medi-Cal into both a health insurer and a social services provider, one that relies not only on doctors and nurses, but also community health workers and nonprofit groups that offer dozens of services, including delivering healthy meals and helping homeless people pay for housing .

Shots - Health News

She has medicare and medicaid. so why should it take 18 months to get a wheelchair.

Should Medicaid pay to help someone find a home? California is trying it

These groups are redefining health care in California as they compete with businesses for a share of the money, and become a new arm of the sprawling Medi-Cal bureaucracy that serves nearly 15 million low-income residents on an annual budget of $158 billion .

But worker shortages, negotiations with health insurance companies, and complex billing and technology systems have hamstrung the community groups' ability to deliver the new services: Now into the third year of the ambitious five-year experiment, only a small fraction of eligible patients have received benefits.

"This is still so new, and everyone is just overwhelmed at this point, so it's slow-going," said Kevin Hamilton, a senior director at the Central California Asthma Collaborative.

Fabiola Sandoval's son, Jorge Sanchez, got a relief from his asthma thanks to Medi-Cal's new funding for social services. Angela Hart/KFF Health News hide caption

The collaborative has served about 3,650 patients, including Sandoval's son Jorge, in eight counties since early 2022, he said. It has years of experience with Medi-Cal patients in the Central Valley and has received about $1.5 million of the new initiative's money.

By contrast, CalOptima Health, Orange County's primary Medi-Cal insurer, is new to offering asthma benefits and has signed up 58 patients so far.

"Asthma services are so difficult to get going" because the nonprofit infrastructure for these services is virtually nonexistent, said Kelly Bruno-Nelson, CalOptima's executive director for Medi-Cal. "We need more community-based organizations on board because they're the ones who can serve a population that nobody wants to deal with."

Meet basic needs, reduce health care costs down the line

Newsom, a Democrat in his second term, says his signature health care initiative, known as CalAIM , seeks to reduce the cost of caring for the state's sickest and most vulnerable patients, including homeless Californians, foster children, former inmates, and people battling addiction disorders.

In addition to in-home asthma remediation, CalAIM offers 13 broad categories of social services, plus a benefit connecting eligible patients with one-on-one care managers to help them obtain anything they need to get healthier, from grocery shopping to finding a job.

The goal, Newsom says, is to keep people healthier and avoid costly care such as emergency room visits, ultimately saving taxpayer money.

The 25 managed-care insurance companies participating in Medi-Cal can choose which services they offer, and contract with community groups to provide them. Insurers have hammered out about 4,300 large and small contracts with nonprofits and businesses.

So far, about 103,000 Medi-Cal patients have received CalAIM services and roughly 160,000 have been assigned personal care managers, according to state data , a sliver of the hundreds of thousands of patients who likely qualify.

"We're all new to health care" and learning to navigate the bureaucracy "is such a foreign concept," said Helena Lopez, executive director of A Greater Hope , a nonprofit organization providing social services in Riverside and San Bernardino counties, such as handing out baseball cleats to children to help them be active.

Fabiola Sandoval (left) has struggled to help her son, Jorge Sanchez, control his asthma since he was an infant. Alondra Mercado (right), a community health worker, secured several items for her including air filters, cleaning products, pillow and mattress covers, and a specialized vacuum that can suck dust out of the air. These items were covered by California's Medicaid. Angela Hart/KFF Health News hide caption

Fabiola Sandoval (left) has struggled to help her son, Jorge Sanchez, control his asthma since he was an infant. Alondra Mercado (right), a community health worker, secured several items for her including air filters, cleaning products, pillow and mattress covers, and a specialized vacuum that can suck dust out of the air. These items were covered by California's Medicaid.

Small nonprofits, big start-up costs to get paid by Medicaid

Tiffany Sickler runs Koinonia Family Services , which offers California foster children mental health and other types of care, and even helped a patient pay off parking tickets. But the program is struggling on a shoestring budget.

"If you want to do this, you have to learn all these new systems," for getting paid through CalAIM, she said. "It's been a huge learning curve, and very time-consuming and frustrating, especially without adequate funding."

Brandon Richards, a Newsom spokesperson, defended CalAIM, saying that it was "on the cutting edge of health care" and that the state was working to increase "awareness of these new services and support."

For nonprofits and businesses, CalAIM is a money-making opportunity — one that top state health officials hope to make permanent. Health insurers, which receive hefty payments from the state to serve more people and offer new services, share a portion with service providers.

In some places, community groups are competing with national corporations for the new funding, such as Mom's Meals, an Iowa-based company that delivers prepared meals across the United States.

Mom's Meals has an advantage over neighborhood nonprofit groups because it has long served seniors on Medicare and was able to immediately start offering the CalAIM benefit of home-delivered meals for patients with chronic diseases. But even Mom's Meals isn't reaching everyone who qualifies because doctors and patients don't always know it's an option, said Catherine Macpherson, the company's chief nutrition officer.

"Utilization is not as high as it should be yet," she said. "But we were well positioned because we already had departments to do billing and contracting with health care."

Middleman companies also have their eye on the billions of CalAIM dollars and are popping up to assist small organizations go up against established ones like Mom's Meals. For instance, the New York-based Nonprofit Finance Fund is advising homeless service providers how to get more contracts and expand benefits.

Full Circle Health Network , with 70 member organizations, is helping smaller nonprofit groups develop and deliver services primarily for families and foster children. Full Circle has signed a deal with Kaiser Permanente, allowing the health care giant to access its network of community groups.

"We're allowing organizations to launch these benefits much faster than they've been able to do and to reach more vulnerable people," said Camille Schraeder, chief executive of Full Circle. "Many of these are grassroots organizations that have the trust and expertise on the ground, but they're new to health care."

One of the biggest challenges community groups face is hiring workers, who are key to finding eligible patients and persuading them to participate.

Kathryn Phillips, a workforce expert at the California Health Care Foundation, said there isn't enough seed money for community groups to hire workers and pay for new technology platforms. "They bring the trust that is needed, the cultural competency, the diversity of languages," she said. "But there needs to be more funding and reimbursement to build this workforce."

Health insurers say they are trying to increase the workforce. For instance, L.A. Care Health Plan, the largest Medi-Cal insurer in California, has given $66 million to community organizations for hiring and other CalAIM needs, said Sameer Amin, the group's chief medical officer.

"They don't have the staffing to do all this stuff, so we're helping with that all while teaching them how to build up their health care infrastructure," he said. "Everyone wants a win, but this isn't going to be successful overnight."

In the Central Valley, Jorge Sanchez is one of the lucky early beneficiaries of CalAIM.

His mother credits the trust she established with community health workers, who spent many hours over multiple visits to teach her how to control her son's asthma.

"I used to love cleaning with bleach" but learned it can trigger breathing problems, Sandoval said.

Since she implemented the health workers' recommendations, Sandoval has been able to let Jorge sleep alone at night for the first time in four years.

"Having this program and all the things available is amazing," said Sandoval, as she pointed to the dirty dust cup in her new vacuum cleaner. "Now my son doesn't have as many asthma attacks and he can run around and be a normal kid."

This article was produced by KFF Health News, a national newsroom that produces in-depth journalism about health issues and is one of the core operating programs at KFF . KFF Health News is the publisher of California Healthline , an editorially independent service of the California Health Care Foundation .

• social determinants of health

This paper is in the following e-collection/theme issue:

Published on 16.5.2024 in Vol 26 (2024)

Person-Generated Health Data in Women’s Health: Scoping Review

Authors of this article:

• Jalisa Lynn Karim 1 , BA, BMath   ; 
• Rachel Wan 1 , BSc, BSN, RN   ; 
• Rhea S Tabet 2 , BSc   ; 
• Derek S Chiu 3 , BSc, MSc   ; 
• Aline Talhouk 1 , BA, MSc, PhD  

1 Department of Obstetrics and Gynaecology, University of British Columbia, Vancouver, BC, Canada

2 Department of Pharmacology and Therapeutics, McGill University, Montréal, QC, Canada

3 Department of Molecular Oncology, University of British Columbia, Vancouver, BC, Canada

Corresponding Author:

Aline Talhouk, BA, MSc, PhD

Department of Obstetrics and Gynaecology

University of British Columbia

593 - 828 West 10th Ave

Vancouver, BC, V5Z 1M9

Phone: 1 604 875 3111

Email: [email protected]

Background: The increased pervasiveness of digital health technology is producing large amounts of person-generated health data (PGHD). These data can empower people to monitor their health to promote prevention and management of disease. Women make up one of the largest groups of consumers of digital self-tracking technology.

Objective: In this scoping review, we aimed to (1) identify the different areas of women’s health monitored using PGHD from connected health devices, (2) explore personal metrics collected through these technologies, and (3) synthesize facilitators of and barriers to women’s adoption and use of connected health devices.

Methods: Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for scoping reviews, we searched 5 databases for articles published between January 1, 2015, and February 29, 2020. Papers were included if they targeted women or female individuals and incorporated digital health tools that collected PGHD outside a clinical setting.

Results: We included a total of 406 papers in this review. Articles on the use of PGHD for women steadily increased from 2015 to 2020. The health areas that the articles focused on spanned several topics, with pregnancy and the postpartum period being the most prevalent followed by cancer. Types of digital health used to collect PGHD included mobile apps, wearables, websites, the Internet of Things or smart devices, 2-way messaging, interactive voice response, and implantable devices. A thematic analysis of 41.4% (168/406) of the papers revealed 6 themes regarding facilitators of and barriers to women’s use of digital health technology for collecting PGHD: (1) accessibility and connectivity, (2) design and functionality, (3) accuracy and credibility, (4) audience and adoption, (5) impact on community and health service, and (6) impact on health and behavior.

Conclusions: Leading up to the COVID-19 pandemic, the adoption of digital health tools to address women’s health concerns was on a steady rise. The prominence of tools related to pregnancy and the postpartum period reflects the strong focus on reproductive health in women’s health research and highlights opportunities for digital technology development in other women’s health topics. Digital health technology was most acceptable when it was relevant to the target audience, was seen as user-friendly, and considered women’s personalization preferences while also ensuring accuracy of measurements and credibility of information. The integration of digital technologies into clinical care will continue to evolve, and factors such as liability and health care provider workload need to be considered. While acknowledging the diversity of individual needs, the use of PGHD can positively impact the self-care management of numerous women’s health journeys. The COVID-19 pandemic has ushered in increased adoption and acceptance of digital health technology. This study could serve as a baseline comparison for how this field has evolved as a result.

International Registered Report Identifier (IRRID): RR2-10.2196/26110

Introduction

The practice of keeping notes to monitor one’s health is not a recent phenomenon. Individuals have long recognized the benefits of tracking various health aspects, including the ability to be more active participants in managing their health, gaining a more complete picture of their health, and reducing the frequency of in-person appointments; however, this tracking was previously done through paper logs [ 1 ]. Today, with the proliferation of digital tools, self-tracking has significantly evolved and become more prevalent. The increasing pervasiveness of technology, particularly mobile phones, has seamlessly integrated it into our daily lives, making self-tracking more accessible and convenient than ever before [ 2 ]. Connected digital health technologies such as smartphones, wearables (eg, smartwatches), sensors, the Internet of Things (eg, internet-enabled weight scales), and web-based applications have permeated society and are increasingly adopted to collect and track health data. In 2021, a total of 87% of Canadians owned a smartphone, up by 73% from 2009 [ 3 ]. With >350,000 digital health apps accessible via these smartphones [ 4 ], approximately two-thirds of Canadians digitally track at least one aspect of their health [ 5 ]; similar statistics have been reported in the United States [ 6 ]. Moreover, since the introduction and popularization of fitness trackers in 2010, sensors and wearable devices have increasingly become part of daily life [ 2 ]. During the global COVID-19 pandemic, self-tracking took on even greater significance [ 7 , 8 ]. With the heightened awareness of health and the need for proactive measures, individuals have turned to self-tracking to monitor their well-being and make informed decisions. With this transformation, self-tracking has transcended its previous boundaries, offering individuals new opportunities to optimize their well-being and ushering in a new era of personalized health care [ 9 - 11 ].

Digital health tools have revolutionized the active and passive collection of health data through various applications and wearable devices. These various digital health tools collect and generate an unprecedented amount of data that can be used to glean insights into one’s health. Person-generated health data (PGHD), which are clinically relevant data captured outside traditional care settings [ 12 ], provide valuable insights that empower users to self-monitor and reflect on their health. PGHD can refer to any data collected from wearable and smart devices as well as self-input information into platforms such as mobile apps and websites. By leveraging digital technologies, individuals can collect and store their health data, enabling them to actively manage their own health and monitor chronic conditions. Furthermore, the integration of these data with research presents an opportunity to improve the patients’ experience and enhance personalized medicine. The recognition of this opportunity has started to take shape with patient-reported outcome measures and patient-reported experience measures being increasingly recognized as essential information to assess quality of care and prioritize patient-centered approaches and with mandatory assessment as part of clinical trials [ 13 ]. Seamlessly linking PGHD that are captured outside traditional care settings with clinical data and disease models can unlock new possibilities for tailored treatments and predictive informatics. The integration of digital health tools not only facilitates patient-provider communication but also offers opportunities for education, increased awareness, self-tracking, and self-monitoring without burdening health care resources. By focusing on the individual’s experience, personalization, and prevention, digital health tools contribute to a patient-centered care paradigm that aims to optimize health care outcomes and improve overall well-being while empowering patients to take charge of their health.

In recent years, the emergence of femtech, defined as technology-driven solutions specifically designed to address women’s health needs and concerns, has revolutionized the landscape of self-tracking and health care for women [ 14 ]. Femtech encompasses a wide range of digital tools, such as period-tracking apps, fertility monitors, pregnancy trackers, and menopause management platforms. These innovative solutions empower women to track and manage their reproductive health, menstrual cycles, and overall well-being with greater accuracy and ease. Femtech has not only provided women with personalized insights into their bodies but has also helped break taboos and encouraged open conversations about topics that were once stigmatized or ignored. The rapid growth of femtech has promoted access to women’s health information, greater autonomy in decision-making, and enhanced overall health care experiences for women worldwide. It has become an integral part of the self-tracking movement, demonstrating the transformative power of technology in promoting women’s health and well-being.

In this study, we reviewed the use of digital tools and PGHD in women’s health research, focusing on articles published between January 1, 2015, and February 29, 2020, before the COVID-19 pandemic. Our review encompassed various connected health devices, which included both passive data collection devices such as wearable sensors and active input devices such as smartphone apps and websites. This review sought to accomplish the following:

• Identify the different areas of women’s health and health-related behaviors monitored using PGHD from connected health devices.
• Explore personal metrics collected through these technologies.
• Synthesize facilitators and barriers that impact women’s adoption and use of connected health devices in managing their health.

This scoping review was conducted based on our previously published protocol [ 15 ]. We adopted the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [ 16 ]. The completed checklist is provided in Multimedia Appendix 1 [ 16 ].

Search Strategy

The search strategy was designed in close collaboration with a reference librarian with input from the authors (JLK and AT). We searched a total of 5 databases: MEDLINE, Embase, APA PsycINFO, CINAHL Complete, and Web of Science Core Collection. Initial searches were completed in early March 2020. Searches were limited to articles published in 2015 or later because publications with the keyword “digital health” started to emerge in the literature around that time [ 17 ], and with the fast evolution of the field, previous articles may not be relevant to the current landscape. Keywords and subject headings were designed to search the literature for the intersection of the following 4 topics: women, health, digital devices, and tracking. The full search strategy, including a full list of search terms, was published with the protocol [ 15 ] and is available in Multimedia Appendix 2 .

Eligibility Criteria

We were interested in digital technologies and interventions targeting women and people assigned female at birth. To be included in the review, studies needed to specifically target women, focus on female-only health topics (eg, menstruation), or only include female participants. We included a variety of publication types but excluded conference abstracts and conference reviews, editorials, letters, and comments due to the limited details in such literature.

We excluded articles that presented digital health tools designed for health care providers as we were primarily interested in devices and apps that women can engage with outside a clinical setting. Articles only discussing the use of real-time consultations, whether through video, phone, or web-based chat, were excluded. We excluded articles that described digital health tools used solely for educational purposes; to maintain the focus of the review on tracking or monitoring one’s data for health, devices must have allowed users to input personal health data.

The complete inclusion and exclusion criteria are presented in Textbox 1 . We decided to retain the original inclusion end date of February 29, 2020, to maintain a focus on the literature before the COVID-19 pandemic and avoid potential complexities caused by pandemic-related disruptions in research and health care practices. Concentrating on prepandemic literature also established a clear baseline for future comparisons and allowed us to maintain feasibility of completion without compromising quality given the broad scope of the review.

Inclusion criteria

• Published between January 1, 2015, and February 29, 2020
• Refers to a health issue that pertains only to women or comprises only female participants of any age
• Includes the use of connected health tools for tracking or monitoring some aspect of health, which could include smartphone apps, wearable devices, the Internet of Things (eg, Bluetooth- or internet-enabled glucometers, blood pressure cuffs, and weight scales), and implantable devices
• Involves data collection from the user of the connected health tool (ie, the user either manually inputs data into the device or they are automatically uploaded)
• The user must be able to interact with the app or device on her own at home (outside a clinical setting)
• Available in English

Exclusion criteria

• Not available in English
• Conference abstracts, conference reviews, editorials, letters, or comments
• Study media releases and user reviews of specific applications
• Research conducted on animals
• Research involving male participants
• Tracking of infants and children unless tracking breastfeeding (because breastfeeding is directly related to the mother’s health and body)
• Devices or apps that are meant for health care provider use or use in a clinical setting only or cannot be used independently without a health care provider present
• Digital health tools that are only for educational or informational purposes and do not allow the user to enter or track her own data (ie, no information exchange)
• Telemedicine services (eg, live video consultations with health care providers)

Study Selection

We imported the results from the database searches to the Covidence systematic review software (Veritas Health Innovation). Covidence detected records believed to be duplicates, and these were manually checked before removing them. In addition, some articles were manually recognized as duplicates during the screening process and were subsequently tagged as duplicates and removed. Screening was conducted independently by at least 2 reviewers (JLK, RST, and AT) at both the abstract screening stage and the full-text screening stage. We attempted to contact the corresponding authors of articles that passed abstract screening when we were unable to locate the full text. Conflicts at either stage were discussed and agreed upon among the 3 authors involved in the screening process.

Data Charting and Deviations From the Protocol

The final list of data charting elements is provided in Textbox 2 . Data charting for all elements except for usability and acceptability was conducted using Google Sheets created by the study team. The categories for different data charting options were initially created based on a small subset of articles and were discussed among the authors involved in the charting process. The team met regularly throughout the data charting process to discuss and refine coding categories that best summarized the data. Starting with more granular categories and later combining them into broader concepts was necessary to summarize the number of articles included in this review. For each article included, data were charted by one reviewer (RW or RST) and verified for accuracy by a second reviewer (JLK). Data were summarized in bar graphs, maps, and tables (JLK, RST, and DSC), as presented in the following sections. For the locations, we recorded the countries from which the participants were recruited (if applicable). If an article did not describe recruiting participants, then the countries of the authors were recorded based on the authors’ affiliations.

Article information

• Year of first publication

Study characteristics

• Country or countries in which the research was conducted
• Research study type

Contexts for women’s connected health

• Health areas of focus

Digital device details

• Types of digital health
• Metrics collected by the devices

Usability and acceptability

• Facilitators of and barriers to the use of the technologies (coded into themes)

For the thematic analysis, articles that mentioned any aspect of usability, acceptability, facilitators, or barriers to the use of digital health tools were imported into NVivo (R1 2020; QSR International). Coding was done independently by 2 reviewers (JLK and RW) and then combined through discussions. As with the data charting process, we initially coded more granularly and then grouped the detailed codes together later in the analytic process. Decisions on how to group the codes into themes and subthemes were made through group consensus (JLK, RW, and AT).

In our protocol, we indicated that we would extract the name of the device or app used in each study. While we did complete this step in our data charting, we have not presented the results in this paper. Several articles either did not specify the brand name (eg, only specified that it was a mobile app) or had digital health tools named after the study, so we did not find this information useful to showcase in our results. There were no other deviations from the published protocol.

The searches identified 14,629 records that were imported into the Covidence software for deduplication and screening. After deduplication, a total of 9102 articles were screened for relevance, and 8545 (93.88%) were excluded based on title and abstract. From reading the full texts of the remaining 557 records, an additional 151 (27.1%) were excluded. The most common reasons for exclusion were the inability of study participants to enter or track their own data (58/151, 38.4%) or because the digital health technology was designed to be used by or with a health care provider (48/151, 31.8%). The remaining 406 publications were included in the scoping review. Some of the included publications reported on the same research project; in those cases, all of them were included. Our search did not encounter any articles that directly addressed or mentioned the inclusion of intersex, transgender, or nonbinary participants. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram detailing the full study selection process is shown in Figure 1 . The list of included articles sorted by health areas of focus can be found in Multimedia Appendix 3 [ 18 - 58 ].

Year and Country

There was an increasing trend in number of publications per year, with 10.1% (41/406) of the articles published in 2015, a total of 13.3% (54/406) of the articles published in 2016, a total of 18% (73/406) of the articles published in 2017, a total of 26.4% (107/406) of the articles published in 2018, and 29.6% (120/406) of the articles published in 2019. Only 2.7% (11/406) of the publications were from 2020 because our cutoff date for inclusion was February 29, 2020.

Articles included in the review covered worldwide research, including every continent except Antarctica ( Figure 2 ). As we only considered articles written in English, most of the articles were published in Western, English-speaking countries, primarily the United States (169/406, 41.6% of the articles), the United Kingdom (34/406, 8.4% of the articles), Australia (33/406, 8.1% of the articles), and Canada (19/406, 4.7% of the articles). Other countries where several included articles were published were China (13/406, 3.2% of the articles), the Netherlands (13/406, 3.2% of the articles), Spain (13/406, 3.2% of the articles), and Sweden (10/406, 2.5% of the articles).

Interestingly, of the 169 articles from the United States, 26 (15.4%) specifically focused on African American or Black, ethnic minority, or low-income women. One study from Singapore specifically included multiethnic women [ 18 ], and a study from Australia included Indigenous Australian women as their participants [ 19 ]. In addition, one review conducted by researchers in Australia looked specifically at studies with women from culturally and linguistically diverse backgrounds [ 20 ].

Study Types

The types of studies that used digital health tools in women’s health research are reported in Figure 3 by year of publication (note that the articles could fall into more than one study category). The most common study type encountered was feasibility or acceptability studies (197/406, 48.5% of the articles, including 9/197, 4.6% protocols), followed by effectiveness studies (146/406, 36% of the articles, including 36/146, 24.7% protocols) and publications reporting on digital tool prototypes (73/406, 18% of the articles). Effectiveness studies reported on outcome measures of an intervention, including randomized and nonrandomized trials with one or more study arms. Reviews (of published literature, apps, or wearables), viewpoints, manuals, case studies, or analytical methods (56/406, 13.8% of the articles combined) were also encountered. Observational or correlative studies (44/406, 10.8% of the articles, including 3/44, 7% protocols) were studies that observed the health behaviors of individuals through digital health technologies without assessing the effectiveness of an intervention or analyzed associations between variables (eg, associations between heart rate and loss-of-control eating) [ 21 ]. Finally, measurement studies (23/406, 5.7% of the articles) reported on the validity, reliability, or accuracy of a digital health tool.

Health Areas of Focus

The analysis of the reviewed articles highlighted research in several recurring women’s health areas of focus. A full breakdown of the health areas is reported in Table 1 (articles could fall into more than one health area). Pregnancy and the postpartum period emerged as the most prominent health area with 42.6% (173/406) of the articles. Within this category, there was a specific emphasis on general care and monitoring (45/173, 26% of the articles), physical activity and diet (34/173, 19.7% of the articles), and glucose monitoring (31/173, 17.9% of the articles). Cancer was identified as the second most common health area, with 19.5% (79/406) of the articles dedicated to its exploration. Specifically, a significant focus was observed on the relationship between cancer and cardiovascular health, with 47% (37/79) of the articles addressing this aspect. The impact of lifestyle on overall health and well-being was also addressed, with 14.3% (58/406) of the articles delving into physical activity, sedentary behavior, diet, weight, and obesity. Menstrual, sexual, and reproductive health were explored in 12.1% (49/406) of the articles to shed light on various aspects of women’s reproductive health and associated concerns, with 76% (37/49) focusing on menstrual cycle tracking or fertility monitoring. Furthermore, 9.9% (40/406) of the articles were dedicated to chronic conditions (such as urinary incontinence, osteoporosis, and diabetes) with the aim of enhancing understanding and developing interventions for individuals living with chronic health conditions. To accommodate articles that did not fit within the primary health areas, an Other category comprising 6.4% (26/406) of the articles was established. This category included articles on athlete monitoring (10/26, 38% of the articles), such as heart rate monitoring during sports tournaments; mental health and quality of life (9/26, 35% of the articles); gender-based violence (3/26, 12% of the articles); and more. Finally, a small subset of 0.5% (2/406) of the articles did not align with any specific health area; these included a publication reporting results from a survey on African American women’s willingness to participate in eHealth research [ 22 ] and a publication analyzing women’s interactions with digital health technologies [ 23 ]. These articles were included because, although they did not discuss a specific health area, they still focused on women’s use of digital health tools in general.

a PCOS: polycystic ovary syndrome.

b CVD: cardiovascular disease.

c COPD: chronic obstructive pulmonary disease.

d SLE: systemic lupus erythematosus.

e IC: interstitial cystitis.

f BPS: bladder pain syndrome.

g ABL: accidental bowel leakage.

Figure 4 shows how the health areas of focus for women’s use of digital health changed over the years that were included in the review (2015-2019 plus January 2020-February 2020). There was an increasing trend from 2015 to 2020 in the number of publications focusing on pregnancy and the postpartum period, as well as cancer and menstrual, sexual, and reproductive health. However, articles focused on women’s use of digital health for lifestyle-related topics and chronic conditions did not see a notable increase over those years.

Type of Digital Health and Metrics Collected

Within the articles reviewed, smartphone, mobile, or tablet apps emerged as the most prevalent type of digital health (295/406, 72.7% of the articles), followed by wearable devices (165/406, 40.6% of the articles) and websites or patient portals (93/406, 22.9% of the articles). Other types of technology were not investigated as much. For example, 13.5% (55/406) of the articles addressed smart devices or the Internet of Things (referring to objects with sensors that connect to a network, such as Bluetooth-enabled glucometers and blood pressure machines). Finally, 7.4% (30/406) of the articles reported on 2-way messaging, 1% (4/406) of the articles reported on interactive voice response telephone calls, and only 0.5% (2/406) of the articles reported on implantable devices. With respect to the metrics collected, we found >250 metrics, such as heart rate, number of steps, mood, ovulation test results, and days of menstruation. A full list of the metrics is reported in Multimedia Appendix 4 .

Thematic Analysis

Of the 406 articles included in this scoping review, 168 (41.4%) mentioned usability, acceptability, facilitators, or barriers to the use of digital health tools at least once. Our thematic analysis identified 6 themes: (1) accessibility and connectivity, (2) design and functionality, (3) accuracy and credibility, (4) audience and adoption, (5) impact on community and health service, and (6) impact on health and behavior. The themes are described in further detail in the following sections.

The thematic analysis detailed in the following sections is primarily based on the views of the participants in the studies we reviewed to provide a user perspective; however, one subsection in theme 5 focuses on the health care provider perspective.

Theme 1: Accessibility and Connectivity

The accessibility and connectivity of digital technologies emerged as an important theme with two subthemes: (1) cost and convenience and (2) connectivity, compatibility, and software issues.

Cost and Convenience

Our analysis revealed that the cost and convenience of digital tools collecting PGHD are important factors that can impact their adoption and use. On the one hand, digital health technologies can be seen as more affordable compared to traditional health care visits and more accessible to a wider range of people, including those of a lower socioeconomic status. On the other hand, they can also be perceived as too expensive and novelty items, and associated extra costs such as data plans can also be a barrier for some people. Because PGHD can be collected and entered throughout the day in real time, and because most people carry a phone around with them every day, these technologies offer greater convenience than traditional in-person health care encounters by providing anytime, anywhere virtual access and putting information at people’s fingertips through smartphones and web platforms. One user spoke about an in-app treatment program:

That was what was so good about this, I can do this at home myself, no need to book an appointment, find the time and suit others, and you know, that process of booking a time. [ 24 ]

Some inconvenient aspects of digital health technologies include uncomfortable wearables that are too bulky, difficulty of use, or not fitting into the users’ lifestyles, as noted in one article:

Women also mentioned that the comfort of the wearable sensors was a barrier. Comfort became a barrier for some women during exercise and hot weather. [ 25 ]

Devices with a short battery life and wearables that are not water resistant are also considered inconvenient as they require the user to frequently remember to charge the device or put the wearable back on after water-based activities. Certain restrictions, such as not being able to wear a device in a workplace, can also create inconvenient barriers for some users.

Connectivity, Compatibility, and Software Issues

Factors related to connectivity and other issues such as device synchronization, freezing, or disconnection can significantly impact the user experience and engagement with digital health tools. For example, the dependence on mobile and internet access can be a disadvantage. Cellphone and network coverage limitations can pose an important barrier in rural areas or during travel. Where mobile data or Wi-Fi connection are limited, people may struggle to use digital health tools that require internet connection; this can create disparities in access to health care resources, particularly for those of a lower socioeconomic status or living in remote communities with limited infrastructure. Incompatibility between operating systems such as Android and iOS, iPhone and iPad, or various browsers can also be an important barrier to accessing digital health technology.

Software issues can significantly impact the user experience of digital health technologies. Broken links can prevent users from accessing valuable information or features within apps or websites. App crashes can discourage users from engaging with the digital health tool altogether. In addition, slow loading times can negatively impact the user experience, making it challenging for users to access information or features quickly and efficiently.

Theme 2: Design and Functionality

The second theme centered on design and functionality and included four subthemes: (1) appearance and design; (2) functionality and features; (3) personalization; and (4) safety, privacy, and security.

Appearance and Design

Appearance and design play important roles in the success of connected health devices. In terms of app design, the color scheme and layout facilitate user-friendliness. Bad formatting can make it difficult for users to read or understand the content of an app or website. Font size that is too small can be challenging for those with visual impairments, and some color combinations can be difficult to read. The quality of the images used in digital health interventions can also impact user experience, with low-quality images potentially making it difficult for users to interpret the information being presented.

Apps that are visually appealing and easy to use are more likely to be successful. When it comes to wearables, women tend to prefer sleek, understated designs that are esthetically pleasing without being bulky. A sleek design can encourage use of the wearable. For example, some women consider their wearable to be a fashion item that sparks conversation, which encourages their continued use of the device, as illustrated in a participant quote:

Um, that it’s, like, kind of stylish, like, I feel, like, cool that I wear one. A lot of people ask me, they’re like, oh, which one is that, like, is that a Fitbit, is that an Apple watch? It has the interchangeable bands and stuff like that, so, you can, like, change the color of it and everything. It’s like a conversation piece. [ 26 ]

Other women prefer more discretion in the design of wearables and their size or in the app icon on their mobile device because they do not want to reveal the purpose of the device to others. People may feel self-conscious when wearing the device or using the app, especially if it reveals their medical condition. For example, the following quote is from a study that incorporated a sensor band worn on the wrist to help female undergraduate students with problematic drinking:

P310 noted that while in class, “my professor commented on it which made me feel awkward.” [ 27 ]

Functionality and Features

In terms of functionality, the availability of clinical interpretation of user data is deemed essential, and health warnings based on recorded PGHD are noted to be helpful. Moreover, notifications and reminders are also useful for improving adherence to self-tracking and maintaining goals, and users appreciate receiving automated SMS text messages and feedback on progress. Actionable advice is seen as very important, and women expressed a desire for more interaction and the ability to integrate with other apps. For example, users want the ability to access information from their health record and to be able to see graphical summaries of their data over time. Regarding the presentation of information, users appreciate concise information written in simple language. Choice of words is also perceived as especially important to ensure that the information is easy to understand. People enjoy the gamification of content, and the graphical presentation of results is found to be informative.

The ability to upload multimedia and the ability to customize the application’s displays and notifications are noted as features that improve user engagement and satisfaction. The ability to record voice notes and consultations within apps is noted as a desirable feature, as well as having the option to book appointments directly through apps. Women also want the option to sync their desktop or phone calendar with apps to remind them of medical appointments and prescription requests, as some researchers noted:

Women could see the potential usefulness of being reminded to order their next prescription through the electronic alerts system. They found managing the monthly prescription requests challenging long-term and found setting up the reminder easy with the alert popping up on their phone or tablet. [ 28 ]

The application also allows women to set appointment reminders to ensure she is not missing her appointments and developing gaps in her care [...] “It allows me to remain organized for my visits to my OB with concerns, questions, symptoms I have experienced since my last visit.” [ 29 ]

Issues that negatively impact user engagement and outcomes are the inability to edit information or unsubscribe from notifications, which are sometimes thought to be either inconvenient or intrusive, as well as the presence of advertisements within the app. Ease of use is essential as apps or websites that are difficult to navigate can discourage users from engaging with them. For example, a study including the use of a mobile phone app reported the following:

A hindrance and disliked aspect was the difficulty in navigating through the app (eg, no back button, clunkiness, and the inability of participants to edit their inputted daily goals) as well as a lack of color and visuals within the app, giving it a clinical appearance. [ 30 ]

Some women are not comfortable answering questions that they consider intrusive, such as those related to sexual health. They rely on applications to provide trusted information about their condition and want suggestions for additional resources such as website links and local information.

Personalization

Women generally expressed a desire for greater personalization across several features within digital health tools. Messages and notifications that are personalized to the user’s health and self-tracking history and goals are more motivational and less likely to be ignored or perceived as irritating. Even factors such as using a first name in messages from the app make women feel like the messages are more personal and supportive.

Users have individual preferences when it comes to the frequency and timing of notifications, and it is important for digital health apps to allow for the customization of these settings as they can greatly impact user engagement and adherence. Moreover, users expressed a desire for the ability to customize their goals and the metrics they tracked. For example, they may want to change their goals in an app when their life circumstances change (eg, moving, starting a new job, becoming pregnant, or sustaining an injury). The ability to customize the dashboard of an app or website according to the user’s goals was also expressed as a desired feature. The ability to make these customizations will improve their adherence in the long run as their goals evolve.

When it comes to wearables, their placement on the body influences users’ preference and adherence to their use. For example, some women may prefer a wrist-worn device, whereas others may prefer a chest strap, a ring, or a device worn on the waist or ankle. The type of activity being monitored may also influence placement preference. A wrist-worn device may be more appropriate for monitoring steps, whereas a chest strap may be better suited for monitoring heart rate during exercise. Furthermore, placement preference may also be influenced by factors such as comfort, convenience, and visibility. A user may prefer a wrist-worn device because it is more visible and easier to access, whereas another user may prefer a device worn on the waist because it is less obtrusive and more comfortable during exercise or sleep. For instance, one study found the following:

Eight of the participants (40%) reported at some point of the long study period that the smart wristbands were uncomfortable to wear, especially at night. The wristbands irritated the skin, possibly due to pregnancy-related swelling. [ 31 ]

Finally, users have different preferences for how they want information to be presented in an app or website. Some people prefer to read content that is written out with citations and links to external websites. Others enjoy learning content from videos or audio recordings. When looking at their trends and progress, some users like to look at detailed graphs showing their daily progress, whereas others prefer to look at the data occasionally and only receive high-level information. The challenges concerning personalization were articulated by several authors:

It’s a difficult one. Some women want the full picture to fully understand what they are taking. Others want a black and white sketch, but not the details. They just want to know enough. Others do not want to see the picture, they just want to get on with it without knowing too much. Catering for all is a challenge. [ 28 ]

Safety, Privacy, and Security

Women are sometimes concerned about the physical safety of certain devices. For example, some mothers worried about their wearable wristbands scratching their babies [ 31 ]. Others worried about the effects of wearable devices on their skin, as expressed by a participant:

It’s weird because it does have a little laser thing on it, and I wonder if that’s, like, harming my skin (laughing). Like, I’ll sleep in it, and when I wake up I’ll have a red spot on my arm, it’s itchy sometime or sensitive, and I think it’s because of the laser thing, but I don’t really know. [ 26 ]

Some women are concerned about the privacy and security of digital health technologies and expect appropriate safeguards to be implemented in the tools they use. However, privacy and data security are not a concern for all women:

As I said, I’m very critical about patient data in general, especially in terms of data security...If you have a free app, it really depends on what happens to the private data. As a matter of fact, usually the information is stored on the app itself, and so other apps might gain access to the data easily. [ 32 ]

The survey revealed a low level of concern about issues relating to privacy or security of personal data. This suggests that privacy concerns were secondary to the benefits offered by uploading personal details into apps to provide the type of customisation they seek. [ 33 ]

Researchers also shared that some users perceived there to be more privacy when using an app as compared to traditional ways of communicating:

Some participants perceived the storage of their glucose levels on the smartphone as more secure than their current registration in a booklet. [ 34 ]

Women, particularly those who worked outside of the home, also commented that they appreciated the added convenience and privacy of this [text-based] communication method over phone-based communication. [ 34 ]

Theme 3: Accuracy and Credibility

In theme 3, we identified accuracy and credibility as important factors for acceptability considerations in digital health technologies.

The accuracy of digital health can impact user trust and adoption. Digital health tools enable users to keep track of their health, symptoms, and behaviors over time without relying on memory recall, which can be inaccurate or incomplete. Many studies reported that digital tracking can lead to more accurate data collection compared to paper-based methods. For example, at-home measurements of blood pressure and other vital signs have been found to be more accurate than those taken in a hospital or clinic setting. In some cases, apps are even able to accurately predict users’ menstrual cycles and mood changes. In addition, food diaries and activity trackers are often found to be more accurate when tracked within the app compared to using traditional paper-based methods. As the following participant conveyed, digital health may also make it easier for patients to tell the truth about their habits or health concerns:

I like this principle because...I know exactly, that via tablet one would admit things you wouldn’t necessarily tell the doctor or nurse. So, for starters, you can state it in the application. Of course, a conversation shouldn’t be missed afterwards, but this might make it easier for you to overcome yourself. [ 32 ]

However, accuracy can still be an issue in digital health. Different devices can produce different measurements, and some devices may miscount steps, the intensity of workouts, or the quantity and quality of sleep. For example, some women reported devices not tracking their steps while pushing a grocery cart or stroller, whereas others found that their steps were overcounted due to arm movements while they were seated. In addition, some users reported that food tracking options in apps were limited and did not include foods from their culture. Therefore, users may perceive digital health tools as not being representative of their true activity, which may lead them to discontinue the use of the devices. The following participant quote refers to a wrist-worn activity tracker:

Out paddling and we’re huffing and puffing and barely breathing and this isn’t even triggering anything. So it shows [...] that our 150 minute goal is like 60 or half of that. But we’ve actually put in the effort and then you just give up after a while. Like there’s no way I can make this. [ 35 ]

Women often prefer evidence-based health information (eg, explanations of conditions and symptoms and health advice) from a trustworthy source, such as an app curated from up-to-date and evidence-based research, over general internet searches. Users reported that the information provided in some apps was incomplete or inaccurate, with gaps in content or contradictory information that diminished their trustworthiness. In such cases, users may still prefer to talk to a health professional for more trustworthy information. Some women may also find it challenging to trust information that does not disclose sources as they are unsure of its reliability. Devices that are endorsed by, cite, and link to trustworthy health sources are more appealing to users. When sharing results from a web-based survey, the authors of one study reported the following:

Some respondents were specific about from where such advice should come, stating that they wanted expert, credible and up-to-date advice while others noted that they would like to see more Australian-specific or locally-based information in apps or apps that were not linked to the manufacturers of pregnancy or baby products. [ 33 ]

Theme 4: Audience and Adoption

Our fourth theme concerns audience and adoption, which includes two subthemes: (1) demographics and inclusivity and (2) timing and circumstances.

Demographics and Inclusivity

One of the challenges with digital health is to avoid one-size-fits-all interventions and to strive to tailor interventions to address the specific needs of different populations. Digital health that targets specific demographic groups or specific health conditions may increase the adoption of digital tools in those populations. That said, even when targeting people with specific health conditions as the audience, attention must be paid to the language and content in apps and websites. Some researchers noted that women did not want to participate or continue in their study because they did not want to constantly be confronted with their disease. Too much of a focus on disease and ill health can deter women from engaging with the tools, as commented on by some authors:

All but one participant preferred text content that focused on health and physical activity rather than content explicit to cancer. [ 36 ]

The women emphasized that less attention should be paid to chronic disease management and medication as the only treatment option. [...] it was important to explain the implications of the result of the scan and the risk of fractures in a way that will not place the women in a sickness role unnecessarily. [...] The knowledge base of osteoporosis should focus on osteoporosis as a common condition instead of a chronic bone disease. [ 37 ]

Younger women are often more familiar with and more comfortable using digital technology and, therefore, are more likely to use and adhere to a digital health protocol. Users with low technology skills want more training on how to use the digital health tools properly. Little provision is made for those for whom English is not their primary language, which can limit the accessibility and usefulness of digital health interventions. Factors such as language barriers, cultural beliefs, or lack of access to technology may lead to less adoption by some people belonging to ethnic minority groups. The relevance and usefulness of digital health may also vary based on geographic location.

Digital health tools are negatively perceived by some users if not designed to be inclusive of attributes such as body type or gender. For example, users prefer applications that use pictures or models that represent a diverse range of body sizes. Digital health technologies may not be gender inclusive and can conflate sex and gender. It is important to consider the unique health needs and experiences of individuals across the gender spectrum, as several researchers reported:

Participants commented on an exercise demonstration video and recommended that the model should have an “everyday-look” (e.g. plain clothes, jewellery). Also a choice of models of different ages to engage a wider range of patients and help them to relate or identify with the model was proposed. [ 38 ]

[Participant quote]: Maybe the body image it presents...like on a lot of apps, the people doing it looked like they were athletes already. And maybe they should have more people that look normal. [ 39 ]

Two women commented on the gendered design of most FTAs. FTA092 commented that “I chose Clue because it’s the only app that wasn’t pink.” FTA051 also found the gendered design of her previous app insulting; “my last app had a pink flower and was called MyDays or something ...I felt like they were trying to lure me in with this kind of ‘women’s’ approach” (FTA051). She subsequently stopped using that app and downloaded Clue. [ 40 ]

Timing and Circumstances

Individuals are more motivated to use digital health tools during times of illness or when they have a specific health goal in mind. The introduction of technology at the appropriate time impacts the utility and effectiveness of digital health interventions, especially when they are integrated into existing health care systems and routines. Digital health apps need to account for existing medical conditions or medical history to ensure accurate and complete information. For example, technologies that do not provide an option to indicate current pregnancy are perceived as frustrating to users as the in-app goals or notifications can be irrelevant and inconsiderate of their current limitations. In a focus group, one mother shared the following:

I get frustrated with the Garmin [smartwatch] because I wear my watch during the night so it tracks my sleeping as well. Then it gives you like an insight—so a little note will pop up and you know whether your sleep has been really regular or you’ve had irregular sleep. I wish that there was a thing that during pregnancy where that I could put in and say I’m pregnant, because I got those notes that your sleep is really irregular, and I was like, “Because I’m pregnant!” [ 23 ]

Users who are not experiencing symptoms or who perceive their health to be good are less likely to adopt digital health tools as they may not perceive any benefit from using them. Moreover, those who are already tracking their health using other methods (eg, paper-based tracking) are less interested in trying a new digital health tool. Similarly, regarding wearables, some people may already have a wearable and be less interested in having an additional wearable device.

Theme 5: Impact on Community and Health Service

This theme considers the impact of PGHD on community and health service, with three subthemes: (1) communication and community support, (2) clinical integration, and (3) health care provider perspective.

Communication and Community Support

One of the many perceived benefits by users of digital health interventions is the sense of community that these platforms enable. Even though some women reported feeling uncomfortable sharing personal information with strangers in a virtual group, most found that the ability to connect with others who shared similar experiences provided a sense of belonging and support that was motivating and reassuring, as shared by one woman:

What I did love about the apps is the forums. So if you have a weird pain or, you know, you have cramp in your legs at three a.m., you can get on your phone straight away, and you can get support by the women who are going through the same thing. [ 41 ]

Discussion forums and social media platforms associated with digital health interventions are perceived as helpful for connecting with others, sharing personal stories, and receiving support. Digital health interventions can also help women elicit support from friends and family to stay motivated and achieve health goals. For example, researchers who reported on women’s experiences of an app for stress urinary incontinence shared that some participants found it easier to talk to friends about an app for pelvic floor muscle training rather than talk about incontinence [ 24 ]. This can enable increased accountability and further encourage adherence to the intervention. One woman spoke about how her family supported her engagement with a digital health intervention for physical activity maintenance among female cancer survivors:

My husband’s a good motivator. When I say I’m going for a walk, he’ll go with me...with my sister-in-law and her kids, it’s they want to go with me; so it’s how many steps have you got today? Or, are we going to go for a walk. That kind of thing. And with my husband and my daughter it’s, “how many steps did you get today, did you do your workout, let us get it going.” [ 36 ]

In addition to support from family, friends, and community members, these digital platforms can provide an alternative to speaking with a health care provider in person. Asynchronous communication with health care providers is helpful especially for those who may not have easy access to in-person visits or for those who are uncomfortable discussing sensitive information face-to-face. Records of PGHD can also improve the ability to gather and share details with health care providers about symptoms that are difficult to remember during an in-person visit.

Clinical Integration

Women are more willing to participate in digital health interventions if they perceive that they have a direct impact on their clinical care. They appreciate the idea that their health is being monitored and that someone is keeping an eye on their data. Furthermore, women want to see more integration of their clinical test results within their digital health apps and websites. This increases their motivation to adhere to the interventions prescribed through the digital health application.

It was noted that physicians and other health care providers play a crucial role in promoting the use of digital health interventions among patients. As noted in the following participant quote, women enjoy being able to communicate with a health care provider through digital health:

I like it because you can tell the doctor what’s going on and submit it to your doctor, that is the main reason I like it because you can talk directly to your doctor and tell them what is going on without going in or calling. [ 42 ]

Women are more likely to adopt and use technology if it is recommended by their health care providers, family members, or friends. Women reported that digital health interventions were more effective when they were supported by a health care team. For example, having access to a health coach or counselor or receiving feedback from a health care provider on their progress increases their motivation to adhere to the interventions. This support also provides reassurance that they are on the right track toward achieving their health goals. However, some patients become frustrated when they receive conflicting advice from the digital health tool and their health care provider.

Health Care Provider Perspective

Some articles included thoughts from health care providers on digital health tools collecting PGHD [ 20 , 37 , 43 - 55 ]. From the health care provider perspective, digital health can offer several benefits, including the ability to monitor patients’ adherence to treatment and interventions. This can be particularly helpful for patients with chronic conditions that require ongoing management. Providers can use digital health tools to track patients’ progress and identify any potential issues that may require further attention, which can lead to improved clinical outcomes and reduce unnecessary consultations. For instance, one provider learned about their patient’s anxiousness through a mobile health intervention:

I didn’t know my patient was feeling anxious...But when she wrote it down, we could talk about it... [ 43 ]

Some health care providers expressed that digital health tracking could give them a more accurate picture of their patients’ activities and adherence to treatments. In a study about perspectives on a sensor attached to pills that can send data such as date and time of ingestion, a provider commented the following:

A positive would be data and getting a better grip on compliance. (...) I’m making sure the patient is adhering - assuming that the patient is taking everything inside of that blister, you can have confirmation of that. [ 44 ]

In addition, digital health can improve the efficiency of care delivery by providing education and resources directly to patients. This can help patients better understand their condition, treatment options, and self-management strategies, which can lead to better health outcomes.

However, it was also noted that digital health interventions should not replace in-person visits but rather complement them. Some health care providers are concerned about overreliance on digital health tools as well as the potential for misinterpretation of the data they provide. There may be a lack of feedback on the correct use of interventions, such as interpretations of medical advice provided, and health care providers have raised concerns about the safety and trustworthiness of the medical advice generated by the digital health tools. Health care providers especially worry about medico-legal effects of having information from digital health tools taken out of context or without considering the full picture of the user’s history and health, as demonstrated in the following quotes:

As a health care professional, I’m just mindful that if there was a video of me up there talking, if that was taken out of context or shared with another person where that information was not appropriate, that’s a concern to me. [ 45 ]

One anesthesiologist raised, “Who has access to the responses that I provide? Because if a patient receives information from me which they hold onto and is taken out of context, in a medical–legal situation, then that’s a big issue as well.” [ 46 ]

Providers may also find that the abundance of information generated by digital health tools can be overwhelming and time-consuming to manage, adding to an already hectic workflow and blurring professional boundaries. Large volumes of alerts and notifications from digital health tools can be disruptive to health care providers, who expressed the need to set boundaries regarding how and when they engaged with digital health tools. In a study reporting on perspectives about digital health from key informants (health care providers and researchers), one participant shared their thoughts on the potential for digital health to increase workload and liability:

Sometimes the more information that we provide for them (doctors), the more work and liability we give them, right? So if they get so much information that becomes actionable but they are overwhelmed, now they would be obligated to do something with this patient, they are in a chain of distribution, a chain of liability. [ 44 ]

Theme 6: Impact on Health and Behavior

Finally, our sixth theme describes the impact of PGHD on health and health behaviors.

Several studies reported that digital health interventions helped users stay motivated and, in turn, improved their health habits and behaviors, such as adherence to medication, physical activity, and healthy eating. The ability of users to look back at their data helps them identify patterns in their health and behaviors, which increases their awareness of their health and habits. The awareness then allows them to be more mindful of their habits and encourages self-reflection, thus promoting a deeper understanding of their health and well-being. The tracking of patterns in their health, combined with the educational component of some digital health tools, helps users come up with better self-management strategies and feel more confident in their ability to reach their health goals, giving them a greater sense of self-efficacy and control over their health. In a digital health intervention aimed at treating lymphedema following breast cancer treatment, a participant spoke of changes in her awareness of symptoms and improvements:

It helped me realize that I had excess fluid. My arms got lighter each time I did the exercises. My arms began to feel less heavy. It noticed it in my clothes as well. [ 56 ]

Digital health interventions are often reported to positively impact the mental health and well-being of individuals. Women reported improvements in their mood, emotional state, and coping abilities. They also reported a reduction in stress and anxiety levels, which can lead to improvements in overall health outcomes. The digital health tools provide users with a sense of support and accountability as well as feelings of accomplishment when meeting their goals.

However, it is important to note that, while digital health interventions can have many benefits, they may not be suitable for everyone and may even have negative effects on some individuals. For example, some users reported increased anxiety due to excessive monitoring or notifications, and others reported negative effects on their thoughts or worsening of symptoms related to health conditions. Some users found that self-tracking made them more attached to their phones, less likely to engage in social activities, and more isolated overall. Care should be taken to ensure that users do not become obsessive about self-tracking as this can be counterproductive or even harmful. Being hyperfocused on their symptoms or health condition could be distressing and even detrimental to their overall well-being. Therefore, it is important to carefully monitor the use of digital health interventions and adjust them as needed to ensure the best possible outcomes for each individual. One woman spoke about her overreliance on an app used to track breastfeeding:

I stopped using it because um I thought I’m being too anal about this...being too concerned about it, I just need to stress less, and just go with the flow and just be a bit more relaxed about it...so, that’s why I stopped using it completely, and then I think the breastfeeding improved from there ’cause I was worrying about it less. [ 57 ]

Table 2 provides a summary of the thematic analysis grouped into barriers and facilitators. It is worth noting that many things are both a barrier and a facilitator (eg, cost) depending on the individual. In addition, the presence of a specific feature may be a facilitator, whereas the absence of it may be a barrier.

Principal Findings

In this scoping review, we summarized information from 406 articles on digital technologies collecting PGHD and how they have been used in women’s health research. We found a steady increase in articles meeting our inclusion criteria from 2015 to 2020, indicating an increasing trend in the uptake and use of digital health tools in women’s health research before the COVID-19 pandemic. Most included studies (310/406, 76.4%) were feasibility or acceptability studies, effectiveness studies, or reports of digital tool prototypes. Most studies (299/406, 73.6%) focused on tracking conditions related to pregnancy or the postpartum period, cancer survivorship, or menstrual, sexual, and reproductive health. Several types of digital health were represented, with the most common being apps, wearable devices, and websites or patient portals. Through our thematic analysis, we found several considerations of facilitators of and barriers to using digital health tools, including the accessibility and convenience of the tools, visual appearance, device functionality and ability to personalize the user experience, and accuracy of the algorithms and information provided. It is also important to consider the target audience to optimize the adoption of the tools. Engagement with digital health tools may help users improve their health and health-related behaviors and gather support from friends, family, and other digital health users. Women are more likely to use digital health if it is recommended by a health care provider, but there are both benefits and challenges that health care providers may face if considering integrating digital health technology into clinical practice.

A previously published scoping review focused on information and communications technologies as a tool for women’s empowerment [ 59 ]. They reported that the concept of empowerment appeared in various ways with no clear consensus on the definition, with some studies mentioning terms such as self-concept, self-esteem, self-worth, and self-efficacy. Our thematic analysis also found that some women’s use of digital health tools increases their self-efficacy in managing their health. Another systematic review of 13 digital health interventions for midlife women found that many interventions did not use a specific behavior change theory [ 60 ]. Our scoping review did not examine the effectiveness of the interventions described, but those designing digital health tools and interventions may want to carefully consider behavioral theories in the design to increase adoption and retention rates and adherence to interventions.

Overall, digital health technology to collect PGHD has gained popularity over the past several years. The integration of wearables, smartphones, and digital health technologies has enabled the integration of passive data collection. This wealth of data provides valuable insights into various aspects of health, enabling informed decisions and the adoption of proactive measures to improve well-being. The uptake of this technology will usher in a new era in how we manage our health and well-being. This transformation has changed how we engage with our health and shifted our perception of health and the approach we take toward maintaining it.

Femtech, as a subset of digital health technology, has grown in popularity. This was evidenced by the large increase in the number of articles published between 2015 and 2020 that used digital health tools to track metrics during pregnancy and the postpartum period as well as metrics related to menstrual, sexual, and reproductive health. These technologies empower women and people assigned female at birth to take charge of their health. This is particularly relevant for people with conditions that are not diseases or health concerns per se but are nevertheless part of managing their overall health and well-being. In this way, femtech can provide a greater sense of control over reproductive health and choices, which can be precarious in many settings worldwide. However, in a previous scoping review, researchers reported that many mobile health apps do not follow data privacy, sharing, and security standards [ 61 ]. Issues related to the privacy and security of personal health data may be especially important when it comes to tracking reproductive health in settings where sexual and reproductive health rights are not guaranteed. This focus on pregnancy and reproductive health is consistent with the fact that women’s health research has largely focused on reproductive health topics [ 62 ]. Researchers and digital health developers must address gaps in women’s health regarding areas that are not strictly related to reproductive health. Women’s health encompasses much more than obstetrics and gynecology; even for health conditions that affect men and women, there may be sex or gender differences in disease presentation, personal experiences, and treatment plans. While using gendered language and design in femtech has the potential to reinforce stereotypes regarding femininity that could cause harm [ 63 ], there is a need for apps to provide content relevant to female populations while being gender inclusive and conscious of biases in the language and advice presented.

When analyzing themes related to acceptability, personalization emerged as a key aspect influencing the adoption and sustained use of digital health tools. People respond positively and want to engage with tools that cater to their unique needs and preferences. The ability to customize elements such as the frequency of notifications, specific health measures tracked and displayed, goal-setting options, and the amount of health information provided enhanced user engagement and motivation. However, offering too many personalization options might overwhelm users, making apps or devices cumbersome to use and navigate. Simplicity and ease of use should not be compromised in the pursuit of personalization. Creating personalized experiences that are intuitive and user-friendly while integrating multiple functionalities into a given device is an important consideration. Recognizing that a “one-size-fits-all” approach is inadequate, digital intervention designers need to define their target audience clearly. Apps that cater to specific groups, such as those with certain chronic health conditions, may inherently provide a sense of personalization by addressing their unique requirements. We have also learned the importance of ensuring that the design is inclusive and accessible to everyone within the target audience. Our findings that some tools are not sensitive to certain circumstances such as pregnancy are consistent with those of a systematic review of digital health interventions for postpartum women, in which the authors reported that barriers related to postpartum status could make it more difficult to engage with the interventions [ 58 ]. Tools designed with these circumstances in mind may be more engaging for women during pregnancy and the postpartum period, leading to greater adoption and quality of the technologies. Attrition can be high among users of digital health interventions [ 64 , 65 ], but most participants were willing to self-track when motivated by a specific health condition.

An important finding of this review was the growing demand and expectation that PGHD are integrated with clinical care. As digital health continues evolving, patients seek more seamless interactions between digital health data and health care providers. Moreover, services delivered through digital health technologies were not expected to replace the role of health care professionals but rather to be a useful tool to support health care management. Maintaining the human touch during communication for health care delivery was seen as important, with technology complementing clinical care to enhance the overall experience for patients and providers.

One of the critical considerations in clinical integration is the accuracy of PGHD collected from digital health tools. Ensuring the reliability and validity of the data is essential for effective clinical decision-making. Striking a balance between patient empowerment and health care provider oversight is crucial to achieving the best possible outcomes. In general, it is important for health care providers to actively propose digital health during patient visits and encourage its use. While challenges and concerns associated with the use of digital health are noted from health care providers’ perspective, such as concerns about medico-legal effects, maintaining professional boundaries, and not adding an abundance of work, the benefits of these tools in supporting patient care and improving outcomes are perceived as important.

Strengths, Limitations, and Future Directions

There are some limitations to this scoping review. Our inclusion criteria did not cover conference abstracts, conference reviews, editorials, letters, comments, or gray literature. Our review also did not include articles written in languages other than English. Therefore, there may be other uses of PGHD in women’s health that were not captured in this review. The assessments of the quality of included articles, the effectiveness of the interventions, or the accuracy in validating PGHD were outside this review’s scope and were not performed. Our aim was to provide a broad overview of PGHD in published women’s health research literature rather than evaluating the quality of the digital technologies or intervention effectiveness. Another limitation is the rapid growth of digital health and femtech, especially during the COVID-19 pandemic. It is important to note that this scoping review only captures the use of PGHD in women’s health before the emergence of the pandemic. We suggest that this review may provide a baseline for comparison in a future scoping review that captures articles published in March 2020 or later. The strengths of this review include the large number of publications analyzed and the data charting process conducted in duplicate by 2 reviewers. The broad scope of this review also helps provide an overall picture of digital health for women and highlights gaps in the research literature.

Future endeavors in this space should consider digital health tools for women for nonreproductive topics such as chronic health conditions that primarily affect women or conditions that have sex or gender differences in presentation and treatment. Within reproductive health, there was a large focus on pregnancy, but there is an unmet need for research and digital health tools appropriate for women in perimenopause and menopause. A previous literature review found <5 articles published between 2010 and 2020 about digital health technologies that meet the psychosocial needs of women experiencing menopause [ 66 ]. There may also be further opportunities for digital health tools geared toward specific racial or ethnic groups that are culturally sensitive and available in multiple languages. A systematic review found that barriers to the use of digital health among culturally and linguistically diverse populations include lower literacy levels and the use of complex medical terminology in some apps, lack of recognition of cultural concerns, stereotypes, and inaccurate portrayals of cultural groups [ 67 ]. Previous scoping reviews in the space of women’s digital health have identified the need for femtech to pay more attention to cultural appropriateness and consider cultural contexts in their design [ 68 , 69 ].

Conclusions

In conclusion, the integration of wearables, smartphones, and other forms of digital health has revolutionized how we approach and engage with our health. Personalization, inclusivity, and integration with clinical care are vital aspects of developing effective digital health solutions. By understanding the needs of the target audience, providing meaningful personalization, and ensuring data accuracy, digital health can truly transform health care and empower individuals to take charge of their well-being while maintaining a collaborative relationship with health care professionals.

Acknowledgments

Thank you to Shannon Cheng, reference librarian, for her work in developing the search strategy and conducting the database searches. A big thank you to Dr Beth Payne for reviewing the manuscript draft and providing helpful feedback. AT is funded by a Michael Smith Health Research British Columbia Scholar award.

Data Availability

The data sets generated during this study are available in the OSF repository [ 70 ].

Authors' Contributions

JLK and AT conceived the study and designed the study protocol. JLK, RST, and AT conducted the article screening. JLK, RW, and RST conducted the data charting. JLK, RST, and DSC created the visualizations and tables of the charted data. The thematic analysis coding was conducted by JLK and RW, with additional discussions with AT in refining the themes. JLK, RST, and AT wrote the draft of the manuscript. All authors reviewed the manuscript before submission.

Conflicts of Interest

None declared.

PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist.

Full search strategy.

List of included articles by health area.

Metrics collected in the included studies.

• Conceptualizing a data infrastructure for the capture, use, and sharing of patient-generated health data in care delivery and research through 2024. The Office of the National Coordinator for Health Information Technology. Jan 2018. URL: https://www.healthit.gov/sites/default/files/onc_pghd_final_white_paper.pdf [accessed 2024-05-03]
• DeSilva J, Prensky-Pomeranz R, Zweig M. Digital health consumer adoption report 2020. RockHealth. Feb 26, 2021. URL: https://rockhealth.com/insights/digital-health-consumer-adoption-report-2020 [accessed 2024-05-03]
• Taylor P. Penetration of mobile devices in Canada as share of the population from 2009 to 2021. Statista. URL: https://www.statista.com/statistics/462386/mobile-device-penetration-canada [accessed 2024-05-03]
• Olsen E. Digital health apps balloon to more than 350,000 available on the market, according to IQVIA report. MobiHealthNews. Aug 4, 2021. URL: https:/​/www.​mobihealthnews.com/​news/​digital-health-apps-balloon-more-350000-available-market-according-iqvia-report [accessed 2024-05-03]
• Paré G, Leaver C, Bourget C. Diffusion of the digital health self-tracking movement in Canada: results of a national survey. J Med Internet Res. May 02, 2018;20(5):e177. [ CrossRef ] [ Medline ]
• Fox S, Duggan M. Tracking for health. Pew Research Center. Jan 28, 2013. URL: https:/​/www.​pewresearch.org/​internet/​wp-content/​uploads/​sites/​9/​media/​Files/​Reports/​2013/​PIP_TrackingforHealth-with-appendix.​pdf [accessed 2024-05-03]
• Peek N, Sujan M, Scott P. Digital health and care in pandemic times: impact of COVID-19. BMJ Health Care Inform. Jun 2020;27(1):e100166. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Budd J, Miller BS, Manning EM, Lampos V, Zhuang M, Edelstein M, et al. Digital technologies in the public-health response to COVID-19. Nat Med. Aug 2020;26(8):1183-1192. [ CrossRef ] [ Medline ]
• Bruining N. The post-pandemic legacy: the breakthrough of digital health and telemedicine. Cardiovasc Res. Jul 27, 2021;117(9):e118-e119. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meskó B. COVID-19's impact on digital health adoption: the growing gap between a technological and a cultural transformation. JMIR Hum Factors. Sep 19, 2022;9(3):e38926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kim KK, Jalil S, Ngo V. Improving self-management and care coordination with person-generated health data and mobile health. In: Edmunds M, Hass C, Holve E, editors. Consumer Informatics and Digital Health. Cham, Switzerland. Springer; Jan 18, 2019.
• Shapiro M, Johnston D, Wald J, Mon D. Patient-generated health data. White paper. RTI International. 2012. URL: https://www.rti.org/publication/patient-generated-health-data-white-paper [accessed 2024-05-03]
• Vodicka E, Kim K, Devine EB, Gnanasakthy A, Scoggins JF, Patrick DL. Inclusion of patient-reported outcome measures in registered clinical trials: evidence from ClinicalTrials.gov (2007-2013). Contemp Clin Trials. Jul 2015;43:1-9. [ CrossRef ] [ Medline ]
• Femtech—time for a digital revolution in the women’s health market. Frost & Sullivan. Jan 31, 2018. URL: https://www.frost.com/frost-perspectives/femtechtime-digital-revolution-womens-health-market [accessed 2024-05-03]
• Karim JL, Talhouk A. Person-generated health data in women's health: protocol for a scoping review. JMIR Res Protoc. May 28, 2021;10(5):e26110. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. Oct 02, 2018;169(7):467-473. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ahmadvand A, Kavanagh D, Clark M, Drennan J, Nissen L. Trends and visibility of "digital health" as a keyword in articles by JMIR publications in the new millennium: bibliographic-bibliometric analysis. J Med Internet Res. Dec 19, 2019;21(12):e10477. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lau Y, Cheng LJ, Chi C, Tsai C, Ong KW, Ho-Lim SS, et al. Development of a healthy lifestyle mobile app for overweight pregnant women: qualitative study. JMIR Mhealth Uhealth. Apr 23, 2018;6(4):e91. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maxwell H, O’Shea M, Stronach M, Pearce S. Empowerment through digital health trackers: an exploration of Indigenous Australian women and physical activity in leisure settings. Ann Leis Res. Oct 04, 2019;24(1):150-167. [ CrossRef ]
• Hughson JP, Daly JO, Woodward-Kron R, Hajek J, Story D. The rise of pregnancy apps and the implications for culturally and linguistically diverse women: narrative review. JMIR Mhealth Uhealth. Nov 16, 2018;6(11):e189. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ranzenhofer LM, Engel SG, Crosby RD, Haigney M, Anderson M, McCaffery JM, et al. Real-time assessment of heart rate variability and loss of control eating in adolescent girls: a pilot study. Int J Eat Disord. Feb 2016;49(2):197-201. [ FREE Full text ] [ CrossRef ] [ Medline ]
• James DC, Harville C2, Whitehead N, Stellefson M, Dodani S, Sears C. Willingness of African American women to participate in e-Health/m-Health research. Telemed J E Health. Mar 2016;22(3):191-197. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lupton D, Maslen S. The more-than-human sensorium: sensory engagements with digital self-tracking technologies. Senses Soc. Jul 25, 2018;13(2):190-202. [ CrossRef ]
• Asklund I, Samuelsson E, Hamberg K, Umefjord G, Sjöström M. User experience of an app-based treatment for stress urinary incontinence: qualitative interview study. J Med Internet Res. Mar 14, 2019;21(3):e11296. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huberty J, Ehlers DK, Kurka J, Ainsworth B, Buman M. Feasibility of three wearable sensors for 24 hour monitoring in middle-aged women. BMC Womens Health. Jul 30, 2015;15:55. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haney AC. Young female college millennials' intent for behavior change with wearable fitness technology. Walden University. 2018. URL: https://scholarworks.waldenu.edu/dissertations/5197/ [accessed 2020-04-11]
• Leonard NR, Silverman M, Sherpa DP, Naegle MA, Kim H, Coffman DL, et al. Mobile health technology using a wearable sensorband for female college students with problem drinking: an acceptability and feasibility study. JMIR Mhealth Uhealth. Jul 07, 2017;5(7):e90. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brett J, Boulton M, Watson E. Development of an e-health app to support women prescribed adjuvant endocrine therapy after treatment for breast cancer. Patient Prefer Adherence. Dec 11, 2018;12:2639-2647. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chaudhry BM. Expecting great expectations when expecting. Mhealth. Jan 10, 2018;4:2. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mann D, Riddell L, Lim K, Byrne LK, Nowson C, Rigo M, et al. Mobile phone app aimed at improving iron intake and bioavailability in premenopausal women: a qualitative evaluation. JMIR Mhealth Uhealth. Sep 28, 2015;3(3):e92. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grym K, Niela-Vilén H, Ekholm E, Hamari L, Azimi I, Rahmani A, et al. Feasibility of smart wristbands for continuous monitoring during pregnancy and one month after birth. BMC Pregnancy Childbirth. Jan 17, 2019;19(1):34. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Goetz M, Müller M, Matthies LM, Hansen J, Doster A, Szabo A, et al. Perceptions of patient engagement applications during pregnancy: a qualitative assessment of the patient's perspective. JMIR Mhealth Uhealth. May 26, 2017;5(5):e73. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lupton D, Pedersen S. An Australian survey of women's use of pregnancy and parenting apps. Women Birth. Aug 2016;29(4):368-375. [ CrossRef ] [ Medline ]
• de Mooij MJ, Hodny RL, O'Neil DA, Gardner MR, Beaver M, Brown AT, et al. OB nest: reimagining low-risk prenatal care. Mayo Clin Proc. Apr 2018;93(4):458-466. [ CrossRef ] [ Medline ]
• Kokts-Porietis RL, Stone CR, Friedenreich CM, Froese A, McDonough M, McNeil J. Breast cancer survivors' perspectives on a home-based physical activity intervention utilizing wearable technology. Support Care Cancer. Aug 2019;27(8):2885-2892. [ CrossRef ] [ Medline ]
• Gell NM, Tursi A, Grover KW, Dittus K. Female cancer survivor perspectives on remote intervention components to support physical activity maintenance. Support Care Cancer. May 2020;28(5):2185-2194. [ CrossRef ] [ Medline ]
• Ravn Jakobsen P, Hermann AP, Søndergaard J, Wiil UK, Clemensen J. Development of an mHealth application for women newly diagnosed with osteoporosis without preceding fractures: a participatory design approach. Int J Environ Res Public Health. Feb 13, 2018;15(2):330. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Harder H, Holroyd P, Burkinshaw L, Watten P, Zammit C, Harris PR, et al. A user-centred approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J Cancer Surviv. Dec 2017;11(6):732-742. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Depper A, Howe PD. Are we fit yet? English adolescent girls’ experiences of health and fitness apps. Health Sociol Rev. Jul 04, 2016;26(1):98-112. [ CrossRef ]
• Gambier-Ross K, McLernon DJ, Morgan HM. A mixed methods exploratory study of women's relationships with and uses of fertility tracking apps. Digit Health. Jul 25, 2018;4:2055207618785077. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lupton D. ‘It just gives me a bit of peace of mind’: Australian women’s use of digital media for pregnancy and early motherhood. Societies. Sep 15, 2017;7(3):25. [ CrossRef ]
• Logsdon MC, Lauf A, Stikes R, Revels A, Vickers-Smith R. Partnering with new mothers to develop a smart phone app to prevent maternal mortality after hospital discharge: a pilot study. J Adv Nurs. Jan 2020;76(1):324-327. [ CrossRef ] [ Medline ]
• Wright AA, Raman N, Staples P, Schonholz S, Cronin A, Carlson K, et al. The HOPE pilot study: harnessing patient-reported outcomes and biometric data to enhance cancer care. JCO Clin Cancer Inform. Dec 2018;2:1-12. [ FREE Full text ] [ CrossRef ] [ Medline ]
• de Mendoza AH, Cabling ML, Dilawari A, Turner JW, Fernández N, Henderson A, et al. Providers' perspectives on adherence to hormonal therapy in breast cancer survivors. Is there a role for the digital health feedback system? Health Technol (Berl). Mar 2019;9(2):175-184. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Willcox JC, van der Pligt P, Ball K, Wilkinson SA, Lappas M, McCarthy EA, et al. Views of women and health professionals on mHealth lifestyle interventions in pregnancy: a qualitative investigation. JMIR Mhealth Uhealth. Oct 28, 2015;3(4):e99. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ke JX, George RB, Wozney L, Chorney JL. Patient-centred perioperative mobile application in Cesarean delivery: needs assessment and development. Can J Anaesth. Oct 2019;66(10):1194-1201. [ CrossRef ] [ Medline ]
• Firet L, de Bree C, Verhoeks CM, Teunissen DA, Lagro-Janssen AL. Mixed feelings: general practitioners' attitudes towards eHealth for stress urinary incontinence - a qualitative study. BMC Fam Pract. Jan 26, 2019;20(1):21. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garnweidner-Holme LM, Borgen I, Garitano I, Noll J, Lukasse M. Designing and developing a mobile smartphone application for women with gestational diabetes mellitus followed-up at diabetes outpatient clinics in Norway. Healthcare (Basel). May 21, 2015;3(2):310-323. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grassl N, Nees J, Schramm K, Spratte J, Sohn C, Schott TC, et al. A web-based survey assessing the attitudes of health care professionals in Germany toward the use of telemedicine in pregnancy monitoring: cross-sectional study. JMIR Mhealth Uhealth. Aug 08, 2018;6(8):e10063. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pais S, Parry D, Petrova K, Rowan J. Acceptance of using an ecosystem of mobile apps for use in diabetes clinic for self-management of gestational diabetes mellitus. Stud Health Technol Inform. 2017;245:188-192. [ Medline ]
• Ragavan MI, Ferre V, Bair-Merritt M. Thrive: a novel health education mobile application for mothers who have experienced intimate partner violence. Health Promot Pract. Mar 2020;21(2):160-164. [ CrossRef ] [ Medline ]
• Runkle J, Sugg M, Boase D, Galvin SL, C Coulson C. Use of wearable sensors for pregnancy health and environmental monitoring: descriptive findings from the perspective of patients and providers. Digit Health. Feb 06, 2019;5:2055207619828220. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sadigursky A. Move my mood: development and evaluation of a mobile mental health self-help app using behavioral activation for women with postpartum depression. Alliant International University. 2018. URL: https://books.google.co.in/books/about/Move_My_Mood.html?id=dYtP0AEACAAJ&redir_esc=y [accessed 2020-03-30]
• Scherr CL, Feuston JL, Nixon DM, Cohen SA. A two-phase approach to developing SNAP: an iPhone application to support appointment scheduling and management for women with a BRCA mutation. J Genet Couns. Apr 2018;27(2):439-445. [ CrossRef ] [ Medline ]
• Tommasone G, Bazzani M, Solinas V, Serafini P. Midwifery e-health: from design to validation of “mammastyle — Gravidanza Fisiologica”. In: Proceedings of the IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom). 2016. Presented at: Healthcom 2016; September 14-16, 2016; Munich, Germany. [ CrossRef ]
• Fu MR, Axelrod D, Guth AA, Wang Y, Scagliola J, Hiotis K, et al. Usability and feasibility of health IT interventions to enhance Self-Care for Lymphedema Symptom Management in breast cancer survivors. Internet Interv. Sep 2016;5:56-64. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dienelt K, Moores CJ, Miller J, Mehta K. An investigation into the use of infant feeding tracker apps by breastfeeding mothers. Health Informatics J. Sep 2020;26(3):1672-1683. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lim S, Tan A, Madden S, Hill B. Health professionals' and postpartum women's perspectives on digital health interventions for lifestyle management in the postpartum period: a systematic review of qualitative studies. Front Endocrinol (Lausanne). Nov 8, 2019;10:767. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mackey A, Petrucka P. Technology as the key to women's empowerment: a scoping review. BMC Womens Health. Feb 23, 2021;21(1):78. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sediva H, Cartwright T, Robertson C, Deb SK. Behavior change techniques in digital health interventions for midlife women: systematic review. JMIR Mhealth Uhealth. Nov 09, 2022;10(11):e37234. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alfawzan N, Christen M, Spitale G, Biller-Andorno N. Privacy, data sharing, and data security policies of women's mHealth apps: scoping review and content analysis. JMIR Mhealth Uhealth. May 06, 2022;10(5):e33735. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hallam L, Vassallo A, Pinho-Gomes AC, Carcel C, Woodward M. Does journal content in the field of women's health represent women's burden of disease? A review of publications in 2010 and 2020. J Womens Health (Larchmt). May 2022;31(5):611-619. [ CrossRef ] [ Medline ]
• Figueroa CA, Luo T, Aguilera A, Lyles CR. The need for feminist intersectionality in digital health. Lancet Digit Health. Aug 2021;3(8):e526-e533. [ CrossRef ]
• Jabir AI, Lin X, Martinengo L, Sharp G, Theng YL, Tudor Car L. Attrition in conversational agent-delivered mental health interventions: systematic review and meta-analysis. J Med Internet Res. Feb 27, 2024;26:e48168. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Meyerowitz-Katz G, Ravi S, Arnolda L, Feng X, Maberly G, Astell-Burt T. Rates of attrition and dropout in app-based interventions for chronic disease: systematic review and meta-analysis. J Med Internet Res. Sep 29, 2020;22(9):e20283. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cronin C, Hungerford C, Wilson RL. Using digital health technologies to manage the psychosocial symptoms of menopause in the workplace: a narrative literature review. Issues Ment Health Nurs. Jun 2021;42(6):541-548. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Whitehead L, Talevski J, Fatehi F, Beauchamp A. Barriers to and facilitators of digital health among culturally and linguistically diverse populations: qualitative systematic review. J Med Internet Res. Feb 28, 2023;25:e42719. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Birati Y, Yefet E, Perlitz Y, Shehadeh N, Spitzer S. Cultural and digital health literacy appropriateness of app- and web-based systems designed for pregnant women with gestational diabetes mellitus: scoping review. J Med Internet Res. Oct 14, 2022;24(10):e37844. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Woodley SJ, Moller B, Clark AR, Bussey MD, Sangelaji B, Perry M, et al. Digital technologies for women's pelvic floor muscle training to manage urinary incontinence across their life course: scoping review. JMIR Mhealth Uhealth. Jul 05, 2023;11:e44929. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Person-generated health data in women’s health: scoping review. OSF Home. Mar 7, 2024. URL: https://osf.io/3eync/ [accessed 2024-05-07]

Abbreviations

Edited by T de Azevedo Cardoso; submitted 04.10.23; peer-reviewed by D Liu, M Herron; comments to author 23.02.24; revised version received 15.03.24; accepted 26.03.24; published 16.05.24.

©Jalisa Lynn Karim, Rachel Wan, Rhea S Tabet, Derek S Chiu, Aline Talhouk. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 16.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Download PDF
• Share X Facebook Email LinkedIn
• Permissions

Prevalence of Cardiovascular-Kidney-Metabolic Syndrome Stages in US Adults, 2011-2020

• 1 Brigham and Women’s Hospital Heart and Vascular Center, Harvard Medical School, Boston, Massachusetts

Cardiovascular, kidney, and metabolic (CKM) diseases are pathophysiologically interrelated, 1 have affected more than 25% of US adults between 2015-2020, 2 and were the leading causes of death in 2021. 3 In 2023, the American Heart Association introduced a novel staging construct, termed CKM syndrome , 1 to enhance multidisciplinary approaches to prevention, risk stratification, and management of these disorders. Based on risk factors and established disease, the stages range from 0 (no risk factors) to 4 (established cardiovascular disease [CVD]).

Read More About

Aggarwal R , Ostrominski JW , Vaduganathan M. Prevalence of Cardiovascular-Kidney-Metabolic Syndrome Stages in US Adults, 2011-2020. JAMA. Published online May 08, 2024. doi:10.1001/jama.2024.6892

Manage citations:

© 2024

Artificial Intelligence Resource Center

Cardiology in JAMA : Read the Latest

Browse and subscribe to JAMA Network podcasts!

Others Also Liked

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing
• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts