Development and validation of management practices scorecard in Nigeria

Photo by Chokniti Khongchum on Pexels.com

Background

In low- and middle-income countries (LMICs), numerous initiatives have been introduced with the aim of improving the quality and performance of primary health services. However, determinants of primary health facility performance in resource-limited settings have not been well understood [1]. Among many factors that potentially determine the performance of primary health facilities, one of the most under-researched areas is health facility management. Several studies [15] have identified specific health facility management practices that are associated with the improvement of health facility performance including: (i) engaging and problem-solving with local stakeholders [3], particularly with community leaders [5]; (ii) building a system of accountability [4], through performance management activities [5]; (iii) motivating health workers for change [2,3,5]; (iv) building work around teams and creating a sense of belonging, trust and respect [1,3,5]; (v) providing management support [1,3,5]; and (vi) improving health facility managers leadership competency to build a supportive environment for staff [2,4,5].

Much of the research on primary health care (PHC) management practices in low and middle-income countries (LMICs) is qualitative in nature, employing case studies and realist evaluations to understand which types of management practices work in which settings [15]. Quantitative empirical studies aimed at measuring these critical health facility management practices and exploring the relationship between such practices and health facility performance are limited primarily to inpatient settings in developed countries [6–17]. Without a validated and reliable measurement tool, many interventions to improve the management and performance of primary health care facilities in LMICs are designed without an adequate evidence base on what management practices are critical for improving health facility performance. The lack of measurement tool for management practices further limits the assessment of health system strengthening (HSS) interventions in LMICs.

Performance-based financing (PBF) is an example of such HSS interventions and has increasingly been used as a major approach for improving health facility performance in LMICs. It is defined as “fee-for-service conditional on quality of care’ [18]. In Nigeria, PBF was rolled out in 3 out of 36 States (Adamawa, Nasarawa, and Ondo States) to improve the quality and quantity of primary health care centers (PHCCs) services in 2014. The major design features of the PBF scheme in Nigeria include: (i) the provision of finance to health facilities on top of existing budgets based on the quantity and quality of services provided; and (ii) increased autonomy for PHCCs so that they may use the received funds to improve health services. Part of the funding received can also be allocated to motivate health workers based on their performance and responsibilities [19]. Given that PBF provides resources and autonomy for PHCCs to manage resources, management practices at the PHCCs covered by such PBF schemes will be even more important than facilities without such a platform. How PBF works at the health facility level and how health facility management influences performance under PBF have been regarded as a ‘Black Box’ [20]. A reliable measurement tool is required to unveil the dynamics between PBF, health center management, and performance.

The objective of our study was to develop and validate a scorecard to measure management practices at primary health care facilities under the PBF scheme in Nigeria. While the scorecard was designed to include a few measures of PBF-specific activities (such as the development of business plan and use of PBF funds), broader measures of management practices applicable to diverse PHCC programs were also included. We hope that the scorecard will be useful for the assessment of management practices at PHCCs, and improve our understanding of how PBF and similar HSS interventions work to influence health facility management in LMICs.

Methods

This study was conducted in 2014–2016. The management practices scorecard was developed through a two-step process. In the first step, we conducted a literature review to: (i) examine existing tools; and (ii) develop management practices areas and sub-areas that could be built on to develop a management practices scorecard for PBF in Nigeria. The authors reviewed literature collected through a PubMed and Google scholar search for the years 1996–2016. The search combined terms related to: (i) measurement tools; (ii) facility management practice and management competencies; and (iii) primary health care. We applied this search to literature from both developed and developing countries. We selected a tool to form the foundation for the scorecard based on the following criteria: (i) the tool had gone through a validation process that linked management measures to health facility performance; and (ii) the indicators and measurement approach were consistent with findings from a qualitative case study at PHCCs in Nigeria [5] that identified community engagement, performance management and staff management as key management practices. Further, based on over 40 relevant publications in peer-reviewed journals on management competencies, we developed a list of important management practice areas for PHCCs. We also reviewed the PBF literature to identify additional management practice areas for managing PBF schemes in health facilities. We extracted all the elements of management practices mentioned in the relevant publications, and organized them into a shortlist of management practice areas and sub-areas under each area. We also selected relevant indicators under each sub-area based on the review of existing tools.

As the second step, we developed additional indicators, scoring criteria using ordinal responses, and a scoring grid based on the key findings of the qualitative study [5]. For example, the case study research found that high-performing PHCCs carry out frequent outreach activities, visit targeted households in each outreach, change services based on the feedback from patients, and carry out many strategic activities to recruit and retain patients such as creating incentives for women to deliver at PHCCs, individualized follow-up of pregnant women, year-end celebrations. These practices were converted to indicators to measure community/client engagement, and ordinal responses for each indicator were developed by comparing relevant practices between high versus low-performing PHCCs observed in the qualitative study.

We tested the face validity of the scorecard through a review of health system personnel in Nigeria including staff of the National Primary Health Care Development Agency (NPHCDA), and State Primary Health Care Development Agencies (SPHCDAs) in Adamawa, Nasarawa, and Ondo states of Nigeria; PBF technical assistance consultants; the World Bank’s Nigeria health team; and health systems experts at the Johns Hopkins University. We then divided the scorecard into two separate questionnaires to be administered (i) to officers in charge (OICs) and (ii) non-OIC health workers. We pilot-tested the questionnaires in six PHCCs under PBF in Nasarawa state that were not included in the main phase of data collection and revised the questionnaires for clarity and interpretation based on feedback from the pilot-test. The revised questionnaires, evaluation criteria and scoring grid were scripted into electronic templates on hand-held devices to facilitate data collection and reduce errors in scoring and aggregation of the scores.

Sampling and data collection

We proposed a sample size (N) of 111 PHCCs to explore a scorecard with 32 indicators or questions (p) for a N:p ratio of 3 as described by Arrindell and van der Ende [21] to be adequate for demonstrating the validity and reliability of questionnaires designed for identifying latent constructs with alpha level of 0.05. We selected the 111 PHCCs from 457 PHCCs that have been implementing PBF in the three Nigerian States (Adamawa, Nasarawa, and Ondo). We used a stratified random sampling technique to allocate the 111 PHCCs (Adamawa: 54, Nasarawa: 21 and Ondo: 36) based on the number of PHCCs under the PBF scheme in each State.

In the EFA presented below, average communality of selected items was 0.45. The items per factor ratio (p:r ratio) in the model presented in Table 4 is 17:6. For EFA with low communality case (less than 0.4), an empirical study suggested that N = 100 and N = 200 are needed to have over 95% convergences for 20:3 and 10:3 p:r ratios respectively [22]. They also found that for p:r ratio 10:3 or 20:3, a sample size (N = 60) could still result in over 99% convergence if the level of commonality is wide or high. As the commonality of this EFA result is higher than 0.4, 17:6 p:r ratio may require a little over 100, which is consistent with the sample size of our study, N = 111.

Advertisement

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: