Statistical analysis
Mean and distribution of scores for each scorecard indicator
Complete responses for all indicators on the scorecard were obtained from 111 PHCCs, including 222 respondents (111 OICs and 111 non-OIC workers). Of the 111 PHCCs initially selected, two PHCCs in Ondo state refused the interview because they had just started PBF, and six PHCCs in Adamawa were not accessible due to an insurgency. They were replaced by other PHCCs through random selection. As shown in Table 2, mean scores of the 32 indicators ranged 1.5–2.77, with average 2.16. The standard deviation of the scores ranged 0.33–0.83, with average 0.6.
Validity and reliability of the scorecard
Experts review suggested high face validity of the scorecard. Table 3 presents the result of the EFA with PROMAX rotation. Five factors had eigenvalues more than 1, and screeplot shows flattened line at the Factor 7. Given that the commonality was higher for the six-factor model than for the five-factor model, and that our qualitative and literature review described above suggested a six-factor model, we chose the six-factor model dropping 12 indicators with loadings less than 0.4. Since all items except for S5, S22, S31, and S32 had uniqueness higher than 0.50, we kept the items with uniqueness higher than 0.50 as long as their factor loadings were 0.40 or above. Also, three cross-loading items with values ≥ 0.32 on at least two factors were dropped (S9, S17, S20) from this model. As a result, 17 indicators were kept for analysis.
Table 4 summarizes the analysis of the factors presented in the EFA and indicators. The EFA result consists of six factors and 17 indicators. The six factors were named based on the discussion among the authors on grouped items under each factor: – A: Stakeholder engagement and communication; B: Community-level activities; C: Update of plan and target; D: Performance Management; E: Staff attention to planning, target, and performance; and F: Drugs and financial management. The reasoning by the authors behind the names of the six factors is summarized in Table 4. For the ICC, the correlation among mean ratings for each team of judges is 0.94, showing high inter-rater reliability.
Discussion
We developed a novel scorecard that measures management practices in PHCCs in Nigeria. We highlighted financial management, community and stakeholder engagement as key additional elements of management practices for PHCCs in LMICs in addition to the Management Practices Measurement tool developed by Dorgan et al. [6], Bloom et al. [7], and McConnell et al. [8] for use in high income countries. Our scorecard also introduced a more specific definition of scoring criteria than the original instrument, and questions for non-OIC health workers to enable local data collectors to rate practices and to reduce social desirability bias. These are new and original features of the scorecard that would facilitate its adaptation to capacity-constrained contexts in LMICs.
Original scorecard vs. EFA results
The developed scorecard was further refined through the EFA. The EFA reduced the number of items from 32 to 17. It also provided a different grouping of items from the originally proposed management practices scorecard based on our qualitative study [5] and literature review. Table 5 compares the originally proposed management practices scorecard with findings based on the EFA results. There are a few notable differences. First, although community engagement is to some extent covered by the latent factor A and B in Table 3, a set of items related to building the relationship with and attracting patients (S1-S4) were not included in the EFA results (see Table 5, right-hand column ‘New Groupings’). These dropped items were however highlighted as key differentiating factors of PHCC performance under the related qualitative study [5]. This may suggest that there are slight differences between factors that relate to PHCC performance and factors that represent health center management (suggested through EFA). Hence, the factors that explain health center management on the one hand and health center performance on the other, may be overlapping but not identical. For example, drugs and financial management are not a factor that directly differentiated high and low performers in the qualitative case study [5], whereas this is an important element of health center management based on the EFA. Likewise, outreach, household visits, and strategies to attract patients may not be a direct element of health center management, though they are key specific approaches that influence the performance of the PHCC. It is noted that community/client engagement is not included in the management practices measurement tool by Dorgan et al. [6], Bloom et al. [7], and McConnell et al. [8], and synthesized key elements of critical primary health facility management (Table 1).
Another possibility is that the indicators for community/client engagement did not measure the practices sufficiently well. These indicators were developed specifically for this scorecard and could have been flawed. For example, the frequency of outreach last week (S1) may be too short a time period to get a reliable picture of outreach, or this measure may put too much emphasis on frequency and not enough on the quality of outreach. Further formative research, elaboration and testing of the scorecard questions may be needed in this area.
Second, most of the items related to Staff Management in the original scorecard were dropped, and the items kept were assigned to separate groups (i.e. ‘A. Stakeholder engagement’ and ‘D. Performance management’). This is not consistent with the synthesized key elements of critical primary health facility management (Table 1) where activities to assign appropriate roles and responsibilities, create opportunities for learning, motivate and coach health workers, and promote cohesion and teamwork were highlighted as a key element of health facility management. The Management Practices Measurement tool also has ‘Talent management’ in the instrument [6–8]. This may suggest the challenge of scoring such activities in the scorecard, and points to the need for further review and adaptation. At least, however, some dimensions of staff management, such as responsiveness to staff feedback as a part of broader stakeholder engagement, and handling of poor performing staff as a part of performance management are covered in the final factors.
EFA results vs. literature
The EFA results are consistent with the developed management practices scorecard and literature in other settings. Latent factors ‘C. Update of plan and target’, and ‘D. Performance management’ and the items grouped in these factors are consistent with the Management Practices Measurement tool. Also, the factor ‘F. Drugs and financial management’ is consistent with the synthesized key elements of critical primary health facility management (Table 1), as well as the key management practices for the health facilities to manage the PBF scheme [19]. The latent factor ‘E. Staff attention to plan, target, and performance’ is a different grouping from the original management practices scorecard. However, this demonstrates the importance of communication, involvement, and incentives to motivate staff to be attentive to plan, target and performance, which is consistent with findings in the qualitative case study [5] and the Communication element of the synthesized key elements of critical primary health facility management (Table 1).
Value and use of the research
This research added significantly to the literature on health center management in developing countries. A careful review of prior studies and application of existing instruments with adjustments, expert review of the scorecard, and high inter-rater reliability are signals of the validity and reliability of the developed measurement approach. The EFA also provided a refined management practices scorecard, despite some differences between the results that it offered and findings from the literature and the related qualitative case study [5].
Capacity building of health facilities is included in most primary health care interventions in developing countries. However, there has been no instrument to help assess management practices and provide critical feedback to improve health facility management to-date. Recent systematic reviews of researches on primary health care systems in LMICs suggest that major research gaps exist in how to improve facility management [31], and that routinely used performance measurement and management strategies are implemented without sufficient knowledge of their effects [32]. This scorecard can help address these critical gaps thus strengthening primary care services. The resulting scorecard is relatively simple, encompassing just 17 different indicators, and includes clear scoring criteria, meaning that it would be relatively straightforward for the central and local government officials to apply the scorecard as part of routine supervisory visits, and not just as part of a research project. This scorecard was used in Nasarawa state of Nigeria to measure baseline and follow-up management scores of the PHCCs under PBF funded by the World Bank to design/guide and measure the result of management strengthening interventions. This indicates high acceptability of the scorecard. Wider application of this scorecard would in turn help to further strengthen the scorecard and guidance associated with it.
Limitations and areas for further study
As suggested above, one of the limitations to this research and the scorecard is that some of the scorecard questions and scoring criteria, notably those related to community/client engagement, and staff management would benefit from further investigation and refinement. Given the limited literature seeking to assess management practices quantitatively, we were unable to compare our findings to other studies from LMICs.
The scorecard was designed to serve the needs of primary health care facilities under PBF or similar schemes that provide autonomy and funds for the health facilities to improve health services. and it was designed for use in the Nigerian context, drawing in particular on a qualitative case study previously conducted in Nigeria [5]. In order to understand how this scorecard may apply in other contexts, both with and without PBF, further studies may be required using confirmatory factor analysis (CFA) to assess the model fit of the scorecard. Adaptations would also be necessary to assess management practices in settings where there are more limited management autonomy and discretionary funds. Differences in health system structure and function, for example the structure of drug supply systems, or the extent of decentralization, may also influence items and constructs to be included in the scorecard.
academicscores.com 