www.snrs.org

Interactive comment section

Abstract

Ruby Shaw Morrison, DSN, RN, Associate Professor, Capstone College of Nursing, The University of Alabama, Tuscaloosa; Charlotte Burroughs, BSN, RNC, Tuscaloosa Veterans Affairs Medical Center; Melanie Witt, BSN, RNC, Tuscaloosa Veterans Affairs Medical Center; Jessie Redden, BSN, RNC, Tuscaloosa Veterans Affairs Medical Center; James D. Leeper, PhD., Professor and Chairman, Department of Behavioral and Community Medicine. The University of Alabama, Tuscaloosa.

In this pilot study, the NOC instruments Knowledge of Medication, Caregiver Home Care Readiness, and Health Promoting Behavior were evaluated for use in measuring outcomes of a nurse care manager program with chronically ill veterans. The convenience sample included 10 chronically ill veterans who were diagnosed with at least three of the high priority diagnoses targeted for care management: congestive heart failure, ischemic heart disease, chronic obstructive pulmonary disease hypertension, and diabetes. The typical patient was a 68-year-old married Caucasian male. The Caregiver Home Care Readiness and Health Promotion Behavior instruments demonstrated strong correlations for inter-rater reliability and overall reliability using Cronbach’s alpha. The correlation coefficient for inter-rater reliability was below the acceptable .80 for Knowledge of Medications. This instrument had been modified considerably from the original NOC instrument because many of the indicators were in the scope of responsibility of pharmacists rather than nurse care managers in the Veterans Administration Medical Center (VAMC). All of the instruments showed acceptable inter-item reliability of .71 to .87. Use of chart audits to collect data regarding health promotion behavior and knowledge of medications was compared to the procedures of data collection with modified NOC instruments. The results from this pilot study demonstrated that NOC instruments provided more reliable data than the chart audits.

 Key Words       Nurse Care Manager, Outcomes, NOC instruments, Chronic illness

¹Maas, M. L., Johnson, M., Moorhead, S. (1996). Classifying nursing-sensitive patient outcomes. IMAGE: Journal of Nursing Scholarship, 28(4), 295-301.

²Johnson, M., & Maas, M. (1998). The Nursing Outcomes Classification. Journal of Nursing Care Quality, 12(5), 9-20, 85-87.

 ⁴ibid.

Outcomes sensitive to the influence of specific disciplines are necessary for the discipline to judge the effects of interventions.¹ Evaluation of outcomes is essential to test the effectiveness of nursing interventions and appropriateness of decision making.² Nursing-sensitive outcomes have been identified and developed, but require testing to ascertain reliability as measures of effective nursing practice. The Nursing-Sensitive Outcomes Classification (NOC) system has evolved through the efforts of a dedicated team of nurse scholars and clinicians participating in the Iowa Outcomes Project.³ Because the NOC instruments have not been tested as outcomes measures, a pilot study was deemed necessary to evaluate the reliability of the instruments and feasibility of their use.

An ideal opportunity to pilot the instrument arose when a veterans facility developed a nurse care manager program with specific mandates to improve quality and improve patient outcomes. The purpose of this study was to evaluate the usefulness of (NOC) instruments⁴ to measure outcomes of chronically ill veterans in order to evaluate the effectiveness of the nurse care manager program. In order to accomplish the purpose of the study, data collected through patient interviews using the NOC instruments were compared to data obtained from audits of the computerized medical record.

⁴Maas, Johnson, & Moorhead, op.cit.

⁵Iowa Outcomes Project, op.cit.

Development of the NOC outcome measurement instruments has been well documented in the literature.^4-7 NOC labels and indicators have been used to document outcomes in home care, cardiac rehabilitation, obstetrical clinical pathways, and pain management programs.⁸ Cox⁹  used the NOC framework to link nursing diagnoses, nursing interventions and nursing outcomes in a long-term care facility, but extensively modified the applications to that clinical setting’s specific needs. Outcomes were measured dichotomously, rather than the 5-point original NOC scale. Similar alterations, either in the items themselves or in the manner of scoring, were used in the cardiac rehabilitation, obstetrical setting, and other studies. Variability of the NOC outcome measurement use makes them difficult to compare across settings, despite their general acceptability. No references on testing of the original NOC instruments and scoring scales were found. Communication with the Iowa Outcomes Project Research Team indicates that the pilot study reported in this article is the first of which they are aware.¹⁰

Background of NOC: The NOC system was developed using inductive and deductive reasoning as well as qualitative and quantitative research methods. Sources of data were written documents that included textbooks, care planning guides, nursing diagnosis manuals, critical paths, nursing information systems, and research instruments.¹¹ From these sources, a list of more than 4500 outcome statements and 282 nursing-sensitive outcome labels were generated. These statements were then grouped and labeled by outcomes for further development of measurable outcome indicators by the NOC research team. Eight broad categories of the Medical Outcomes Study¹² were combined with categories found in the nursing literature to classify outcomes according to physiological status; psychological and cognitive status; social and role status; physical functional status; safety status; family care giver status; health attitudes, knowledge, behavior; and perceived well being. Nurses in clinical practice developed indicators for each outcome.¹³

Items included in the inventory of indicators reflect the following theoretical base. Patient outcomes are influenced by the diagnosis, the discipline providing care and the health care system in which care is provided.¹⁴ Outcomes are also influenced by such individual patient factors as age, gender, diagnosis, functional status, psychological/cognitive status, psychosocial function, cultural/ethnic factors, resources, and severity of illness. Care provider factors include provider type, technical competence, interpersonal style, and discipline standards. System factors include type of organization, geographic setting, human, and financial resources and organizational structures.¹⁵

¹⁶ibid.

Content validity of the nursing-sensitive patient outcomes indicators was established through an extensive process that included refinement of indicators by focus groups, sensitivity rating by nurse experts, and calculating indicator importance and sensitivity to determine inclusion of indicators.¹⁶

¹⁷Department of Veterans Affairs. (1998). Functional statement, nurse II care manager. Unpublished document. Tuscaloosa, Alabama: Department of Veterans Affairs, p.1.

²⁵American Diabetes Association. (1997). ABC’s of preventing vascular disease. Clinical Diabetes, 15(6), 257.

The Nurse Care Manager Program. Health care demands to control costs, improve quality and improve patient outcomes prompted the development of a nurse care manager program at the Veterans Administration Medical Center (VAMC) study site. The nurse care managers were responsible for coordinating care of a select panel of patients across the continuum of care. They were facilitator/advocates for the patient and collaborated with the interdisciplinary team to assess, plan, organize, implement, coordinate, monitor, and evaluate care for veterans. They were also to document and report any variances in care in order to produce optimal health outcomes.¹⁷ In this VAMC model, nurse care managers coordinate care for patients who have a primary diagnosis and 2 comorbidities of the following chronic illnesses: congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), ischemic heart disease (IHD), hypertension, and diabetes. These diagnoses were selected because of their high volume, high cost for treatment, and high risk.¹⁸ These diagnoses are also the leading causes of death in the United States. Heart disease, including CHF and IHD, was the leading cause of death, COPD was the fourth, and diabetes the seventh leading cause of death in 1997.¹⁹ The costs of treating chronic illness for individuals with the illness and society at large are estimated at $10-38 billion annually for CHF^20-24 and $105 billion annually for diabetes.²⁵

  Sample and setting: The site of the study was a VAMC that uses a nursing care management program to manage care of chronically ill veterans with multiple diagnoses in an outpatient setting. The convenience sample included 10 chronically ill veterans who were diagnosed with at least three of the high-priority diagnoses targeted for intervention: CHF, IHD, COPD, diabetes, and hypertension. The co-investigators in the study, also nurse care managers of the VAMC, identified qualified subjects from the clinic schedule. The nurse care manager explained the study and obtained signatures on the informed consent form prior to data collection. Data were collected in face-to-face interviews in an office adjacent to the clinic where the clients received care. Follow-up comparison data were obtained from audits of the computerized medical record. A limitation of the study is the possibility of researcher bias because the nurse care managers served as data collectors.  

²⁶(Iowa Outcomes Project, 1997, op.cit. p. 191.

²⁸ibid, p.159.

Instruments: The demographic data section of the instrument was developed by the investigators and included items regarding age, gender, race, marital status and diagnoses. Following a review of the NOC instruments, nurse care managers identified three NOC instruments that could potentially evaluate the effectiveness of the nurse care manager program. The instruments selected included Knowledge of Medications,²⁶ Care Giver Home Care Readiness"²⁷  and Health Promoting Behavior.²⁸ Modifications were made, for the reasons indicated, by the investigative team.

²⁹ibid, p.191.

Knowledge of Medications. The NOC instrument Knowledge of Medications,²⁹ used to quantify the variable knowledge of medications, was modified to remove 7 of the 17 indicators that were part of the pharmacists’ responsibility at the study site. Examples included the indicators "description of appearance of medication" and "description of proper disposal of unused medication."

³⁰ibid, p.103

Caregiver Home Care Readiness. The NOC instrument Caregiver Home Care Readiness³⁰  was used to evaluate the level of preparedness of the caregivers for their responsibilities. This instrument was modified by deleting the indicator "demonstration of positive regard for care recipient"30 and by revising "caregiver well-being"³⁰ to "caregiver physical well being." The nurse care managers did not believe the type of regard the caregiver had for the veteran was within their scope of influence and that their assessment of the caregiver's well being was limited to the physical capability of providing care.

³¹ibid, p.159

Health Promoting Behavior. The NOC instrument Health Promoting Behavior³¹ was expanded to specify 3 risk avoidance behaviors of particular importance at the study site: smoking, the use of seat belts, and drinking alcohol. Two other indicators "monitors environment for risks" and "supports healthful public policy"³¹ were deleted because the nurse care managers did not believe those appropriate to evaluate their role in influencing the patient outcome health promoting behavior.

The revised NOC instruments included 10, 19 and 11 indicators respectively. Indicators were rated on a scale of 1 to 5. For the Knowledge of Medication and Caregiver Home Care Readiness instruments, the discriminators were: 1= none, 2=limited, 3= moderate, 4= substantial, and 5= extensive. For the instrument Health Promoting Behavior the discriminators were 1=never, 2=rarely, 3=sometimes, 4=often, and 5=consistently. The scales of the 3 NOC instruments permitted a summed score for the outcome measure. Permission was obtained from the Iowa Outcomes Project Research Team to use the modified instruments.

³²Fleiss, J. L. (1985). The Design and Analysis of Clinical Experiments. New York: Wiley.

Procedure: The procedures in this pilot study were designed to evaluate the reliability of the instruments. Several experts in research and outcomes measurement who reviewed the instruments expressed concern that the rating scale of 1 to 5 for each indicator needed greater clarity than the existing labels to achieve consistency in data collection. Therefore, the initial phase of data collection and analysis was designed to evaluate inter-rater reliability. This was accomplished by having one nurse care manager interview the subject and complete the NOC data collection instruments. Concurrently, a second nurse care manager observed the interview and completed the data collection instrument independently. Inter-rater reliability was considered acceptable if the correlation coefficients were .80 or higher.³² If the inter-rater reliability coefficients were less than .80, further instructions regarding the rating scale would be needed. In addition, Cronbach’s alpha was calculated for each of the instruments on data collected by each of the care managers (interviewer and observer) to evaluate reliability of the instruments.

The second phase of this pilot study was to compare data obtained from audits of the computerized medical records to data from the NOC instruments of the same subjects. No data relating to the variable "caregiver home care readiness" was routinely documented in the medical record. Therefore, only the variables "health promoting behavior" and "knowledge of medications" could be used for comparison. Audit instruments were developed by the researchers for each variable with five indicators recorded on a scale of 1-5 where 1=never, 2=rarely, 3=sometimes, 4=often, and 5=consistently. Independent audits were conducted by two nurse care managers. Reliability was evaluated by Cronbach’s alphas for each scale and correlation coefficients for inter-rater reliability.

Prior to beginning data collection, the proposal was reviewed for protection of human subjects by the Institutional Review Board of The University of Alabama and the VAMC site. Only the minimal risk of inconvenience to veterans was anticipated for this study involving the time needed to answer the questions asked by the investigators. Identity of patients was confidential with no identifying information reported. A separate list with subject name and corresponding patient identification numbers was retained by the co-investigators who were employed by the agency. Data and the list of participants were kept in a locked cabinet and designated to be destroyed five years after publication of the findings. Care was taken to assure participants that participation in the study was voluntary and care received by the veterans would not be affected by their decision to participate or not. Data were grouped for analysis.

The convenience sample of 10 chronically ill veterans ranged in age from 50 to 83 years with a mean age of 68 (SD=11.15) and was predominantly male (n=9, 90%). Half were Caucasian (n=5, 50%), 4 (40%) were African Americans, and 1 (10%) was Hispanic. Half (n=5, 50%) were married, 3 (30%) were widowed, 1 (10%) was divorced, and 1 (10%) was separated. All had 3 of the five targeted diagnoses and were distributed as follows: hypertension (n=9), diabetes (n=8), COPD (n=7 ), IHD (n=3), and CHF (n=3).

Scores for the variables knowledge of medication, caregiver home care readiness, and health promotion behavior were calculated by adding the responses and dividing by the number of indicators, resulting in a score between 1 and 5. Only 3 subjects had caregivers present in the clinic on the day of data collection. The other 7 subjects reported managing their care independently. Rather than omit the caregiver home care readiness variable, the data collectors proceeded to interview the subjects for information relevant to indicators on the caregiver home care readiness scale. Only 6 of the 19 indicators on the Caregiver Home Care Readiness instrument had to be omitted when the subject was independent in caregiving. The remaining 13 indicators were used to calculate caregiver home care readiness scores for all 10 subjects.

NOC Instruments: The mean scores for health promoting behavior were 4.05 (SD=0.51) for the interviewer data and 3.91 (SD=0.63) for the observer data. Mean scores for caregiver home care readiness were 3.80 (SD=0.28) and 3.88 (SD=0.27) respectively. Means for knowledge of medication were 3.55 (SD=0.28) and 3.44 (SD=0.32). Correlations for inter-rater reliability were calculated using Pearson’s correlation coefficients. Correlations between the interviewer and observer for knowledge of medications (r=.66, p=.04) were statistically significant but did not meet the pre-established criteria of .80. Correlations for caregiver home care readiness (r=.89, p=.001) and health promotion behavior (r=.85, p=.002) were both statistically significant and met the criteria. Further analysis of the instruments involved use of Cronbach’s alpha to evaluate reliability as follows. Knowledge of Medications produced an alpha=.71 for both the interviewer and the observer. Caregiver Home Care Readiness produced an alpha=.83 for the interviewer and an alpha=.82 for the observer. Finally, Health Promotion Behavior yielded an alpha=.84 for the interviewer and .87 for the observer.

Chart Audit Data: Descriptive statistics were computed for the chart audit data. Mean scores for health promoting behavior were 4.18 (SD = 0.40) and 4.44 (SD = 0.26) for the first and second data collectors. For the variable knowledge of mediation, mean scores were 4.22 (SD = 0.50) and 4.62 (SD = 0.39).

Inter-rater reliability was analyzed using Pearson’s correlations coefficients. Correlations between the two data collectors were not significant for the scale Health Promoting Behavior (r = .59, p = .071) or Knowledge of Medications (r = .11, p = .76) nor did they meet the .80 criterion for inter-rater reliability. Scale reliabilities were calculated using Cronbach’s alpha resulting in alphas for the scale Health Promoting Behavior of .26 and .30 for the first and second data collectors. The scale Knowledge of Medications was considerably more reliable with alphas of .74 and .80 for the first and second data collectors.

In order to compare results from the interview data using revised NOC instruments and the chart audit data using researcher-developed instruments, correlation coefficients were calculated. Neither the correlations between the NOC instrument and chart audit for health promoting behavior nor knowledge of medication were significant. Although chart audits of the computerized records were anticipated to take less time than interviews for data collection, both methods required approximately 20 minutes for the data collectors.

³³op.cit., Iowa Outcomes, 1997, p. 191.
³⁴ibid., p.103.
³⁵ibid, p. 159.

In this pilot study, the NOC instruments Knowledge of Medication,³³ Caregiver Home Care Readiness,³⁴ and Health Promoting Behavior³⁵ were evaluated for use in measuring outcomes of a nurse care manager program with chronically ill veterans. All of the instruments showed inter-item reliability with Cronbach’s alphas of .71 to .87.

The correlation coefficient for inter-rater reliability was below the acceptable .80 for Knowledge of Medications. This instrument had been modified considerably from the original NOC instrument because many of the indicators were in the scope of responsibility of pharmacists rather than nurse care mangers in the VAMC. Those modifications may have contributed to the weaker reliability results in both the inter-rater correlations and the Cronbach’s alpha.

³⁶ibid., p. 197.

This is an interactive article. Here's how it works:

Reader Comments/Questions and Author Responses

Posted August 31, 2001

Question #1: "Where can I get copies and permission to use this instrument?"

Response: "Instruments are available in the book:  Johnson, M. & Maas, M. (1997).  Iowa Outcomes Project:  Nursing Outcomes Classification (NOC). St. Louis:  Mosby.  Interested researchers should contact the authors for permission to use the instruments.

Question #2: "What further work should be done before  the instruments could be generally used to measure outcomes?"

Response: "Decisions should be made about the discrimination on a five-point scale for each of the items on the instruments. For instance, some appear that a simple "Yes" or "No" would be better than a 1-5 scale. Instruments need testing with large groups of nurses and patients to develop more confidence in the generalizability and applicability. Currently the Iowa team is continuing this work. Researchers should contact them regarding progress on testing of specific instruments from the NOC book. The project team also maintains a reference list of all publications known to them using the NOC instruments. This is a very helpful resource."

Question #3: "How have you worked with the organization that continues to develop the "NOC Indicators" to give them feedback to refine the indicators?"

Response: "Our contact with the Iowa team. We sent them our article and pointed out the difficulties we had had. We acknowledge that our revisions in the instruments may not yield the same results that other users have. They posted the reference to our article on their master list. We hope to conduct further studies that will further the NOC work in the future."

Question #4: "In your opinion, what would you consider current dilemmas with nursing-sensitive outcomes?"

Response: "For a variety of reasons, we have not continued with research using the NOC instruments. The issue we raised in the article regarding response sets--1,2,3,4,5 --when a "yes" "no" response would have been more logical. The NOC researchers may have improved the response sets since our work in 1998-1999.

The second dilemma we faced, and probably would still be true today, is that
adding a burden of extra docummentation to nursing staff is problematic. Our
data collectors had to collect data within their care giving time with clients
resulting in their having to complete other work on their own time. Having
reimbursement to nurses for their time to collect the data could improve this
situation.

We were quite excited about the NOC work and the instruments. Results of our
evaluation were favorable and we encourage continued use and development.