CHAPTER FOUR
OUTCOME MEASUREMENT

I. Introduction

In designing and conducting an evaluation of responsible fatherhood interventions, the evaluator must first determine the primary program outcomes of interest. As fatherhood programs vary greatly, so do the outcomes these programs seek to achieve. Some programs may have only one or two primary outcomes, while others may address an array of factors related to responsible fatherhood.

The outcomes chosen for the evaluation should be those that are most directly related to the program goals and must be amenable to measurement. Some programs may already systematically document information on particular outcomes, which can serve as a starting point for determining those that should be included in a formal evaluation. A review of more than 300 fatherhood programs, however, found that the vast majority did not document outcomes in their programs.⁽¹⁾

In the following section, we describe potential outcomes of fatherhood interventions, suggest specific measures that may be used in an evaluation, and discuss difficulties that may be encountered when developing measures for outcomes of fatherhood interventions.

II. Potential Outcomes and Methods of Measurement

Discussions with experts and examination of relevant literature yielded several potential outcomes for fatherhood interventions that may be categorized into five broad categories: ⁽²⁾

• responsible behavior;
• father's relationship with child;
• father's support capabilities;
• child well-being; and
• the co-parental relationship.

The most common methods for measuring outcomes used in published studies are self-reports by the subjects (e.g. father, child, mother) and interviews conducted by trained staff. Other methods include observation and coding of behavior by a trained observer, and examination of public records (e.g. paternity status, employment, criminal activities); these activities, however, are done with less frequency given the significant resources necessary to conduct them.

There are several issues to be aware of when designing and using outcome measures for fatherhood programs. Subjective measurements, such as closeness and quality of the father/child relationship as measured by self-reports, are likely to differ depending on the person that is reporting the measure. For instance, a father might report feeling close to his child, but the child may report not feeling close at all. In addition, what the measures mean may differ across respondents, so that one child, for example, may view closeness very differently than another child. Deciding how to use responses from different groups, either separately or in combination, as well as standardizing responses from individuals are important issues to consider when designing outcome measures.

Matching outcome measures to program goals and characteristics is also crucial. Care should be taken to ensure that the outcomes to be measured are directly related to the actual goals of the program. In addition, how outcomes are measured can significantly affect how the results should be interpreted. For instance, having the mother, as opposed to the father, provide responses can greatly change the results.

It is also important to consider the time and resources necessary to measure the outcomes. For instance, having an expert observe and code interactions between father and mother may be a desirable way of measuring quality of co-parental interactions, but given resource constraints, it may not be feasible.

Below, we discuss a variety of outcomes that might be associated with fatherhood interventions and how these outcomes can be measured. We organize the discussion by the five broad categories of outcomes: responsible father behavior, father's relationship with child, father's financial capabilities/support, child well-being, and the co-parental relationship.

A. Responsible Behavior

Examples of outcomes that might be indicative of responsible father behavior include:

1. Reduced Substance Abuse: Whether the father uses/abuses drugs or alcohol, as reported by mother and/or father. Use and/or abuse may be defined in terms of frequency, quantities, and types of drugs or alcohol used, or in terms of the clinical criteria for a substance abuse diagnosis.⁽³⁾
2. Reduced Criminal Involvement: The nature and frequency of arrest and convictions of the father, as reported by father or as ascertained from public records.
3. Reduced Unplanned Child-bearing: Whether the father has subsequent children that are unplanned and/or out-of-wedlock, and whether they are with the same mother or different mothers, as reported by father and/or mother.
4. Marriage/Stable Relationships: Whether the father has married the mother, as reported by mother and/or father, married someone else, is involved in a stable relationship, or has reduced the number of sexual partners.
5. Community Connectedness: Whether the father participates in community activities or organizations (e.g. voting, church, philanthropy, community).
6. Safe Sex Behavior: Whether the father has knowledge of and practices safe sex behavior.

B. Father's Relationship With Child

Examples of outcomes that illustrate the nature of the father's relationship with his child include

1. Contact/Visitation: How often the father visits or has contact with the child and the duration of the visits, as reported by mother, father, and/or child.
2. Paternity Status: Whether the father has established paternity, as reported by father or mother, or as ascertained from government records.
3. Type of Child-Related Activities in which the Father Participates: A measure of how active the father is in his child's life, as reported by mother and father. This could include how regularly the father engages in activities such as providing child care, disciplining, dressing and grooming, moral training, running errands for and with the child, celebrating holidays/special occasions with the child, attending school/church activities with the child, engaging in recreational activities with the child, discussing the child's problems with the child, and taking the child on vacation.
4. Parenting Skills: Father's knowledge of child development; provision of prenatal and well-baby care and immunizations.
5. Closeness: A measure of how close a father and child are, as reported by father, mother, and/or child. For example, the child could be asked, "How close do you feel to your father?", and responses could be "not very close," "fairly close," "quite close," or "extremely close." A measure of closeness could be taken from ratings on a variety of scales, such as a child's rating of "parental understanding," "trust," "respect," "fairness," and "affection." Alternatively, trained staff could be used to observe and record interactions between father and child.

C. Father's Support Capabilities

Outcomes that illustrate a father's ability to support himself and his child, financially and otherwise, include:

1. Employment and Earnings: Whether or not the father is employed and the level of his wages, earnings, and income, as reported by father.
2. Education/Training Activities: Whether or not the father has completed a given level of education, or is engaged in education and/or training activities (e.g., GED classes, enrolled in high school, college prep courses, vocational training).
3. Child Support: How much formal or informal child support and how regularly the father is paying, as reported by mother, father, and/or public records. Any other types of non-monetary support/services provided by the father. Whether the father has an understanding of and the ability to navigate the formal child support system.
4. Other Responsibilities: Does the father have a driver's license, library card, insurance, comply with local regulations/pay fines, etc.
5. Work Ethic/Attitudes: Father's attitudes toward work and relations with employers. Job duration and reasons for leaving employment may be indicators or work ethic and attitudes.
6. Housing: Whether the father has adequate housing and a permanent address.
7. Physical Health: Father's physical health and nutrition. Physical health can be measured as an overall rating (excellent, good, fair, poor) as reported by the father, and/or as the presence or absence of specific health conditions.
8. Mental Health: Father's mental and emotional health. There exist a number of scales available to measure depression and anxiety.
9. Self-Awareness/Self-Esteem: Father's level of self-awareness and esteem, engagement in self-development activities.
10. Anger Management: Father's ability to control anger and constructively address emotional problems.
11. Ability to Deal with Racism: Father's ability to cope with racism and racial discrimination.

D. Child Well-Being

Outcomes that reflect aspects of the child's well-being might include:

1. Academic Achievement: How well the child is doing in school, as reported by teacher, father, mother, and/or child. Alternatively, the child's performance can be measured through scores on achievement tests, which might be a more accurate measure.
2. Social Behavior: A measure of how the child interacts with others, as reported by the father, mother, and/or teacher. Measures could include types of behavior at home and in school, such as peer sociability, autonomy, aggression, attitudes towards strangers, obedience, leadership, self-confidence, cooperation, and communication skills. This measure could be difficult to interpret due to potentially biased reports from the father, mother, and teacher, as well as variation in how behavior is interpreted.
3. Problem Behavior: Type and frequency of delinquent behavior, as reported by the father, mother, and/or teacher. This could include such indicators as deliberate damage of school property, truancy, lying to parents about something important, taking something without paying for it, and injuring another person seriously enough to require a visit to a medical facility. Components of this measure could be difficult to interpret due to potentially biased reports from the father, mother, and teacher, as well as variation in how behavior is interpreted.
4. Child's Economic Status: The level of household income and poverty status of the child.
5. Safety in the Household: The child's home environment, appropriate supervision, and personal safety.
6. Physical Health: Indicators of the child's physical health might include reports of the child's general health status by the mother, age appropriate level of development as measured by height and weight, days lost from school due to illness, whether the child has been immunized, and the child's access to health care and health insurance.
7. Emotional/Mental Health: A measure of how the child views himself/herself and the child's level of distress or dissatisfaction, based on responses to a self-esteem and/or a self-assessment questionnaire, reports by the mother, and/or whether the child has seen a psychiatrist or other professional about behavioral or emotional problems.

E. Co-parental or Team Relationship

Evaluators may be interested in determining the effect of a program on the relationship between a father and the mother(s) of his child(ren). Examples of outcomes that may reflect that relationship include:

1. Agreement/Cooperation Concerning Child-Rearing: A measure of how synchronous the mother's and father's views on child-rearing are, as reported by mother and father. Some components of this measure could include discussion of school problems and planning special events for the child.
2. Father's Relationship with Child's Significant Others: Father's relationship with mother's partner(s) and ability to deal with mother's attitudes towards his own partner(s). Also, his relationship with grandparents and other relatives of his child.
3. Quantity and Quality of Communication Between Parents: A measure of the parents' ability to communicate with each other about both parental and non-parental issues, as reported by mother and father. Alternatively, an expert could observe the interactions between the parents in order to more accurately assess the quality of communication; this, however, would require significant time and resources.
4. Arrangement for Child Access: Are there formal or informal arrangements for the father's access to the child and are the arrangements adhered to.
5. Agreement on Child Support: Whether the mother and father agree on the level of financial and non-financial child support provided by the father.
6. Parents' Feelings Toward Each Other: A measure of how parents feel about one another, as reported by mother and father. This could include such affect measures as guilt or anger toward the other parent, as well as measures of conflict between parents, including incidents of spousal abuse.

III. Summary

The outcomes and measures described in the preceding sections are only generic suggestions of possible fatherhood program outcomes. The actual set of outcomes and measures used in an evaluation will depend on the nature of the intervention being evaluated and the specific circumstances under which the evaluation is being conducted. It is unlikely that any particular program's impact evaluation would include all or even most of the outcomes described here.

The programs we visited varied somewhat in terms of the specific outcomes each program was designed to affect. For example, one program has a particular focus on reducing infant mortality and improving child health by increasing the involvement of the father in pre-natal and child health care. This is a very specific objective not shared by the other fatherhood programs we visited. Another program, through its arrangement with the county court system, has as one of its primary objectives, increasing the level and consistency of child support payments. This is only a secondary objective of the other programs we visited. There were, however, a number of outcomes the programs did have in common. These include (see also Exhibit 4.1):

• increased education and employment;
• reduced alcohol and drug use;
• improved parenting skills;
• increased father involvement with his child(ren);
• improved attitudes or feelings toward children; and
• improved social and family interactions.

The above outcomes represent those that fatherhood program managers believed to be the most important outcomes that their programs attempt to impact. Through our conversations with government agencies and private funders we gained a sense of the outcomes that they, as funders, believed to be most important for fatherhood programs to address. From the funder's perspective, the most important outcomes include (see also Exhibit 4.1):

• reduced unplanned child-bearing;
• reduced criminal involvement;
• increased paternity establishment;
• increased contact with child;
• increased formal or informal child support;
• increased employment and earnings;
• increased education or training;
• improved child behavior; and
• increased cooperation with mother concerning child-rearing.

Regardless of the outcomes chosen for inclusion in the evaluation, they should be ones that are directly related to the program's activities (i.e., there is a hypothesized relationship between program services and the outcome of interest) and they should be important and meaningful to the intended audience of the evaluation findings, whether that audience be program managers, funders, policymakers, or all of the above. Once the desired set of outcomes to be measured is established, the evaluator must develop survey questions to address each outcome.⁽⁴⁾ We recommend the use of questions and measures from existing survey instruments to the greatest degree possible, especially if such instruments have proven validity.⁽⁵⁾ The use of existing instruments and measures also facilitates the comparison of findings across studies.

Return to Contents

Endnotes

1.  See Levine, Jim and Pitt, Ed (1995). New Expectations: Community Strategies for Responsible Fatherhood, Family and Work Institute. New York, NY.

2.  See Appendix A for a list of experts with whom we have discussed fatherhood intervention outcomes.

3.  See American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, for the criteria for a substance abuse diagnosis.

4.  In this chapter, we have generally expressed the outcomes and measures as levels in the discussion. In some cases it may be more appropriate to measure the change in outcome variables, rather than the level.

5.  For a review of a wide variety of survey instruments designed to measure attitude and personality, see Robinson J.P. et al. (eds.) (1991), Measures of Personality and Social Psychological Attitudes, Academic Press, Inc. San Diego, CA.