Welcome to the Institute for Digital Reseach and Education
Institute for Digital Reseach and Education Home
Help the Stat Consulting Group by giving a gift
Stata FAQ:
How can I use countfit in choosing a count model?
When modeling a count variable, there are several available models to choose
from. Stata supports Poisson, negative binomial, zero-inflated Poisson,
zero-inflated negative binomial models, zero-truncated Poisson, and
zero-truncated negative binomial models. Among these, you can likely
narrow your data down to one or two models based on how the data were collected
and the distribution of your outcome variable.
Some important questions to ask yourself before you start building a count model
are:
- Does my outcome variable contain zeroes?
- If yes, what does a zero mean?
- If no, why not?
- How is my outcome variable distributed?
- What is the variance of my outcome variable?
- How does the variance compare to the mean?
Deciding on a final model
You may find that, after making all of the above considerations and deciding which predictors should be included in your model, you are still torn. You have two count models that both seem substantively reasonable, but that are clearly mathematically different. At this point, the countfit function in Stata (written by Long and Freese) can be a helpful tool. You can download it by typing findit countfit (see How can I use the findit command to search for programs and get additional help? for more information about using findit) and following the appropriate links.Countfit runs user-specified count models (Poisson, zero-inflated Poisson, negative binomial, and zero-inflated negative binomial) with user-specified variables and compares the model residuals. If you do not indicate which models you wish to compare (nbreg, zinp, prm, or zip), countfit will default to running and comparing all four. One if the quirks of countfit is that it displays results for a given predictor next to the predictor's label rather than name. So an unlabeled predictor will lead to both empty space in the output and possible confusion. This can be prevented by labeling ALL predictors before running countfit.
In this example, we will be looking at academic information on 316 students. The response variable is days absent during the school year (daysabs). We have narrowed our model choices down to two negative binomial models, one with zero-inflation and one without. We believe that the daysabs can best be predicted with math standardized tests score (mathnce), language standardized tests score (langnce) and gender (female). We suspect certain zeroes can be predicted with bilingual status (biling) and language score (langnce). We will run the countfit command indicating these predictors and the two models we wish to compare, then discuss the output.use http://www.ats.ucla.edu/stat/stata/notes/lahigh, clear generate female = (gender == 1)label variable female `"female"'countfit daysabs mathnce langnce female, inf(biling langnce) nbreg zinb---------------------------------------------------------- Variable | NBRM ZINB ---------------------------------+------------------------ daysabs | ctbs math nce | 0.998 0.998 | -0.33 -0.34 ctbs lang nce | 0.986 0.987 | -2.57 -2.38 female | 1.539 1.502 | 3.09 2.94 Constant | 9.825 9.795 | 10.89 10.94 ---------------------------------+------------------------ lnalpha | Constant | 1.288 1.191 | 2.65 1.62 ---------------------------------+------------------------ inflate | bilingual status | 10.132 | 1.53 ctbs lang nce | 1.094 | 1.92 Constant | 0.000 | -2.16 ---------------------------------+------------------------ Statistics | alpha | 1.288 N | 316.000 316.000 ll | -880.873 -879.131 bic | 1790.525 1804.307 aic | 1771.746 1774.262 ---------------------------------------------------------- legend: b/t Comparison of Mean Observed and Predicted Count Maximum At Mean Model Difference Value |Diff| --------------------------------------------- NBRM 0.011 1 0.004 ZINB 0.018 1 0.005 NBRM: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.201 0.004 0.031 1 0.146 0.135 0.011 0.278 2 0.098 0.104 0.006 0.093 3 0.082 0.083 0.001 0.004 4 0.070 0.068 0.001 0.008 5 0.063 0.057 0.006 0.230 6 0.047 0.048 0.000 0.001 7 0.038 0.040 0.002 0.046 8 0.028 0.034 0.006 0.318 9 0.035 0.029 0.005 0.319 ------------------------------------------------ Sum 0.804 0.799 0.044 1.327 ZINB: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.207 0.011 0.176 1 0.146 0.127 0.018 0.850 2 0.098 0.101 0.003 0.022 3 0.082 0.082 0.000 0.000 4 0.070 0.068 0.001 0.007 5 0.063 0.057 0.006 0.194 6 0.047 0.048 0.001 0.006 7 0.038 0.041 0.003 0.077 8 0.028 0.035 0.007 0.393 9 0.035 0.030 0.005 0.240 ------------------------------------------------ Sum 0.804 0.798 0.055 1.965 Tests and Fit Statistics ------------------------------------------------------------------------- NBRM BIC= -28.290 AIC= 5.607 Prefer Over Evidence ------------------------------------------------------------------------- vs ZINB BIC= -14.507 dif= -13.783 NBRM ZINB Very strong AIC= 5.615 dif= -0.008 NBRM ZINB Vuong= 0.858 prob= 0.195 ZINB NBRM p=0.195
Making sense of the output
First, let's discuss the graph. Countfit produces a graph that plots the residuals from the tested models. Small residuals are indicative of good-fitting models, so the models with lines closest to zero should be considered for our data. This may be useful in eliminating a model, but be careful about deciding on one model over all the others based solely on this graph. In the graph above, we see that our two models perform very similarly for counts greater than two, and that they both differ most from the actual values and each other at the zero and one counts. At the zero and one counts, the negative binomial model appears slightly better than the zero-inflated negative binomial model.Model parameters and fit
The first table in the output summarizes the parameter estimates from each of the tested models. For each of the models, we see the exponentiated coefficients and their t-statistics in the first block of the table. Then, for any negative binomial models, we will see the estimated dispersion parameters. Next, for any zero-inflated models, we see the estimates from the logistic model predicting the certain zeroes. In the last block of the table, a set of fit statistics is provided for each of the four models. This includes the log-likelihood, BIC, and AIC.
----------------------------------------------------------
Variable | NBRM ZINB
---------------------------------+------------------------
daysabs |
ctbs math nce | 0.998 0.998
| -0.33 -0.34
ctbs lang nce | 0.986 0.987
| -2.57 -2.38
female | 1.539 1.502
| 3.09 2.94
Constant | 9.825 9.795
| 10.89 10.94
---------------------------------+------------------------
lnalpha |
Constant | 1.288 1.191
| 2.65 1.62
---------------------------------+------------------------
inflate |
bilingual status | 10.132
| 1.53
ctbs lang nce | 1.094
| 1.92
Constant | 0.000
| -2.16
---------------------------------+------------------------
Statistics |
alpha | 1.288
N | 316.000 316.000
ll | -880.873 -879.131
bic | 1790.525 1804.307
aic | 1771.746 1774.262
----------------------------------------------------------
legend: b/t
From the last block, we can see that the two models are extremely
close. The parameter estimates are nearly identical. We can continue to look at
the rest of the output.
Residuals by count
Next, we see a table with one line per model showing the maximum and mean differences in observed versus predicted counts.
Comparison of Mean Observed and Predicted Count
Maximum At Mean
Model Difference Value |Diff|
---------------------------------------------
NBRM 0.011 1 0.004
ZINB 0.018 1 0.005
This confirms what we observed in the graph: both models performed worst when
predicting a count of 1. Between these two, we see that the negative binomial
did better at this prediction and, overall, had a lower mean absolute difference
between predicted and observed values. At this point, the negative binomial
model is looking more appropriate than the zero-inflated negative binomial
model. Next, we have one table for each of the models containing count-by-count
information.
In these two tables, we are able to see, for counts 0-9, the actual proportion of our data records with the given count and the predicted proportion from each models. The absolute difference is included, as is the given count's contribution to a Pearson Chi-Square statistic comparing the actual distribution of the data and the distribution proposed by the model. For a given row, the Pearson statistic can be calculated as N(|Diff|2)/Predicted, where N is the number of observations in the dataset. Looking at the sum of the Pearson column gives us a sense of how close the predicted proportions were to the actual proportions. Using this method to compare, the negative binomial appears better than the zero-inflated negative binomial.NBRM: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.201 0.004 0.031 1 0.146 0.135 0.011 0.278 2 0.098 0.104 0.006 0.093 3 0.082 0.083 0.001 0.004 4 0.070 0.068 0.001 0.008 5 0.063 0.057 0.006 0.230 6 0.047 0.048 0.000 0.001 7 0.038 0.040 0.002 0.046 8 0.028 0.034 0.006 0.318 9 0.035 0.029 0.005 0.319 ------------------------------------------------ Sum 0.804 0.799 0.044 1.327 ZINB: Predicted and actual probabilities Count Actual Predicted |Diff| Pearson ------------------------------------------------ 0 0.196 0.207 0.011 0.176 1 0.146 0.127 0.018 0.850 2 0.098 0.101 0.003 0.022 3 0.082 0.082 0.000 0.000 4 0.070 0.068 0.001 0.007 5 0.063 0.057 0.006 0.194 6 0.047 0.048 0.001 0.006 7 0.038 0.041 0.003 0.077 8 0.028 0.035 0.007 0.393 9 0.035 0.030 0.005 0.240 ------------------------------------------------ Sum 0.804 0.798 0.055 1.965
Summary comparisons
Tests and Fit Statistics
-------------------------------------------------------------------------
NBRM BIC= -28.290 AIC= 5.607 Prefer Over Evidence
-------------------------------------------------------------------------
vs ZINB BIC= -14.507 dif= -13.783 NBRM ZINB Very strong
AIC= 5.615 dif= -0.008 NBRM ZINB
Vuong= 0.858 prob= 0.195 ZINB NBRM p=0.195
In this table, the tested models are compared to each other head-to-head using
the tests appropriate to each comparison. Each line can be boiled down to
the last three columns. They suggest which model is preferred by the given
comparison and the strength of the evidence supporting this preference.
When we compare our two models using the BIC and AIC,
the negative binomial is preferred over zero-inflated negative
binomial. The Vuong test prefers zero-inflated negative binomial model over the
negative binomial model, but not at a statistically significant level.
Thus, these model fit statistics support what we have seen in the model
residuals.
Full model output
There may be times when you wish to see the full model output for each of the four models. This can be especially helpful in choosing between two models because it allows you to see the significance levels of the parameters, which are not included in the countfit output. For instance, in this example, we might be interested in the significance levels of our zero-inflated variables. To do this, you can add the noisily option to the countfit command:This will print the full output from each of the four models, then the summary output provided without the noisily option.countfit daysabs mathnce langnce female, inf(biling langnce) nbreg zinb noisily
Replace option
When you run countfit, variables are generated and added to your dataset. If you ran the same countfit command a second time, you would encounter error messages telling you that variables have already been defined. Thus, if you are running multiple countfit tests, you can add the replace option to your command to replace the variables generated by prior runs with the variables generated by the current run.Final thoughts
Countfit presents several nice ways for you to compare models that are not necessarily easy to compare. It presents side-by-side fit statistics and parameter estimates, it gives you a graphical representation of the model residuals, as well as a table of the residuals by counts. All of these pieces of information are worth taking into consideration when choosing a model. For your purposes, some of these measures may be more relevant than the others. Think carefully about what you are aiming to do in the model and which measures should be prioritized over others. While countfit is very convenient and provides lots of information, sorting through the information it provides must be done carefully.The content of this web site should not be construed as an endorsement of any particular web site, book, or software product by the University of California.