Gordon Smyth Home: Research: Publications
Smyth, G. K., and Verbyla, A. P. (1996). A conditional approach to residual maximum likelihood estimation in generalized linear models. Journal of the Royal Statistical Society B, 58, 565-572.
Gordon K. Smyth
Department of Mathematics, University of Queensland, Q 4072, Australia.
Ari P. Verbyla
Department of Statistics, University of Adelaide, SA 5005, Australia
Residual maximum likelihood estimation (REML) is often preferred to maximum likelihood estimation as a method of estimating covariance parameters in linear models because it takes account of the loss of degrees of freedom in estimating the mean and produces unbiased estimating equations for the variance parameters. In this note it is shown that REML has an exact conditional likelihood interpretation, where the conditioning is on an appropriate sufficient statistic to remove dependence on the nuisance parameters. This interpretation clarifies the motivation for REML and generalizes directly to non-normal models in which there exists a low dimensional sufficient statistic for the fitted values. The conditional likelihood is shown to be well defined and to satisfy the properties of a likelihood function, even though this is not generally true when conditioning on statistics which depend on parameters of interest. Using the conditional likelihood representation, the concept of REML is extended to generalized linear models with varying dispersion and canonical link. Explicit calculation of the conditional likelihood is given for the oneway layout. A saddle-point approximation for the conditional likelihood is also derived.
Keywords: residual maximum likelihood, restricted maximum likelihood, conditional likelihood, exponential dispersion model, modified profile likelihood, saddle-point approximation, oneway layout.
Patterson and Thompson (1971) introduced residual maximum likelihood estimation (REML) as a method of estimating variance components in the context of unbalanced incomplete block designs. REML is often preferred to maximum likelihood estimation because it takes account of the loss of degrees of freedom in estimating the mean and produces unbiased estimating equations for the variance parameters. Alternative and more general derivations of REML are given by Harville (1974), Cooper & Thompson (1977) and Verbyla (1990). In all of these the residual likelihood is presented as the marginal likelihood of the error constrasts. This makes generalization of the residual likelihood principle to nonlinear models or non-normal distributions difficult since zero-mean error contrasts do not generally exist. Cox and Reid (1987) give an approximate conditional likelihood which reduces to REML when used to estimate covariance parameters in normal linear models. Although Cox and Reid's conditional likelihood is approximate, and is based on a simplification of Barndorff-Nielsen's (1983, 1985) modified profile likelihood which reduces to REML only in special cases, it does suggest a conditional interpretation for REML. In this paper we show that REML has an exact conditional likelihood interpretation in which the conditioning is on an appropriate sufficient statistic to remove dependence on the nuisance parameters. This interpretation clarifies the motivation for REML and generalizes directly to non-normal models in which there exists a low dimensional sufficient statistic for the fitted values.
Consider the linear model y = X b + e where y is an n x 1 vector of responses, X is an n x p design matrix of full column rank and e ~ N(0,M) is a random vector. The covariance matrix M is a function of a q-dimensional parameter g, and is assumed positive definite for g in a neighbourhood of the true value. For any fixed value of g, the statistic t=AXTM-1y, where A is any nonsingular p x p matrix function of g, is complete sufficient for b. We show that the residual likelihood can be viewed as the conditional likelihood of y given t. We show, given the above form for t, that the conditional likelihood is well defined and satisfies the properties of a likelihood function, even though this not generally true when conditioning on statistics which depend on parameters of interest.
Using the conditional likelihood representation, the concept of REML is extended to generalized linear models with varying dispersion. We assume that y1,...,yn follow a generalized linear model with canonical link, design matrix X and weights wj/sj. The wj are known prior weights and the sj are assumed to depend on g. The REML estimator of g is defined to be that which maximizes the conditional likelihood of y given t=AXTM-1y where in this case M=diag(sj/wj). Explicit calculation of the conditional likelihood is given for the oneway layout. A convenient saddle-point approximation is derived for use in other cases.
The idea of conditioning to remove nuisance parameters goes back at least to Bartlett (1936, 1937), and is discussed extensively by Kalbfleisch and Sprott (1970). Our conditional likelihood is direct and differs from that suggested by Kalbfleisch and Sprott and motivated by their ``Euclidean assumption''. The difficulties that the Euclidean assumption was intended to overcome do not occur when the conditioning statistic is of the form given above.
McCullagh and Tibshirani (1990) give an estimating equation method of adjusting profile likelihoods for nuisance parameters, which reduces to REML when estimating covariances in normal linear models. Again this is not generally equivalent to our conditional likelihood but, because it produces unbiased estimating equations, may approximate our approach in large samples.
Preprint 30 May 1995 (pdf)