Semi-parametric nonlinear mixed-effects (SNM) models extend
current statistical nonlinear models for grouped data in
two directions: adding flexibility to a nonlinear mixed-effects
model by allowing the mean function to depend on some non-parametric
functions, and providing ways to model covariance structure and
covariates effects in an SNR model. An SNM model assumes that

where the first-stage model () is the same as a SNR model (), and the second-stage model is the same as one for a nonlinear mixed-effect model. Specifically, is the response of subject at design point , are independent variables in a general domain , is a known function of which depends on a vector of parameter and a vector of unknown non-parametric function ; random errors ; is a -vector of fixed effects, is -vector of random effects associated with subject ; and are design matrices of sizes and for the fixed and random effects respectively. It is assumed that the random effects and random errors are mutually independent. Each function is modeled using an SS ANOVA model ().

Let
,
,
,
,
,
and
.
The SNM model () and () can then be written in a
matrix form

(44) |

where , and . In the following we assume that and depend on an unknown parameter vector .

For fixed
and ,
we estimate
,
,
,
as the minimizers
of the following double penalized log-likelihood

Denote , and as solutions to (). Let , and . We estimate and as minimizers of the approximate profile log-likelihood

Since may interact with and in a complicated way, we have to use iterative procedures to solve () and (). We proposed two procedures in Ke and Wang (2001) for the case when is linear in . It is not difficult to extend these procedures to the general case. In the following we describe the extension of Procedure 1 in Ke and Wang (2001).

**Procedure 1**: estimate
,
,
,
and
iteratively using the following three steps:

(a) given the current estimates of , and , update by solving ();

(b) given the current estimates of and , update and by solving ();

(c) given the current estimates of , and , update and by solving ().

Note that step (b) corresponds to the *pseudo-data* step and step
(c) corresponds to part of the *LME* step in Lindstrom and Bates
(1990). Thus the `nlme` can be used to accomplish (b) and (c).
In step (a) () is reduced to () after certain
transformations. Then depending on if is linear in
,
the `ssr` or `nnr` function can be used to update
.
We choose smoothing parameters using a data-adaptive criterion such as
GCV, GML or UBR at each iteration.

To minimize () we need to alternate between steps (a) and (b)
until convergence. Our simulations indicate that one iteration is
usually enough. Figure shows the flow chart of Procedure 1 if we
alternate (a) and (b) only once. Step (a) can be solved by `ssr` or
`nnr`. It is easy to see that steps (b) and (c) are
equivalent to fitting a NLMM with
fixed at the current estimate
using the same methods proposed in Lindstrom and Bates
(1990). Therefore these two steps can be combined and solved by S
program `nlme` (Pinheiro and Bates, 2000).
Figure suggests an obvious iterative algorithm by calling
`nnr` and `nlme` alternately. It is not difficult to
use other options in our implementation. For example, we may alternate
steps (a) and (b) several times before proceeding to step
(c). In our studies these approaches usually gave the same results.
For details about the estimation methods and procedures, see
Ke and Wang (2001).

Approximate Bayesian confidence intervals can be constructed for (Ke and Wang, 2001).