Deep neural networks have demonstrated impressive performance in predictive tasks.  However, these models have been found to be opaque, brittle, and data-hungry---which frustrates their use for scientific, medical, and safety-critical applications.  In this talk, I describe how imposing additional probabilistic structure on the network makes it more amenable to the best practices of traditional statistical modeling.  For instance, I show that the deep learning regularization strategy known as “dropout” can be interpreted as a Bayesian structured shrinkage prior.  Taking this perspective better illuminates modeling assumptions as well as improves performance in small-data settings.  For a second example, I show how to constrain the deep neural network to encode only bijective functions.  Under this constraint, the network can then be interpreted as a reparameterized linear model, which in turn improves the tractability of inference and criticism.  I close the talk by arguing that this latter example presents a modest but clear step towards next-generation probabilistic models.  Ideally, these models should be capable of making complex, autonomous decisions, but at the same time, can be probed and criticized within the traditional statistical modeling workflow.