Abstract: We propose a new methodology for solving an uncertain stochastic Markovian control problem in discrete time. We extend the robust control approach by incorporating a learning mechanism on the unknown model dynamics. The learning algorithm is meant to reduce the uncertainty on the true probabilistic structure while observing dynamics of some driving state variables. We prove that the optimal strategies are solutions of robust Bellman equations. The methodology is numerically illustrated on a portfolio selection problem under model uncertainty.