We use a derivative expansion for gently curved surfaces to compute the leading and the next-to-leading curvature corrections to the Casimir-Polder interaction between a polarizable small particle and a nonplanar surface. While our methods apply to any homogeneous and isotropic surface, explicit results are presented here for perfect conductors. We show that the derivative expansion of the Casimir-Polder potential follows from a resummation of its perturbative series, for small in-plane momenta. We consider the retarded, nonretarded and classical high-temperature limits.