We study the Casimir-Polder interaction at finite temperatures between a polarizable small, anisotropic particle and a non-planar surface using a derivative expansion. We obtain the leading and the next-to-leading curvature corrections to the interaction for low and high temperatures. Explicit results are provided for the retarded limit in the presence of a perfectly conducting surface.