The reaction is fastest when $X$ is:

The reactivity of acid derivatives depends on the leaving ability of $X^{-}$.Leaving group ability follows: ${\text{Cl}}^{-}>{\text{OCOR}}^{-}>{\text{OR}}^{-}>{\text{NH}}_2^{-}$.Thus, acid chlorides are the most reactive.