As a first step towards optimizing solid state electrolyte materials, we have constructed realistic models of various types of isolated defects in crystalline Li3PO4 involving O vacancies and N and Si dopants, and have used first principles calculational methods to study their effects on the production and migration of mobile Li ions. We find that mobile Li ion vacancies are stabilized by removing oxygen from the lattice which, in turn, causes the rebonding of nearby phosphate groups to form O3P-O-PO3 (POP) structures embedded in the crystal. In the presence of nitrogen in the system, the POP structures can be further stabilized by replacing the bridging oxygen with nitrogen. We examine the electronic and geometric structures of these stable POP and PNP defects which are prototypes of chain structures found in thin film lithium phosphorus oxynitride (LiPON) electrolytes. We also find that mobile interstitial Li ions are stabilized by N or Si dopants substituting for O or P host ions and forming PO3N or SiO4 defects, respectively. In all of these cases, the defect stabilizes extrinsic mobile ions, but also provides traps for the ions to escape into the bulk regions of the crystal by as much as 1.5 eV for the vacancies and 0.9 eV for the interstitials. On the other hand, migration barriers for diffusion steps near the defects are as small as 0.4-0.6 eV for the vacancies and 0.2-0.3 eV for the interstitials. Extrapolating our results to crystals with appreciable concentrations of defects, our results compare favorably with experimental migration energies reported in the literature.