The effective core potential (ECP) approach provides a successful way to calculate the electronic structures of transition metal systems. The performance of several typical ECP basis sets on calculating the nonlinear optical properties of transition metal systems has been evaluated. The static first hyperpolarizabilities β₀ of some typical organometallic compounds have been studied at the HF, MP2 and DFT levels by using different ECP basis sets for metallic atoms. To calculate properties of transition metal systems accurately, the electrons of ns and np orbitals should act as valence electrons as well as those of nd. Alternatively speaking, the effect of spin-orbit-coupling is uppermost in relativistic effects. By comparison, the ECP basis set, MHF, was good for computing the first hyperpolarizabilities of organometallic systems, and the Stuttgart RSC 1997 and the SBKJC VDZ were relatively good. The Lanl2dz, the Hay-Wadt MB (n＋1) and the Hay-Wadt VDZ (n＋1) did not apply the spin-orbit-coupling, so their calculating accuracy was a little poorish. Whereas the CRENBL and the CRENBS provided larger deviations between the computed and experimental values than other ECP basis sets.

Key words: ab initio, ECP basis set, transition metal organometallic compound, the static first hyperpolarizability