The study of the processes that occur in the nucleus of atoms, including fission, fusion, and radioactive decay.