I remember if you changing the jobs and change it back again you'll lose your stats levels?...
Your stats will remain the same, but depending on your job. Some stats will increase more as well as hp (when lvling up normally).
The only time you suffer is when you change a job in the beginning, what they call 'crystal sickness'. But only the job will suffer, like if you change to black mage. Then you will deal less magic damage till the turns required are fullfilled.