I did following but still problem exists;
I moved all contacts to Phone Memory
I deleted cache of Contact Manager
I deleted app data of Contact Manager
Did Factory Reset
I deleted all contacts from SIM/Gmail/Phone etc. And then exported to Phone Memory (Total Contacts: ~500)
Still no luck! It is really really annoying, contact manager often stopped working (~10 times a day) and it takes 15 minutes to start working to normal.