So , you don't want to move out, for pretty good reasons, too, but you can't stand being around him. You have to CHOOSE . You pays your money and you takes your choice. Either you find a way to avoid him as much as possible, like staying at school and studying in the library, working part time, etc.; or, you table your other priorities and make it your topmost one to become independent and live elsewhere. If you have an unpleasant encounter with him, you can choose to say "I won't stand here and let you talk (treat) me this way, and walk away. Refuse to engage, answer, etc.
Apparently, your current choice is to put staying in college ahead of moving out, which is legitimate. Only you can make such a choice. There are alternatives, however: loans and scholarships, joining the military and having them pay for your education in return; becoming a live in nanny in exchange for room and board, room mates, etc. Good luck, good wishes, you can look forward to greater freedom now or in the future, depending upon the choices you make.