That's a good call. Versatility in film and TV will probably come in handy... Because, yes, like your boss said, there's no guarantee that you'll have work in between animation projects.

I just work in a kitchen as a fry cook. It's really mindless work and it pays pretty well for... you know, what it is. (My coworkers are perpetually high, so the bar is set pretty low.). It's not challenging, but it'll suffice for now. I never have difficulty paying my bills on time and I almost always have money for art supplies and most other things I want. If I'm serious about a project getting done, I'll set a deadline and make it second priority. Right now my art is limited mostly by my temporary living situation, lack of space and being suddenly understaffed at work. But yeah, those things happen, too.

So no, it's not really hard, and it's not the worst situation ever by any means.