I tried for a while to make those small changes, but I always found it too hard to do, until I finally just decided to cut out all animal products one night, and I never really went back.
I think the difference was how I framed it, mentally. I always saw it as an act of willpower to not eat animal products, like I have to overcome my cravings in the same way I would if I was cutting calories. But quitting animal products altogether allowed me to frame it differently for myself -- instead of telling myself "I shouldn't eat this", I can just say "I don't eat this." Like, it's not on the table as something I have to consider. I don't even have to recognize animal products as food.
Maybe if you cut things out one at a time you could do a similar thing. Though one problem is that it's a series of changes and commitments you have to make, instead of just one thing. I feel like that could be harder, depending on who you are.
Animals eat food, too. If you eat meat, you're actually creating more demand for crops than you would if you ate the crops directly. Furthermore, migrant workers also work on animal farms, in slaughterhouses, etc. I hear it's not always great.
I guess what I'm saying is, I'm fairly sure that going vegan helps both animals and the children of migrant workers.
One caveat is that I'm assuming you'd eat the same classes of crops that an animal would, namely things like corn and grains. But honestly that sounds about right for most people, vegans included. Many vegans eat a lot of processed shit too lol. (me included)
Edit: I should add that the most commonly suggested vegan diet that I've heard from other vegans is to have rice and lentils as your staple foods. I'm fairly sure those aren't typically harvested by hand, but I could be wrong.