If you switched back to T after being on E for a couple years, your body would just begin to redistribute fat back in masculine patterns. Just as the fat distribution changed when you went on E, it would change again when you switch to T.
Maybe this isn't relevant to you, but I personally thought I didn't want any breasts before I transitioned, and when I started E I was really worried about whether my breasts would be too big, whether I would hate having breasts, etc. That worry melted away after a few months on E, and after 6 - 8 months my perspective was completely opposite of where I started, that I really wanted more breast growth and that they are one of the best parts of my feminine body.
I would guess if you want a feminine body you might also find that after some time you realize you actually like having breasts, like me. Maybe not, though - but either way, you can't really have one without the other. Breast growth is a consequence of estrogen.
If it helps, lots of trans women have small breasts, and depending on your genetics you may not end up growing much. A common rule of thumb is that you will probably be a size smaller than your cis female relatives.