Depends on how fast you memorize and understand its 5 rules for ownership (3) and borrowing (2), and how much effort you put into memorizing its types.

The biggest hurdle I had with it is not reflexively understanding how the intermediate types it has work and how to bounce between them. For example: String -> &str -> String. Collection -> filtered/split slice of it -> back to Collection.. It's often not just 1 type either, it's multiple types which you get by functions declared in traits.. Just typing this is giving me a headache.

Combine unintuitive types/not knowing them all by heart with not fully understanding borrow/ownership rules, and you're going to have a bad time.

Long story short: it's a fantastic language, and I hope I never touch it ever again. I don't really need types or memory safety for what I do, but I appreciate it for what it is.

I don't like this type of question. In my experience knowing one language has little impact on learning another. What matters much more is understanding the underlying concepts.
If you grok OOP it doesn't matter if you go from Java to C# or from C++ to Python. Yes, there are differences, but they're mostly syntactic in nature.
So assuming you got the hang of imperative programming and maybe had some exposure to functional programming, too, the concept you're likely to struggle with the most is ownership. Simply because it's a concept that's fairly unique to Rust.
Having come from Java, via C++ and Python and having dabbled with Haskell a bit, I feel like The Book does a decent job of explaining Rust in general and its oddities in particular.