An HRV tempers the incoming air with the air going out no matter the season. So, in winter, it recovers some heat from the exhaust air and exchanges it with the cold intake air, keeping heat inside.
In the summer, it's the opposite. The exhaust air cools the hot intake air, keeping the heat outside.
It's essentially always working for your benefit (except for one edge case that I can expand on if you'd like), always providing fresh air and exhausting stale air while helping to keep the interior temperature where desired.
I'd recommend keeping it on all the time, no matter if you open windows or not.
A reminder that cities aren't loud, cars are loud.