I know that feeling lol.
I have a love/hate relationship with my Google speaker. Some examples that have happened:
"Hey google, set the temperature to 21 degrees"
google: "Sure. Setting the heater to 71 degrees!"
And
me: hey google, start a timer for ten minutes
google: okay, starting a 10 minute timer
me: perfect
google: okay, consider it cancelled
me: ...
I would definitely unplug it. If you, someone else, or perhaps even a pet accidentally closes the door and it turns on with nobody realizing it bad things can happen. It will basically microwave itself and if you leave it running long enough it creates a big fire risk. It is a small mistake to make with a high risk.