Pulse audio gets stuck on application?

I have a Dell XPS 9343, and I frequently dock it to a monitor. Recently, I noticed that if I have an application running(say google-chrome) then plugging in the monitor will continue to make my audio for that application run through the laptop speakers until I manually change. Is there a way for Pulseaudio to forget an application output? Restarting the daemon did not help. This also happens when I unplug the monitor and move back to the laptop(any application that was set to play on the monitor is muted until I manually change it).