I do not have a problem like you're explaining. Actually I feel the way the Pacifica handles the navigation voice to be the best I've ever used. The voice navigation comes out of the left from speaker (from what I can tell). While voice is active you can adjust the volume up or down with std volume controls or you can adjust volume through the UConnect menus. Mine lowers the radios volume when speaking directions.
Maybe it is your trim level, what trim do you have? What speaker system? I have the Limited Trim with Alpine (13 Speaker 506 watt)system.
There is no way to turn the radio completely off. But the volume control can be "pushed" to mute it.