My best friend has been riding his DR650 to work, and hardly ever did he hit any gravel. He's been loving it, just as I love my SV650S. If the dirt is packed and kinda even, with proper tires it should be just fine, but I have to say that if I had to choose a bike for dual-purpose, I'd go with a dual-purpose bike like the DR650. I have taken my heavy cruiser on dirt roads before, that were not maintained, with rocks, ruts and tree roots sticking out... it's fine when I only do a few miles, a few times a year when I go camping. If I were to do it daily, nah, I'd take a pass.
Okierider: go with your gut. The worst that will happen if you go with the SV, is that you won't love its handling in the dirt, and you'll go buy something else and keep the SV for highway riding, or sell it and make someone else happy. If you change the tires to 80/20 type, keep the old ones to toss them in if you end up selling the bike.