IMO, the real advantage of clip ons is not whether or not you get faster with them or slow (some may think that is true) or if you are passing people. The REAL advantage for me is that it puts me in a very natural position on the track. Makes it easier to hang off and most importantly, if I low side, I can just get a spare bar and fix my bike and go back out there. With handlebars, you have to carry a spare.
If you have to swap then yes, I agree. It's a painful thing. If you keep the clip ons permanently, it will make your life easier.
You will have to get shorter S lines (if you were planning on getting SS lines, this is a good time), you need to shorten the headlight brackets and you need a set of clip ons. After that you just need to drill a couple of holes and you are done.
There was an instructor with a KTM on my last track day. He was running handlebars and he's fast as hell. He was running on the advanced group. So yes, some people feel fine with them. I ran my Duc's with handle bars and I didn't like it.