From purely an annoyance point of view, the horse and carriages cause much more disruption than the busses.
I don't think either are really an issue, mind you. I like it when Victoria's full of people, and busses, and cars, and motorcycles and bicycles and horse carriages in the summer.