Put the sugar and lavender in a spice grinder and grind to a powder. In a large bowl, combine the lavender sugar with the apricots, nectarines, cherries, lemon juice, orange zest and 1 tablespoon of the olive oil. Toss well and season lightly with salt and pepper. Let stand for 10 minutes.
Meanwhile, in a small bowl, stir the mascarpone with the honey and season with pepper. Brush the bread slices on both sides with olive oil.
Light a grill. Thread the fruit onto the rosemary skewers; reserve any juices in the bowl. Grill the bread over high heat until lightly charred and crisp, about 30 seconds per side. Grill the fruit skewers over moderate heat, basting a few times with the reserved juices, until lightly charred and just tender, about 4 minutes per side.
Spread the honey mascarpone on the grilled bread and halve each slice. Transfer to a platter. Slide the fruit off the skewers onto the bread, drizzle some of the juices over the fruit and serve.