1. Preheat oven to 350°.
2. Weigh or lightly spoon flours, potato starch, and flaxseed meal into dry measuring cups; level with a knife. Combine flours, potato starch, flaxseed meal, xanthan gum, and next 3 ingredients (through salt) in a medium bowl; stir with a whisk. Stir in olives and rosemary; make a well in center of mixture. Combine buttermilk and next 3 ingredients (through eggs); stir with a whisk. Add to flour mixture, stirring just until moist.
3. Pour batter into an 8 x 4-inch loaf pan coated with cooking spray. Bake at 350° for 30 to 40 minutes or until a wooden pick inserted in center comes out clean. Cool 15 minutes in pan on a wire rack; remove from pan. Cool completely on wire rack.