Heat oil in a large nonstick skillet over medium heat. Add rind, rosemary, fennel seeds, and fennel wedges; cook 5 minutes, stirring frequently. Add olives; cook 1 minute. Remove from heat; stir in vinegar. Place fennel mixture in an 8-inch square baking dish. Cover and let stand at least 2 hours.
Preheat oven to 250°.
Uncover olive mixture. Bake at 250° for 10 minutes or until heated, stirring once.