Combine all ingredients except rosemary in a large bowl; stir gently. Cover and chill at least 8 hours or up to 24 hours.
Divide cheese and olives into 6 glass containers. Place 1 rosemary sprig in each container. Pour remaining dressing evenly into containers. Cover and refrigerate up to 2 weeks.