Place first 3 ingredients in a spice or coffee grinder; process until finely ground. Combine spice mixture with salt and pepper in a small bowl. Rub pork with 1 teaspoon oil. Sprinkle spice mixture evenly over pork; wrap with plastic wrap. Refrigerate 2 hours.
Place pork on a grill rack coated with cooking spray; grill 16 minutes or until a thermometer inserted in the thickest portion registers 155°, turning once. Let stand 10 minutes; slice pork crosswise into 1/4-inch-thick slices.
Place plum halves in a shallow dish. Drizzle with remaining 1 1/2 teaspoons oil; toss gently to coat. Place plums cut sides down on a grill rack coated with cooking spray. Grill 3 minutes or until golden. Serve with cheese and pork. Garnish with rosemary sprigs, if desired.