1. Heat grill to medium-low (350° to 400°).
2. Combine 1 tbsp. chopped rosemary, the salt, and pepper in a small bowl. Peel potatoes and put in a bowl of water. Coat a large cast-iron skillet with 1 tbsp. oil. Using a mandoline or knife, cut half the potatoes into 1/8-in. slices and arrange them in tightly overlapping circles in skillet. Sprinkle with half of salt mixture. Repeat with remaining potatoes and salt mixture, making a second layer. Drizzle remaining 3 tbsp. oil over potatoes, shaking pan to distribute it well. Press potato cake with hands to compact.
3. Cook potatoes with grill lid down until well browned on bottom (lift edge with a spatula to check), 35 to 40 minutes. Remove from grill. Loosen potatoes with a wide spatula. Invert a platter over pan, carefully hold both together (use oven mitts), and invert potato cake. Scatter a little more chopped rosemary over potatoes, plus rosemary flowers if you like. Cut into wedges.
Make ahead: Keep potato cake warm in pan or on platter in a 200° oven up to 1 hour.