1. Preheat panini press or grill pan.
2. Using a pastry brush, brush oil on both sides of bread. Spread jam evenly over 1 side of each slice of bread; sprinkle with rosemary. Cut mozzarella into 8 slices; place 2 slices of cheese on each of 4 bread slices. Sprinkle a pinch of salt over cheese, if desired; top with remaining 4 slices of bread, jam side down. Sprinkle tops with brown sugar.
3. Grill panini in a panini press until cheese has melted and bread is golden and crispy (about 3 minutes). If you do not have a panini press or indoor grill, use a ridged grill pan: Put sandwiches in pan (in batches, if necessary); place a weight, such as a brick wrapped in aluminum foil or a heavy cast-iron skillet, on top to press them down. Brown the first side (about 2-3 minutes), flip the sandwich, replace the weight, and brown the other side (about 2-3 minutes) to finish melting the cheese. Cut paninis in half; serve.