Place marshmallows on a baking sheet lined with parchment paper. Broil 45 seconds or until puffed and toasted. Cool on pan 2 minutes.
Crush 4 graham cracker sheets into a blender. Add milk, 4 toasted marshmallows, chocolate, and ice cream; process until smooth. Pour milk shake into 4 glasses. Top each shake with 1 marshmallow. Serve immediately.