Grease an 8-inch square baking pan. Line bottom of pan with a strip of aluminum foil or waxed paper long enough to hang over 2 sides of pan by about 2 inches. Grease foil or waxed paper.
In a 2-quart saucepan, stir condensed milk, chocolate and fresh ginger over medium-low heat until chocolate has melted and mixture has thickened, about 5 minutes. Remove from heat and stir in crushed gingersnaps and half of candied ginger.
Spread mixture evenly into prepared pan and sprinkle top with remaining candied ginger. Refrigerate until completely set, about 2 hours.
Loosen foil sides from pan with a knife. Lift fudge out by grasping foil, and then transfer fudge to a cutting board. Cut into 16 squares.