Combine water and sugar in a small saucepan; bring to a boil, stirring until sugar dissolves. Add tea; cover and steep 5 minutes. Strain tea mixture through a fine sieve into a bowl; discard tea leaves. Stir in lemon juice; chill completely.
Stir in milk. Pour mixture into the freezer can of an ice-cream freezer; freeze according to manufacturer's instructions. Spoon the ice cream into a freezer-safe container; cover and freeze 2 hours or until firm.