Place potatoes in a large pot; add cold water to cover potatoes by 1 inch. Bring to a boil and cook until potatoes are just fork-tender, 5 to 7 minutes; drain. Gently toss in a large bowl with 3 Tbsp. cider vinegar. Let cool slightly.
Add onion, celery, pickles and eggs to bowl with potatoes and gently mix. In a small bowl, whisk mayonnaise, mustard, chives and remaining 2 Tbsp. cider vinegar. Season with salt and pepper. Pour mixture over potatoes and toss gently. Cover and chill for at least 5 hours.