I'm wondering where Milton said he knew Philip before the apocalypse. All I remember is him saying in "Prey": "I knew him before he became what he is [The Governor]." or something similar to that. That doesn't necessarily mean he knew him before the apocalypse occurred.
I don't think there's enough evidence to claim he knew him pre-apocalypse, unless there's something I'm missing. I think he meant that he knew him before he became the monster he is.