Double minor or suck it up and focus on one?

Go with physics. You can always take a few philosophy classes, but it's completely useless as a major.