If either major party's expected Presidential nominee were for some reason to decline or be unable to accept the nomination, the convention delegates would select a replacement. In recent decades convention outcomes have usually been locked in well before the formal process, as the primary process has driven the decision.
If a nominee were for some reason to be unable to continue after the convention, each party has rules. The GOP's Rule 9 states that the party has to either reconvene the convention or put the choice to its national committee. In the case of the Democrats, a decision on whom to put forward falls to the national committee.
In 1972, the Democratic vice-presidential nominee dropped out some days after the convention. The committee approved the candidate's choice of another running mate.