Trump spoke just minutes after Pelosi and Senate Minority Leader Sen. Chuck Schumer were supposed to be meeting with him to find common ground on infrastructure.