Banking group UBS and many of its U.S.-based affiliates agreed to pay $1.43 billion in penalties to settle a civil action alleging fraud in the sale of residential mortgage-backed securities, the company and the Justice Department said Monday.