Nearly four million Americans have stopped working or looking for jobs, a 2.2% contraction of the U.S. work force. A smaller labor market leaves fewer workers to build machines and clean tables, restraining the economy’s long-term prospects.