A shakeout in the private-equity industry is forcing firms to adapt or get left behind.

The landscape of successful firms is increasingly divided between behemoths with several business lines such as Blackstone BX -0.30%decrease; red down pointing triangle and Apollo Global Management and top performers that have focused on their roots of buying and selling companies for a profit. Firms that fall in the middle—those that are big, but not big enough, or whose returns are good but not great—are finding it difficult to raise money.