Unlike the hottest amateur shows airing on YY Inc.’s online entertainment platform, the company’s own performance in the third quarter was more of a flop than a hit.