Rajapakse left his party with his followers