I'm not overly familiar with the show, but it would be like having Seinfeld without Seinfeld. Sure, you have Jason Alexander and some other talented cast members, but the focal point of the show is gone.
The Thick Of It was more fortunate, in that when the main star of the first two seasons was convicted of having child pornography, they could then focus the series on Malcolm Tucker; because the show was not predicated on Chris Langham's character.