"I don't come to these things and I realized last night why that is: we've annoyed too many people," he said at the White House Correspondents' Brunch -- one of the many events surrounding Saturday evening's White House Correspondents' Association dinner.

Oliver stopped by multiple parties Friday night, which got the weekend's festivities started.

But he wasn't welcomed with open arms.

"They've been passive aggressive," he said of his run-ins with D.C.'s political class.

"They say they really love the show with clinched teeth," he noted, adding that their faces are "not suggesting that's true."

Oliver acknowledged there are several politicians the show would love to book, but can't get to come on: former Alaska Gov. Sarah Palin, former Vice President Dick Cheney, and former President George W. Bush, among others.

But he won't be asking anyone he bumps into over the weekend.

"I don't know if this is a good time to say 'Do you want to come on the show?' And have them say 'yes' with their face saying 'no.' "

Follow The Hill's In The Know columnist Judy Kurtz (@JudyKurtz) and Features Editor Emily Goodin for (@Emilylgoodin) this weekend for live updates from the White House Correspondents' Association dinner and related parties.