Specter famously switched parties in 2009, from Republican to Democrat, after some 44 years as an elected official for the GOP. Casey said that decision took a toll on Specter, who counted many Republicans among his best friends.
"That was a tough decision," Casey said. "I knew he was doing it because he was thinking how he could best serve Pennsylvania."
Casey also said he would miss Specter's sense of humor; the senator would go on to pursue stand-up comedy after leaving the Senate in 2011.
"He had a great sense of humor," Casey said. "Sometimes someone who's that capable and that serious and that determined, you can forget the fact that they have a sense of humor, and he did. Not only as he exhibited as a stand-up comic, but also just in the interactions we would have. He would tell stories about campaigns and politics, and he was a delight to be around, especially on the floor when you could get him a couple of minutes to have him tell a war story or two."