As employees lined up outside the entrance, Biden immediately began to shake hands and exchange hugs.
EPA chief Gina McCarthy joined Biden and began to hand out the muffins.
"168,000 people laid off, no inspections on water and air and the Everglades," Biden said to reporters. "These guys not only took a hit and weren't sure -- I mean, the anxiety of knowing whether they'd get back or paid."
"But now they're back and they've got all that work piled up so they've got a lot to do, so I'm not going to hold them up very long."
When asked if there was any guarantee another shutdown would be prevented, Biden said nothing is guaranteed.
"Look, what you saw last night is Democrats and a significant number of responsible Republicans saying enough," Biden said to reporters. "There was no economic rationale for any of this."
"And I hope, I hope everybody walks away with a lesson that this is unnecessary and I hope we can regain the trust of the American people."
Employees were handed a one-page memo with a welcome back message from EPA chief Gina McCarthy.
"I know the past couple of weeks have been difficult, especially for families. But we are working to ensure your transition back to the office is smooth and as painless as possible."
The memo also said all government travel is canceled through Oct. 20 and offered seven points on how to transition back to work.
"It was depressing," Ettereteen Welch, the morning security officer, said of the shutdown. It made her realize "how much I enjoy doing what I do."
Interior Secretary Sally Jewell also greeted returning employees outside her department Thursday morning.
"Proud to welcome back @interior teammates to work this morning," Jewell tweeted along with a photo.
White House chief of staff Denis McDonough was spotted with his sleeves rolled up and a No. 2 pencil behind his ear as he greeted workers outside the Eisenhower Executive Office Building on Thursday.