Stephen Colbert has caused a lot of controversy over the years - both as the irate caricature of right-wing pundits on 'The Colbert Report,' and by unleashing outrageous zingers while hosting 'The Late Show.' The comedian rarely lets criticism ruffle his feathers, but there was one joke that made Colbert think his career might be over.

In 2014, Washington Redskins owner Dan Snyder responded to criticism over his NFL team's incendiary nickname by creating The Washington Redskins Original Americans Foundation - a charity designed to support indigenous peoples in America. Colbert ridiculed the move on air by saying, ”I am willing to show the Asian community I care by introducing the Ching-Chong Ding-Dong Foundation for Sensitivity to Orientals or Whatever.”

As if the satirical jab wasn't inflammatory enough, the Twitter account for 'The Colbert Report' made things worse by tweeting the joke without the context, which led to Colbert getting pilloried on social media.

"That’s when I thought, 'Oh, well, it’s been a good career,'” Colbert recalled in a recent interview with Rolling Stone. "I was getting in a car to go home, and I saw that it exploded. And I went, 'Uh-oh.' What happened was, just that one line, absent any context, was tweeted out by someone who the week before had been an intern. There was nothing I could do; I wasn’t on the air for three days. And I went, 'I’ve lost complete control of the context of my joke, and maybe I’ve lost a 25-year career with a single line.'”

Things only got worse when Comedy Central tried to do damage control.

"It was the only time I ever really got mad at the network," Colbert explained. "Because they took the tweet down, and I go, 'What’re you thinking? Now you’ve apologized before I can contextualize my response, and now I’m 100 percent fucked. By putting that thing up there without the context of the character — and the story being that the Redskins were starting the [foundation] — without that context, you’ve given people who really would like to stick a knife in me a place to stand, and an 11-inch bowie knife.'”

And while his career endured, Colbert now looks back on that moment as a deciding factor in his decision to move beyond playing a grotesque embellishment of right-wing commentators.

"It was just that this was the sort of thing that could happen. It could happen to any performer, but because I was in character, and part of the game was how much could you get away with in character — trying to make a pure expression of something that you disagree with and having that intention come through. I began to doubt my ability to do that, and not that this actual moment had been my fault; the lack of context was the fault. But that definitely reinforced my sense that I had to stop."

And while it's clear that the incident left him scarred, he added that there's no hard feelings for the person who responded to the controversial joke by launching the #cancelcolbert movement on Twitter. 

"That young woman’s feelings about that joke, in context or out of context, are perfectly valid, even if I don’t agree," Colbert said. "It doesn’t mean that people’s feelings about it are not valid. All you can do is control your intention, not people’s interpretations. Everyone’s feelings are valid, especially from any community that has been marginalized and has been told habitually their concerns are not valid. So I hold nothing against anybody who is offended by what I say."

But he's probably not a huge fan of Twitter after that platform nearly transformed him from a pop-culture phenomenon to a cautionary tale.