The 29-year-old sprinter suffered the injury when he fell heavily in a high-speed crash after colliding with Australian Simon Gerrans close to the finishing line of 190.5km opening stage from Leeds to Harrogate.
Although his Omega Pharma-Quick Step team said they would wait until to decide if he would continue, the sprint ace, a 25-time Tour stage winner, said he never held out much hope.
"I knew, normally I bounce well when I crash, I always have done," Cavendish told reporters outside his team's bus at the start of second stage in York.
"When I was on the floor yesterday I knew something was wrong. It's the first time in my career I knew something was wrong.
"I wanted to cross the finish line, I was in Harrogate, the fans had come to see me, I got on my bike but it was impossible to hold the handlebars
"I saw there was something wrong with my shoulder, it was sticking out a bit so we went to get it checked out. I was in a lot of pain, I can't move my shoulder"
"I held a bit of optimism that maybe it was just a bit of swelling and would go down this morning, but it's actually worse this morning. I'm gutted, I'm major disappointed but I guess it could be worse."
Cavendish said he expected to face a lengthy spell on the sidelines and maybe even an operation.
"I've got to go and get an MRI and see if it needs surgery, the chances are it probably does and unfortunately I'm likely to be out for a few weeks now."
But while he said he was "devastated" to be out of the Tour, he said some perspective was needed.
"It could be worse, I've got friends like Taylor Phinney who's out for the rest of the season (with a broken leg)," added the Manx man.
"I've got friends who've come back from Afghanistan with the Armed Forces. I think you saw my 'Help for Heroes' helmet yesterday and my friend Josh was messaging me yesterday, he's a double amputee on his legs and amputee on his arm as well. He sent me a message yesterday joking 'I've got a spare arm for you'.
"Things could be worse, I've got everything. It was my fault at the end of the day."
Cavendish had undergone X-rays and an ultrasound in hospital night, which revealed he had not broken anything but merely suffered a dislocated collarbone.
Cavendish had been fired up to win the opening stage in Harrogate, where his mother was born.
He had been well-placed on the run-in before the dramatic crash, which saw him hit the tarmac heavily on his right shoulder and brought a number of other riders tumbling down with him, including Gerrans.
Cavendish was seen slumped on the ground, clutching his collarbone and clearly in pain.
But he held his hands up and admitted it was his fault.
"I spoke to Simon, I spoke to him after the stage and asked if he was alright, I spoke to him on the phone and apologized.
"I hope he's OK, today's a good stage for him and I really wish him the best for the rest of the tour.
"We came up and I think there were two of us going for Peter Sagan's wheel.
"There was room to the left, I went to go but Gerrans was there, I was using my head to not go into him and take us across the road but Simon wasn't budging, I wasn't budging, but at the end of the day at any other race I'd have stopped. The stage was lost but I wanted it that bad that I tried to find a gap that wasn't there."
The Tour continues with a punishing 201km ride from York to Sheffield with nine categorized climbs.