The appeals court in the western city of Rennes dismissed a last-ditch bid from the families to re-open the case of the Bugaled Breizh.
The fishing boat sank suddenly off Britain's Lizard Point in the Channel on January 15, 2004, in circumstances that have never been fully explained.
The families believe that a submarine could have been involved as traces of titanium — often used in the hulls of subs — were found on the ship's cables.
In 2013, French authorities ruled out the theory when two expert reports said neither the Royal Navy's HMS Turbulent nor any other submarine was involved in the shipwreck.
The US has also denied it had a nuclear attack sub in the area.
French authorities have said that the sinking was most likely caused by a cable snagging on the sea bed.
In a crunch hearing in March, state prosecutor Pascal Bougy called for the case to be shelved, saying that the hypothesis of a sea accident was as credible as the involvement of a submarine.
He said the court should have “the courage to say that we cannot carry out an enquiry indefinitely when we know for sure that there isn't the slightest chance of proving anything.”
“There is an enormous level of doubt. I don't know the truth,” said Bougy.
Speaking to AFP before the decision, a lawyer for the families Dominique Tricaud vowed to take the case to the European Court of Human Rights if necessary.
“I think we are getting close to the time when someone in the know, especially in the crew of the (French submarine) Rubis and the Turbulent, will have the courage to tell the truth.”
“One is a seaman before one is a soldier and I think there are secrets that a mariner will not take with him to the grave.”