Justice Arturo Zaldivar had recommended Cassez's release due to irregularities in her trial, after a long legal battle which has strained diplomatic relations and divided opinion in a country where kidnapping is rife.
Five Supreme Court justices ruled 3-2 against granting Cassez her immediate freedom, but four also said her rights had been violated.
"I'd have liked to see her release but I think we have a great victory," Cassez's lawyer, Agustin Acosta, said.
The decision "opens the debate to discuss the irregularities (in the case) of Florence Cassez."
A court official later told AFP that Justice Olga Sanchez, who voted in favour of Cassez's release, would make a new recommendation to the five-member panel, without giving a timeframe.
Mexican authorities have presented the arrest of Cassez and her ex-boyfriend Israel Vallarta, in late 2005, as a spectacular success in the fight against rampant kidnappings.
The attorney general's office hailed Wednesday's ruling as a "victory" for the victims and stood by its prosecution of Cassez for organized crime, kidnapping and weapons possession.
But Zaldivar wrote that the proceedings had been flawed, criticizing the police's role in a TV setup showing a "live" raid on a ranch to free three kidnap victims. The raid actually took place hours after the suspects were detained elsewhere.
He also criticized the failure to respect the presumption of innocence or to immediately present Cassez to consular and legal officials.
Cassez's mother, speaking in Paris, said she was "sad, disappointed and shocked" at the decision by the Mexican court.
"We can't lose hope... We must once more keep up the fight," a visibly emotional Charlotte Cassez told reporters in the French capital.
Her daughter's French lawyer Frank Berton called the situation "a massive judicial impasse."
"I am convinced that (President Felipe) Calderon made this a personal matter," Berton insisted. "This is a real international scandal."
The case has strained diplomatic relations between Mexico and France.
Mexico's government even canceled a year of Mexican cultural celebrations in France after French President Nicolas Sarkozy said he would dedicate them to Cassez.
Families of Mexican kidnap victims have meanwhile expressed outrage at Cassez's possible release.
"Today, we the victims are confident that justice will finally prevail and that a person accused of kidnapping has not been let out free," said Isabel Miranda de Wallace, director of the organization "Stop Kidnappings" and candidate for mayor of Mexico City from Calderon's conservative party.
Cassez, who is now 37 and has spent more than six years in a Mexican jail, has denied any knowledge of the crimes and claims the authorities made an example of her.