The zoo in Thoiry ( see map below) outside Paris said it was the first such incident in Europe.
The perpetrators forced the main gate and broke through at least two other security barriers on Monday night, without disturbing five people who live on the grounds.
The animal, a four-year-old southern white male named Vince, was attacked inside an area where at least two other rhinos are kept.
"Staff left the rhino enclosure on Monday. When they returned on Tuesday, an animal had been killed and its two horns had been sawn off," a police spokeswoman told AFP.
She added that the horns were "probably cut off with a chainsaw".
The rhino had been shot three times in the head.
"Only the main horn was stolen," the spokeswoman said.
The two other rhinos in the enclosure with Vince which were unharmed were a 37-year-old female, Gracie, and a five-year-old male, Bruno.
Thierry Duguet, the manager of the zoo, told AFP: "This has never happened before in a zoo, either in France or in Europe.
"We are extremely shocked and upset -- this is supposed to be a sanctuary for the animals."
Thoiry zoo is equipped with video surveillance, but cameras are not installed in the area where the rhinos live.
Staff were left distressed by the attack.
Colomba de Panouse, part of the family which set up the zoo, told AFP: "The rhinos' warden, Elodie, is very distressed by what's happened.
"She was the one who made this macabre discovery and now she can't talk" because of the shock.
The zoo said in a statement: "This was carried out despite the presence of five members of staff who live on the site and (despite) security cameras."
Black market rhino horn sells for up to $60,000 per kilo -- more than gold or cocaine -- with most demand from China and Vietnam where it is coveted as a traditional medicine and aphrodisiac.
International trade in rhino horn has been banned since 1993, but thieves have realized that, in Europe at least, museums are a quick way to snag some ivory.
A spike in museum thefts around five years ago in France saw increased security measures, although brazen thieves went as far as to gas guards at a Paris museum to pilfer a rhino horn.