Seeming to exclude the possibility of paying a ransom, Fabius told the National Assembly: "We must do the maximum (to free the hostages) but nothing would be worse than yielding. We will not yield to terrorist groups."
The family - a couple, their children aged five, eight, 10 and 12 and an uncle - were snatched in northern Cameroon by six gunmen on three motorbikes on Tuesday and officials said they had been taken across the border into Nigeria.
France has said it suspects Nigerian Islamist group Boko Haram was behind the abduction.
"There is every indication they were kidnapped to Nigeria. Everything also indicates that the perpetrators of this abduction" were Boko Haram, Fabius said.
"This adds to the other hostage-takings. Sadly France is one the countries that is perhaps most affected by this," he said.
President Francois Hollande condemned the seizure as an "odious" act, saying: "This is the first time that children have been taken hostage in this manner."
With the latest abduction, France has overtaken the United States as the country with the most number of hostages held abroad, with 15 nationals in captivity against nine Americans.