The result will have major implications for Europe as it struggles to emerge from a financial crisis and for France, the eurozone's second-largest economy and a nuclear-armed permanent member of the UN Security Council.
Hollande won the vote with about 52 percent, according to several estimates from polling firms based on ballot samples, becoming France's first Socialist president since Francois Mitterrand left office in 1995.
Sarkozy quickly conceded defeat and signalled that he intends to step back from frontline politics.
"The French people have made their choice... Francois Hollande is president of France and he must be respected," the outgoing leader told an emotional crowd of supporters, adding that he had wished his successor well.
"In this new era, I will remain one of you, but my place will no longer be the same. My engagement with the life of my country will now be different, but time will never strain the bonds between us," he told supporters.
Sarkozy stopped short of confirming his retirement, but leaders in his right-wing UMP party told AFP that he had told them he would not lead them into June's parliamentary elections.
Hollande was due to speak later and joyful crowds had already gathered in his adopted hometown of Tulle and in Paris to celebrate his victory.
"We are rid of a poison that was blighting our society. A normal president! It gives us a lot to dream about," said Didier Stephan, a 70-year-old artist who was among throngs of supporters at Paris's Place de la Bastille.
Hollande led in opinion polls throughout the campaign and won the April 22 first round with 28.6 percent to Sarkozy's 27.2 percent – making the right-winger the first-ever incumbent to lose in the first round.
Grey skies and rain showers greeted voters across much of France, but turnout was high, hitting 71.96 percent at 5:00 pm (1500 GMT) according to interior ministry figures. More than 46 million people were eligible to vote.
The election was marked by fears over European Union-imposed austerity and globalisation, and Hollande has said his first foreign meeting will be with German Chancellor Angela Merkel – the key driver of EU budget policy.
The 57-year-old Socialist has vowed to renegotiate the hard-fought fiscal austerity pact signed by EU leaders in March to make it focus more on growth, but is facing resistance from Merkel.
The French vote coincides with an election in Greece, where exit polls showed the country's two main parties suffering big losses for landing the country in its bleak economic state.
Anger over sputtering economies has brought down leaders from Ireland to Portugal since the debt crisis washed over the European continent.
Berlin moved quickly to establish ties with Hollande, with Foreign Minister Guido Westerwelle vowing "close partnership" between the two nations and saying: "We will work together on a growth pact."
Hollande has said he will move quickly to implement his traditionally Socialist tax-and-spend programme, which calls for boosting taxes on the rich, increasing state spending and hiring some 60,000 teachers.
Sarkozy fought a fierce campaign, saying a victory for Hollande would spark market panic and financial chaos and calling him a "liar" and "slanderer" in the final days of the race.
But Sarkozy failed to overcome deep-rooted anger at meagre economic growth and increasing joblessness, and disappointment after he failed to live up to the promises of his 2007 election.
Sarkozy, 57, was also deeply unpopular on a personal level, with many voters turned off by his flashy "bling bling" lifestyle – exemplified by his marriage to former supermodel Carla Bruni – and aggressive behaviour.
Hollande has vowed to be a "normal president" in contrast with Sarkozy, but some have raised concerns over his lack of experience.
Hollande, a long-time Socialist party leader and local lawmaker from the central Correze region, has never held a top government post.
The first round of the election last month was marked by a record score for Marine Le Pen of the far-right, anti-immigrant and anti-Europe National Front, when she took nearly 18 percent of the vote.
Sarkozy turned increasingly to the right ahead of the run-off – vowing to restrict immigration and "defend French values" – but Le Pen refused to call on her supporters to back him and she cast a blank ballot.
Hollande won the backing of centrist Francois Bayrou, who took nine percent in the first round, and Communist-backed Jean-Luc Melenchon of the Left Front, who took 11 percent.
"This is a very big failure (for Sarkozy) against a candidate who has no experience in government," said political analyst Stephane Rozes.
"It is not so much for the content of his policies that he has been punished, but for his way of being and acting," Rozes said.
Hollande is expected to be sworn in by May 15 and after seeing Merkel will quickly set off for a series of international meetings, including a G8 summit in the US on May 18-19 and NATO gathering in Chicago on May 20-21.
The Socialists, Sarkozy's right-wing UMP and France's other political parties will now be focused on a parliamentary election to be held over two rounds on June 10 and June 17.