French sailor Louis Burton, whose boat Bureau Vallée was hit by a trawler, pulled out after wind conditions made it impossible for him to return to Les Sables d'Olonne in western France to repair a damaged shroud — rigging that holds up the mast.
The collision happened about 400 nautical miles (460 miles, 741 kilometres) off the coast of the Portuguese capital, Lisbon. The 27-year-old instead had to go to Corunna in northwest Spain, organizers said.
British sailor Samantha Davies, 38, dismasted on Thursday evening in strong winds about 130 nautical miles northeast of Madeira.
"I'm fine — I was inside the boat when it happened," she told organizers on a conference call on Friday morning.
She confirmed that she would take no further part in the race, adding that she was disappointed.
Davies finished fourth in the 2008-9 edition of the competition.
Twenty skippers set off from Les Sables d'Olonne last Saturday but now only 16 remain.
Two, including Burton, have pulled out following collisions with trawlers, while one of the favourites, Marc Guillemot, also quit after the keel of his boat was badly damaged.
France's Armel Le Cleac'h — nicknamed "The Jackal" — was leading the prestigious non-stop race at 0400 GMT, 7.5 nautical miles ahead of compatriot Francois Gabart and Switzerland's Bernard Stamm.
Britain's Alex Thomson, currently in fourth position, was quoted as saying in an interview with the Guardian newspaper published on Friday: "To my mind, it's the most difficult sporting challenge that exists on the planet today . . .
"Something like 100 times more people have climbed Everest than have sailed single-handed around the world, so that shows you how hard it really is."