Around 120 migrants were aboard the inflatable craft when it ran into trouble, according to survivors taken to Al-Hmidiya, east of the capital Tripoli.
Sixteen people were rescued, while the missing included two babies and three children under the age of 12.
Coast guard spokesman Ayoub Gassim quoted a Yemeni survivor as saying the boat carried around 120-125 migrants, including women and children, before it capsized.
The Libyan coast guard also intercepted three smuggling boats carrying around 345 migrants east of Tripoli.
Libya is the main departure point for most migrants trying to reach Europe.
More than 1,000 people have drowned or gone missing in the Mediterranean already this year.
The deaths came as EU leaders came to an agreement on how to deal with unauthorised migration following marathon talks overnight in Brussels.
The 28 leaders agreed to consider setting up "disembarkation platforms" outside the bloc, most likely in North Africa, in a bid to discourage migrants boarding EU-bound smuggler boats.
