About 60 percent of Barbuda's roughly 1,400 people were left homeless, Gaston Browne, Antigua and Barbuda prime minister, told the Associated Press news agency, when the eye of the storm passed almost directly overhead early on Wednesday.
"Either they were totally demolished or they would have lost their roof," Browne said after returning to Antigua from a plane trip to the neighbouring island.
"It is just really a horrendous situation." Browne said roads and telecommunications systems were destroyed and recovery will take months.
A two-year-old was killed as a family tried to escape a damaged home during the storm, he said.
Puerto Rico was buffeted by powerful winds and heavy rain as authorities struggled to get aid to small Caribbean islands already devastated by the storm.
The US National Weather Service said Puerto Rico had not seen a hurricane of Irma's magnitude since Hurricane San Felipe in 1928, which killed a total of 2,748 people in Guadeloupe, Puerto Rico and Florida.
But as the storm moved west, it devastated the small islands in its path.
On St Thomas, in the US Virgin Islands, Laura Strickling spent 12 hours hunkered down with her husband and one-year-old daughter in a boarded-up basement apartment with no power as the storm raged outside.
They emerged to find the lush island in tatters, with many of their neighbours' homes damaged and the once-dense vegetation largely gone.
Significant effects were also reported on St Martin, an island split between French and Dutch control.
Photos and video circulating on social media showed major damage to the airport in Philipsburg and the coastal village of Marigot heavily flooded.
By late Wednesday, the centre of the storm was about 135km north-northwest of San Juan, Puerto Rico, and heading west-northwest at 26 kph.
The US National Hurricane Center said Irma's winds would fluctuate, but the storm would probably remain at Category 4 or 5 for the next day or two as it moves past just to the north of the Dominican Republic and Haiti on Thursday, nears the Turks & Caicos and parts of the Bahamas by Thursday night and touches Cuba on Friday night.
It will then probably head north towards Florida, which rushed to prepare for a possible direct hit on the Miami area with potentially catastrophic 300km per hour (kph) winds.
Meanwhile, the UN estimates as many as 37 million people could be affected by Hurricane Irma.
Stephane Dujarric, UN spokesman, said the UN has deployed a humanitarian team to Barbados to work with the Caribbean Disaster Emergency Management Agency to help hurricane victims, and additional teams are on standby.