A 66-year-old man from Mount Macedon was driving a white BMW SUV, which mounted the kerb in Albert Street, ploughing into the lawn area of the hotel beer garden.

It's understood the vehicle struck diners eating in the area, killing four people at the scene, including two men in their 30s, a woman in her 40s and a small child. A fifth person, a teenage girl, died a short time later in hospital.

Of the injured, an 11-month-old boy was taken to Ballarat Base Hospital in a stable condition, along with the driver of the SUV.

A boy, possibly aged six, was flown to the Royal Children's Hospital with broken legs and internal injuries. On Monday, he was in a stable condition.

A 35-year-old woman was flown to the Royal Melbourne Hospital and is in a serious condition in intensive care.

A 43-year-old woman and a 38-year-old man were also flown to the Royal Melbourne Hospital in stable conditions.

It's believed the victims were members of two families, known to each other and visiting Daylesford for the long weekend.

Victoria Police Chief Commissioner Shane Patton said the driver of the SUV was treated for shock and minor injuries in hospital.

MCIU detectives are expecting to interview him on Tuesday to get his version of events.

The man is unknown to police and has no prior convictions.

"The Major Collision Unit has attended and done everything you would expect them to do - a reconstruction of the scene to establish the movement of the vehicle, speeds and those types of things," Chief Commissioner Patton said.

"It is still very early days in this investigation."

Police found no alcohol in the man's system during a breath-test following the crash. They were yet to rule out any other potential factors in the crash.

Police were yet to track down all family members who needed to be notified and difficulties identifying everyone given the "horrendous scenes" at the crash site, Chief Commissioner Patton said.

"What I can say in essence is that there were two different family groups that were known to each other, who were significantly involved and impacted by this," he said.

Police have rejected accounts of a delayed response to the crash, saying two members of the Daylesford police station arrived on foot almost immediately.

The scene was described as "horrific", with bystanders rendering medical treatment until emergency crews arrived.

Bodies were quickly covered in white sheets and debris was scattered all over the road. A single shoe could be seen lying nearby, while a light pole was also knocked over.

Roads were blocked off and ambulances took patients by road to the local sporting oval, where four were airlifted.

Ambulance Victoria said others were taken by road to hospitals in Ballarat and Melbourne.

Addressing the media at 11pm on Sunday, Victoria Police Superintendent John Fitzpatrick from Road Policing Command said police were yet to identify the victims.

"[The vehicle] has driven through an area where there were patrons eating dinner and enjoying themselves. As a result, we have people deceased at the scene,'' he said.

Supt Fitzpatrick said it was a really complex scene and police were trying to work through what had occurred.

"We'll canvas a whole range of places around that scene to try and get some CCTV footage," he said.

Ambulance Victoria regional director Trevor Weston said initial crews arriving at the hotel found "chaotic" and "confronting" scenes.

Mr Weston said his thoughts are with Daylesford paramedics who first responded to the emergency.

"You never want to respond to any incident like this but certainly not in your hometown," he said.

"Any incident involving children is that next level of distress for our paramedics."

A number of locals rushed to the scene, where there were multiple casualties, to render medical treatment before emergency services arrived.

Bystanders reported hearing the vehicle making strange noises and jolting right before it crashed into the beer garden, which is a popular venue for locals and tourists, especially on a long weekend.

Witness Rhonda White, 67, told media that immediately after the crash, people came from everywhere to help the injured.

"All I remember is people just doing CPR ... there were so many people that ran to the rescue," she said.

"It was about 15 minutes before the police arrived and half an hour before the ambulance ... so in that time, it was just bedlam.

"[It was] horrible, horrible and I can't believe what I saw. We were pitching in and helping people; I don't know where they came from."

A Melbourne couple who arrived in the area a couple of minutes after the accident, spoke to ACM reporter Gabrielle Hodson.

The pair had driven past the hotel earlier in the day and described the area as packed with people.

"We arrived probably five minutes after the accident and white sheets were up, so we knew it was very serious," the couple said.

"We're visiting and are just in total shock. We can't believe the whole town has to go through this and what we've seen, it's absolutely horrific.

"We just feel so distraught for the family and friends, it's an absolute tragedy."

Anyone who witness the incident, has a dashcam/CCTV footage or information is urged to contact Crime Stoppers on 1800 333 000 or report online at crimestoppersvic.com.au.