Prince HarryandMeghan Markleknow theywant to start a family, but don’t expect them to have a big brood!

The royal brushed off a suggestion that he should follow in the footsteps of a well wisher in Ireland — and have five children.

As the couple greeted fans in Dublin, Elaine Adam-Stewart, 43, told Harry, “My husband also has red hair and he gave me five children — when are you and Meghan going to get going?”

Harry saw the funny side, Adam-Stewart tells PEOPLE. “He laughed and said ‘five children? — too many.’ ”

Samir Hussein/WireImage

The Duke And Duchess Of Sussex Visit Ireland

Chris Jackson - Pool/Getty Images

The Duke And Duchess Of Sussex Visit Ireland

Elaine’s husband, Simon Stewart, who runs an IT consultancy, was not in the crowd outside the EPIC museum of Irish Emigration, but the English-born homemaker and her five kids — Nathan, 10, Adelaide, 8, Malachy, 7, Siwan, 5, and Portia, 22 months, were — along with an England flag as they back the team for today’s big World Cup match.

RELATED VIDEO:Prince HarryandMeghan MarkleTouch Down in Dublin! See the Newlyweds’ First Ever Airport Arrival

Earlier in the day, they sharedadorably candid moments with a couple of toddlersat Croke Park. Both Meghan and Harry share a love of children — and they’ve spoken out about their desire to start a family.

The Duke And Duchess Of Sussex Visit Ireland

PA Images/INSTAR

Prince Harry and Meghan Duchess of Sussex Royal Visit to Ireland-Day 2 Appearances

Last year, Harry said he was looking forward to fatherhood. “I would love to have kids,” he toldThe Telegraph.

In a 2016 interview with Canada’sBest Healthmagazine, Meghan was asked about her bucket list items and said, “I want to travel more and I can’t wait to start a family, but in due time.”

source: people.com