Three children, product of ages is 72

A census taker asks a father the ages (whole numbers) of his three children. He says: "The product of their ages is 72." The taker says that's not enough. He adds: "The sum of their ages equals my house number." The taker looks at the house number, thinks, and says it's still not enough. The father finally says: "My oldest child is asleep upstairs." Now the census taker knows the ages.

What are the three ages, and how does that last clue settle it?