Patent applications from Amazon and Google have revealed how the technology giants are planning to use the information to help build a profile about a person.
The patent filed by Google says their future devices could monitor thinks such as hygiene habits, meal schedules and general activities whilst a patent from Amazon claims new could listen out for things such as 'I love skiing' and then send out targeted adverts accordingly.
A spokesman for Amazon said: "We take privacy seriously and have built multiple layers of privacy into Echo devices. We do not use customers' voice recordings for targeted advertising. Like many companies, we file a number of forward-looking patent applications that explore the full possibilities of new technology. Patents take multiple years to receive and do not necessarily reflect current developments to products and services."
The patents were unearthed by Consumer Watchdog, who warned of the problems this could cause.
John Simpson, Consumer Watchdog's privacy and technology project director, said: "Google and Amazon executives want you to think that Google Home and Amazon Echo are there to help you out at the sound of your voice. In fact, they're all about snooping on you and your family in your home and gathering as much information on your activities as possible.
"You might find them useful sometimes, but think about what you're revealing about yourself and your family, and how that information might be used in the future. Instead of charging you for these surveillance devices, Google and Amazon should be paying you to take one into your home."