The coach is set to begin his first full season as Hammers boss and explained that he is taking a methodical approach to his task at the London Stadium.
Potter told BBC Sport: "The coach's job is to build something.
"You've got to try and improve your team, improve the club, improve the players. That's the trick.
"I've always tried to do what I think is the right thing for the club. At the end of the day, West Ham is the most important thing. I'm an employee."
The club hasn't spent much in this summer's transfer market but Potter has warned against splashing the cash just for the sake of it.
The manager, who joined the Hammers in January, said: "I don't think we needed drastic change.
"It's tempting to think that the solution is going to be external but, from my perspective, the focus should always be on the players you have – with an eye on improving.
"When we arrived there was, for different reasons, a low-trust environment. When you change the manager halfway through the season, there's a feeling something isn't quite working.
"If you look back over a 12-month period up to that point, the team had conceded a lot of goals which is an indication that, maybe, something wasn't quite right on a cultural or foundation level.
"It was something we started to do a lot of work with last season. We did make some progress but pre-season gives you a chance to make more of an inroad, which I think we have done."
Potter is also wary of spending big money in the transfer window after doing so during his ill-fated seven-month tenure at Chelsea.
He said: "I've experienced what £300 million can do. It's not always positive.
"You just deal with what you have, work with the people in the club and represent it the best way, by staying as close to the person I am as I can.
"I think I've done that pretty much all my career."