That depends.
If you have 20mbps and it’s consistently that high then it would be fine unless someone is also using your bandwidth and causes it to hit its limit. This would cause many issues. However, depending on your internet type, you may experience lower bandwidth and inconsistent performance during certain hours of the day. A DSL connection is known for this and there’s really no way around this unless you switched to Cable or Fiber. I’m not sure what connection type you have so I can’t really say for sure.
In my case, on Fiber, I don’t share a line with anyone. I get a consistent Gigabit connection that isn’t affected by others. On Cable and DSL, you do and if one has an issue in their house line it could create an issue for everyone connected to them. I experienced this on Cable myself.

