Generally no, atheists do not believe that the universe came from nothing. However, they are sometimes clumsy in how they express this.
Atheists generally defer to science when it comes to questions about the origin of the universe. The current best scientific theory about the origin of the universe is the Big Bang Theory.
The Big Bang Theory asserts that, at some point in the distant past, all matter in the universe was much closer together – the universe was far denser and hotter than it is today. This idea is based on the observation that the universe is, at present, expanding. (Which in turn is based on the observation that distant galaxies all appear to be moving away from us and away from each other.) If the universe is currently expanding, everything must have been closer together in the past.
The Big Bang Theory is usually taken a step further, and it is asserted that at some point in the distant past, the entire universe was concentrated at a single point – known as a singularity. This singularity would have been infinitely dense.
However, the Big Bang Theory includes no assertions about what came before the singularity. It’s possible that the singularity always existed, or it’s possible that the singularity was the result of a previous universe collapsing in on itself. There are many ideas about what produced the singularity, but as yet there is no evidence for any of them.
The very early universe remains mostly a mystery to science, and a rational scientist – and also a rational atheist – would take no position on what came ‘before’ the singularity – they would simply say that they don’t know.
So atheists, generally, do not believe that the universe came from nothing. They would simply say that the Big Bang Theory offers the most likely view into the origin of the universe, but that it is an incomplete theory, and there is much about the origin of the universe that remains unknown.