Stock wood-dowel diameters are often slightly too large for the holes that you've drilled.

To ensure a snug fit--and avoid the possibility of splitting your workpiece--chuck the dowel into a drill press or portable drill and sand the dowel as it spins. To maintain a consistent diameter (and keep your fingers cool), move the sandpaper up and down the dowel as you sand. Don't oversand!
—From the WOOD® magazine shop