Although DNA replication is often imagined as a regular and continuous process, the DNA polymerase enzyme is a complicated machine and can pause upon encountering physical and chemical barriers. We used single molecule measurements to make a detailed characterization of this behavior as a function of the template's secondary structure and the sequence context. Strand displacement replication through a DNA hairpin by single DNA polymerase molecules was measured in real time with near single base resolution and physiological concentrations of nucleotides. These data enabled the measurement of the intrinsic "speed limit" of DNA polymerase by separating the burst synthesis rate from pausing events. The strand displacement burst synthesis rate for Escherichia coli DNA Polymerase I (KF) was found to be an order of magnitude faster than the reported bulk strand displacement rate, a discrepancy that can be accounted for by to sequence specific pausing. The ability to follow trajectories of single molecules revealed that the burst synthesis rate is also highly stochastic and varies up to 50-fold from molecule to molecule. Surprisingly, our results allow a unified explanation of strand displacement and single strand primer extension synthesis rates.