statistical significance

Variant B converted at 4.2 percent, A at 3.8 percent. Significance is the check on whether that gap is a real difference or the same coin landing heads a few extra times. The p-value answers one narrow, conditional question: assuming the two variants are truly identical and the rest of the test's assumptions hold, how often would I see a gap this big or bigger? Below your chosen threshold (alpha, conventionally 0.05) most teams are willing to call it. A small p-value says the data sit badly with the no-difference assumption; it does not hand you the probability that chance was at work, and it cannot tell you which assumption broke.

Keep two numbers apart that everyone mashes together. Alpha is the false-positive rate you agreed to tolerate on this one test. The 95 percent on a confidence interval is coverage: over many repeats of the same procedure, that many intervals would contain the true effect. They line up numerically for a standard two-sided test, which is why people conflate them, but they answer different questions, and neither one is 'a 95 percent chance B is better'. Significance also says nothing about size: with enough traffic a 0.05 percent lift goes green and is still worth nothing. Read the confidence interval on the lift, not the yes/no badge, because 'somewhere between minus 2 and plus 14 percent' is not a decision.

The gotcha that ruins most tests is peeking: checking every morning and stopping the moment it turns green. That turns a 5 percent false positive rate into something closer to 30 percent, because you gave noise twenty chances to cross the line. Fix it by computing the sample size up front from your baseline rate and minimum detectable effect, running to it, and covering whole weeks so Tuesday traffic does not outvote Sunday. If you must watch it daily, use a sequential or Bayesian method built for exactly that.