Short answer: current evidence is unclear.
As Contrarian mentions, formation of a synapse does not always mean 'learning' as synapses are formed and eliminated all the time, however -- both -- formation and elimination have been found to be linked with learning (interestingly the same spines that are 'learnted' can be 'unlearnted'). So in this sense, the formation of a new synapse or the elimination of an existing one could be 1) 'learning' or it could be 2) a stochastic process unrelated to any learning.
Furthermore, we know that synapses have variable efficacy, which is commonly assumed to underlie learning (evidence on this is less clear). As Contrarian mentions, Hebb's postulate is a nice place to start. It is far from complete however as much evidence suggests that sometimes when 'Neuron A' takes part in the firing of 'Neuron B' a conjoining synapse may be weakened. Hebb's postulate is a very simplified view (and originally didn't even include weakening at all).
When it comes to memory, many people assume that synapses are the substrate for memory, and this may be accurate however it is still an assumption as current evidence is not concrete (for example, here is a paper with evidence that CaMKII, a protein kinase, could be a candidate for the molecular basis of memory, or perhaps a combination of several molecules -- a nice review paper). It is usually assumed that strong synapses represent memories however this may not be the case, it could be that a weak synapse represents a memory.
An important note to this, is the time scale which you elude to. Synapse formation/elimination will take longer than the time for you to remember clicking the 'Ask Question' button. In this case it might be that a group of neurons (commonly called assemblies) are 'remembering' this with their activity, independent of any changes in synapses.
Substantial evidence has been collected to suggest that the basis of a memory is a group of neurons (often termed a cellular assembly) and their connections, simply put: see papers from the Tonegawa Lab.
(Edited to include some references.)