Very many references may easily be found with a Google search for "mathematical model memory". Probably the most classic and iconic reference is Atkinson and Shiffrin (1965), which is also described on Wikipedia. Its three components and their relationships are nicely encapsulated in this figure:
Many other, lesser-known mathematical models of memory exist, including but not limited to these:
- A model of recognition memory (Atkinson & Juola, 1974)
- A model of the circular process in short-term memory (Min'ko & Petunin, 1981)
- An application of interference theory to memory retention failures (Anninos, 1972)
- A model of hippocampus activity involving affine and chaotic rules (Yamaguti, Kuroda, Fukushima, Tsukada, & Tsuda, 2011; see also Tsuda, Yamaguti, Kuroda, Fukushima, & Tsukada, 2008; Fukushima, Tsukada, Tsuda, Yamaguti, & Kuroda, 2009)
- A computer simulation model of short-term learning (Anderson, 1983; see also a simpler model proposed by Hicklin, 1976, cited in a review of Anderson's article by Preece & Anderson, 1984)
- A computer simulation model of operant conditioning (Bower, 1967; Mandler, 1967)
- The Hopfield (1982, 1984, 2006) neural network model of associative memory (Rojas, 1996; see also Wikipedia; thanks @Memming!) describes neural activity as binary, the attributes of memories as state space locations, and the system as governed by a Lyupanov function. Much more could be said about artificial neural network models of memory in general (e.g., LSTM, Hochreiter & Schmidhuber, 1997; ART, Carpenter & Grossberg, 1987a, 1987b).
You may also be interested in the following references:
And a related question here, in which "memory" comes up twice:
- Anderson, O. R. (1983). A neuromathematical model of human information processing and its application to science content acquisition. Journal of Research in Science Teaching, 20(7), 603–620.
- Anninos, P. A. (1972). Mathematical model of memory trace and forgetfulness. Kybernetik, 10(3), 165–167.
- Atkinson, R. C., & Juola, J. F. (1974). Search and decision processes in recognition memory. In D. H. Krantz, R. C. Atkinson, R. D. Luce, & P. Suppes (Eds.), Contemporary developments in mathematical psychology: I. Learning, memory and thinking. Oxford, England: W. H. Freeman.
- Atkinson, R. C., & Shiffrin, R. M. (1965). Mathematical models for memory and learning. Technical Report No. 79: Psychological Series. Institute for Mathematical Studies in the Social Sciences, Stanford University. Retrieved from http://www.rca.ucsd.edu/selected_papers/IMSSS_79.pdf.
- Bower, G. (1967). A multicomponent theory of the memory trace. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: I. Oxford, England: Academic Press.
- Carpenter, G. A., & Grossberg, S. (1987). ART 2: Self-organization of stable category recognition codes for analog input patterns. Applied Optics, 26(23), 4919–4930. Retrieved from http://www.opticsinfobase.org/ao/fulltext.cfm?uri=ao-26-23-4919&id=30891.
- Carpenter, G. A., & Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1), 54–115. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.70.9588&rep=rep1&type=pdf.
- Fukushima, Y., Tsukada, M., Tsuda, I., Yamaguti, Y., & Kuroda, S. (2009). Coding mechanisms in hippocampal networks for learning and memory. In Advances in Neuro-Information Processing (pp. 72–79). Springer: Berlin Heidelberg.
- Hicklin, W. J. (1976). A model for mastery learning based on dynamic equilibrium theory. Journal of Mathematical Psychology, 13(1), 79–88.
- Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. Retrieved from http://web.eecs.utk.edu/~itamar/courses/ECE-692/Bobby_paper1.pdf.
- Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC346238/pdf/pnas00447-0135.pdf.
- Hopfield, J. J. (1984). Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the National Academy of Sciences, 81(10), 3088–3092. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC345226/pdf/pnas00611-0151.pdf.
- Hopfield, J. J. (2008). Searching for memories, Sudoku, implicit check bits, and the iterative use of not-always-correct rapid neural computation. Neural Computation, 20(5), 1119–1164. Retrieved from http://arxiv.org/ftp/q-bio/papers/0609/0609006.pdf.
- Mandler, G. (1967). Organization and memory. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: I. Oxford, England: Academic Press.
- Min'ko, A. A., & Petunin, Y. I. (1981). Mathematical modeling of short-term memory. Cybernetics, 17(2), 287–298.
- Preece, P. F., & Anderson, O. R. (1984). Mathematical modeling of learning. Journal of Research in Science Teaching, 21(9), 953–955. Retrieved from http://onlinelibrary.wiley.com/doi/10.1002/tea.3660210910/pdf.
- Raaijmakers, J. G. W. (2008). Mathematical models of human memory. In Learning and Memory: A Comprehensive Reference, Vol. 2: Cognitive Psychology of Memory (pp. 445–466). Elsevier.
- Restle, F. (1971). Mathematical models in psychology. Harmondsworth: Penguin.
- Rojas, R. (1996). Neutral networks: A systematic introduction. Springer. Retrieved from http://page.mi.fu-berlin.de/rojas/neural/neuron.pdf.
- Tsuda, I., Yamaguti, Y., Kuroda, S., Fukushima, Y., & Tsukada, M. (2008). A mathematical model for the hippocampus: Towards the understanding of episodic memory and imagination. Progress of Theoretical Physics Supplement, 173, 99–108.
- Yamaguti, Y., Kuroda, S., Fukushima, Y., Tsukada, M., & Tsuda, I. (2011). A mathematical model for Cantor coding in the hippocampus. Neural Networks, 24(1), 43–53. Retrieved from http://ow.ly/tLewI.