What causes an individual to not be able to hold all the details of a large project in memory?

Background: First of all, I'm not in any kind related to cognitive science, I'm a programmer. I'm not even sure that it's the right place for asking my question, but I'll try anyway. So, first of all, I'll describe the initial problem that motivated me to ask this question. There are some concrete examples.

• I join the programmers team that have rather big (but not huge) project. I'm trying to figure out how it works, how each part of the system cooperates with each other. I want to see how it works as a whole, so that I could say that I understand what's going on: what goes first, what goes next, without going into pretty much detail. I want to have this image in mind and to discuss the project with no need to diving into code, or to schemes describing the project, or any additional info. The thing that does not let me do this -- I fail to remember the system components, their execution precedence that may vary, and their interactions between each other.

• I got explanations of some project use case. Everything seems to be clear, but there comes a time when I realize that I can not hold all this in my head, can not keep all precedence of actions, the interactions between them. The whole thing becomes blurry.

I tend to see these cases having the same reason. As this reason results in my wish for better results achievement, I call it a problem. It may or may not be a "problem" from medical or any other point of view, but this is a problem for me anyway. So, when I talk about solving this problem, I talk about improving my results in area described above.

Questions

• What is this problem called where a person can not hold all the details of a large project in memory?
• Is this a failure of memory?
• Is the problem due to lack of intelligence?
• Would any of the following reduce the problem: chess, poker, solving math problems, nootropics?
-
sounds like "seeing the forest for the trees" or the "Gestalt" of it all. The "big picture". These are all terms we use in society, anyway. Not sure of the formal psychology term. I think your question could be more concise and include initial research once we find it, though. –  Keegan Keplinger Apr 14 '13 at 11:14
What's the time frame we are talking about here? How long have you been on this project? If this is a large and complex project that has been built over the course of several months or even years, I'd be surprised if anybody could remember it all in a few days or weeks. Just take notes: draw the structure, note down the key elements and their function, etc. That is, create your own manual for this project. With time, you'll remember all the parts that you regularly work on. –  user1196 Apr 14 '13 at 13:12
@what Thx, I do the schemes and draw the structures, but I want to operate with all this stuff in memory and do it quickly. Of course I understand that it can not be achieved in a few days, but anyway I want to shorten this time frame, to improve the ability of the details remembering, of predicting what effect one component changing would have, etc. –  Zapadlo Apr 14 '13 at 13:55
How well is the code commented out? If the creators didn't take the time to put a short note next to the functions when they were created then this would be a great time to do so. It will both serve as a way to familiarize yourself with each function and to provide helpful hints when your memory does fail you. If possible take the code home with you, homework doesn't just end when you get out of school. I think one of the problems you are facing is simply one of differences in familiarity. The other members of the team have probably been together since the beginning of the program, or at least –  user3290 Jul 18 '13 at 12:44

I just completed an extensive study (and corresponding lit review) of how people learn. I think you are running into the classic difference between expert and novice users.

Experts vs Novices

It has been demonstrated that the knowledge structures in experts are different than those in novices. In particular, experts' knowledge is far more structured, and they are capable of immediately connecting concepts that novices are unable to see. (I have articles I can cite if you wish, but I'm just trying to get you on the right track here).

Bascially, people learn by slowly (or quickly) becoming "experts," that is, by organizing their conceptual models better around various concepts, such that the distinctions between them disappear into this web of schematic knowledge.

It has also been observed that novices remember learning "episodes" - e.g. hands-on exploration activities or (sometimes) lectures. As the novice becomes an expert, the details of these individual episodes tend to fuzz together and a unified cognitive schema emerges.

Short-term Working Memory

As was mentioned in a previous answer, short-term memory can hold 5-9 discreet "chunks" of information. To use programming terms, a chunk can either be a struct, where it represents actual data stored in working memory, or it can be a pointer to the location in long-term memory where a concept is stored. This helps explain why experts can perform so much more quickly than novices, because their working memory can be much more efficiently utilized.

People rely on working memory to learn. As you learn, new information is incorporated into your existing mental model (this is known as constructivism). This is an active process requiring the information to be reprocessed in working memory before it is stored into LTM.

Implications

The research above suggests that you need to take the system in pieces. Learn what each piece does via a series of hands-on episodes, then move on to the next piece. Then, come back and re-start this process. As you do so, you will find yourself becoming more and more familiar with this complex system.

-
By far better description than mine on short term memory. Thanks for your answer. –  Jaime Ivan Cervantes Apr 23 '13 at 16:27
Thanks! Learning is a complex process, not well-understood, so any little tidbit, if used properly, should help :) –  rmayer06 Apr 23 '13 at 17:51
rmayer06, thx for great answer. Some questions. 1. Can not find decent translation to "hands-on exploration activities", would you please explain what do you mean by this? 2. >their [experts'] working memory can be much more efficiently utilized. More efficiently utilized -- because of their "knowledge is far more structured"? I.e., because of they have more experience? –  Zapadlo Apr 26 '13 at 9:51
1. Hands-on in terms of giving students a very structured problem/scenario and asking them to answer specific questions (e.g. a laboratory), some call it problem-based learning (see papers by Hmelo-Silver). 2. The knowledge structures are more efficiently organized, and more connections are made to relevant concepts, thus making recall easier. –  rmayer06 Apr 26 '13 at 18:47

I don't know if this is helpful, but I would chime in as a programmer. I would argue that few people truly know all the parts of a large system by heart, instead they know how to find out/debug what's happening within the project quickly.

A lot of large, complex programming projects are a mess. They are difficult to understand and maintain/navigate/debug. An Apple engineer once joked:

"If you were as smart as you could be when you wrote the program, and it's twice as hard to debug something than to write it, how are you ever going to debug it?"

Large projects did not originate over a single day, but may be a result of an "evolution" over a few months or years. So the basic components which are simple get burdened by exceptions, special use cases, subclasses, multiple modes of operation and precedents as you have said.

Additionally, programmers in general rarely write useful, clear, up to date documentation of large complex systems (there's no energy left to do this at the end).

It is easier for the original developer to navigate such project without having to keep all details in memory, because the programmer knows what to search for. The name of the class, instance variable or a method would bring the original creator to the proper point within the project. But as a "new guy", you do not have the luxury of remembering many "landmarks" within the project. So it takes longer for you to find the solution or understand what's happening.

-

The problem that you are describing appears to be a problem of short-term memory. It is well known among psychologists and neuroscientists that most people can retain 7 +/- 2 numbers or words in short-term memory. For a programmer, I would describe short-term memory and the process of "recalling", as a process similar to searching in a tree structure.

I am a software developer too. I understand your frustration, and I assure you that most programmers have similar problems when dealing with complex projects. What I recommend you to do, is to find a good systems engineering book, and learn how to work with complex software projects. If you are working with a team, I recommend you to break apart your project into modules with high cohesion and low coupling. Assign a module to each member of the team, and workout the interfaces between the sub-systems. I also recommend you to set some kind of version control.

-
I believe you mean 5-9 discreet "chunks" - they don't have to be numbers or words. This is why experts learn more effectively than novices (see my answer below). –  rmayer06 Apr 23 '13 at 15:12
You are right, something like "Discrete Elements/Chunks" is probably the word that I was looking for. –  Jaime Ivan Cervantes Apr 23 '13 at 16:20