There are many, many more neuromodulators in the brain than that. Essentially anything that binds to G-protein coupled receptors could be considered a neuromodulator. Even "classic" fast neurotransmitters like GABA and glutamate have "neuromodulatory" receptors (GABA-B receptors for GABA, 3 different families of "metabotropic glutamate receptors"). Peptides (e.g., endogenous opioids, orexin, oxytocin...) also commonly bind to G-protein coupled receptors.
There are even "orphan receptors" that exist, but we haven't identified the chemical that naturally binds to them.
As you get deeper into pharmacology, you'll learn that each of the common neurotransmitters has MANY kinds of receptors that are located in different places and do different things. On the other hand, G-protein coupled receptors can be be classified according to which G proteins they bind to. "Second messengers" would be a good thing to search for to learn more about this (cAMP, phospholipase C.....).
To answer your question, though, we can measure the release of some neurotransmitters using PET. The idea is that you pick a drug that binds to a particular receptor for that neurotransmitter (e.g., dopamine). Then, you manufacture some of that drug such that one of the atoms in the drug is radioactive (i.e., the drug is made with a radio-isotope of carbon, fluorine, etc.). The PET scanner will pick up the radioactivity released by the drug, which will be localized to places that contain that type of receptor (e.g., D2 receptors). Drugs don't stick to receptors perfectly, though. The natural ligands for the receptors ALSO bind to the receptors.
You can take advantage of this by having two conditions in your experiment. In one, subjects do some control task while you measure the level of radioactivity in various parts of the brain. In the other, they do the task you're interested in. If that task is associated with an increase in dopamine release, you should see a DECREASE in the amount of radioactivity, since some of the receptors will now be occupied by normal, nonradioactive dopamine instead of the drug. The more dopamine release, the less radioactivity.
Neurotransmitter levels can also be measured directly using a technique called "microdialysis". The idea is that you've got an animal implanted with a probe that ends in a particular brain region you're interested in. The end of the probe is a "semi-permeable membrane", and you've got liquid flowing through the probe. At regular intervals, you take some of the liquid and use a chemistry technique called high performance liquid chromatography (HPLC) to measure the levels of various things that dissolved into the probe. In rare cases, you can do microdialysis experiments in humans (patients with severe epilepsy).
Another way of measuring the release of particular neurochemicals, with better time resolution than microdialysis, is fast scan cyclic voltammetry (sp?). I'm not really familiar with it, but it exists.
In terms of interactions among different neurotransmitter systems, there are many ways this can happen. Cells that release transmitter A might be excited or inhibited by transmitter B. Transmitters A and B might act on different G proteins in the same cell, with opposite effects on some downstream system ("Gi coupled receptors" and "Gs coupled receptors" have opposite effects on cAMP, for example). Receptors for two different transmitters may be physically right next to each other and affect each other's functioning (D2 receptors and adenosine A2A receptors, for example). Neurotransmitters may bind to each other's receptors or use each other's uptake proteins (cortical dopamine is taken up by norepinephrine transporters; there are norepinephrine receptors in the striatum despite the absence of norepinephrine terminals, presumably because they can be activated by dopamine). Single neurons may release more than one transmitter at a time (glutamate and dopamine can be released from the same cell). The downstream effects of one receptor may cause a second receptor to be phosphorylated, affecting its function in some way. Downstream effects could also change gene expression.
My suggestion would be to start searching PubMed for review papers related to things you're interested in. At least some of them should be available without a university library.
