Education Reform

This post is about Kindergarten through High School education, generally referred to as K-12.  The U.S. has a reputation for providing more elite post-secondary (e.g. college and grad school) institutions than any other country in the world.  It also has a very good reputation for its non-elite schools in this category.  So the general consensus is that this category doesn't need fixing.  K-12, however, is a different matter.  This category has been argued over for generations and has been a political punching bag for at least a generation.  I have no special expertise in this area.  But that's not going to stop me from pitching my two cents in anyhow.

I am, of course, a consumer of this product.  I received my K-12 education in the U.S.  There are probably few people whose trajectory through this system is exactly typical.  Mine isn't completely typical but it's not very different and it is a common trajectory.  I was educated in a Parochial School run by the Roman Catholic parish where my parents attended church for my first 8 years.  Then I attended public schools for the last four years.  In my opinion I got a good education.  And I am in a position to personally compare the Parochial School experience with the Public School experience.  The comparison is very enlightening.

Parochial School was a "stick to basics", "no frills" experience.  The curriculum was completely standard except for the addition of a one hour religion class each day.  When I hit Public School I found I was well prepared.  I could read well.  My mathematics was up to snuff.  My social studies abilities and skills were up to snuff.  I can't say what kind of shape I would have been in had I attended Public School for those eight years but I see no reason to believe that it would have been much different.

But physically the experience was different from what I would have experienced in a Public School.  There were no shop classes.  There was no PE (Physical Education) classes.  There were no music classes.  I remember that one day a teacher brought in a Chemistry Set.  It was the kind a parent would buy for a child.  No particular use was made of this.  All of these types of amenities cost money.  You need a Gym and showers for PE.  My school had an auditorium but no showers.  You need musical instruments to do music.  My school had no musical instruments.  The school had no lab space suitable for biology or chemistry.  The physical plant of the school consisted of the aforementioned auditorium, class rooms, and a playground with a couple of basketball hoops and a few tether ball poles.  No other athletic equipment was provided.

But wait, there's more.  I do not remember attending a class with less than thirty other students.  One teacher was supposed to maintain discipline, teach all the classes, and provide whatever one-on-one attention students needed.  In short this school did a number of things that are supposed to be exactly the wrong way to educate students.  There were too many students in the class room.  There was little or no "enrichment" (e.g. music, shop, athletics).  But I got an excellent education anyhow.  And I was not an anomaly.  Many of my fellow students followed my same path.  They did a number of years in Parochial School, transitioned to Public School, and did just fine.  I was a typical exemplar of students turned out by Parochial School, not an outlier.

And this is generally true.  Parochial Schools have a very good reputation in the US.  It is so good that many Parochial Schools have a large percentage of their students coming from non-Catholic families.  Parochial Schools are the only alternative to a Public School option that is financially possible for many families.  Parochial Schools have a reputation for providing a high quality low cost option to the standard Public School option.  And they prove that a lot of the conventional wisdom about how to fix the public school system is bunk.

I certainly enjoyed my time in Public School and felt I got a good education there too.  But it is important to remember that this school district was in a well off suburban area.  The school system had and still has a very good reputation but it also has more money and a more stable social environment than many public school systems.

So if many of the standard nostrums for fixing the public school system are wrong then what's right?  Here I am very disappointed with what the Bill and Melinda Gates Foundation have come up with so far.  You can read a position paper from them here:  http://www.gatesfoundation.org/postsecondaryeducation/Documents/nextgenlearning.pdf.  I didn't think much of it.  It is  mishmash of jargon almost completely devoid of clear thinking and any kind of data driven foundation for what little it contains.

One of the ideas that the Gates Foundation and others push is Charter Schools.  Charter Schools have been around long enough so that if they did a substantially better job of educating kids we should see some clear data to support this.  But what little data I have seen indicates that Charter Schools perform about on a par with Public Schools.  Another is reducing class size.  This too has been tried a lot.  I have seen no strong evidence that this works particularly well either.  Another idea is technology in the classroom.  As a computer guy I should be all for this.  But again there is no strong evidence supporting the idea that this makes a big difference.

Parochial Schools are not much different than Charter Schools.  My Parochial School experience argues against expecting much from reduced class sizes or introducing a lot of technology.  It also argues against the great benefit of a richer experience (e.g. sports, music, labs,. etc.).  Now some of you may be about to argue that I have just contradicted myself.  If Charter Schools are a lot like Parochial Schools then they should be working well.  And they should.  But the data says they don't.  And I remember seeing a "60 Minutes" (I think it was 60 Minutes) episode where they talked to a Parochial School Principal.  She said she would not be able to do even as well (her school was rated noticeably better than the surrounding Public Schools) if she had to follow the same rules and regulations as the Public School administrators did.  So what do I think works?

One of the problems with most of the analysis of what's wrong stems from looking in the wrong places.  If you are not looking in the right places the answer to your question contains a high percentage of noise and little or no pattern will emerge.  I think the wrong things are being measured to try to find the success factors.  Here's my list of success factors:

1.  The most important thing is whether a kid comes to school willing and able to learn.  Key to this is whether the kid thinks it is important to learn.  And key to this is whether the parent(s) think education is important.
2.  If we have met the first criteria, the second criteria is a good teacher who is allowed to do her job the way she thinks it should be done.
3.  The school environment must be safe from violence and from bullying and other activities that discourage a kid that wants to learn from learning.
These are the keys.  Of lesser importance are:
*  A good and safe physical plant (e.g. no pealing paint, broken windows, lights that work, etc.)
*  Smaller classes.
*  A richer experience.
And so on.

Someone somewhere has succeeded in teaching whatever "unteachable" group you can think of.  It's not the native intelligence of the kid.  In most cases parents seek out these experiences once a program attains a good reputation.  And that is important.  Because if a kid has parents (or parent) that care how well the kid does in school then they keep on top of how the kid is doing.  This results in the kid being expected to learn.  If the kid fails the parent(s) get on him or her to improve.  In this environment most kids most of the time end up willing to learn.  And almost all kids are able to learn.  These "success story" situations also usually involve good teachers and a safe environment.  I think you can find successes that lack one or more of the remaining criteria.

How about this for an idea?  As far as I know it has never been tried.  Evaluate kids and pick out the under performing ones.  Then look at the home environment.  Look for kids with parents that are uninvolved or have low expectations.  Try to educate the parents to be better at motivating and monitoring their kids.  Then there are homes where the parents would like to do the right thing educationally by their kids.  But they can't due to poverty, language skills, violence, etc.  Here it would be nice to provide help.  But except in the language situation this should be a job for social services, not schools.  As far as I can tell low or non-existent educational attainment of parents (e.g. parental illiteracy) is not a factor. Kids of illiterate parents do very well even if the parents never learn to read.

With this as a foundation it is instructive to look at efforts to improve education.  Little or no effort is devoted to my most important item.  There have been many efforts devoted to item number two but they usually involve more not less interference with the teacher doing things the way she would prefer.  In fact there is a large industry dedicated to standards, tests, evaluations, etc., all of which have the effect of telling the teacher how to teach.  Efforts to address item number three are sporadic and not up to the task.  Instead most effort is devoted to the lower priority items or to items I didn't even list.

It is also important to pay attention to what ought not to be done.  One idea usually associated with liberals is to pass students whose work does not merit it.  I think this is a mistake.  If a kid is not doing fourth grade work or eighth grade work he should be flunked until he can demonstrate the proper proficiency.  Unqualified students provide a distraction to both teachers and other students.  In the long run they don't even do the kid any good.  For a short period of time his self esteem is left undamaged.  But society and the kid eventually figure out that the kid is not a high school graduate in terms of what he is capable of and things go rapidly down hill from there.

And there is the issue of focus.  The more things you try to do the less likely you are to do a good job of all of them.  I think schools and school districts should be focused completely on education.  In the same way that they should provide an honest evaluation of how educated a student is by not passing him along, they should not be responsible for public safety or caring for the needs of people with physical or mental problems.  If a kid is misbehaving then he should be kicked out of school and turned over to the juvenile justice or police system.  Let schools teach and let public safety organizations provide for the public safety.  Similarly, I am sympathetic to the plight of people with physical or mental handicaps.  But at some point they should become social service problems not problems to be solved by our educational systems.

Years ago the standard was to keep people with physical or mental handicaps out of sight.  This was wrong, particularly for people with mild handicaps.  The standard flipped over to "mainstreaming" everyone.  I believe this is an over reaction.  I think it is good to mainstream people with mild handicaps.  It's good for the individuals themselves.  It is also good for the other students and staff.  It broadens their experience base and makes them more tolerant, which is good.  But it is not the job of school systems to deal with all of these people, particularly with those with severe problems.  This means a line needs to be drawn.  How do you decide who is mild and who is severe?  I am not confident I have the right answer.  But I do have an answer.  How much does it cost to place and maintain a particular individual in a Public School environment.

I think you set a financial threshold.  My suggestion is six times.  This is a completely arbitrary line and it may be that some other cut off would work better.  But it is the one I have come up with.  If the additional cost is six or less times the cost of a normal student then the handicapped individual should be placed in school and become the responsibility of the school district.  If the expense will be higher then the responsibility should rest with social services.  If social services can come to an agreement with the school district and pay the school district whatever the additional cost over that of a normal student then it may be a good idea to place the individual in school.

And this is the usual "bright line" rule.  This may result in the school people and the social service people (or others) trying to game the system to get the individual over or under the threshold.  Bright line rules always result in these kinds of issues.  I am perfectly willing to entertain a "sliding degree of responsibility" where there is a sliding scale of financial and other responsibility for the individual.  But I think my idea is a good place to start the discussion from.

There is also an elephant in the room that I have not brought up yet.  That's politics.  Educational policy is a bigger political football than it has ever been before.  This is very bad for education.  It draws energy and money away from the current system.  In almost all cases more resources are better than fewer resources.  So whatever resources are eaten by the squabbling end up being taken away from where they are needed.  And if the fight gets hot enough a consensus may develop to starve the beast.  Certainly the time and effort that a good teacher spends dealing with paperwork, bureaucracy, and politics is time and efforts that can not be applied to teaching.  And good people don't like working in a politically charged atmosphere.  It's just not worth the aggravation.

Now let's look at Teach for America.  Teach for America is good intentioned.  It is designed to help solve a very real problem.  Our society depends heavily on what is generally called STEM, Science, Technology, Engineering, and Mathematics.  So our educational system needs to do STEM well.  But there is a large shortage of STEM qualified teachers.  And, as I said, this is a real problem.  Teach for America attempts to address this problem by finding STEM qualified individuals, running them through a fast "teaching boot camp" and putting them into the classroom.

It is better than nothing.  But it is a "paper it over" solution rather than an effort to address the problem directly.  The direct way to address the problem is to have teachers who are STEM qualified.  How do we do this?  Money!  If we paid teachers with STEM skills more money then we would find teachers getting STEM training and we would find STEM trained people getting teaching degrees.  But that would cost too much money.  So we have Teach for America.  The way you know that Teach for America is not the right solution is by looking at retention.  Do people who get in the program stay in teaching?  No!  Large numbers of them are gone after two years or less.  And remember the job market currently sucks.

I just don't believe that finding the right solutions to the education problem is that hard a problem.  So why aren't we well on the way to solving it?  Because the fundamental problem is money.  We need to spend more money and we need to spend it more effectively.  No one wants to spend what it would take to solve the problem so it becomes a big political football.  And in a highly politicized environment what money gets spent where becomes all the more important.  And those with the most political power, not those with the best ideas, tend to win the fights.  The overall result is that more and more money and effort is invested in making and enforcing rules, and in the kind of bureaucracy that grows up in a highly politicized environment.  This leaves less and less money to actually do what works.  And so things get worse and we start another round of political fighting that eventually makes things even worse.

Let me make a final observation on unions.  There is a large group of people invested in the idea that unions are evil and wasting lots of money and standing in the way of education reform.  In the current environment what are teachers supposed to do?  They are buffeted by every "trend du jour" and generally speaking no one cares what they think about anything.  In that environment a strong union with a mission to make teachers impossible to fire makes a lot of sense.  I think teachers unions have stood in the way of a lot of educational reform.  But it is hard to get angry at them.  There are a lot of others trying to mess up the educational system and harass teachers.  Perhaps if teachers didn't feel so much like the football in a Superbowl game they would be willing to be more flexible.

One thing Bill Gates has come to believe is that you can tell if a teacher is going to be a good teacher by seeing how they do in their first three years on the job.  If this is true then all you need to do is put in a "three year probation" rule.  If the teacher makes it through the first three years (assuming the evaluation procedure is a good one - not the teacher's responsibility) then it makes no sense to worry very much about how to get rid of teachers that have more than three years on the job.  They should be good teachers in almost all cases.  It is probably cheaper to carry the few "dead wood" teachers that make it past three years than it does to put in a lot of effort into a procedure for terminating experienced teachers.  And this would have a great benefit.  The many good teachers who made it past their three year probation could relax and focus on teaching for the rest of their carrier.  The morale boost would more than compensate for whatever it cost to keep the dead wood.

50 Years of Science - part 4

This is the fourth in a series.  The first one can be found at  http://sigma5.blogspot.com/2012/07/50-years-of-science-part-1.html. Taking the Isaac Asimov book "The Intelligent Man's Guide to the Physical Sciences" as my baseline for the state of science as it was when he wrote the book (1959 - 1960) I am examining what has changed since. For this post I am starting with the chapter Asimov titled "The Death of the Sun".

Again Asimov starts with a review of thought on the subject, starting with Aristotle.  He starts out with a general discussion of whether the sky as a whole is unchanging.  He notes several instances of changes in the sky that would have been visible to the naked eye and, therefore, noticeable to the ancients.  The Greeks either didn't notice them or decided to ignore the changes.  But other ancients did notice some of these changes.  This leads to a quite general discussion of several stellar phenomenon.  He then starts moving toward a discussion of our nearest star, the Sun.  As part of this discussion he introduces the Hertzsprung-Russell diagram and the concept of the "main sequence".

The reason for this is that these ideas form the basis for understanding how stars evolve.  This, in turn, allows us to predict the life history and eventual fate of stars.  In short, large stars burn brightly and don't last very long.  Small stars burn much more dimly but last a very long time.  Our Sun is in the middle.  It is in the middle in terms of how bright it is and also in terms of how long it will last.  The H-R diagram also allows us to predict how our Sun will age.

According to this analysis our Sun is middle aged and will stay that way for several more billions of years.  Then it will become a Red Giant, a very large, very cool star.  Asimov then relates recent (relative to 1960) developments.  Stars burn Hydrogen to make Helium.  But then they can burn Helium to make Carbon.  This chain can continue so that stars can create large amounts of Oxygen and Neon.  Asimov also reports that Magnesium, Silicon, and Iron can also be created in the heart of a star.  If a star explodes (e.g. in a Supernova) then these elements can be spread throughout space.  This was the start of solving the problem of where these other elements come from.  Only Hydrogen, Helium, and and a very small amount of Lithium are created in the Big Bang.  Of course it did not solve the mystery of where all the other elements came from.  It turns out this mechanism can not create any of the elements heavier than Iron.  Research that took place after Asimov's book came out suggests that the Supernova explosion itself creates the other elements.

Once most of the Hydrogen is burned the evolution of a star speeds up tremendously.  All the other stages happen very quickly compared to the billions of years the Hydrogen stage takes for a star the size of the Sun.  And once a star hits the Iron stage it quickly runs out of energy.  A star like the Sun goes from a Red Giant to a White Dwarf in the blink of an eye at that point.  Asimov then moves to the Chandrasekhar limit.  A star with the mass below the limit (1.4 times the mass of the Sun) will relatively gently settle into the role of a White Dwarf.  Those above the limit, however, explode as the Crab Nebula did.  This supernova explosion was observed in 1054.  But current estimates put the nebula between 5 and 8 thousand light years away.  That means the supernova actually occurred between 3,000 BC and 6,000 BC.  The best guess is that it exploded about 4.300 BC.

Asimov wraps the chapter up with the observation that White Dwarfs last tens of billions of years.  So the Sun will be a White Dwarf for much longer than it will look the way it currently does.

Missing from the discussion are Black Holes and Neutron Stars.  These existed at the time as theoretical speculation.  A few years after the book was published Astronomers concluded that Cygnus X-1, an X-ray source in the constellation Cygnus, was a black hole.  There still exists no direct observations of Black Holes.  But out understanding of them has continued to improve.  Many likely Black Holes are now known.  And there is a class of Black Holes whose existence was not even suspected at the time of Asimov's book.  Astronomers now believe that many galaxies, including our own Milky Way and our nearest large neighbor galaxy, Andromeda, contain supermassive Black Holes.  These Black Holes weigh in at millions to billions times the mass of our Sun.  It is early days in terms of our understanding of these entities.  But they seem closely bound up in the formation and evolution of galaxies.

And in 1967 something magical was found.  A radio beacon was flashing once every 1.33 seconds.  No natural source of such a bright and quickly changing entity occurred to the Astronomers who discovered it.  So they initially christened it LGM-1 for the first signal from what might be Little Green Men or more formally space aliens.  As other sources were detected the name was changed to Pulsars.  Pulsars are Neutron stars.  They are small enough that they can rotate 1.33 times per second without violating the laws of physics.  So if they have an energy source somewhere on their surface it can flash like the rotating beacon in a lighthouse.  What makes it possible to have a very small very energetic object is the collapse of a star.

nucleuses jammed right up against each other with no surrounding cloud of electrons to keep them far apart.  If this happens you end up with what Astronomers have come to call a Neutron Star.  Such a star would be only a few miles in diameter but it would weigh more than the Sun.  It is easy to imagine such a small object rotating in a full circle in about a second.

So why were Neutron Stars, Pulsars, and Black Holes not discovered by the time Asimov wrote his book?  The answer is that a lot of the evidence for these objects can not be gathered from the surface of the Earth.  You have to put a satellite into orbit.  From there it becomes possible to observer the many kinds of electromagnetic radiation that are blocked by the earth's atmosphere.  Much of the early evidence for the existence of these objects and for the data that resulted in insight into their structure came from satellites launched in the '60s after the book was written.  Since then we have launched more sophisticated satellites that have been able to gather more and better data.  We have also improved our ability to make ground based observations.  We have learned how to tie multiple radio telescopes together.  We have even succeeded in tying multiple optical telescopes together in some cases.

50 Years of Science - part 3

This the third in the series.  The first one can be found at http://sigma5.blogspot.com/2012/07/50-years-of-science-part-1.html.  Taking the Isaac Asimov book "The Intelligent Man's Guide to the Physical Sciences" as my baseline for the state of science as it was when he wrote the book (1959 - 1960) I am examining what has changed since. For this post I am continuing with the chapter Asimov titled "The Birth of the Universe".

In part 2 I discussed the age of the Earth.  In discussing the age of the Earth Asimov broaches the subject of "the solar paradox".  Cutting to the chase, Lord Kelvin did a calculation in the late 1800s that indicated that the Sun could be no more than 50,000 years old.  Why?  Because there was no known energy source that could keep it burning any longer.  The two main candidates:  "It's all coal", and gravitational collapse couldn't provide enough energy to explain the steady output of the Sun for any longer.  The discovery of radioactivity in 1896 provided an alternate energy source powerful enough to save the day.  Radioactive decay could provide enough energy to keep the Sun shining at its current level for billions of years.  Over the next forty years subsequent scientific progress allowed scientists to conclude that the Earth and Sun were each about 5 billion years old, very close to the modern figure of 4.7 billion years.  (Modern cosmology posits that the Sun and all the planets, including Earth, were created at almost the same time).

In examining the question of the age of the universe as a whole Asimov gives us a nice description of the Doppler effect.  Let's say you are driving on a road and an emergency vehicle is coming the other way.  Before it reaches you the siren will have a slightly higher than normal pitch.  After it has passed the siren will have a slightly lower pitch.  This shifting of the pitch as a result of motion is called the Doppler effect.  There are many references, including Asimov's book that can give you more detail.  But the bottom line is that this change in pitch can be used to calculate the speed of the other object.

Doppler, the physicist the phenomenon is named after, decided in 1842 that this effect could be used to calculate the speed toward or away from the earth of celestial objects by examining the "spectrum" of these objects.  For reasons that were not well understood until Quantum Mechanics were developed about 1930 when you heat something to an appropriate temperature it will glow.  The intensity of various colors in this glow are called the spectrum of the object.  An individual spectrum will contain features.  At some frequencies the intensity will be particularly bright (emission features) and at other frequencies the intensity will be particularly dim (absorption features).   An object that has the same composition and temperature will always have the same spectrum with the same emission and absorption features.  And the combination of the temperature and the atomic and molecular composition precisely determines the details of these spectral features.  In short, from the spectrogram of an object you can determine its precise composition and temperature.  The process may be very complicated for objects with a complex composition but that's the idea.

Note that I indicated above that an object's spectrum depends solely on its temperature and its composition.  But if the object is moving with a speed that is a noticeable percentage of the speed of light (and the amount of speed that is needed to qualify as "noticeable" keeps dropping as scientific instruments keep getting better), the spectral features will shift.  If the object is traveling toward the earth the frequency will shift higher and the wavelength will shift lower.  Astronomical short hand for this is "blue shift".  If the object is traveling away from the earth the frequency will shift lower and the wave length will shift higher.  The astronomical short hand for this is "red shift".  The amount of shift allows the relative speed to be calculated precisely.  Astronomers make very precise measurements of the spectrum of an object.  Then they identify well known features in the spectrum.  Then they calculate how far and in which direction (higher or lower) the feature has shifted.  From this information a simple calculation yields the speed at which the object is moving and whether it is moving toward or away from the earth.

Now if astronomical objects moved randomly you would expect that about half would show red shift and half would show blue shift.  But it turns out that almost every astronomical object shows a red shift.  Almost everything is moving away from us.  An astronomer named Silpher was the first to notice this in 1914.  Before going on let me discuss the issue of "standard candles".

How do you figure out how far away something is?  Well the simplest and most reliable method is to simply pace it off and measure it.  But what if the distance involved is too great to measure directly?  For longer distances there is a trigonometry based technique called parallax.  Again assume you are in a car.  You are driving down a straight rural road and staring sideways out the window.  This is OK because you are a passenger, not the driver.  Notice that the sections of fence near the road whiz by quickly.  But if you look out across a field at a house or barn it will move slowly as you drive along.  Finally if you look at a mountain a long ways away on the horizon it doesn't move at all.  That's the basic idea behind parallax.  You need to dress it up with trigonometry and careful measurements but if you measure the distance you travel down the road and the change in the angle of the barn or house you can calculate the exact distance it is from the road.  Taking the basic idea and applying the proper measurements and trigonometry is how astronomers can measure distances across space.  But before continuing let me take a second digression and talk about astronomical distances.

People really don't understand how big space is.  Say you get in a car and drive for an hour on a straight road at 50 miles per hour.  (I know, I know, no road is straight for that long but work with me on this).  Everyone has done something like this and it gives them some emotional idea of how far 50 miles is.  Now imagine driving at 50 miles per hour (I have picked this speed because it makes the math easier) for ten hours straight.  You have now gone 500 miles.  Now most people who are stuck in a car for ten hours straight tend to day dream a good part of the time even if they are the driver.  So even a distance of 500 miles, while intellectually comprehensible in terms of our ten hour trip at 50 miles an hour, loses a lot of its sense of concreteness.  I contend that 500 miles is about as far as people can realistically have a concrete feel for.  It is possible to get in an airplane and go thousands of miles.  But you get in the plane.  You may even look out the window for the whole trip.  But a plane ride is emotionally like using a teleporter but with a couple of hours of delay thrown in.  You don't get a real sense of the distance involved.

Now imagine a trip around the world at the equator, a distance of 25,000 miles.  In our car this would require 50 days of 10 hours per day driving.  If people tend to zone out in one 10 hour drive there is no way they are going to be paying attention every day for 10 hours for 50 days in a row.  So I contend that 25,000 miles, the circumference of the earth, is such a great distance that it is not really comprehensible.  But 25,000 miles is infinitesimal in terms of typical astronomical distances.  So all astronomical distances blur together and become "so large as to be unimaginable" in concrete terms to a person.  Scientists can do the math but the numbers are so large as to be meaningless to us.  And since we can't in any real sense comprehend these numbers we make really wild mistakes all the time.  Some numbers are really a lot bigger than other numbers.  But they are all so large that our emotions misread them and we think of them as being nearly the same size or we get it wrong as to which is really the larger and which is really the smaller.  Back to the subject at hand, namely parallax.

Parallax works well enough to be able to estimate distances within the solar system with a reasonable degree of accuracy.  The most useful "baseline" for measuring these kinds of distances is the orbit of the earth around the Sun.  It is about 100 million miles.  Compare this to the circumference of the earth at 25 thousand miles, a distance I said was too great to be emotionally comprehensible.  Well this distance is 40 times as great.  It seems inconceivably large.  But it is actually quite small.  And things get a very slight bit better.  The earth goes all the way around the Sun.  So at one point it is 100 million miles this way and six months later it is 100 million miles that way.  So the distance between the extremes is 200 million miles, a number that is twice as big.

If we want to use the parallax technique to figure out how far away something is then what we want to do is wait for the earth to be on one side of the Sun and then carefully measure the angle to the "something".  Then we wait 6 months and measure again.  We are now 200 million miles away from where we started so the angle should change a lot, right?  Well, this is where our intuition goes wrong because we are comparing these giant numbers.  The closest star to us that is not the Sun is Proxima Centauri.  Most people think it's Alpha Centauri because that's what a lot of people say.  Alpha Centauri and Proxima Centauri are very close together but Alpha Centauri is a lot brighter so people usually go with it.  But Proxima Centauri is actually a little closer.

Anyhow with this giant baseline of 200 million miles it should be a piece of cake to do the parallax thing to find out how far away it is.  And the parallax trick actually works for Proxima Centauri (and Alpha Centauri too) but just barely.  The reason is because the star nearest our own is actually a very long way away.  Let's see how long "very long" is.  To do this I am going to figure distances in "light minutes".  A light minute is the distance traveled by a photon of light in a minute.  Trust me, it's a very big number.  Now the light from the Sun takes a little over 8 minutes to get here from there.  So a hundred million miles is about 8 light minutes.  And 200 million miles is about 16 light minutes.

Now Proxima Centauri is 4.25 light years away (the distance light goes in 4.25 years).  Again this is a really big number if we put it in terms of miles.  But let's put it in terms of light minutes.  It still turns out to be a pretty big number.  Proxima Centauri is about 2.2 million light minutes away.  So to do the parallax thing to figure out how far away Proxima Centauri is we create a triangle.  One side of the triangle is 16 light minutes long.  The other two sides are 2.2 million light minutes long.  In geometry there is a concept called "similar triangles".  By using similar triangles we can throw all the "million" parts away.  So imagine a triangle with one side that is 16 inches long and the other two sides are 2.2 million inches long.  It turns out that the 2.2 million inch sides are each over 35 miles long.  Now to get the parallax thing to work we need to measure the tiny angle between the two 35 mile long sides.  Remember on one end they meet and on the other end they are 16 inches apart.  It is a brilliant piece of work that Astronomers have actually been able to measure that super tiny angle.

Now let's try to do the parallax technique on a star that is twice as far.  That means that we need to measure the angle between the two sides that are now 70 miles long.  Remember that they meet at one end and are separated by the same 16 inches on the other end.  Astronomers have only been able to use the parallax technique to measure the distance to only a few of the nearest stars.  I think you now understand why.

So if the parallax technique only works for a few very close stars what do we do about the rest?  The answer finally gets us back to the "standard candle" technique that I mentioned a long time ago.  Imagine having a 100 watt light bulb.  Now measure how bright it is from 100 yards away.  There is a standard mathematical formula that tells us exactly how bright it will be when viewed from 200 yards away or a thousand yards away.  So if we know we are looking at our 100 watt light bulb (so we know exactly how bright it is) and we can very accurately measure how bright it appears to be (called its "apparent brightness") then we can calculate how far away it is.  That's the idea behind "standard candle".  If we know how bright something is from close up, its "intrinsic" brightness, and we can measure its apparent brightness, and we know that everything is clear between it and us, then we can calculate how far away it is.

Now most of space is pretty empty.  So it conforms to the "everything is clear" requirement to use the technique.  Sometimes this is not true.  There are dust clouds and other things that get in the way.  And these present real problems for some measurements scientists would like to make.  But in a lot of cases there appears to be no obstruction and in other cases scientists come up with techniques to allow them to adjust for the amount of obscuring going on.  So a lot of the time this "everything is clear" requirement is met.  That leaves the problem of knowing the intrinsic brightness of what you are looking at.

A solution to this problems is discussed by Asimov.  It involves the use of Cepheid variables.  These are a kind of variable star. The brightness of the star varies in a predictable way.  What makes this important is that Astronomers came to understand enough about how Cepheid variables worked that they could predict the intrinsic brightness of the star based on the specifics of its pattern of variability.  Originally work determined that specific types of Cepheids all had the same intrinsic brightness.  This allowed the development of a relative distance scale.  This item is twice as far away as that item, that sort of thing.  Soon a large number of relative distances were known.  But to turn the relative distance scale into an absolute distance scale it was only necessary to determine the actual distance to one Cepheid.  That was only achieved recently when high precision measurements using the Hubble Space Telescope and other techniques became available.

At the time Asimov wrote the book only relative distances were known for sure.  Astronomers used a number of techniques to estimate the intrinsic brightness of Cepheids with more or less success.  At the time the book was written there was still a lively discussion as to what the correct value for the intrinsic brightness was.  This resulted in a number of respected Astronomers using a number of different estimates of intrinsic brightness.  As time went by Scientists also determined that there were several classes of Cepheids and members of each class displayed a different intrinsic brightness than apparently similar members of a different class.  General agreement as to how to place a specific Cepheid into the right class and the correct intrinsic brightness for each class are now pretty much sorted out.  But bringing everything into alignment was not completed until many years after Asimov's book was written.  Astronomers were very aware that there were problems with Cepheids at the time the book was written.  But there was no better way to determining distances at the time.  And Astronomers of the time were careful to acknowledge these kinds of issues.

Also at the time Asimov wrote the book Cepheid variables were the brightest standard candle available.  But for really large distances they are too dim to work.  Since then Astronomers have developed another standard candle called a "Type 1A Supernova".  As a supernova it is way brighter than a standard star like a Cepheid so it works for much greater distances.  All of the details of how the intrinsic brightness of a type 1a supernova has been worked out are different.  But the general idea is the same.  Certain attributes that can be measured from far away allow the intrinsic brightness to be determined.  There have been problems to work through with the type 1A supernova as a standard candle.  But Astronomers think they have things worked out pretty well at present.  Now back to the main line of the story.

In 1929 Edwin Hubble, who had been studying Galaxies published Hubble's Law.  Using Cepheids as standard candles Hubble had found that if you ignored a few close in Galaxies it appeared that the farther away a Galaxy was the faster it was moving away from the earth.  He posited that there was a single "Hubble Constant" that was the ratio between the recession speed and the distance from the earth.  Do to the problems with the Cepheid standard candle he couldn't establish the specific value for the Hubble Constant but he established that it appeared to be a constant across the range of relative distances he could measure.

This turned out to be a very remarkable observation.  Using Hubble's Law one could project to the point where galaxies would be receding from each other at the speed of light.  This in turn meant that the universe had a specific age.  This idea was shocking.  Before, scientists had not spent much time thinking about the age of the universe.  They knew it was vastly older than the 6,000 or 10,000 years that biblical scholars had calculated.  Other than that, most thought, when they thought about it at all that the universe was either a vast unspecified age or that it had always been there in something similar to its current state.  Hubble's ideas ushered in the modern era of cosmology.

As these ideas spread among first the Astronomical community and then to the broader scientific community speculation soon settled down into two main competing theories.  The "steady state" theory was championed by among others Einstein and a British Astronomer named Fred Hoyle.  It stated that the universe had always looked pretty much as it does now.  The competing theory, named by Hoyle with the most ridiculous name he could think of, was called the "big bang" theory.  By the time Asimov's book was written the evidence against "steady state" had become overwhelming.  So "big bang" was winning by default.

It didn't take scientists long to note some convenient features of Quantum Mechanics in their efforts to flesh out the big bang theory.  The most relevant item was something known as the Heisenberg Uncertainty Principle.  Most simply (and I don't want to get into yet another diversion so this is all you get) the Principle said that there was a fundamental uncertainty about the universe and that the smaller the thing you were studying the more uncertain its characteristics were.  Astronomers latched on to this and posited an extremely small piece of space.  It was so small that the energy content was vastly uncertain.  This was taken as the seed out of which the whole universe would explode.  As the universe exploded it would cool (that's what gasses naturally do as they expand) and eventually the temperature would drop to what we see today and the size of the universe would grow to the size we see today.  That was roughly the state of the big bang theory at the time Asimov wrote his book.

You are probably thinking that seems inherently implausible.  Scientists slowly came to the same conclusion.  And the big bang theory has evolved considerably from the humble roots I outlined above.  The biggest change is to add something called "inflation".  The subject is complex and I again want to avoid digression.  But the basic idea is that from its early tiny seed (which may have looked greatly different than our tiny exploding point) the universe inflated to a fantastic size in a fantastically short period of time.  This may sound even weirder than the very weird original big bang theory I outlined above.  But it turns out that there is actually some evidence for inflation.  Yet again in an attempt to avoid another large diversion I will note that the most compelling of this evidence consists of the measured variations in something called the Cosmic Microwave Background and leave it at that.

Asimov does a nice job of going into Hubble's work and that of subsequent scientists up to the time he wrote the book.  Given all the uncertainties scientists supported age estimates for the universe ranging from about 11 billion years to 42 billion years.  Since then the uncertainties have been greatly reduced and the consensus number today is 13.7 billion years.

Since then another startling development has transpired. It looks like the Hubble Constant is not constant.  There is evidence that the rate of expansion of the universe has changed over time.  There have also been related developments in scientist's views on the constitution of the universe.  At the time Asimov wrote the book what Astronomers could see were bright things like stars.  Generally this is referred to as Baryonic matter.  A couple of decades ago Astronomers noticed a problem.  They could roughly weigh a galaxy by doing some calculations based on the light the galaxy generated.  They could then use Newton's theory of gravitation to predict how fast portions of the galaxy should rotate.  Everything came out wrong.  Eventually Astronomers decided that there was a large amount of what they dubbed "dark matter" surrounding galaxies.  They still have no idea what dark matter is but there seems to be a lot of it.  The recently measurements that have led to the idea that the Hubble Constant is not constant has let scientists to posit something called "dark energy".  They know even less about dark energy than they do about dark matter.  But their current thinking is that the universe consists of about 4% Baryonic matter, 26% dark matter, and 70% dark energy.  So scientists in 1960 knew about only 4% of the mass that current day scientists think the universe actually contains.

And this leads me to my final subject for this post.  Scientists in 1960 envisioned three basic fates for the universe.  The first option was that the universe would explode (big bang), expand for a while, then collapse back on itself.  This was dubbed the "cyclic universe" theory.  At the other extreme the universe would explode then keep on growing.  It would get bigger and bigger.  Everything would spread farther and farther apart until each component of the universe was an island so far away from any other component as to be completely isolated.  The third option was the happy medium one.  The universe would explode and expansion would gradually slow down due to gravity but everything would be on a balance point,  It wouldn't expand forever but it wouldn't collapse back either.  Which would be the fate of the universe?  Well it all depended on the density of the universe.  If it was too dense it would expand then collapse.  If it was not dense enough then it would expand forever.  And if the density was just right if would end up on the balance point.

In 1960 these calculations had been done and it appeared that the universe had exactly the right density to end up at the balance point.  But scientists were completely at a loss as to why the density of the universe was exactly right.  Even a little too much or a little too little would tip the universe one way or the other.  Since then we have this whole dark matter / dark energy thing going.  Factoring everything in, Baryonic matter, dark matter, dark energy, the universe seems to have exactly the correct density.  But current measurements indicate that the density is so ridiculously exactly the correct amount that they are even more puzzled by the whole thing than they were in 1960,

And that gets us to the end of the chapter.  

50 Years of Science - part 2

This is the second in a series of posts.  The first one can be found at http://sigma5.blogspot.com/2012/07/50-years-of-science-part-1.html.  Taking the Isaac Asimov book "The Intelligent Man's Guide to the Physical Sciences" as my baseline for the state of science as it was when he wrote the book (1959 - 1960) I am examining what has changed since.  For this post I am starting with the chapter Asimov titled "The Birth of the Universe".

In this chapter Asimov reviews in more detail than in previous chapters what science has determined about the age of things.  He reviews various "origin stories" for the Earth, including the one in the Bible.  He then moves quickly on to scientific attempts to determine how old Earth is.  Based on the salt content of the ocean the earth is at least a billion years old.  Based on various radioactive decay-based measurements the Earth is at least 3.3 billion years old.  Both of these estimates contradict "young earth" creationists.  Asimov doesn't mention them anywhere in the book because at the time the book was written no one took them seriously.  They did not have a political home in the Republican party and a well established network of religion channels on cable and mega churches to support and maintain their belief system.  In the decades since this book was written science has developed and enhanced the lines of reasoning Asimov lists, along with dozens of others, all indicating that the Earth is billions of years old.

No one has come up with any credible evidence that even one of these multiple lines of reasoning is wrong.  But we live in a world where people's knowledge of science has diminished to the point where most people are unfamiliar with the reasoning or the evidence that supports the reasoning.  Instead they are drowned in a sea of "facts" that are factually wrong, and people whose idea of a scientifically valid argument is " I believe it because my faith demands I believe it" or "I believe it because I wish it were so and 'wishing it were so' is enough to make something true".

In any case, the basic methods Asimov discusses have been refined and extended so that we now know that the Earth is 4.7 billion years old.  The primary line of evidence for this is radioactive decay.  Why is the modern number different from the number in 1960?  The big reason is that a concerted effort has been made to date lots and lots of rock formations.  When rocks melt then many radioactivity "clocks" reset resulting in a misleadingly young estimate of how old the rocks are.  Scientists have now located rock formations that are substantially older than the oldest ones they were familiar with in the '60s.  The scientific methods of radioactive dating have also gotten better.  More isotopes can now be used as the basis for these radioactive studies.  The amount of material necessary to make an accurate measurement is now much smaller.  And the overall accuracy and sensitivity of the measurements have improved.  Scientists are now also able to measure different "isotope systems" in the same rock and compare the results.  This makes it easier to identify situations where a sample appears to be pristine but has actually been processed (e.g. heated up by a geologic process).

Now is probably a good time to spend some time explaining how radioactive clocks work.  The thing that makes an atomic element what it is is the number of protons in the nucleus.  Hydrogen is Hydrogen because its nucleus has one Proton.  Helium is Helium because it has two Protons in its nucleus.  But there are actually multiple kinds of Hydrogen, Helium, and other elements.  Each kind is called an isotope,  The three isotopes of Hydrogen are called "Hydrogen", "Deuterium", and "Tritium".  Regular Hydrogen has a nucleus consisting of one Proton.  That's it.  Deuterium has a Proton but also a Neutron in its nucleus.  The name references the two (deu) nucleons.  Tritium has a Proton and two Neutrons, hence the "tri" in the name.  The isotopes of other elements don't have such cute names.  Chemists and Physicists also have various superscripts and subscripts they use to indicate isotopes but it is essentially impossible to get these to print correctly in the blog.  So I am instead going to use H-1 to indicate Hydrogen with just the one nucleon in its nucleus, H-2 to indicate Deuterium, the isotope of Hydrogen with two nucleons, and H-3 to indicate the three nucleons in Tritium.

Now from a chemical point of view H-1, H-2, and H-3 are indistinguishable.  They all behave like Hydrogen in every way when it comes to chemical reactions.  The same is true for the isotopes of Helium:  He-2, He-3, and He-4.  In each case there are two Protons in the nucleus along with 0, 1, or 2 Neutrons.  As a result each isotope acts just like the others from a chemical reaction point of view.  But in other ways each isotope differs.  For one thing the weight of each differs.  An atom of H-2 weighs about twice as much as an atom of H-1.  Both have one Proton and, in normal circumstances one electron.  But the electron weighs about one 2000th as much as a Proton, whereas a Neutron weighs roughly the same as a Proton.  So H-1 has one Proton and 1 electron and weighs about as much as a Proton.  But H-2 has a Proton, an Electron, and a Neutron.  So it weighs about the same as two Protons.  When you get to heavy atoms like U-235 versus U-238 the difference is much smaller.  U-235 weighs roughly as much as 235 Protons and U-238 weighs roughly as much as 238 Protons.  But here the difference in weight is roughly 1%.  In some cases the weight difference can be important but in most cases the difference in not enough to make a big difference.  And in any case that is not what we are interested in.

The difference that matters to us is that the stability of various isotopes varies considerably.  H-1 is stable.  If you sit around and watch a H-1 atom for a very long time it won't do anything.  H-2 is also stable.  But if you watch H-3 for about 12 years there is a 50-50 chance that it will "decay" into something else.  It will stop being H-3 and become a different isotope of a different element.  It will spontaneously become He-3.  One of the Neutrons will turn into a Proton.  If you have a bunch of H-3 atoms and wait a little over 12 years 50% of it will spontaneously decay into He-3.

There are 92 naturally occurring elements.  They range from e.g. H-1 to e.g. U-238.  Hydrogen comes in three isotopes as does Helium.  Other elements like Uranium come in a dozen or so isotopes.  All together there are hundreds of isotopes.  Many like H-1 are stable.  They never decay into something else.  But most isotopes are like H-3 and U-235 and U-238.  They decay spontaneously into other isotopes.  This is a complicated process.  U-235, for instance, can decay into one of several isotopes.  And sometimes the isotope it decays into is radioactive (e.g. unstable) so it decays into something else.  But scientists have carefully studied many isotopes and for the radioactive ones they have studied what they decay into.  H-3 always decays into He-3.  And for a combination like H-3 to He-3 there is a single magic number called the "half life".  In the case of the H-3 to He-3 decay the half life is exactly 12.32 years.  This means that if you put 10 lbs of H-3 into a container and wait exactly 12.32 years, when you look into the container you will find 5 lbs of H-3 and 5 lbs of He-3.

U-235 is more complicated.  It can decay into a number of different isotopes.  But most of the time it decays into Th-231.  The half life of this decay is 700 million years.  U-238 has three different decay paths.  The most common one is to Th-234 and its half life is 4.5 billion years.  What's important is for each decay path (e.g. H-3 to He-3 or U-235 to Th-231) you have three things:  the starting isotope, the ending isotope, and a very specific half life.  As we have seen half lives can be relatively short (e.g. 12.32 years) or very long (e.g. 4.5 billion years).  They can even be much shorter.  The half life of some isotopes is less than a second.  And they can be even longer than 4.5 billion years.  But, since 4.5 billion years is about as long as the Earth has been around, decay paths that have a half life longer than 4.5 billion years are not very useful as radioactive clocks.

And this whole half life thing is a little more complicated than it looks.  If we look in on our container of H-3 after 12.32 years we have half as much H-3, namely 5 lbs.  But what if we seal it back up and wait another 12.32 years?  Is it all gone?  No!  "Half life" means the amount of time it takes for half the remaining material to decay.  So after a total of 24.64 years we will have 2 1/2 lbs of H-3 (half the 5 lbs we had at the 12.32 year mark).  Radioactive decay is what mathematicians call an exponential process.  After one half life we have half the material.  After two half lives we have a quarter of the material.  After three half lives we have an eighth of the material.  And so it goes to a sixteenth (4 half lives) a thirty-second (5 half lives) a sixty-fourth (6 half lives).  If a large number of half lives are involved there is a shortcut that can be used.  After ten half lives we will have about a thousandth of the material left.  After twenty half lives we will have about a millionth, etc.  Every additional ten half lives will reduce the amount of original material by a factor of a about a thousand.

This whole "isotopes and half lives" thing gives us a clock for measuring times.  If we know how much of a specific isotope we started with and we know how much we have now then we can measure time.  For periods of hundreds to tens of thousands of years C-14 (carbon fourteen) works really well.  Lots of things like wood have carbon in them.  Most Carbon is stable C-12.  There is also some C-13, which we will ignore.  But there is usually a small amount of C-14 mixed in with the other isotopes of Carbon.  The half life of C-14 is 5,730 years.  If by careful analysis we find that exactly half the C-14 we started with is gone we can conclude that the artifact containing the Carbon is 5,730 years old.  If a quarter remains then the artifact is a little over 11,000 years old.  If a little less than a thousandth of the C-14 is left then the artifact must be about 57,000 years old.  In theory the process is that simple.  In actual practise it is more complicated than that.

The most obvious problem is with an artifact that we suspect is a little over a hundred thousand years old.  In this case we expect to measure about a millionth of the C-14 we started with.  That's not very much.  So C-14 dating is not very good for artifacts that are more than about 50,000 years old as the remaining amount of C-14 is so small.  But there is an even bigger problem for an artifact that we suspect is say 20,000 years old, what should be in the butter zone where we should have enough C-14 left over to get an accurate enough measurement to produce a pretty sharp age estimate.  Now the issue hangs on the question of how much C-14 we started with.  And that turns out to be a much harder question than it would seem.

Originally scientists just assumed that everything started out with pretty much the same percentage of C-14.  So they would measure the total carbon, apply the magic percentage to estimate how much C-14 there originally was, and go from there.  But it turns out the magic percentage trick doesn't work very well.  C-14 comes from high altitude cosmic rays hitting the upper atmosphere.  If the rate of cosmic rays stays constant then after a while the carbon in the atmosphere will contain a specific percentage of C-14.  This C-14 will end up in carbon dioxide in the air.  And plants will absorb the carbon dioxide and end up with a specific percentage of C-14 in their tissues.  If the plant lives for a very short time compared to the 5,730 year half life of C-14 then we will end up with plant material with a predictable initial C-14 percentage and we are good to go.  But this process is complicated and it turns out that there are variations in the efficiencies of some of the steps.  So the percentage of carbon in plant material that is C-14 varies somewhat.  And this introduces errors.  We can still measure what is now called the C-14-age of material containing carbon.  Scientists have developed elaborate adjustment procedures that work pretty well most of the time for turning C-14 age into real age.  But they are complicated and don't work all the time.

So some times there are problems with C-14 based radioactive dating.  Scientists have reacted to this in two ways.  First, they have developed and continued to refine their C-14 adjustment procedures.  The second way is to come up with other isotope systems.  That way they can compare the results for the C-14 isotope system with the results of the other isotope system.  Other isotope systems also allow artifacts to be dated that are much older than 50,000 years.  For instance, if you can find some Uranium in a rock and you can estimate how much of that Uranium was originally U-238, you can use radioactive dating on a very old rock.  If you measure the remaining U-238 and it turns out to be half of the amount you calculated was originally there you can estimate that the rock was 4.5 billion years old.  Other isotope systems can be used in situations where your age estimate is different.  If you can use an isotope system that has the right half life you can get an accurate and reliable date for a range of from hundreds of years to billions of years and anything in between.

This digression has turned out to be much longer than I originally planned.  So let me stick with it just a little longer and explain how scientists figured out that the C-14 isotope system had problems.  They didn't match it against a different isotope system.  Instead they matched it against a completely different dating system called dendrochronology.  This is just a fancy name for counting tree rings.  People have known for a long time that if you cut tree down you will see rings.  And each ring represents a year in the life of the tree.  The rings represent wood of different colors.  And the explanation is simple.  In the Spring when the weather is nice the tree grows quickly and typically creates light material.  In the winter the tree grows more slowly and typically creates darker material.   This idea of annual tree rings has been around a long time and was certainly not invented by scientists.  But scientists took this basic idea and built on it.

Scientists observed that a wide ring represented a year with good growth weather and a narrow ring represented a year with poor growth weather.  Originally this idea was used to determine weather patterns for times and places where there weren't good weather records.  But scientists found a way to do even further.  All the trees in a specific stand experience the same weather so they will have the same pattern of narrow rings for poor growth years and wide rings for good growth years.  This allows the pattern of rings to be synchronized between different trees.  Specifically, if you can find the stump of an old tree in a stand with younger trees you can match rings from late in the life of the stump with rings early in the life of the younger trees.  This allows you to establish the time period when the old tree was alive.  You now have access to weather information going farther back than the age of the oldest tree still alive.

This idea can be extended to trees in different stands as long as the stands are subject to similar weather.  And this method can be used to develop a weather record that spans not just two trees but several trees.  So a record can be developed that spans hundreds, in some cases thousands of years.  And the method does not require a whole tree.  A beam from a house or any piece of wood big enough to contain a number of rings can be used.  So a beam from a building or a piece of furniture can be dated.  You know the object containing the piece of wood was constructed some time after the tree that originally contained the piece of wood died (e.g. was cut down).  This allows you to date the piece of wood as being after the newest date represented by the newest ring in the piece of wood.  This can be very useful.

Specifically, wood contains carbon.  You can take a small sample from piece of wood and C-14 date it.  You can then compare this C-14 date to the tree ring date for the larger piece of wood.  You may even know the exact year the rings were laid down that ended up in the small piece that was C-14 dated.  Scientists did that.  They had complete confidence in the tree ring dates.  They found, however, that the C-14 date did not match.  That caused them to go back and look harder at the C-14 system and decide it had problems.  They now know what these problems are.  But there is not always a method of correcting the C-14 date that works.

Scientists do this kind of thing all the time.  They test one method against another method to see if they agree.  It's nice when the do but they don't always.  When there is disagreement they go back and look at both methods to see if they can figure out what went wrong.  Most of the time when it turns out that something is wrong it is scientists and not the critics that figure out that there is a problem.  When it comes to legitimate criticism, criticism that turns out to be justified when all the facts are in, Scientists are much harder on Science than critics are. 

50 years of Science - part 1

A few months ago I was rooting through my book shelf and I came across "The Intelligent Man's Guide to the Physical Sciences" by Isaac Asimov.  Asimov initially became famous during the "Golden Age" of Science Fiction and is considered one of the Grand Masters of the genre.  He is famous for inventing the "The Three Laws of Robotics".  This ushered in the era of good robots to supplement the previous trope of evil monster robots.  He also wrote the "Foundation" trilogy (and eventually added additional books to the series).  This was an early entry in the sub-genre of Future History and posited that crowd psychology would eventually become a hard science, thus allowing broad historical trends to be forecast and possibly manipulated.

After many years of success as a Science Fiction author he branched out into several other areas.  One of these was writing about science for a general audience.  The "Guide" was written in 1959 and 1960, making it roughly 50 years old.  Asimov did a good job of summarizing the state of the art at that time.  I thought it would be interesting to do a series of posts comparing the state of Science then and now.  But first let me set the scene by looking at the more general situation in 1960 as compared to the present.  Let me start with a long list of "no"'s.

There were no integrated circuits.  The transistor had been invented about 10 years earlier but was not in wide use.  Something called a "transistor radio" would be introduced at about this time.  Previously radios, and electronics in general, were powered by vacuum tubes (essentially a small light bulb with a bunch of extra wires and other stuff jammed inside the glass shell).  A simple device like a radio would have less than 10 tubes.  A very complex device would usually have less than 100.  Modern electronics, by contrast, have the equivalent of millions of tubes combined into a single small chip costing a few dollars.  Computers existed at this time but they had the processing power of a digital watch and cost millions of dollars each. 

There was no Internet.  The very beginnings of what would be eventually become the Internet (called ARPANET at the time) was begun in the late '60s.  Since there was no Internet there was no E-Mail (invented in the '70s) or web pages (invented in the '90s) or Twitter or Facebook (both invented in the '00s).  In fact, you couldn't call someone on your phone by pushing buttons.  Telephones of the time had "rotary" dials with 10 holes (one for each digit).  There were no Area Codes or International codes.  To make a long distance call you had to contact an operator, an actual person, who would make arrangements.  Long distance calls within the U.S. were possible but expensive (a dollar or more per minute).  International calls were just barely possible.  The sound quality was terrible.  It could easily take 20 or 30 minutes to set one up and they were fantastically expensive.  At the time almost no one had actually participated in one due to the cost and difficulty.  You could look up a local phone number in a "phone book" (still around). To get a number for someone not in your town you had to contact an operator (again, a person) in that area who could look it up for you.

And there were no cell phones.  You rented phones from the phone company.  You couldn't even buy one.  There were only a few models to choose from and they were all hard wired to the phone system (e.g. no "walking around" portable phones - you had to go where the phone was and it was by the wall where the phone man had wired it in).

There were TVs but there were no color TVs.  Almost no one had cable so all you typically had were the few channels that broadcast over the air in your area.  There were a few satellites but there were no communications satellites so there were no extra channels like ESPN or C-SPAN or USA or HBO.  You were stuck with what came from your local TV stations.  There were no TiVos or DVRs so you had to watch the show when it was broadcast.  And a particular episode was only broadcast once (except for some reruns in the summer) so if you missed it there was no going back.  At this time there were also no VCRs so you couldn't "Tape" anything.  Nor could you rent movies (or download them).  If you wanted to see a movie, you had to go to a theater while it was in town.

Cars all ran on leaded gas with no ethanol in it.  They had no seat belts or air bags.  Air conditioning was available on a few luxury models.  Cars were cheaper but tended to wear out quicker.  A car was old at 50,000 miles and a junker at 100,000.  But car repairs were much simpler and a lot of people did their own.  SUVs hadn't been invented yet and pickup trucks were only driven by people who needed them for work.  The only in-car electronics were radios and they only got AM.  No FM.  No CD player or entertainment package.  There were no navigation systems. You got free paper maps at gas stations.  There was no "self serve".  Someone (a "gas jockey") pumped your gas for you and checked the tire pressure, and oil and radiator levels.  Cars didn't have fuel injection.  They also didn't have any anti-pollution or other complicated stuff like a diagnostic computer.  A mechanic had to figure out what was wrong on his own.

Finally, books were a lot cheaper.  At $0.90 (1969 price), "Guide" was a little  more expensive than a typical book of the time.  Of course, most things were cheaper then.  But you also got paid a lot less too.  And, if you had a paying job, you were almost always a man.  Few women worked outside the house.  With that as an introduction, let's jump into the book.

The biggest telescope of the time was the 200" Hale telescope located on Mt. Palomar in California.  It used large photographic plates, about 1' by 1'.  They were covered with an emulsion that was sensitive for the time and designed for maximum sharpness.  After processing the plates were examined by eye or perhaps with a small magnifying glass.  CCDs had not been invented so there was no electronic alternative to photographic methods.  And photographic methods were better than staring through the telescope with the naked eye.  Another problem was that the atmosphere introduced small distortions. This was one reason no larger telescope had been built.  The Hale was about as big as it made sense to go.  Modern telescopes use adaptive optics (and other tricks) to deal with this (and other issues) but the technology to make a bigger telescope work better than the Hale did not exist then.  There were other limitations imposed by the atmosphere.  It is opaque to Infrared, Ultraviolet, X-rays, and Gamma Rays.  And there were no telescopes in space (e.g. Hubble) and no big telescopes in the Southern hemisphere.  So Astronomers knew little about how the Universe looked in the Southern hemisphere and nothing about how the universe looked at these other wavelengths.  There were a few radio telescopes like Jodrell Bank in the U.K. but big dish radio telescopes like in the movie Contact or at Arecibo in Puerto Rico had not been built yet.

So with these limitations, how did Astronomers of the time do?  They got the size of the Solar System right.  They got the size of the Milky Way and our rough location in it right.  But they did not know that there was a giant Black Hole in the center of the Milky Way.  Black Holes at this time were an entirely theoretical concept.  No one had any evidence that they actually existed.  Astronomers were also able to estimate the size and distance of the Andromeda Galaxy with reasonable accuracy.  That, and other observations, led them to believe that the universe was at least 5 billion years old.

The current estimate for the age of the universe is 13.7 billion years.  So Astronomers of the time got that wrong.  But they knew they did not have good data.  Instruments of the time only allowed Astronomers to see out about 2 billion years.  (The constant speed of light makes distance and time equivalent.  If you are looking at something that is one million light years away, you are seeing it as it was one million years ago).  So they stated their estimate as "at least 5 billion years".  The 2 billion year observational limit was why they had such a poor estimate of the size and age of the universe.

Asimov does a great job of explaining how Astronomers knew what they knew.  Much of it was hard to figure out given the tools they had to work with at the time.  This is generally true.  It is easy for us to figure out a lot of things now because we now have tools that are so much better.  Given the "it's easy to do now" phenomenon it's easy to fall into the trap of unconsciously thinking we are so much smarter now than they were then.  But in many ways the opposite is true.  They were so much smarter then than we are now because they had to be so clever and creative to figure things out with such poor tools.  So I recommend picking the book up, if you can find it.  It is a useful exercise in humility to see what they had to go through to figure out what they were able to.

 I am going to end things here.  I will pick things up starting with the next chapter in the next installment.

Risk

We are now 3-4 years into our current economic problems.  I have read several books on the subject.  I think I have come up with a simple explanation (e.g. one much less than book length) of how we got into this mess.  I think the whole thing can be explained by focusing on a single word:  risk.  Now the more enlightened among you know Risk as a board game.  But I am talking about "risk" with a lower case "r".  At its most basic, risk is about whether or not things are going to go badly wrong.

Businesses don't like things to go badly wrong.  So they have long sought out "risk mitigation" strategies.  Let's say a business is worried about a large investment going bad.  And let's say further that the business thinks that the chances of this happening are 1 in 100.  Say this business can pay a premium of 2% of the size of the investment for someone else to cover the loss if the investment goes bad.  On paper this might seem like a bad investment as in a probabilistic sense the cost of avoiding the problem is twice the "expected cost" of the problem.  But the business might decide it's willing to pay the 2% anyhow just to avoid having to worry about the investment going bad.  If this sounds like insurance that's because it is.

There are many complexities in actual specific situations but this "what will it cost and what is the probability it will happen" captures the core concept of risk.  And businesses engage in risky behavior as a normal part of doing business.  Any business deal, investment, loan, etc. can go bad.  So one of the critical skills of a good businessman is to be able to properly manage risk.  If a business engages in a lot of low cost high risk transactions then the business most likely self insures.  They build expected losses into the cost of doing business and eat the loss directly.  This saves them the additional cost of farming the risk out.  If they have judged the risk correctly then the mark up on the successful transactions will more than pay for the losses on the ones that go bad.  Businesses often put a lot of thought into situations where the risk is very low but the cost of failure is very high.  These are the situations where they are likely to buy some form of insurance.  But the key to doing this sort of thing correctly is to be able to accurately judge risk.  If something is low risk but you think it is high risk then you waste money on insurance.  If something is high risk but you think it is low risk then you may choose to under-insure or get no insurance at all.  These situations can lead to very bad things as we all now know only too well.  So let's take a look at events of the past few years from the perspective of risk.

For most of my lifetime a mortgage was a low risk investment for a company.  But this was not always so.  The house my father grew up in was built in 1910.  For various reasons my father took control of the family's finances about 25 years later.  The house originally cost $5,000.  He was shocked to learn that after payments had been made for 25 years $5,000 was still owed on the mortgage.  And this being the middle of the great depression, the house was still worth only about $5,000.  His parents had taken out an "interest only" mortgage.  He quickly moved to start paying down the principal and had the house fully paid off a few years later.  If my father's family had been a little less lucky they would have ended up with nothing.  Many depression era families did.  As a result of the experience of those other families FDR was able to introduce mortgage regulation which banned interest only mortgages.  For many years a 30 year 20% down mortgage was the standard.

And a 30 year 20% down mortgage is a very safe investment for the investor.  If something goes wrong the investor can repossess the house and sell it.  They will likely get enough to cover the mortgage balance and any additional costs.  And there was a good chance the home owner who got in trouble would sell the house and use the proceeds to pay off the mortgage in full before the mortgagor even knows there's a problem.  In only a few rare cases did the investor take a loss on the mortgage.  So the risk of any loss was low.  And even in the case of a loss the investor was likely to get most of his money back.  An investor was fully justified in assigning a very low "risk premium" (the amount he needed to mark the transaction up by) to a mortgage transaction.  This situation continued for 40 years or so.

Then people looked at how to make more money in the mortgage business.  The first thing they asked themselves was "what if we reduced the down payment minimum below 20%?"  The answer turned out to be "not much".  Few mortgages went into foreclosure.  When they did the potential loss was usually well below 20%.  So by changing the down payment requirement from 20% to say 15% or 10% nothing much really happened on the risk side.  The number of mortgages that went bad stayed low and the losses, when there was a loss, stayed small.  But all of a sudden a lot more people could afford a mortgage.  So costs went up but not by much and volume went up a lot.  And down payments went down some more to 5% or 3%.  And even more people flooded into the market.



Now what if I write a mortgage to a bad person?  This is someone who is unable or unwilling to pay the mortgage.  What's my risk?  Well, even if the bad person is in the house for only a year the investor will get all his money back if housing prices rise enough.  So why bother with credit checks if you will not lose money if the credit is bad?  In this environment an accurate credit check is not worth much.  If the mortgagee is a good person it's a waste of money.  If the mortgagee is a bad person you still get your money back eventually.  So a entire segment of the mortgage business appeared that specialized in writing mortgages for bad (only in the sense of credit risk) people.  And investors made a lot of money.

It's not just that they failed to lose money.  They actually made money, more money.  This is because of a perverse situation.  If a person has a bad credit rating then you are justified in charging them a higher interest rate (see "risk premium" above).  Investors love getting a higher rate of return (e.g. the higher interest rate) for the same risk (e.g. chance of a loss).  So investors encouraged mortgagors to find problem mortgagees because they could be talked into a higher interest rate.  In fact, it got so bad that many people who could qualify for a low risk low interest rate mortgage were sold a high risk high interest rate mortgage because that was he kind of mortgage investors wanted to buy (and were willing to pay a higher commission on).  Then there was the balloon.

Mortgages began to be structured with a low "teaser" interest rate for the first few years (typically three years).  Then the interest rate would "balloon" up to a much higher rate.  From a mortgagee's point of view this was not as bad as it seemed.  They could refinance, presumably into a new mortgage with another 3 year teaser rate.  If they did this often enough they'd never get to the part where the interest rate ballooned.  If that didn't work they could just sell the house and pay off the mortgage.  So it looked like the mortgagee always had an out, if the mortgagee was smart enough to figure all this out.  If not then he was in for a big surprise but that was his problem.  And on the other side of the deal, the investor side, there was a con job going on.  Mortgages were rated on their return.  Since a typical balloon mortgage would have a low interest rate for 3 years but a high interest rate for 27 years the average interest rate was pretty close to the high interest rate so the mortgage looked like a pretty good investment.  Now this ignores the whole "refinance or sell" thing but investors went with the "ignorance is bliss" strategy in large numbers and pretended the "refinance or sell" option did not exist.

The mortgage industry evolved quickly.  Down payments went down quickly.  Teaser rates and other gimmicks appeared quickly.  So there wasn't a lot of history that accurately represented the current market.  Wall Street took advantage of this.  They would trot out lots of statistics about how slowly mortgages turned over and that only a small percentage of mortgages defaulted.  But most of these statistics covered a different market, a market where down payments were higher and most mortgages did not contain gimmicks.  Averaging a bunch of good old mortgages with some bad new mortgages gives a distorted picture of what is likely to happen with the new mortgages.  But investors in general ignored all this and the marker for bad mortgages was hot, hot, hot.

Now let's step back a little.  For more than 40 years mortgages were a modest boring business.  Volume was relatively low because of the 20% down requirement and the fact that mortgagors usually did a thorough credit check and did not loan to problematic potential customers.  As a result mortgages were in fact very low risk.  Then the market started evolving.  Each evolution was in the direction of higher risk.  The early changes (down payment requirement reduced to 15% or 10%) increased risk but by only a small amount.  But they increased the volume by a lot.  Wall Street loved this.  More transactions meant more profit.  So Wall Street pushed for even more loosening of mortgage standards.  During this period risk increased more quickly than before but the increase in risk was still relatively small and by adding more risk premium (e.g. higher interest rates) loses could be managed and volume increased even more.

This led to a vicious cycle.  Wall Street pushed for more loosening of now already loose standards and volume increased.  And by adding gimmicks the apparent profit margin increased.  Default risk was still small because by this time home prices were increasing by leaps and bounds.  By the end of the process anyone could get a mortgage.  In one book the author met a Los Vegas stripper who owned four houses as investments.  In many cases gardeners and dishwashers were buying McMansions.  There was no way these people could keep up on their mortgage payments even though the vast majority wanted to.  But it didn't matter whether a mortgagee couldn't or wouldn't keep their payments up because any problem could be fixed by flipping the house.  So it appeared if you didn't look closely at what was going on that mortgages were still a low risk investment.  In fact the risk associated with the mortgage market had climbed to the point that it was very high.

This finally became apparently when housing prices stalled out.  They stopped rising quickly.  Then they stopped rising at all.  Finally, they started falling.  If someone has made a 20% down payment and the value of the house drops by 10% the investor will still come out OK.  But if the assessment on the house was inflated (as too many were) and the mortgagee paid 0% down and the value of the home drops then the investor is going to lose a serious amount of money.  And that's what happened.  And that fed into a substantial downturn in the economy.  So that people who could normally have afforded their mortgage payments lost their jobs and defaulted.  And this put a lot of distressed houses into the market, which further depressed housing prices.  And many of the people who were employed in the construction business or the appliance business or many other businesses that saw sales drop off dramatically were let go.  So more mortgagees got in trouble and housing prices got depressed some more.

At the peak most mortgage backed securities were rated AAA.  This means they are very low risk investments.  And right up to the end they behaved like they were very low risk.  Losses up to the peak were very small.  There is a branch of mathematics that describes these kinds of situations.  It is called "catastrophe theory".  Imagine a Popsicle stick.  It is placed on the edge of a table so that half of it is sticking out past the edge of the table.  Now imagine holding the stick down on the table and pushing gently on the other end.  The stick will bend slightly.  Push harder and it will bend some more. Let go and it will straighten out.  Now push much harder.  The stick will bend even more and, if you push hard enough, it will break.  Now stop pushing at all after the stick has broken.  What happens?  The stick will not return to being straight.  It will stay bent at the broken spot.  Catastrophe theory deals with these "bend till it breaks" situations.  Fortunately, we don't need to be catastrophe theory experts.  The broken Popsicle stick tells it all.  After the stick breaks a little change like not pushing on the stick any more does not bring the stick back to being straight.

The mortgage business ended up like the broken Popsicle stick.  Once it broke small fixes like lower interest rates did not put it back to where it had been.  And before it broke a number of people made conscious decisions to push the mortgage market harder and harder.  There justification was "well, it hasn't broken yet".  But they kept pushing harder and harder toward higher and higher risk behavior.  On the front lines were the mortgage sellers.  Once they sold a mortgage to a customer they wholesaled it out to Wall Street.  They made their money on volume and retained no risk once the mortgage was sold off.  And they were pushed by Wall Street to make more and more and riskier and riskier mortgages.  As long as nothing went wrong Wall Street made more and more money.  Much of the mortgage origination market is unregulated.  To the extent that it is regulated some regulators tried to push back.  But they were opposed by Wall Street and their powerful lobbying operation.  The regulated mortgage originators also opposed the regulators because they were losing business to unregulated originators.  They added their lobbying muscle to Wall Street's.

A partner in crime were the securities rating firms like Moody's and S&P.  They rated these investments AAA right up until the end.  But they were captured by Wall Street.  If one firm gave a security a bad rating then Wall Street would hire a different firm if the new firm would promise to give the same security a good rating.  Everyone knew how the game worked.  So the ratings agencies would build a paper trail to justify their rating, a paper trail based on the bogus (see above) historical data and other "analysis" Wall Street provided.  The idea was to have plausible deniability.  "We followed accepted industry practises.  We had no idea, honest!"  This, their own lobbying operation, and a "White Shoe" Wall Street law firm on retainer, was judged to be sufficient cover.  And so far their defensive strategy has worked. No one is in jail.  All the firms are still in business and no individuals have lost big law suits.

An argument can be made that mortgage originators are not all that smart.  I don't believe it, but let's just say.  And similarly an argument can be made that the regulators and the ratings agencies are not that smart either.  Again, I don't believe it but let's just say.  Wall Street prides itself on having lots and lots of "smartest guy in the room" types.  But if one of these Wall Street smart guys had applied their intelligence and pointed out the problems in the mortgage industry, what would have happened?  They would been chastised for not being a team player.  If their firm acted on their conclusions they would have stopped making the kind of money other firms were making.  Instead what the smart guys on Wall Street adopted (or in some cases tried to adopt) a different strategy, the "musical chairs" strategy.

In musical chairs there are a number of people walking around a circle of chairs with one less chair than people.  When the music stops everybody tries to sit down.  Whoever does not make it safely to a chair is the loser.  Many people on Wall Street knew there were problems.  But they also knew that were many players involved.  So it was like musical chairs with lots of people but not enough chairs.  As a Wall Street smart guy the strategy I (and pretty much everybody did this) adopted was to make sure I (or my company) always had a chair I could definitely make it to when the music stopped.  Since I'm the smartest guy in the room some other schmuck will get stuck without a chair.  The problem, of course, turned out to be that the whole roof caved in and there were no chairs left for anyone.  For many on Wall Street AIG was the designated schmuck.  Unfortunately no one thought it was their job to make sure AIG had enough money to fulfill its role.  They didn't.  A couple of beats after AIG went under the music was stopped by the roof falling in.

"No chairs" was not a possibility that anyone had considered.  And it turned out that the government and its middle class taxpayers ended up having to come in and bail Wall Street out.  So for the most part it didn't matter to Wall Street that it had screwed up.  A couple of firms went under and a lot of employees got laid off but the system as a whole survived just fine.

So Wall Street didn't know (or pretended it didn't know) that mortgage risk was not being calculated correctly.  Some regulators got it right or at least came closer than any other group but an aggressive lobbying campaign in public and behind the scenes caused them to be ignored when they weren't silenced outright.  The ratings firms got it completely wrong.  And the mortgage origination industry got it completely wrong.  But several of the largest mortgage origination firms (e.g. Countrywide) sold out to Wall Street at very high prices a year or so before the collapse. So the senior executives of these firms did very well.  And ask yourself why so many sold out when things were going so well?  Maybe some people in the mortgage origination business did have a clue.  They differed from Wall Street only in adopting a "sell out at the top" strategy instead of a "musical chairs" strategy.

If it had just been the mortgage meltdown things would have been bad enough.  But unfortunately, it wasn't.  Wall Street has always made money by selling advice.  This is a steady business with a nice profit margin but there isn't a big enough market for it to make the kind of out sized profits Wall Street craves.  Wall Street's main moneymaker used to be buying and selling securities.  And by "securities" I mean Stocks and Bonds.  That used to be a very lucrative business.  But deregulation set in a few years ago and the amount of money a firm can make per transaction plunged.  The fee on a Stock or Bond transaction used to be enough to buy a dinner at a fancy restaurant.  Now it would be lucky to cover the cost of a small Coke at McDonald's.  Squeezing a half a cent out of the cost of a transaction is just not what a Wall Street Master of the Universe dreams of.  They looked around and found derivatives, the big moneymaker for Wall Street for some time now.

A derivative buy or sell transaction is like a Stock or Bond buy or sell transaction from an execution point of view.  So Wall Street didn't have to invent anything new to move into this business.  Stocks and Bonds are traded on exchanges.  The commissions are negotiable and everyone knows what is going on. So the market is fiercely competitive and the fee for executing the transaction is tiny because people can shop around for the best deal.  Derivatives historically have not been traded on exchanges.  For a specific derivative typically only one firm knows what the security is really worth so that firm can add a nice markup into the transaction.  So the profit per transaction can be like the old days with Stocks before deregulation.  In some cases it can even be much better.  The profit potential is awesome with derivatives.  So what is a derivative?

The answer turns out to be pretty much anything.  All you have to do is find two people, one for each side of a bet and you can package it up as a derivative.  In practise Wall Street deals in money.  So derivatives are generally about something financial.  As an example let's talk about mortgages.  As I said above, Wall Street bought bails of mortgages.  Now Wall Street was only interested in the money part of the mortgage.  So the mortgage was split.  A "mortgage servicer" would worry about collecting the payments and all that nitty gritty stuff.  Wall Street left this part alone and concerned itself only with the cash flow.  They would provide the money to buy the house.  Then they would get the cash flow generated by the mortgage payments from the servicer and marry it back to the mortgage.  And a single mortgage for a single house, even if it was a million dollar McMansion, was too small to interest Wall Street.  Typically thousands of mortgages were bundled into a single "mortgage backed security".

And Wall Street could stop there and just market the mortgage backed security.  But that was not interesting (read profitable) enough.  Different potential investors had different investment objectives.  Some investors were high risk - high reward types.  Others were low risk - low reward types.  It was just too difficult to build special packages for each investor type.  Then someone came up with a brilliant idea called "tranches".  I have no idea where the word came from but the idea is simple.  Say your bundle of mortgages has 1,000 mortgages in it.  Divide them into 10 tranches.  Tranche 1 would contain the 100 riskiest mortgages.  Tranche 2 would contain the next 100 riskiest mortgages.  Tranche 10 would contain the 100 least risky mortgages.  Now we can sell tranche 1 to a high risk investor and tranche 10 to a low risk investor.  The details of which tranche a specific mortgage ended up in are complex and might vary from security to security.  But this "tranche 5 of mortgage package xxx" is a derivative.  Its value is derived from some package of underlying securities.  And a derivative can be created that is a bundle of other derivatives.  So you can have layer upon layer upon layer.

And now the magic happens.  In a normal market what are the probabilities that 10% of a typical package of mortgages will default?  The answer is practically zero.  So a ratings agency might easily rate tranches 2 - 10 as AAA.  Anyone can buy a AAA investment.  Lots of people (pension funds, insurance companies) are required to only deal in "investment grade" securities.  AAA securities are investment grade.  So we have just created a bunch of AAA securities.  If the interest rate on the tranche 1 mortgages is high enough then someone will take a flier on it.  And let's say you are one of those "AAA only" investors.  If you are given a choice between an investment that returns 5 1/4% and one that returns 5 1/2% what do you do.  They are both AAA so of course you buy the 5 1/2% one.  By bundling and tranching Wall Street was able to create giant piles of AAA investments.  They could get the ratings agencies to rate lots of stuff AAA by saying "there is only a one in a million chance that more than 10% of mortgages will default".

Now this "bundle and tranche" strategy was applied to all kinds of stuff besides mortgages.  If you tranche Credit Card Accounts Receivable (your outstanding balance on your credit card) you can create a bunch of AAA securities.  Everybody wants to buy the AAA security that has a little higher return than the run of the mill AAA security.  And "due diligence" for many investors started and finished with "is it AAA?"  And for a fee Wall Street would customize the security.  Typically the Wall Street firm that created one of these derivative securities was the only one who had full information on it.  So if they bundled in some fees and added some markups they could still make the security look like a good deal.  So they did and made fantastic amounts of money doing so.

One more piece was necessary to make the whole scheme work.  Wall Street firms needed a way to assign a risk rating to each of these derivatives.  Now we can ask how many Boeing 575 airplanes fall out of the sky in a given year.  There are lots of 757's around so we can come up with a pretty good estimate for how much risk is built into a 757.  (Fortunately for all of us the answer in most years is none).  But in the case of derivatives many of these securities are unique.  Even when this is not true there is not much to go on.  Wall Street was saved a few years ago.  A math genius came up with an "algorithm" (essentially a computer program) that would pop out a single magical "risk" number no mater how many moving pieces the derivative contained.

The details are for the most part irrelevant to our discussion because we can understand why things went so wrong by concentrating on something easy, what information went into the algorithm.  The first kind of information was the specific details of each underlying security (e.g. mortgage amount, interest rate, duration, FICA number (credit score), etc.).  The other kind of information was historical information for similar securities. For these securities additional information on default rate, delay till default, loss amount, etc. were added to the basic "security" information.  The algorithm would use the historical data to make an estimate of what would happen to the specific set of securities in the package in question.  It would then boil that down to a single magical "risk" number.  With this algorithm a uniform procedure could be applied to any kind of derivative.  Every derivative could be assigned a risk based on the magic algorithm.  And all the major Wall Street firms bought off on this system.  With it all these derivatives could be boiled down to three numbers.  What's the price of the derivative?  What's the return (interest rate) of the derivative?  What's the risk of the derivative?  Now Wall Street was ready to sell large quantities of derivatives.  All the complexity was safely hidden out of sight and everyone could concentrate on "sell, sell, sell".  It looked like a good deal to customers because Wall Street was offering all these AAA securities with these wonderful returns.  What's not to like?

At this point we all know this story did not turn out well.  After the fact the magic algorithm was roundly criticized.  A lot of the criticism was fairly technical (e.g. "tail effect").  But most of the criticism ignored the question of whether the algorithm was properly applied.  Frequently the answer was no.  The original work was done using mortgage data.  The author was only interested in proving that the algorithm worked.  So he grabbed the data that was most readily available.  This consisted of a few years of mortgage data from the boom times.  There was no "bad things happening" data in the historical data he used.  This weakness was unimportant in terms of proving that the algorithm was mathematically correct.  But it was critical in real world applications.  Wall Street never expanded the historical data they used to included a broader sample.  The algorithm with the crippled historical data gave them the answer that allowed them to sell the security so they were happy.  But it gets worse.  The magic algorithm was applied to all kinds of stuff, not just mortgages. Was a custom set of historical data developed for these other types of securities?  No!  The mortgage data was handy.  It had a proven track record of generating a low risk number so why mess with success?  In fact the whole magic formula procedure as implemented on Wall Street was one giant con.  There was nothing wrong with the underlying math.  But the algorithm has to be applied properly and frequently it was not.

If you talk to Wall Street today they claim that they have fixed everything.  They still use the magic algorithm but they now claim that they are using it correctly.  But I see no reason to believe this.  Wall Street went wrong in the first place because it was too hard to make the kind of money they wanted to make without getting creative.  If anything, it is now even harder to make money the old fashioned way.  Margins in traditional lines of business are tighter than ever.  Competition is fiercer than ever.  So there is still a strong incentive to get creative.  And there is still money to be made by selling something that is risky but claiming it is not.  We have just seen clear evidence of this.  A few weeks ago JP Morgan Chase announced that they had taken a 2 billion dollar trading loss.  Many news reports claim the actual figure is 3 billion.  I have even seen one claim that it is 7 billion.  Jamie Dimon, the CEO claims that he now has things under control and that JP Morgan can absorb the loss without significant harm to the company or to the economy at large.  But it highlights the fact that a fundamental problem still exists.

The working assumption of all these Wall Street firms is that all these derivatives are "liquid".  In short this means if you decide to sell the instrument you can easily find a buyer.  And the price you get will be predictable.  But there is no reason to believe this.  In 2008 during the worst of it there were lots of derivatives that could not be sold for any price.  People came to believe they did not know what was in the underlying bundle of securities and, to the extent that they could tell what the underlying securities were, they could not accurately estimate their value.  So they said "I'm staying on the sidelines while I wait for things to shake out".  And the "value" of many derivatives went from 100% of purchase price to 90% or 70% or 50% or less, sometimes in a few days.  Some of these "AAA" rated derivatives ended up being worth 30% or less of face value.  The very fact that the content of these derivative bundles varies from instrument to instrument means that each one is unique or nearly unique.

This has been a problem for JP Morgan.  They have not released details on just what went wrong with which securities.  But traders think they have a pretty good idea.  And if a trader knows (or suspects) that a particular security being offered for sale is a JP Morgan problem child he knows JP Morgan is in a bind.  And that means he can low ball his price knowing JP Morgan might have to accept it anyhow.  Outsiders speculate that traders taking advantage of JP Morgan's bind is why the estimates for how much this will eventually cost JP Morgan vary so much.

In some sense the JP Morgan case is a special case.  But in another sense it is not.  Traders might chose to avoid a particular derivative or class of derivatives at any time for any reason or no reason.  Mostly what they care about is getting the best return for the least risk.  The fact that derivatives are generic because mostly all people care about is price, return, and risk but are custom in the sense that each derivative has a different set of underlying securities makes any specific "risk" number unreliable.  There are always other derivatives that have a similar price, return, and risk but a different set of underlying securities.  And this means that all derivatives are risky all the time.

In summary, we got here by doing a terrible job of estimating risk.  All this was started by various groups working together to drive risk in the mortgage market through the roof, all the while loudly claiming that no such thing was happening.  The market was driven so far from stability that it broke in a catastrophic (in the mathematical sense and in its impact on society) manner.  This firestorm swept through the markets on Wall Street and exposed more and larger failures to properly calculate risk.  We are probably doing a better job now, both in the mortgage market and on Wall Street.  If nothing else, I think customers now take Wall Street "risk" estimates with a 5 pound bag of salt.  The question is are we, and here I primarily mean Wall Street, now doing a good enough job of correctly calculating risk.  The JP Morgan "problem" argues strongly that we are not.