Foundations of Science

I have written a couple of posts on the most basic questions that surround Science.  This is a continuation of the series after a long delay.  The last post was http://sigma5.blogspot.com/2011/12/what-is-science.html, a post I made back in 2011.  In these previous posts I have argued that you can't prove Science is true using scientific methods.  In fact, in the post I link to above I argue that Science is a popularity contest that is scored using peculiar rules.  In this post I want to get into areas that are not generally spoken of.  These ideas are so foundational that they are accepted without even acknowledging their presence.  That makes them critically important.  Frequently in these posts I jump into a digression.  The digression serves the purpose of setting up what follows.  This time around I am going to do the opposite.  I am going to state the key items right up front.  Then I will spend some time discussing them.

Three key unspoken beliefs that all Scientists share

There is a single real world that is external to each of us but shared by all of us.

There are rules that govern how the real world works.  At least some of them are discoverable.

The rules that govern the real world all interact with each other in a consistent manner.

Three corollaries to the three beliefs

Whatever works is right.

Whatever doesn't work is wrong.

There are degrees of wrongness.

Discussion

The first belief is akin to the "there is only one God" idea in religion.

Science is grounded in observation.  If some of us experience a different world (i. e. a world with different rules) then what is observed depends on which group the observer is a member of.   Yet Science depends heavily on the concept of "repeatability".  Two people, any two people, each of whom does the same experiment are expected to get the same result.  In fact, if other scientists are convinced that both people did the experiment correctly but got different results they decide there is a serious problem that needs to be addressed.  Now it is sometimes found that one or both did the experiment poorly.  Then the next thing to do is do the experiment yet again and try to get it right this time.

If scientists decide that both did the experiment correctly then they start looking for something else to explain the situation.  They have found in many cases that "all other things being equal" is not true.  There is some influence that has not been properly accounted for.  Early experiments to determine "G", the Gravitational constant, ran into this.  People would make very precise measurements of how fast things fell.  Good experimenters got differing results.  This led to small adjustments for altitude.  But there were still problems.  This led to the idea that the density of the earth differed from place to place.  This density difference has since been turned inside out.  Geologists now use sensitive gravitational measurements to determine some of the attributes of rock that may be buried way below the surface of the ground.  This is one of the tricks in the bag of people who look for oil for a living.

Now this is definitely a belief.  It is not a provable proposition.  If you find a thousand or a million or a billion examples of everyone seeing the same thing that doesn't mean that there is not some situation just around the corner that will be the exception that disproves the rule.  And to some extent this is a self fulfilling prophecy.

In quantum mechanics there is a theory called the "many worlds" theory.  (Do not confuse this with the "many worlds" theory in cosmology.  It's a different theory that I do not have time to get into.)  In quantum mechanics we have all these situations where (to keep things simple) there are two possible outcomes.  The "many worlds" theory says that both outcomes actually occur.  But the world splits into two copies.  In one world the first thing happens.  In the other world the second thing happens.  We (or at least the copy of "we" that we are following) exist in only one copy of these many worlds.  As we look back we see, for instance, the first outcome of the most recent event.  And we also see a specific outcome of the previous event.  And so on.  There is a world in which each possible outcome of each event happened.  We see only the world as it is in our "world line" but there are zillions of other world lines in which events resulted in different outcomes.

In this conception of quantum mechanics there are in fact many worlds with a copy of each of us occupying many of them (remember, there may be events that caused us to not come into existence in some of these other world lines).  But we can only see the world and the world line we are in.  Scientists (and science fiction writers) have spent a lot of time working out detailed "many worlds" theories.  No one has come up with proof that the theories are wrong.  How could you?  But most scientists go about their daily business as if the theories are wrong.  And so far they have not come across any evidence that they are right.

Another line of thinking is the "Matrix" (from the popular movie) idea.  Maybe the real world looks quite different but we live inside a giant sophisticated computer simulation of the world we see.  Again, if the simulation is sophisticated enough and well enough done, there is no way to determine that it is a simulation.  Here the proper response is "who cares?"   The problem for Science just becomes "what are the rules in the simulated universe?"  We can't beat the simulation.  Therefore we have no access to the "real" real world.  So it does us no good to try to study it.  So we might as well stick to trying to figure out the simulated rules in our simulated world.  Those are the rules we can use to advantage.

The second belief is one that philosophers have spent a lot of time on.  There are systems of philosophy and religion that say "you can't so don't".  There are also religious systems that say "the anointed get into heaven no matter what they do and the rest don't get in no matter what they do so why should anyone try to figure things out?"  And, if your objective is to get into heaven, then it makes sense to not try if you subscribe to the "anointed" school of religion.  But here we are talking about your efforts to achieve salvation in the next world.  Most of us spend a lot of time trying to navigate around in the current world.  For those science says "Science can be at least of some help".

And Science has a pretty good track record.  Science has found out a lot about how the world works.  Some scientists believe that all of the rules are discoverable.  Some don't.  But all scientists believe that there are undiscovered rules that can be discovered.  So it doesn't really matter if they can all be discovered or not.  As long as there are more rules that are currently undiscovered but haven't yet been discovered there is reason to continue the endeavor.

No scientist believes that all the rules that can be discovered have already been discovered.  Nor do they believe that knowing the rules of science allows you to find the correct solution to all problems.  There is even a shorthand for this.  "Science is about 'what'.  Religion/philosophy is about 'why'".  Put another way, Science tells you a lot about what is possible but it doesn't tell you which choice among the possible choices is the "right" choice.

The third belief is actually quite a subtle one.  Scientists believe that all the rules apply all the time.  Now each rule can have a specific range of operation.  So most rules don't apply in most situations.  They can be effectively ignored.  But part of a proper scientific rule is to define what situations it applies to.  If a rule is applicable to a particular situation then it must be applied.  Put another way, you must look at all the data when trying to create a rule.  If you can't say why the rule applies here but doesn't apply there then there is a problem with your rule.  This trips nonscientists up all the time.  A rule may look perfectly good if you apply it to only a limited number of situations.  But if it doesn't apply universally you better have a "whereas" that says in effect "but it doesn't apply in these other situations".  Let me give you an example of this sort of thing done right.

Newton said "f=ma".  As far as he knew this was a universal rule.  But then Einstein came along and said in effect "f=ma + a relativistic correction".  And he showed that the relativistic correction was such a tiny number in a lot of "normal world" situations that it could be ignored.  Scientists and engineers know where the boundary between "normal world" and "relativistic world" is so they can keep it simple most of the time but fall back to the more complex relativistic methods when they need to.

A more common situation is where someone makes a "bold new" discovery.  But no one else can get the method they claim to have used to work.  This is common in the world psychic phenomenon.  They say essentially "I did this and here's what happened".  For instance they find certain people that can identify a card selected at random "at a rate above chance".  But others make what appear to be the same test but don't get the "above chance" result.  This situation has come up repeatedly.  And the same explanation has been made repeatedly:  shyness.  Apparently the phenomenon is shy.  If there are too many "negative vibrations" around the phenomenon goes into hiding.  Many scientists have a different explanation: bad experimental procedure.

The same thing happens in more traditionally scientific situations.  Check the Wikipedia articles on "N rays" or "cold fusion" for two examples that happened more than 80 years apart.  The "shyness" explanation was trotted out in both of these examples.  Eventually scientists concluded that the original discoverers just got it wrong.

Now let me move on to my corollaries.

The first one ("whatever works is right") seems obvious and sensible.  But scientists have gotten really good at coming up with really weird results.  The classic example is photons.  People have been studying light for a long time.  As I pointed out in my previous post, Newton did some wonderful work and in 1704 published his results in a book whose title rendered in English is "Optics".  A lot of good work was done before this and a whole lot was done afterwards.  In spite of this no one could decide whether light was composed of waves or particles.  That is until Einstein came along in 1905.  His answer was "neither".  Instead light was composed of things called photons.  Photons behave like waves in some situations and particles in other situations.

To a lot of people this was a completely unsatisfactory solution.  An answer of "particles" or an answer of "waves" would have been natural and satisfactory. But the fact is that the extremely weird theory of "quantum electrodynamics" (the modern successor to Einstein's original work) does an excellent job of telling us how light will behave in all kinds of situations.  QED (the standard shorthand for quantum electrodynamics) works.  "Waves" doesn't work.  "Particles" doesn't work.  And science has now built up a long list of truly weird theories that work.  Evolution (which is not actually that weird) works.  Quantum Mechanics (actually its modern descendent "The Standard Model") works.  Relativity works.  The list goes on.

The second corollary ("whatever doesn't work is wrong") also seems obvious and sensible.  But in practice it is not.  As I just got done pointing out, in the case of light, "waves" doesn't work and "particles" doesn't work.  But they are both connected to our day to day experience.  Similarly, the Bible as an accurate historical reference doesn't work.  Pre-QM atomic theories don't work.  What is now called "Newtonian Mechanics" (e.g. "f=ma" and other non-relativistic theories) doesn't work.  In some cases the old theory that doesn't work is readily discarded.  If it strikes people as weird or unnatural they are happy to let it go.

Most people had no idea what "N rays" were about.  They didn't much care when they were "discovered".  And they didn't much care when they were debunked.  So it was pretty easy to discard the whole "N Ray" theory.  And the whole "N Ray" battle was scientists versus scientists.  The general public felt it didn't have a dog in the fight.  But in the case of the other now discarded theories I mentioned, people had a lot more invested in the old theories.  There are large, well funded groups invested in the whole Evolution "debate" so leaving the discredited alternatives behind has turned out to be much harder.  But that's why this corollary is so important.  Except . . .

There are degrees of wrongness.  All scientific theories have problems.  The gold standard of scientific theories is "The Standard Model".  It is the governing model for particle physics and is generally characterized as "spectacularly successful".  It has problems.  The biggest and most obvious problem is that no one knows how to reconcile it with Relativity.  So it fails the test for perfect scientific theory.  In fact there are no known perfect scientific theories.  So how do scientists deal with this?  The short answer is they fudge.  That's why scientific theories are always referred to as "theories".  It is always possible that the theory will be modified or even completely replaced by a quite different theory.

Scientists have now had hundreds of years of experience dealing with this issue.  And part of that experience is with dealing with theories that are known to be wrong but have proved to be useful anyhow.  They try to balance the utility of each theory with the problems its inadequacies can cause.  This has resulted in an informal "wrongness" scale:

Degrees of wrongness

 Best - These are well established theories.  Scientists believe that the core of the theory is correct but that some modifications may be needed in the future around the edges.  Also, any theory that is in contravention to the core of the theory is easily demonstrated to be wrong.  A classic example of this is Evolution.  Evolution grew out of problems with all bible based theories.  So all bible based theories are known to be wrong.  The core of evolution is known to be solid.  Some tweaking (the relationship of two species, evolutionary mechanisms, details of the evolutionary record) may need tweaking but the core is solid.

Very good - These are theories that have had a great deal of success.  But there are known problems for which no known solution currently exists.  A classic example of this is The Standard Model.  The relationship of this theory to Relativity is currently a big problem.  There are other issues where scientists are not sure they have everything right.  Note:  Finding the Higgs Boson substantially reduced the problem areas seen with the Standard Model.

Good - The theory works well in a lot of situations but has a number of known problems.  Like Very Good theories the core looks solid but here there are a lot more problems around the edges.  A classic example of this is Plate Tectonics.  This theory solved a lot of problems.  It gives us a good way to understand a lot of geology.  But there are a number of open problems having to do with how some plate boundaries work.  There are a lot of areas where current understanding can best be described as "incomplete".  But there are a number of areas where the theory works very well and there is currently no better theory on the horizon.

Ok - Science has had a lot of success with an incremental approach.  A bad theory is better than no theory at all.  A lot of theories don't work very well.  But they work some of the time and the fact that we know where they work and where they have problems is helpful in determining what additional experiments might yield useful information or what problems a new theory might solve.  A classis example of this kind of theory is the "Cooper pair" explanation for superconductivity.  It works pretty well some of the time and not well at all a lot of the time.  Nobody currently has any ideas that seem to work better.  No one would be surprised if a new theory came along that was quite different to do a better job of explaining Superconductivity.

Poor - These theories are typically found where scientists really don't have much understanding.  A good example of this would be planetary formation.  There used to be what appeared to be a good theory.  Small "rocky" planets formed close to stars where gasses were boiled off.  Gas giants formed further away where it was cooler.  Then came all the data on exoplanets from the Kepler spacecraft.  It turns out there are all kinds of planets of the wrong kind and size in the wrong places.  Scientists have some ideas but mostly it's early days.

Not even wrong - This phrase is attributed to Wolfgang Pauli, the great physicists.  In describing ok and poor degrees of wrongness I emphasized that these theories all were useful to the extent that they were helpful in pointing to where a better theory might be found.  "Not even wrong" theories are so bad that they just add noise to the situation.

So given all this wrongness why do scientists get so exercised about theories or criticisms from outsiders?  Scientists are very aware of how problematic some of their theories are.  And they dearly love it when a new better theory comes along.  But is generally pretty easy to slot a lot of "new" theory into one of the above categories.  Scientists get exercised when a new theory slots into a category that is lower than the one their current favored theory occupies.  They also get exercised when a theory is advanced or a criticism is leveled that they have seen before (frequently many times) and have already thoroughly investigated.

The evolution "debate" falls into this category.  All of the early scientists that worked on evolution were practicing Christians.  They were very familiar with the bible.  Where the bible appeared to get it right, those ideas were incorporated in evolution.  But in many areas the bible got it wrong.  Darwin's "Origin of Species" was first published in 1859.  It was immediately criticized.  Several new editions were quickly put out that included additional material addressing these criticisms.  The sixth edition, the one commonly available now, was published in 1872.  It should be shocking how many "new" arguments against evolution are in fact old arguments that Darwin addressed in great detail over a hundred years ago but apparently it is not.  But most people who oppose evolution have never bothered to read Darwin or familiarize themselves with subsequent developments.  I recently published a post on "Ken Ham Creationism" (http://sigma5.blogspot.com/2014/02/ken-ham-creationism.html).  Many of Hams "modern" criticisms of evolution find well supported refutations in Origin, a book that is now over a hundred years old.

Fixing healthcare.gov

This is the second post I have done that is in response to material published by Steven Brill.  My first post is at http://sigma5.blogspot.com/2013/02/medical-costs.html.  Both were published in Time magazine.  His more recent article is in the March 10, 2014 issue and is titled "Code Red".  Brill is an astute observer of the Health Care industry.  But he is not a techie.  This did not get in the way of his previous article.  With his substantial business background he was able to dissect the hospital business to devastating effect.  I recommend both his original story and my analysis of it.  And, unfortunately, all of the defects he found then are still with us.

His lack of computer expertise shows in his recent outing.  But he still has a lot of useful things to say.  So it is a worthwhile endeavor on his part.  And don't forget the news environment.  Problems with healthcare.gov were wretchedly excessively over covered by all parts of the media.  The "it's fixed" story has been covered, although perhaps less than it should have been given the overkill that preceded it.  But Brill's story is unique in my experience in that he addresses the "how it got fixed" story.  If you are tempted to say "that's not news" then check out any sports section from any paper on any day.  Every one is jammed full of "how we won/lost" stories.  On to Brill's piece.

The obamacare.gov saga can be broken down into 4 eras.  Before October 1 the story was "It will work great from the get go on October 1.  Trust me."  Era two ran from October 1 through October 17.  This was the "It's broken.  Nothing serious.  We'll have it fixed any minute now." era.  The third era (October 17 through December 23) is the one Brill focuses most of his attention on.  This is the period when the site transitioned from seriously broken to working well enough for the moment.  The fourth era (after December 24) is the modern "It's working in the main.  There's more to fix but we have things under control." era.

Brill throws in some "tech talk" tidbits to spice things up but does not get into that much nitty gritty.  Instead he puts more focus into a management level perspective.  This turns out to be very useful.  At a low level all tech projects are different.  But at the management level there is a lot of commonality and a lot that can be usefully applied to the next tech project.  So let's take a look at what Brill found. 

And let me start by picking something out of the middle.  I believe it's the most important thing Brill reports.  Three rules for behavior were quickly adopted.  1.  Meeting are for solving problems.  Blame games can be played somewhere else.  2.  The ones who should be talking are the ones who know the most, not the ones with the highest rank.  3.  We need to stay focused on the most urgent issues.

If you take the opposite of these rules you will have the rules politics are usually played by.  But these rules are used over and over to pull big technical projects off successfully.  Politics has a big effect in the short run.  Technical changes have an even bigger effect in the long run.  Political effects tend to cancel each other out fairly quickly.  Technical changes tend to change everything and endure.  As an aside, it would be nice if the media, particularly the beltway media, stopped reporting everything through the lens of political rules and instead reported things through the lens of technical rules.  This change would help fix the mess our national politics are currently in.

The bottom line is that by adopting the above behavior (and some other things) a small team was able to fix the web site in six weeks or less.  Brill focuses on the contributions of the small team of techie superstars that the Obama Administration assembled, once they figured out they had a big problem.  And, as Brill points out, the Obama people had another big problem that happened to exactly coincide with era number two.  This is the time when the government shutdown was in effect.  Imagine the hew and cry if the administration had diverted its efforts away from getting the government shutdown ended and had instead focused on the "minor" (certainly how it would have been characterized) problem represented by the web site.  I don't think that it is a coincidence that the administration took major action to fix the site immediately after the shutdown crisis was resolved.

If you want lots more details of how the site was fixed, read the article.  But let me cherry pick some more.  A lot of time has been spent on the "blame the contractors" game.  This game has been played all over the place, not just with healthcare.gov.  But the first thing Brill reports is that the technical people employed by the site's contractors were not "defensive or hostile".  In fact the small team of fixers and the bigger teams of contractor people quickly developed a good working relationship and worked together well.  And once the fixers took over management of the project the contractor management also quickly fell into line.  In fact one of the contracting companies solved a big administrative problem by putting the fixers on their payroll.  This made them kosher with respect to various government rules and regulations.  The same turned out to be true of the Obama administration types.  No one started playing politics or putting up unnecessary roadblocks.  This broad cooperation up and down the line was one of the most important keys to getting the site fixed quickly.

The only administration side management flaw Brill found was that there seemed to be no one individual that was in overall charge.  Responsibility was diffuse and it was unclear who was responsible for what.  On the other hand, Obama himself comes off very well.  Early and often he was asking everyone if things were on track.  He was repeatedly reassured that they were.  Once the site started failing he held daily meetings to try to get a handle on the situation.  And as soon as the shutdown was over he tasked a single individual, the one person in his inner circle who actually had technical expertise, to get 'er fixed.  Things immediately started moving quickly from there.

The fixer team consisted of the best.  And they had deep and broad expertise in big technology projects.  Several were silicone valley heavyweights.  The fixer team immediately set out to fix things.  Some of this involved hands on work by members of the team like fixing code.  But they also applied their hard earned knowledge of how you do this kind of thing.  The first thing they did was put together something called a "dashboard".  It performs for software what the dashboard in your car does for your car.  It tells you what's working and what isn't (e.g. "check engine").  It also tells you things like how fast you are going or, in the case of the web site, how many people are accessing the site.  Then their are general health indicators (gas gage, engine temp).  In the case of the site things like the access times of the several databases the site depended on.  This allowed the team to translate "its broken" into "these specific things are not working right".  The more detailed information from the dashboard allowed engineers to investigate specific components looking for specific problems.  And that in turn led to specific fixes that made the site work better.  To boil it all down, six weeks worth of the right specific fixes resulted in a site that worked pretty well and did all the "needs to be working right now" things correctly.

Now let me back off and apply my technical expertise to fill in some blanks.  I'm sure the fix team fixed a lot of stuff.  But I suspect that most of the fixes were actually done by the engineers employed by the contractors.  One of the fix team observes "these guys want to fix things".  I suspect that the problem was not a lack of talent on the part of the contractors.  I suspect that the main problem was poor project management.  I have been involved in enough IT projects to know from personal experience that any IT project can be screwed up by managing it badly enough.  And it doesn't matter what anyone else does.  The best management has a good understanding of the technology (in this case web sites and databases) and a good understanding of the business (in this case health insurance).  Given enough good will (and time and money) shortcoming in one or both areas can be overcome.  In this case the basic technology side did not look that hard to me.  But, as I will explain below, the business side was actually a lot harder than it would appear.

Plenty of money was spent on the site.  Brill reports the amount was $300 million.  That should have been more than enough.  One problem that was identified early was that the hardware was inadequate.  But that problem was identified almost immediately.  Hardware was added in early October, well before the 17th.  And more hardware kept being added well into December.  It may be that more hardware is still being added.

Now let me provide some broader perspective, perspective that is beyond the scope of the subject matter of Brill's article.  I am going to take a quick look at two state sites, Washington and Oregon.  Both of these state sites had a simpler task.  They only had to deal with the idiosyncrasies of one state.  The federal site had to deal with the idiosyncrasies of more than 20 states.  Both states are blue states that are in decent shape from a budget perspective.  So there presumably was enough money in each case and there were no political roadblocks.  But the experience of the two states couldn't be more different.

The Washington site went up on time.  It has had a few glitches. But for the most part it has worked from the beginning and worked well.  Oregon's site has been a disaster.  And it has been a far bigger disaster than the federal site.  It never worked at all during the entire October - November time period.  The last time I saw any news coverage about the site (a couple of months ago now) reports were that it was still completely broken.  Oregon gave up and declared some time in October that they were going to an all manual system.  So why the differing outcomes?  Well, there were apparently some differences between how each state behaved that turned out to be critical.

Washington decided a couple of years ago that the state systems and databases needed to support the state web site were not up to the task.  So they rolled out new modern state systems before October 1.  This meant that the interfaces between the state web site and the state systems could be simple and that the state systems would be able to deliver exactly what the state site needed.  The web site would not have to "crutch around" inadequacies in the state system.  Washington also bid out the web site in the usual manner but was able to provide substantial state supervision "in house".  The result, as I said, was a site that worked pretty well pretty much from day one.

As you might expect, Oregon did things differently.  I don't know whether Oregon did a major upgrade of their state systems like Washington did.  I certainly haven't heard that they did.  The other thing Oregon did was outsource pretty much the whole thing including project management to Oracle Corporation.  I have a lot of personal experience dealing with Oracle.  They have good people and they have bad people.  The company I worked for was not big enough to justify always getting the good people so we frequently got the bad people.  Oracle missed out on a big business opportunity by so mismanaging a project (an outsourcing as it happens) that we cancelled the project and kept that functionality in house.  I don't know anything about the specifics of the Oregon project.  I definitely know nothing about who was or was not assigned to the project by Oracle.  But I do note that things failed.  And they failed spectacularly and in a very public manner.

So how does this relate to the federal project?  First, as I noted above, the federal project has to interface with a large number of state systems.  That is much more complicated than what Oregon had to pull off.  And the federal system always has worked better than the Oregon system, even in those bad old days of early October.  The only thing you can say in favor of the Oregon experience is that the site worked so badly that it was easy to decide early on to just scrap it.  So the Oregon experience is obviously apt.  The task was harder to pull off and easier to screw up that it would appear from the outside.  But what does the Washington experience tell us?

Remember the part where I pointed out that the state decided to do a major upgrade to the state systems the web site would depend on.  Many of the states that depend on the federal site are southern states.  These are states that are generally technology averse and poor.  What kind of shape do you think the state systems are in that have to tie into the federal site?  My guess is that they are in very poor shape.  This means that the federal site would have to have a lot of "crutch around" capability.

Add to this the fact that many of the states depend on the federal site are red states.  The GOP has been adamantly opposed to Obamacare.  I think the anti-Obamacare vote count in the U.S. House is now up to 50.  I pointed out above that it is easy for bad management to mess up IT projects.  It is also easy for hostile state governments to put all kinds of road blocks in the way of a successful web site implementation.  So the federal site has to deal with the complexity of servicing multiple states.  I don't know for sure but it is a good guess that it has to deal with state systems that are more or less inadequate.  And it has to deal with state administrations that are hostile to Obamacare, and therefore the web site, for political reasons.  Finally, I will note that some states made the decision to not do their own site very late in the game.  This meant that the federal site had to be reconfigured to handle these states very late in the game.  In short, the federal site had a number of things to contend with that neither Washington (success) nor Oregon (failure) did.

The site should have worked on October 1.  It seems apparent in retrospect that the biggest cause was that the project was poorly managed by the Obama administration.  One specific cause was that no single individual was in charge of getting the web site right.  But, if the site could be fixed in 6 weeks, it wasn't really all that broken.  And there were a number of contributing factors.  The government shutdown and the long run up to the shutdown were very disruptive.  And the knee jerk hostility of Republicans were definitely contributed.  As just one example, how much time was spent by senior administration officials trying to move other senior officials through the Senate confirmation process.  The time they spent in this necessary but essentially useless task could perhaps been spent doing a better job of managing the web site rollout instead.  And it is important to remember that the Oregon experience shows us that it was easier to bungle this process than it appeared.      

Caller ID is sick and getting sciker

Caller ID is a relatively new technology.  It only dates back to the '90s.  Caller ID was not possible until telephone switching equipment became computerized.  If you look at old black and white crime movies you will see what it used to take to "trace" a call.  You literally had to have people standing by at telephone "central offices", buildings full of complex electromechanical equipment.  When he got the signal the technician would literally look at the equipment to follow the path of the telephone call.  According to dialog in these old movies it typically took a minimum of two minutes to trace a call. And tracing was only possible if it was a local call.

The advent of computerized telephone equipment changed all this.  First the industry had to change over to computer control of the telephone equipment.  Then standardized methodology for doing a trace had to be developed.  Finally a method of passing the trace information to a customer's phone had to be developed.  But all that eventually happened. And when caller ID first rolled out it was very cool.  A little display on your phone would show you the name and phone number of the caller.  Back not so long ago this little trick seemed totally awesome.  Now in the era of smart phones and Skype it doesn't seem like a big deal.

And it turns out that caller ID depends on a technology that is rapidly disappearing, the phone book.  It took me a while to figure out that this connection existed but it does.  I have a phone book but I believe it is now about three years old.  (The phone company has stopped dropping a new one off every year.)  Phone books used to be a cash cow.  You got a set of "white pages", which cost the phone company a bundle of money to produce.  But in exchange you also got a set of "yellow pages".  And not so long ago yellow pages were a very effective method of advertising.  So businesses signed up and paid through the nose to be prominently featured.  And they frequently had to buy multiple ads to cover all of what we now call "search terms".  You would want your pizza joint listed under "Pizza" and "Restaurant", for instance.

But it is now easier and more effective to do a search, on your computer (if you are an old fart like me), or on your smart phone (if you are not).  So the value of putting an ad in the yellow pages has plummeted.  And most businesses have figured that out.  You are far better off putting your money into Google "adwords" instead.  And with the plunge in revenue from yellow pages has come a plunge in interest in complete and accurate white pages by the phone companies.  And that, in turn has become a problem for caller ID.

I am new to caller ID.  First I didn't have a phone that would display caller ID information.  Then it didn't seem worth the trouble.  Finally I got fed up with all the nuisance phone calls I was getting.  So I bit and got caller ID.  I am looking at it with fresh eyes compared to someone who has had it for some time.  And my conclusion is that it doesn't work nearly as well as it should.  And it looks to me like the situation is only going to continue to get worse.  So what's the problem?

Well, technically caller ID works just fine.  Every single time the phone rings a "caller ID" informational display pops up on my phone.  But the actual information is not very informative a lot of the time.  The obvious problem is with "blocked" calls.  A provision was put into the regulations that allowed a caller to block caller ID.  The justification was to protect the privacy of single women living alone.  But as far as I can tell anyone can have caller ID blocked.

This is actually a far smaller problem than you would expect.  Most women have found that blocking caller ID is more trouble than it is worth so they don't bother.  So I only occasionally see a "blocked" message come up on my display.  It turns out their are better methods for hiding your identity.  And I have trained the one person I know who has caller ID blocked to enter an "unblock caller ID for this call" code before she calls me.  So "blocked" calls are not much of a problem.  And to the extent this is a problem the impact is going down because fewer and fewer callers have blocking turned on.

There is a technical term for a much bigger problem with caller ID.  The term is "spoofing".  Spoofing covers any situation where the number shown is not the actual number of the calling phone.  You would think that spoofing would be limited to spies and crooks.  But it is actually in widespread use by many companies large and small.  An obvious example is an "800" number.  Free to the caller long distance has been around for a long time now.  I even know of a situation where an individual had children scattered across the country.  In order to encourage them to call home he got a personal 800 number.  It cost little and had the desired result of causing his children to call home more often.

He was happy with the result (this was several years ago) but cost of long distance has dropped so much it would hardly be worth the trouble now.  So what's the point?  The point is that no phone line is ever actually assigned an 800 number.  The 800 number is an "overlay".  Some computer somewhere in the phone network replaces the 800 number you dial with an actual phone number.  The call is completed to this normal number, which you never see.  But many calls I get from companies show an 800 number on the caller ID display.  Any time I see an 800 number I know the call has been spoofed.

Now this is not necessarily a problem.  In most cases if I call the 800 number I will get the company that called me.  So this example of spoofing is harmless.  The problem is that the "name" part of the caller ID display often does not show the name of the company.  It often dumbly duplicates the same 800 number shown on the "number" line.  So I don't know who's calling me without answering the phone.  This even happened to me with a call from Bank of America.  You'd think they would want me to know it was them calling.

So, to recapitulate, phone numbers are frequently spoofed by businesses.  And I have even seen it done by very small businesses.  And the number displayed is often an 800 number.  That can be justified on the basis that it can make it more convenient for me, the customer, to get back to the company in question.  But the point is that spoofing is common.  So in many cases you can't trust the number caller ID displays.  I know that I am being spoofed when I see an 800 number on the display.  So I know a large number of caller ID displays are showing spoofed numbers.  Do I know that the other numbers I see displayed are the number of the caller?  No.  It is possible (and likely in the case of the frequent "marketing" calls I get) that other numbers are being spoofed too.  It's just that I can't immediately tell that this is happening.

So there is a big problem with spoofing the caller's number.  There is an even bigger problem with the other piece of information caller ID is supposed to give you.  There is frequently no name in the "name" field.  I indicated above that my call from Bank of America just repeated the 800 number rather than saying "Bank of America".  This is a particularly bad idea on their part.  Over the years I have received many calls from "Rachel from cardholder services".  I even wrote a blog post about it (see http://sigma5.blogspot.com/2012/02/rachel-from-cardholder-services.html).  Rachel fronts for a scammer that wants me to switch to a different credit card.  You would think that Bank of America would want to do everything they could to discourage this.  But their legitimate call to me looked very much like a "scam" call.

 So I see a lot of calls where the "name" field just repeats the information in the "number" field.  This seems to happen most frequently with calls from businesses.  But, since I no longer answer calls when I can't figure out who the caller is I don't really know.  So repeating the number in the "name" field is a problem.  But there is another problem with the information in the "name" field.

Frequently it contains just a "City ST" combination.  I recently was called from "Rochester NY", for instance.  I have received a number of calls from "Albuquerque NM" in the past few weeks.  Apparently I have a fan base spread all across the country.  Many of these calls are apparently from places where I know I don't know anyone.  So it is easy to ignore those calls.  But sometimes I bite and answer the phone.  The "City ST" combination happens to match that of a distant friend or relative.  So it might be a call from them.  But when I do bite and answer it has often turned out to be a scam/marketing call.  But not always.

Cell phones (or "mobile" phones, as most of the rest of the world calls them) routinely pop up with this same "City ST" type of display in the "name" field.  In fact, I have never gotten a call from a cell phone where the person's name popped up.  As far as I can tell cell providers never register a name for a cell phone number.  I recently got my mother a prepaid phone.  You can register them.  When you do you provide the usual contact information.  But in this case I could not change the name from "MOBILE USER" to that of my mother.  Now the company involved (it was an AT&T "go phone") certainly knew my mother's address from the contact information I filled in on her behalf.  They even would have had an email address except my mother doesn't have an email account.  But they literally made it impossible to change the name on the account.

And this is where the death of white pages ties in.  It is no longer important to have a phone book to look numbers up in.  In fact, it is of decreasing importance to be able to look a person's number up online.  So apparently cell phone companies have decided as a matter of policy to not publish name and address information for cell phones.  And the world is increasingly going mobile.  The number of people who only have a mobile is growing.  Kids do it.  But I know a number of budget challenged adults who have given up their land line to save money.  They all feel that going "mobile only" is feasible but going "landline only" is not.  My guess is that caller ID gets the name information from public directories (the online version of white pages).  But more and more phone numbers are not listed in any public directories.  So the percentage of the time that the "name" field in a caller ID entry has useful information is low and dropping and this makes caller ID less useful.  The situation has gotten so bad that it is apparently cutting into the ability of the NSA to spy on us.  A recent story (http://gizmodo.com/want-to-avoid-the-nsa-use-a-cell-phone-1518318611) reports that cell phones are giving the NSA fits.

My recent experience getting a prepaid phone for my mother allowed me to answer the burning question "are burner phones for real?"  As anyone who has seen a cop show or a thriller recently knows, there is something called a "burner phone".  It is a phone that can't be traced back to the user.  And, since it is cheap and untraceable, it can be "burned" (thrown away) at any time.  Lots of things that show up in entertainment pieces are fiction.  It is way harder to crack encryption than the movies and TV lead you to believe.  But it turns out that everything they say about burner phones is true.  I bought my mother's phone at a drug store for cash.  I put money into it using a credit card.  So, with a little effort my mother's phone could be tracked back to me because I used my credit card.  But I could have bought a prepaid card for cash and loaded money into the phone that way.

All the company wants is their money.  If you buy the phone (for cash) and load money into the phone (say with a prepaid card bought with cash) then they get their money.  So they don't care who you are or what you use the phone for.  And that's a problem for the NSA.  And its a problem for people like me who want to screen out nuisance calls using caller ID.  The laws and regulations make all this legal.  They could be changed.  But the phone companies (and other businesses) are doing just fine with things as they are.  So their lobbyists will make sure that the laws and regulations stay the way they are.