Computer Chips 101

This post could more accurately be described as "The secrets of Integrated Circuits".  "Integrated Circuit", or IC, is the technical term for what people commonly call a computer chip.  They are the heart of every desktop, laptop, and smartphone.  These devices and many more use all kinds of ICs.  The processor units that computers and smartphones are built around are the most complicated and powerful ICs in existence.  Other ICs can be far less complicated and far less powerful.  Then there are whole classes of special purpose ICs.  But they are all manufactured the same way and all share several important characteristics.  I am going to completely ignore the differences and focus on the commonalities.  In fact, I am going to almost exclusively focus on how they are made.

Don't worry.  I am just going to focus on some key ideas.  It won't require you to have any technical expertise to follow along.  But you will end up with a good overview.  This will enable some surprising insights.  Here's the first one.  The markets for gadgets is supposed to work a certain way.  Take cars, for instance.  If you buy a fancier car it costs more.  And car prices typically increase over time.  I could easily name a hundred other things where the market works the same way.  Fancier is more expensive.  And prices tend to rise over time.  But ICs are different.  Fancier doesn't necessarily mean more expensive.  This is especially true over time.  And over time prices tend to drop for ICs, not rise.  Why is this?

I have been thinking about how to explain this for a long time.  And a long time ago I settled on using picture postcards as the foundation for my explanation.  But between the time I settled on this and now things have changed.  Picture postcards were a part of everyone's shared experience.  I didn't have to explain them.  But the purpose they served for a long time, inexpensively memorializing a place and the fact that you or a friend had visited it, has been replaced by the selfie.  So now I have to worry that when I say "picture postcard" some people will say "what?".

And it's worse.  There is the business of the "picture" part.  In the context of a picture postcard "picture" means a photograph.  But the way people now take pictures has been revolutionized.  Now a digital camera is used, often one that is built into a smartphone.  Understanding how a smartphone takes a picture does not make it easier to understand how an IC is manufactured.  You have to go back to how things worked before digital technology took over the picture-making business.

But with the ubiquity of digital picture-making technology it is also not a given that people understand how things worked previous to this.  So I now feel obligated to throw in an explanation of what a picture postcard is and also an explanation of how photography worked in the time before the digital age.  Fortunately, lots of old movies and TV shows showcase both, so the concepts shouldn't be completely foreign.  Photographs, the non-digital kind, first.

A photograph is in effect a mechanically created painting.  Like a painting there is a supporting medium, something like the canvas painters often use.  In the case of a photograph it is often paper.  But other options like a sheet of clear plastic are available.  And like a painting there is a picture on one side.  In the case of a painting this picture is created by the hand of the artist using, for instance, oil paints.  And the artist frequently introduces various distortions or creates something that never existed in the real world.

With most photographs the idea is to exactly reproduce an image from the real world and to do it without the necessity of someone painting it.  Instead, some very special chemistry and a device called a camera is used.  We seem to have gotten rather far afield at this point.  But from here on out we will head toward not away from our objective.

For our purposes a camera is a simple device.  It has a hollow space inside it and at the back of the hollow space is a film, another word for what will eventually become a picture.  It has a supporting medium, most often paper or clear plastic.  On this supporting medium is spread a thin layer of goo.  A lot of very sophisticated chemistry gives this goo a magical property.  It is photo-sensitive.  If you shine lights on parts of it those parts will have changed chemical properties.  That's the secret that makes it all work.  Special procedures must be used to make sure this special goo is not exposed to light until the proper moment.  The design of the camera assists this.

For our purposes the rest of the camera is quite simple.  It consists of two things.  There is a lens.  The job of the lens is to focus an image from the real world onto the goo on the film.  The image consists of lighter and darker parts.  This pattern of lighter and darker parts is reproduced in the pattern of changed versus unchanged areas of the goo.  The other thing is the shutter.  This is a device that permits light to travel through the lens for only a short period of time.  So you point the camera at a scene you are interested in and trip the shutter. Eventually the scene in question will be reproduced in the picture.

Later the film is removed from the camera and processed.  I am going to skip the details but the result is that the image is "developed" (the part exposed to light change color) and "fixed" (the photograph is no longer changed by subsequent exposure to light).  These subsequent steps are collectively called "finishing" the photograph.  This results in a "negative", a picture where the light parts of the original image are darkened and the dark parts are lightened.  If you "print" the negative (essentially take and finish a photograph of the photograph) you end up with a "positive" a picture of the original image where the parts that were originally dark are dark in the photograph and the parts that were originally light are light in the photograph.

And the process I described above yields a "black and white" photograph.  By what amounts to taking three photographs simultaneously (one for the red parts of the image, a second for the green, and a third for the blue) we can use the same basic process to make color photographs.  "Introduction to photography" is now complete.  What's the point?

Let's say you are on the business of making "picture postcards".  These are cards with a picture on one side, typically of something notable like the Eifel Tower, and a place to put postage, a mailing address, and a short message on the other.  You can buy one, fill out the back side and "snail mail" it to someone.

That used to be the best way to say "I've been to see the Eifel Tower".  Now you can take a Selfie.  With you in the foreground and the Eifel Tower in the background it is much better proof that you were there.  And you can email it or post it on social media immediately.  The level of artistry displayed by the picture postcard is higher.  But the immediacy and convenience of selfies are quickly wiping out the picture postcard business.  But let's ignore all that and go back to picture postcards.

Let's say we want to manufacture a bunch of picture postcards to serve the New York City tourist trade.  (Hang in with me here.  We are about to get to the point of all this.)  We want to give our tourist customers options so we will make postcards featuring pictures of a number of different scenes.  One picture might have of the iconic "LOVE" sculpture.  (It the piece that consists of an "L" and an "O" on top of a "V" and an "E".  The "O" is tilted at an angle.)  The image would be quite simple.  Much of the picture would consist of a neutral background.  Then there would be four blobs, one for each letter.  There would not be a lot of complexity to this picture.

We might decide to use a picture of Broadway during the peak of the afternoon rush as a second option.  This picture would have taxis, other cars of various makes and models, bicyclists, pedestrians, lots of fine detail.  The image here would be a quite complex compared to the one in the "LOVE" picture.  That's all the options I need to complete my explanation so I'll leave it at that.

My point is that it turns out it would cost exactly the same to produce a picture postcard using either of these picture or any other picture we might choose.  We would go through exactly the same process for each option.  We would  start with a piece of paper that had all the stuff we needed on the back.  We would then photograph and finish the appropriate negative to create the picture on the front.  The capital, labor, materials costs, etc. would exactly the same regardless of which picture we chose to put on the front of the postcard.  The only possible place where costs might differ is with the costs associated with procuring the rights to the negatives we want to use.

The secret to why Integrated Circuits keep getting faster, better, and more complex, but the cost stays the same or goes down is that the process of making a simple Integrated Circuit and a complex one is the same.  One might cost more to design than the other but, all things being equal, they are going to cost the same to actually make.  But before diving any further into how ICs are made let's look at why they appeared when they did.

There are certain preconditions that had to be in place before ICs could be invented.  ICs are a type of "solid state device".  The laws governing the operation of solid state devices are called Quantum Mechanics.  So you need to invent Quantum Mechanics in order to be able to invent ICs.  Quantum Mechanics was invented in the nineteen-twenties and -thirties.  Additional refinements had to be added to Quantum Mechanics to be able to figure out how to manufacture transistors and later ICs.  It didn't take long to come up with these refinements.  The first practical transistor was made in 1947 and the first IC followed less than two decades later.

Besides Quantum Mechanics it was also necessary to know how to make super-pure materials.  Twenty-four carat Gold can have impurities representing 1 part in 24 (about 4%).  For most people that's pretty pure.  But there is an even more pure version of Gold called "one thousand fine".  It can only have impurities representing 1 part in a thousand (0.1%).  When it comes to purity, that's about as far as most people's imagination takes them.  But it is far from pure enough to make transistors or ICs.  The least pure materials you can use can only have impurities representing 1 part in a billion.  That's a million times as pure as "thousand fine" Gold.  And for may applications that's not pure enough.  The purity must be taken up another thousand or million fold.  So how do you purify something to that extent?

There's a trick.  Melt the material but keep it just barely hot enough to stay liquid.  Then slowly dip a rod of the same material into the liquid so it is barely touching the surface.  Finally very slowly withdraw the rod.  If you are careful enough very pure material will solidify on the end of the rod causing it to grow.  And the material that grows on the end of the rod will be much purer than either the material in the un-melted part of the rod or in the original pool of liquid.

So you repeat this process.  You draw rods using your originally pretty pure pool of material.  Then you melt the fresh parts of those rods together into a new pool.  The material in this new pool will have far fewer impurities.  If you then draw new rods the fresh part of these rods will be even purer.  If you repeat the process carefully enough and enough times you can create super-pure samples of the material.  And there's a kind of a shortcut you can start using once you have drawn the first set of rods.

You build a special furnace that tightly surrounds a rod.  For most of its length it keeps the rod just below its melting temperature.  But the furnace has a hot spot where it heats the rod to just above its melting point.  If you draw a rod slowly thorough this special furnace you can slowly move a melted area from one end of the rod to the other.  The material melts and re-solidifies as the rod goes through the furnace and past the hot spot.  This concentrates the impurities in the melted part resulting in an increase in purity for most of the length of the rod.

Of course, you eventually end up with a piece on the end of the rod where the impurities are now concentrated.  You just cut that part off and throw it away.  By repeatedly applying this "melt then re-solidify" process you can make rods of extremely pure material.  Companies that specialize in this process have gotten quite good at producing super-pure materials at a surprisingly low price.

Making transistors and ICs depends on an attribute of some materials called semiconduction.  Elements like Copper that are "conductors" of electricity.  Electrons flow freely through them.  Elements like Sulphur are "insulators".  It is nearly impossible for electrons to move around in them.  And an electric current is just electrons flowing.

There are a few special elements called semiconductors that in some circumstances behave like conductors.  But in other circumstances they behave like insulators.  It is possible to manipulate these circumstances on the fly.  This means that at one time one pattern of conductor here and insulator there can be set up.  But a short time later a different pattern of conductor here and insulator there can be set up.  This capability lets ICs do magical things.  With modern ICs circumstances can be changed more than a billion times per second.

The key to this ability to manipulate circumstances is to start with a super-pure semiconductor.  Then a process called "doping" is used to introduce tiny amounts of specific impurities into very small selected areas of the surface of the IC.  Quantum Mechanics tells us that if we have the correct configuration of pure semiconductor and doped semiconductor we can create various kinds of electronic components.  These components can be wired together to perform arithmetic and all the other things a computer (and many other electronic devices) need to be able to do.  An IC is just a whole lot of these components wired together to fly in close formation.

The two most common semiconductors are Silicon and Germanium.  Germanium is relatively expensive and hard to work with.  Silicon, on the other hand, is cheap and much easier to work with.  It is the eighth most common element and is the primary component of beach sand.  Most ICs are Silicon based for cost and convenience reasons.  But Germanium or mixtures of Silicon and Germanium are used in some specialty situations.  And research leading to ICs made from other materials is an active area of investigation.

Okay, so how are ICs made?  The process starts with a rod of super-pure semiconducting material.  It is sliced into very thin "wafers".  Each wafer is processed individually to make many identical ICs at the same time.  Eventually each wafer will be sliced up into many individual ICs.  It is the middle "processing" part that is the interesting part so that's the part I am going to focus on.  And the processing part is an elaboration of the process I outlined above for making pictures.

In the case of the IC the part of the paper or clear plastic is played by our wafer of super-pure semiconducting material.  With a black and white picture we go through a single processing cycle.  We may go through three cycles to make a color picture.  Typically an IC will require many more cycles, sometimes dozens.  But each cycle is just an elaborate version of the processing cycle used for pictures.

At the start of each cycle we first spread some special goo on our wafer.  Then using a bright light, a "mask" (this is what provides the image), and a lens, we project an image onto the goo.  Then we go through a "finish" step.  With photography the finish step is always pretty much the same.  With a color picture we may need to go through a "red" cycle, a "green" cycle, and a "blue" cycle.  But all three cycles are pretty similar.  In the case of ICs things are a bit more varied.  Many more cycles may be necessary and the processing that takes place within a specific cycle varies much more widely than it does with our "red", "green", and "blue" cycles.  But it's the same idea.

The simplest IC process involves doping.  This consists of introducing small amounts of very specific impurities into the semiconductor.  These impurities come in two general classes, P-type impurities and N-type impurities.  The differences are not important to this discussion.  But if the correct pattern of P-type impurities, N-type impurities, and unmodified semiconductor is created in the top layer of our wafer then they work together to create a "gate" or other IC component.  In another process lines of material, most commonly silver or aluminum, are laid down to connect one component to another like a wire would.  In still another process trenches are "etched" into the silicon to separate one component from another.  There are a few more types of cycle but you get the idea.

Repeating myself, the details of each type of cycle are different but the general idea is the same.  A layer of goo, the specifics of which depends on the effect desired, is laid down uniformly over the surface of the wafer.  Then an image of a "mask" is photographed on top of it.  This changes the chemical properties of the parts of the goo that have been exposed to light.  Then another layer of goo is laid down.

This second layer of goo interacts with the first layer of goo .  It may either interact with the parts of the first layer that have been exposed to light or the parts that have not.  The interaction between the two types of goo may result in areas being "doped" with an impurity.  It may result in a line that acts like a wire being laid down.  It may result in a trench being etched in the material.  Whatever changes are necessary to create the IC are made processing cycle by processing cycle.

Typically a cycle finishes with a process that cleans away any unwanted remaining material.  This sets the surface of the wafer up for the next cycle.  Cycle follows cycle until all the layers of patterns necessary for the device to function have been put down on top of the supporting medium.  At this point each device is complete but it is still necessary to cut up the wafer to separate each IC from all the others.

It requires a high degree of knowledge and skill to design the "component layout" on the IC.  It requires a high degree of knowledge and skill to figure out what manufacturing steps (the processing cycles) are necessary.  Then machinery of extremely high precision must be designed, built, and operated to actually produce the ICs.  But I will leave those details aside and move on.  Except that I note that a "fab", a factory for manufacturing ICs, typically costs more than a billion dollars to build.

So the process is essentially the same for all ICs.  ICs with a lot of components require extreme precision machinery to manufacture.  But once the manufacturing process has been set up this has no effect on the "unit cost", the cost to make another identical IC.  And a tiny amount of material goes into each IC.  So material costs contribute little to the cost of the IC.  One factor that does effect unit cost is how many processing cycles are necessary to create the proper components with the proper connections on the IC.  But all ICs that require the same number of processing cycles cost essentially the same.

And the system is set up to make millions of identical ICs.  If you make a million ICs in a billion dollar fab then the fab cost-per-IC runs to $1,000.  That means each IC must be priced at more than $1,000 and that's expensive.  But if you turn out a hundred million ICs, a figure commonly reached over the five year prime lifetime of a fab, then the fab cost-per-IC is $10.  The rest of the process is relatively inexpensive so the fab can make a nice profit turning out chips with a wholesale price of $25 each.

And a fab is not worn out after five years.  Its just no longer a cutting edge facility.  But lots of ICs do not need to be made in a cutting edge fab.  So if the fab that is no longer cutting edge is shifted over to turning out these "commodity" ICs, and it frequently is, then it may now be possible to use that fab to profitably produce ICs that can be sold for far less, perhaps less than a dollar each.

The fact that ICs are made using photographic processes is the big secret to why the IC market behaves so differently than the market for most products.  That's the biggest of the IC secrets.  But there are other, lesser secrets so let's get into some of them.  Another secret has to do with component size.  As the industry has matured it has figured out how to make each component part of an IC, often referred to as a "gate", smaller.  This seems like a good idea in general.  But it turns out to have valuable benefits that are not immediately apparent.

Let's say we want to make an IC with a million components in it.  Then if each component is relatively large the resulting IC will be relatively large.  But there are practical limits on how large you want to make an individual IC.  Turn the relationship around.  If we are making an IC of a certain size then the component size tells us how many components it can have.  Remember, it doesn't much matter how complex the design is.  It still costs about the same amount of money to make it.  So as we reduce component size we can increase the component count and keep the IC size constant.  Components are now so small that some ICs that have tens of billions of components in them.

To recapitulate, fabs use rods of semiconductor, usually Silicon, as their starting ingredient.  These are sliced into thin wafers.  If we can use very small components then we can manufacture a small IC.  And we don't make a single IC from a single wafer.  Instead we divide the surface of the wafer into small rectangular areas and create an identical IC in each rectangle.  The entire wafer is simultaneously subjected to each processing cycle.  So one set of cycles produces many ICs at the same time.  As we make each IC smaller we can fit more of them on a single wafer.  So we can now make more ICs for the same amount of money.  This reduces the cost-per-IC.

In the race to continuously speed ICs up the speed of light is a limitation.  It limits how fast signals can propagate across an IC.  It is a fixed number.  If we make the components smaller then we can put them closer together.  This means the finished IC goes faster because propagation delays, the time it takes for a signal to get from one place to another, are smaller because the components are closer together.  This means we can make an IC faster without changing its design.  All we have to do is make the components (and the final IC) smaller.  For a long time this was one of the big secrets that explained how the industry could keep making their ICs go faster and faster.  They didn't do anything clever.  They just kept shrinking the component size and that reduced propagation delays.

In the early days making components smaller was relatively easy.  You just had to be more and more careful.  But it didn't take long for some interesting problems to start cropping up.  It turns out that you can't focus light infinitely sharply.  There is an inherent limitation based on the wavelength of the light you are using.  At first ICs were manufactured using regular white light.  White light was used because it was the easiest kind of light to come by.  And it worked fine until the components shrank to a certain size.  White light is composed of lots of wavelengths.  That blurred things.  When components were relatively large this blurriness could be ignored.  But at some point components shrank enough that the blurriness started to get in the way.

The obvious fix was to use a light consisting of a single color.  It was easiest to do this with red light.  And that worked for a while.  Red light was sharper than white light so it was an improvement.  But red light has a relatively long wavelength so red light became a no-go when components shrank some more.  Blue light has a shorter wavelength than red so blue lights were used until further shrinkage caused blue to also become a no-go.  Ultraviolet light has a still shorter wavelength so ultraviolet lights were switched in.  And they too worked fine for a while.

But that is pretty much the limit.  The obvious next step would be to go to X-rays.  They have still shorter wavelengths.  But masks by their very nature are full of extremely small details.  After all, they determine the exact shape and location of each part of each component.  Making a mask that has the requisite small detail and stops X-rays has so far proved to be impossible.  So ultraviolet is as good as it gets unless someone comes up with a true breakthrough.  So component sizes can't shrink to the point where they are too small to be manufactured using ultraviolet light.

And it turns out there is another, really strange problem that crops up as you make components unbelievably small.  That is Quantum Mechanics.  It turns out Quantum Mechanics giveth and Quantum Mechanics taketh away.  Quantum Mechanical makes it possible to design and create ICs in the first place.  But new Quantum Mechanical effects come into play when you make components unbelievably small.  Mostly we don't care how many atoms something contains.  That's because even something extremely small still has a fantastically large number of atoms in it.  If lots of atoms are involved then "bulk" Quantum Mechanical effects determine behavior.

But components have now gotten so small they may contain only about 10,000 atoms.  That is where "small scale" Quantum Mechanical effects come into play.  The fewer the number of atoms the greater the small scale effects.  One of these effects is called "tunneling".  If we have a thick insulating barrier the electrons stay on one side.  But if the insulating barrier is thin enough a certain percentage of the electrons "tunnel" through the barrier and end up someplace we don't want them to be.

How thin is thin?  Typically if we have a 10,000 atom component partly surrounded by an insulator only a few electrons will tunnel through the insulator.  But as the atom count drops the percentage of electrons that succeed in tunneling through rapidly goes up.  Components are designed so that they can tolerate a small amount of electricity straying into places where it isn't supposed to be.  But at some point the component stops working properly if the amount of stray electricity gets too high.  No one has been able to figure out how to design components that work properly when they are so small that that tunneling and other small scale effects have to be taken into account.  The solution so far has been "don't go there".  In other words, don't shrink component size to the point where small scale Quantum effects come into play.

These effects are well understood but neither designers nor manufacturers have figured out how to get around them.  The result is that the march toward smaller and smaller components has been drastically slowed.  And this means that the march toward faster and faster components has also been drastically slowed.  Moore's Law predicted that processor speeds would double about every two years.  And for decades they did.  But processor speeds are now little changed from what they were ten years ago.  The dual problems of shrinking components below their current sizes and the design problems caused by small scale Quantum Mechanical effects have combined to bring improvements to a nearly complete halt.

There has been some progress.  But mostly what has happened is that your economy model whatever now has about the same performance as the high end model of five or ten years ago.  This has resulted in PC sales being flat to down.  Why replace an old PC if a new one doesn't go any faster or do anything the older model can't?  For a while laptops and tablets were showing robust sales numbers.  Not so much now.  Again the performance difference between new models and those of a few years ago keeps shrinking.

And the smartphone is now ten years old.  In the early years the new models were much better than similar models from only two or three years previous.  But even the rate of improvements in smartphones has now slowed dramatically.  Manufacturers "maxed out" the performance of PCs first.  Then came tablets and laptops.  Now the same thing is hitting smartphones.

All these devices use ICs.  Power and space requirements were the loosest for PCs so manufacturers found them the easiest to max out.  Specifications are a little tighter for tablets and laptops so manufacturers of ICs for these devices took a little longer to max them out.  Specifications for smartphones are the tightest of all so ICs for these devices got the treatment last.  But they have now gotten the treatment.  The result is a new smartphone that is not much better than the smartphone of a couple of years ago.  Far too many people are now happy with a phone that is one or two generations old.  This is a growing concern among smartphone makers.

What is driving all this is are fundamental laws that can't be easily gotten around.  These limitations that are now inhibiting rapid improvement.  So what's the new,. new, thing?  People have been talking about a couple of things.  Current electronic devices are built out of ICs.  And ICs work by pushing electrons around.  How about replacing electrons with light?  It's the fastest thing in the universe and about three times as fast as electrons moving through wires.

Devices that use light instead of electrons are called "photonic" devices.  There has been a lot of work done in this area.  But no one has been able to figure out how to make a photonic device with more than a few components.  (Remember, processor ICs now have tens of billions of components.)  And light travels in a straight line while electrons are happy follow a bendy wire.  This and other problems have led to photonics being restricted to a very few highly specialized areas.  There seems little chance that will change any time soon.

The other area receiving a lot of attention is Quantum Computing.  Standard computers behave in a "sequential" manner.  They do this then they do that then they do the next thing.  Things proceed a step at a time in sequence.  There are limited exceptions to this.  There are such things as "parallel" computers.  And super-computers are "massively parallel".  But that is somewhat of a cheat.

Parallel computers consist of a few (normal computer) or a whole lot (super-computer) of regular sequential computers all hooked tightly together.  If all these sub-computers are contributing usefully to the solution of the main problem then then these parallel computers will solve that problem way faster than a standard sequential computer could.  This would make everybody happy.  The problem is to get all those sub-computers to contribute productively.  This is called the "partitioning" problem.

If a problem can be partitioned into a number of parts and each part can be assigned to its own sub-computer that can just focus exclusively on that one part while ignoring everything else then a parallel computer can get to the solution much faster than a sequential computer.  Well, there is another thing.  It can't take a long time to do the partitioning on the front or the assembly on the back.  Otherwise, the benefit of doing the middle part quickly will be lost to the cost of doing the front and/or back slowly.

In short, it turns out that partitioning problems is often hard.  In fact, there are lots of problems where no one has figured out how to partition them.  There are only a few super-computers in the world.  There are enough problems that have successfully been partitioned to keep them all busy.  But first let's take a look at a couple of problems that are suited to PCs and other computers that are capable of a relatively modest amount of parallelism.

One PC sized problem that is easy to partition is "rendering" all the stuff we throw up onto our computer screens.  Gaming PCs have a dedicated high end video card.  These cards can do a lot of partitioning and do it very well.  That is a good thing because these cards also feature multiple "graphics processors" that can and do work in parallel.  Gamers prize the improved video experience and are willing to pay for it.  A more limited version of this capability has already started moving down the food chain.

The video IC that came built in to the inexpensive PC I bought a couple of years ago is every bit as capable as were high end video cards from several years ago.  It has a significant amount of parallel processing capability.  And similar ICs are showing up in tablets and laptops.  If they are not already there I'm sure they will soon be found in smartphones.

At the other end of the process, many web pages have multiple components and sub-areas.  It is relatively easy for a good browser to partition all this out and assign each part to a separate sub-processor on a PC.  My main PC has 8 sub-processors.  Most modern PCs, even the inexpensive ones like my newer one, now have at least two.  This allows web pages to load faster.  Unfortunately, a good deal of what loads faster is advertising.  Oh well.

Unfortunately, few other PC tasks partition well.  The same is true for applications that run on laptops, tablets, and smartphones.  A few applications partition well but most don't.  That's why the industry has stopped talking about how many processes a machine can run at the same time.  A higher number generally doesn't result in an improved user experience.  Turning to super-computers.

Weather forecasting turns out to partition well.  That's why so many super-computers are used for weather forecasting.  Meteorologists have figured out how to partition a weather forecast into pretty much as many parts as they want.  If a super-computer with 10,000 sub-processors is available then the forecast is partitioned into 10,000 parts.  When super-computers with 100,000 sub-processors become available they will just partition a forecast into 100,000 parts.

Computer Scientists are getting better at figuring out how to efficiently partition more and more types of problems.  But it is slow going.  There are many more types of problems where they haven't figured out how to partition than there are problems for which they have.  Alas, a problem that does not partition well will run no faster on a super-computer than it will on a regular computer.

100,000 is a large number.  But there are problems that can be partitioned into millions, billions, or even vastly more parts.  Cracking encryption schemes is one of them.  If you can check billions or trillions of keys at once than you can find the key to a particular encryption that would take a super-computer with 10,000 sub-processors an impractically long time to crack.

Quantum Mechanics provides a possible way to solve some problems that would otherwise not be soluble in a reasonable amount of time.  Stories in the press about this sort of thing mix two quite different methods together.  One approach is good for problems where massive partitioning is helpful.  The other approach  depends on Quantum Mechanics but is not appropriate for massively parallel problems.   I will take them separately.

What if it was possible to arrange things so that billions and billions of molecules are all working at the same time to try all the combinations of say potential cryptographic keys?  There are more than a billion molecules in a single drop of water.  So a beaker full of a magic elixir could easily give you a billion times a billion cracks at finding the correct solution.  And the idea is to have all those molecules each try a unique solution at the same time.

Various people have suggested it is possible to mix up special chemicals, say customized DNA, and get them to do just that.  Each molecule would be different so it would test a different possible solution.  Then let's say it is possible to set things up so that the molecule that finds the correct solution fluoresces but all the other molecules don't.  It would then be possible to pick just the "correct" molecule out of the soup, examine it, and from there go straight to the correct solution.  Instead of taking forever it might take perhaps an hour or so to set everything up and get the answer.  That's the idea but no one has come close to making it actually work.

The other and quite different approach depends on Quantum Mechanical phenomena called "superposition" and "entanglement".  It is not appropriate for massively parallel problems.  But it is appropriate for a class of problems that regular computers find hard to solve.  Regular computers are carefully designed so that a particular bit is always either a zero (off) or a one (on).  There are no other possibilities as computer design depends on this.

But superposition and entanglement create situations where something called a Qubit is not either zero or one.  It is in some intermediate indeterminate state.  People have figured out how to use this very indeterminacy to solve certain problems that would otherwise be very hard to solve.  This has the potential of allowing "Quantum" computers to solve problems susceptible to this approach much faster than regular computers operating sequentially.

The theory works just fine.  But the most capable Quantum computer that has been actually built so far can only perform calculations involving a few Qubits.  At this point a simulation of such a small a Quantum computer running on a regular sequential computer would probably get the answer faster than the actual Quantum computer would.  If people can figure out how to build Quantum computers that can calculate using hundreds of Qubits then they would have something.

Lacking some kind of surprise breakthrough I don't expect optical computers or Quantum computers (or chemical solutions) to go anywhere in the near future.  There is nothing on the horizon at this point.  But that's the nature of surprise breakthroughs.  They can arrive out of the blue at any time.  Certainly if a surprise breakthrough materialized I would be forced to change my mind.  I think that's unlikely.  And the same thinking also applies to the prospects of seeing a substantial increase in the speed of standard computers any time soon.  In all likelihood it's not going to happen.

So the likeliest eventuality is boring times ahead in this area.  I see exciting times ahead elsewhere, specifically in biology and medicine.  But that's for another day.  In the mean time you can bore all your friends by wowing them with your deep understanding of computer chips starting with "they are actually called Integrated Circuits".  (At least, that's my fantasy, and we all need fantasies.)  And you can throw shade my way if my pessimistic predictions turn out wrong.  That, at least, is something to look forward to.

George H. W. Bush

This is technically not an obituary.  As I write this Mr. Bush is still alive although his health is poor.  And his beloved wife of 73 years, Barbara, has recently passed away.  It is not uncommon for the surviving spouse of a couple who have been close and who have been together for a long time to follow shortly thereafter.  So that's the reason for this post but not the point of it.

A case can be made (and I'm going to make it) that Mr. Bush (unless otherwise indicated I am referring to George H. W. "Bush 41" Bush and not George W. "Bush 43" Bush) had a front row seat to two extremely important inflection points in society.  In one case I am talking about the evolution of the Republican Party.  In the other I am talking about the Middle East.  I will address the political inflection point first.

I did a post on the history of the two main political parties a couple of years ago (see "http://sigma5.blogspot.com/2016/05/a-brief-history-of-political-parites.html").  But that was a "30 thousand foot" overview and focused primarily on the pre-1960 era.  Here I am going to focus on the period from 1960 on and spend most of my time on the Republican party.  But let me start a little earlier.

Both FDR and Harry Truman were wartime Presidents during World War II.  They fought the war from the White House.  But Eisenhauer was a WW II vet and a war hero.  Kennedy was also a vet and a war hero.  Johnson was not a war hero but he too was in uniform during the war.  And so was Nixon. And so was Ford.  The streak was broken by Carter who was a Navy man but not a WW II veteran.

But it resumed with Reagan and continued with Bush.  None of the post-Kennedy occupants of the White House saw combat until we get to Mr. Bush.  He was a fighter pilot who flew from the decks of aircraft carriers in the Pacific theater.  He was also the last World War II generation President.  As a group the World War II generation Presidents racked up 28 years in office.  That's an inflection point but I am going to fold it into my larger discussion of political parties.

If you review the transcripts of the Kennedy/Nixon debates in 1960 you will find that the policy differences are modest.  Generally speaking the Republican party was more fiscally conservative but neither party could accurately be described as spendthrift.  There was a big discussion about which party was more anti-communist but again both parties were very anti-communist.  There was a bigger difference on social safety net issues like Social Security (Medicare and Medicaid did not exist at the time).  But the Republicans had given up on trying to repeal Social Security or even scale it back.  And the issue of civil rights is quite instructive to the modern eye.

A lot of Republicans of the era were very socially progressive.  A lot of Democrats hailed from the "Solid South" and were very reactionary.  Kennedy won and had solid Democratic majorities in both the House and Senate but was completely unsuccessful in advancing any civil rights legislation.  All that changed when he was assassinated.  Johnson was a southerner and a master political manipulator.  He decided he wanted to pass Civil Rights legislation so he did.  HIs key to success was in talking key southern Democrats into not blocking the legislation.  Then he joined other Democrats together with liberal Republicans to pass the legislation.  He used a similar coalition to pass Medicare and Medicaid.

Johnson was followed in 1968 by Nixon.  By modern lights his political agenda looks down right liberal.  He passed Clean Air and Clean Water legislation.  He created the Environmental Protection Agency.  Johnson was considered a spendthrift for passing budgets with deficits that look tiny to modern eyes.  Nixon balanced the budget in one year and ran very small deficits the rest of the time.  This fiscal conservatism may have led to "stagflation", a stagnant economy with a relatively high level of inflation.  The only President since to deliver a balanced budget is Clinton.  He put Federal spending into surplus but George W. Bush immediately reversed that.

Before Nixon got into office in 1968 the Republicans decided to try something new.  Instead or running a moderate like Nixon in '64 they went with the very conservative for the time, Berry Goldwater.  He got pasted.  That was supposed to kill the conservative movement.  As we now know, it didn't.  It just delayed things.  In 1976 the moderate Ford ran against the moderate Carter.  Carter won.  This caused Republicans to take another look at the whole "conservative" thing.  Reagan, a conservative standard bearer, did well in the '76 primaries against Ford.  This was surprising because technically Ford was an incumbent.  And this sets us up for the showdown of '80.

In 1980 Mr. Bush had a resume to die for.  As mentioned before he was a "war hero" WW II vet.  He also had tons of experience both as an elected official and as a government bureaucrat.  So he had an extremely solid record to run on.  He had served in the House for two and a half terms.  He left his House seat early to become the Ambassador to the United Nations.  From there he moved on to chair the Republican National Committee, serve as Envoy to China before we had an Ambassador, and as director of the Central Intelligence Agency.  These are all high profile governmental positions with a substantial political component.  They showed him to be a committed Republican who knew how to win elections but who also had a high degree of bureaucratic and diplomatic skill.  What better resume could you ask for?

He ran as a moderate Republican and got trounced in the primaries by Reagan.  Reagan knew how to win elections.  But he was well known as a "hands off" administrator who was good at the big picture but poor with the details.  The primary season told Bush in no uncertain terms that he was out of step with the modern Republican voter.  He still had ambitions so he accepted the number two slot and was a loyal Vice President to Reagan for 8 years.  And his gamble paid off.  He secured the Republican nomination in '88 and went on to win the election.

But he had a big problem.  He was still a moderate and frankly didn't understand how conservatives thought.  So he brought in the former governor of New Hampshire and a doctrinaire conservative, John Sununu, to run the Whitehouse staff and did whatever Sununu told him to do.  But his heart wasn't in it.  This resulted in a presidency with a muddled agenda (something about "a thousand points of light" that to this day no one has ever been able to figure out).  And it was obvious to everyone he was faking it.  So conservatives didn't really trust him.  And that let a canny pol from Arkansas named Bill Clinton beat him in '92.  This defeat led Republicans to the conclusion that moderation was for losers.  Since then they have never looked back.

At about this time a canny operator named Newt Gingrich figured out there was a lot of money out there.  A lot of very rich people are very conservative and they are willing to spend a lot of money if they think they can get the results they desire.  So starting in the late '80s Gingrich made the rounds and raised a lot of money.  And he invested for the long run.  He put the money into political infrastructure.  He funded think tanks to come up with slogans and position papers.  He put together schools to teach people how to run campaigns.  He put together schools to teach candidates how to run and win.  He put together groups to find candidates who could front a modern campaign.  They had to look good on TV and know how to stay on message.

Democrats did well in '92 but Gingrich was ready in '94.  By then all this machinery he had put in place was up and running.  Republicans fielded good looking candidates with messages tailored for the TV era.  He even created a national theme, "the contract for America".  It consisted of a bunch of poll tested slogans.  Most of them (i.e. a "balanced budget" constitutional amendment) were bad ideas in reality.  But he didn't promise to implement them.  He just promised that his candidates supported them.  And it worked.  A lot of Republicans won off year elections.  They didn't actually implement anything but they talked about it a lot.

And at the same time conservative talk radio emerged.  This later morphed into the Fox News TV network, innumerable blog posts and web sites, etc.  All this provided a fan base that consistently supported conservative ideas and candidates.  And the whole thing was based on messaging and not actual results.  Mr. Bush was someone who was good at getting results but not so good at messaging.  He was also grounded in the real world with all the complexities, restrictions, frustrations, and inconveniences, that entails.  Catchy slogans win campaigns.  Nuanced arguments lose them.  Mr. Bush was not the last moderate Republican to be his party's standard bearer.  That honor goes to another World War II veteran and war hero Bob Dole.  But Dole also lost to Clinton in '96 and that was that.

Now let me move on to Mr. Bush and the Middle East.  One of the things that caused Carter to lose the '80 election was the Iranian revolution.  And specifically when the revolutionary government kidnapped all of our Embassy staff.  Carter made some bad moves (her should have immediately pushed back at Iran, hard) and had some bad luck (a rescue operation was botched).  Reagan capitalized on this.  So did Saddam Hussein.

In an attempt to make Iraq the preeminent Arab power he invaded Iran.  (Iran is Persian, not Arab.)  This resulted in a war that lasted nearly a decade and did great harm to both countries.  Iran was by far the larger country.  It had substantially more land area, a much greater population, and a much larger economy.  It should have rolled over Iraq with little trouble.  But the revolution had wiped out the Iranian military leadership.  The "human wave" and other extremely primitive tactics employed by Iranian religious leaders allowed Iraq to just barely hang on while costing Iran dearly.  Eventually both sides gave the enterprise up as a bad idea and signed a UN brokered truce.

But this left Iraq in bad shape.  It had essentially bankrupted itself in it's ill fated war with Iran.  In 1990 Saddam decided the solution to this problem was to invade Kuwait.  Kuwait was the opposite of Iran.  It was a small country in size and population but it had a whole lot of oil.  Militarily, this worked out very well in the short run.  The Iraqi army, now battle tested from the Iran war, rolled in and rolled over Kuwait in the blink of an eye.  Bush's initial response was "not our problem".  But he was quickly convinced that he had to act decisively.  And to his credit once the decision was made he did a bang up job.

He first assembled a coalition.  It included the U.S. and its traditional allies.  But it also included a substantial number of Arab countries.  Outside of Iraq, this was one of the least controversial wars.  Everybody was on board with the idea that we were the good guys and that Iraq was the bad guy.  The military offensive was well planned and a large careful buildup took place.  When combat commenced it was a smashing success.  It is generally referred to as "Operation Desert Storm" but it could also be called "the hundred hour war" because that's how long the active phase of the war lasted.

This event has almost completely disappeared from view so let me spend a little more time with it.  It featured some singular characteristics.  The first one is the one I have already noted.  It was a coalition effort.  An extremely wide assortment of countries signed up.  Efforts have been made to duplicate this since in Afghanistan (modest success), Iraq (failure), Libya (again, modest success) and elsewhere.  But Bush is the last US President to actually pull it off.

During the run up to the war I got a map of the region.  To my untutored eye the obvious strategy was what is called a flanking maneuver.  This is where you go around one or the other end of the other guy's army.  To the west of Kuwait is this giant desert.  It seemed obvious to use it to get around to the back of the Iraqi army (which by then was in Kuwait), cut them off from resupply from Iraq, and then attack them from the rear.

The US and allied forces achieved tactical surprise by conning the Iraqis into believing they would do something else, something quite a bit harder militarily to pull off.  The Iraqis fell for it and were completely surprised when the sweep through the desert to their rear took place.  Of course it almost didn't matter.  The US and allied forces were so effective that they almost completely destroyed the Iraqi forces arrayed in front of them even though this was technically a feint.

The other very impressive achievement was how happy all the allies were.  Every force from each part of the world was given a role in the war.  This let them shine.  Each country had no problem making their own forces look good.  So at the end of the war all of the allies were happy with how things came out and how they had been treated.  It was a masterful diplomatic performance.

Unfortunately, there were other not so positive outcomes.  This war featured some very large tank battles.  And what these battled did was what Pearl Harbor did to battleships.  It made it very apparent that the day of the tank was past.  The US and allied tanks did fine but that was because they had all this support from other units.  Pretty much every tank the Iraqis had, and they had a lot, was destroyed.

But only a few of them were destroyed by tank on tank combat.  Airplanes dropping smart bombs turned out to be extremely effective.  As did various shoulder mounted anti-tank weapons.  As did several other weapons.  A Predator drone mounting a Hellfire missile is a great anti-tank weapon.  And the drone/missile combination is much cheaper.  And the tank is ineffective at dealing with drones and missiles.  Nothing has happened in the more than 25 years that have now passed to change this calculus.  But US taxpayers are still buying new tanks for fantastic amounts of money because of pork barrel politics.

The main problem with this war is that it didn't actually settle much of anything.  The old government went back into power in Kuwait and returned to its old ways.  Saddam hung on in Iraq and stuck to his old ways.  And pretty much nothing changed anywhere else in the Arab world as a result of this war.  And one reason for this is something that Bush botched.  And it's something that is often distorted by partisans who discuss it.

As I said, the US and its allies succeeded in flanking the Iraqis.  That means that at hour 100 US forces were in control of the border between Kuwait and Iraq.  From there they could have pivoted and gone into Bagdad, the capital of Iraq.  This is the alternative that is repeatedly brought up and discussed ad nauseum.  Doing so would have made our Arab partners very angry as we promised that our mission was to eject Iraq from Kuwait and not regime change in Iraq.  And we now know what happened when we did go to Bagdad later.  The problem with only discussing this alternative is that it ignores other alternatives.

Once the US military was established and secure in the Iraqi army's rear a decision was made.  That decision was to allow unarmed Iraqi soldiers to return to Iraq unmolested.  They had to leave their equipment behand but they did not risk death or internment by heading for home.  This meant that Saddam lost a lot of equipment but his personnel losses were far smaller than they could have been.  Iraqi soldiers retuned in numbers sufficient to reconstitute the Iraqi military, a key component in Saddam's power structure.  We could have instead rounded up all of those Iraqi soldiers and put them in POW camps for six months or a year but we didn't.

The other decision Bush botched was his failure to institute a no-fly zone over Iraq after the ceasefire.  Saddam's grasp on the reins of power was very fragile in the immediate aftermath of the Kuwait fiasco.  But with his soldiers and his ability to overfly Iraq with jet planes and helicopters he was able to put down the resistance to his regime and retain power.

It is important to understand that there are three major sub-populations in Iraq.  There are Kurds in the North.  There are Shiites in the East (along the Iranian border) and there are Sunnis in the West (along the Syrian border).  Saddam was a Sunni but the largest sub-population were the Shiites.  So he represented a minority government.

As soon as the outcome was obvious the Kurds and the Shiites rose up in revolt.  (They did this in part because we encouraged them to do so in the run up to active combat.)  Saddam was able to put both down using air power.  The US could have bottled the Iraqi army up in Kuwait and enforced a no-fly zone.  If it had it is unlikely that Saddam would have been able to put both factions down.  Most likely he would have been able to put neither down.  Meanwhile, we could have maintained a position of neutrality.  "These revolts are an internal Iraqi matter."

The situation in Iraq quickly fell apart after the end of active combat and this substantially weakened Bush in a manner similar to what had happened to Carter a decade earlier.  And we are living with the international consequences if Bush's actions internationally just as we are dealing with Bush's political failure domestically.  The Republican party has continued to move further and further to the right.  It has also moved further and further away from facts and reality.  But the machine Gingrich created is so effective at delivering political victories anyhow that there is literally no pressure within the GOP to move to more moderate and reasoned positions.

Similarly, the Middle East has continued to deteriorate.  We don't know what would have happened if Saddam's regime had tumbled in 1990.  We certainly wouldn't have had the disastrous war in Iraq that started in 2003 and continues to this day.  Many people believe that the real reason George W. Bush went into Iraq was to prove he could fix what his father had screwed up more than a decade earlier.  Certainly no one has succeeded in assembling and managing a coalition like the one Mr. Bush assembled.  Every subsequent coalition is measured against it and inevitably found wanting.

The 2003 Iraq war has cost the US fantastic amounts of blood and treasure directly.  But our continued involvement has also meant that other options and initiatives have either been impossible or far more costly than they otherwise would have been.  So the Middle East continues to be an extremely unstable part of the world that seems most effective at destroying the dreams of those who live there, those who engage with it, and those who would like to have nothing to do with it but find they can't.  There is an old story from the world of business that ends with "the promotion of the uninvolved".  The big winners in the Middle East seem to be countries like China who have been for the most part uninvolved.  Everybody else has to a greater or lesser extent been a loser.

Not for Identification

I have my original Social Security card.  I applied for it and got it when I was in High School and was about to get my first real job.  Printed at the bottom is "FOR SOCIAL SECURITY AND TAX PURPOSES -- NOT FOR IDENTIFICATION".  This "Not for Identification" business is often misconstrued.  It is usually taken to mean that your Social Security number is not some kind of ID or identification number.  That's not what is meant.  Because the very same card also says "For Social Security and Tax Purposes".  What is actually meant is that  the card itself is not to be used for identification.  That's because the card itself was not designed to provide positive identification.

And that is a nice bridge.  This post is an update to a previous post about Positive Identification (see http://sigma5.blogspot.com/2016/04/positive-identification.html).  If you reread it you will find it links to previous blogs.  I have been weighing in on privacy issues for some time now.  Although the trend continues the details keep changing.  And there have been several important changes since my last post over two years ago.

I spent some time in the post I linked to on the fingerprint identification system Apple had implemented.  You placed your finger on the correct spot on the phone and it would read your fingerprint.  If it recognized it the phone would unlock.  The phone positively identified you by analyzing your fingerprint.

Apple has since moved on to facial recognition.  Smart phones have had cameras in them for some time.  Even my old fashioned flip-phone now comes equipped with a camera.  My phone doesn't have enough computer power to do facial recognition but newer iPhones do.  They take a picture and supplement it by measuring other characteristics of your face.  If it's a match you have been positively identified and your phone unlocks.

It is important to recognize there are limitations.  First of all, what the phone is doing is matching your current face to one that was identified to the phone during the phone's "setup" procedure.  So the phone knows that, relatively speaking, you are you.  But absolutely speaking it doesn't know who you are.  Also, it is possible to fool the recognition system.  This was true of the older fingerprint system and it continues to be true of the facial recognition system.  But everybody expects that the process will be updated and enhanced as time goes by so that it becomes harder and harder to do this.  Even now, it takes a lot of skill and effort to fool either system. It takes deliberate effort and a considerable amount of knowledge.

But the identification is only relative.  The phone recognizes you as the owner because it has been told that you are the owner.  But it doesn't know who you are.  And the same is true more generally.  The analysis I did pointing out that there is really no way the current system can absolutely tie a specific person to a specific birth certificate is still true.  And using CODIS or some other DNA based system to create an absolute connection continues to get easier and easier from a technical perspective.

CODIS has added 7 additional STRs in 2017 so it is now using using 20.  But nothing has moved from a political perspective. There are still tight restrictions on what gets put into a CODIS database.  There has been no move to CODIS newborns, for instance.  And if smartphone makers are thinking about using DNA for identification I don't know about it.

But there has been big developments on the identification front.  These developments are with respect to relative identification.  But they are so pervasive and extensive that they have rendered the difference between relative and absolute identification moot.

We have known for a long time that tech companies were collecting a lot of data about us.  Google famously saves every search ever made.  Initially this was supposed to be so they could analyze it and optimize their algorithms to give you a more useful answer.  But it soon became apparent that they were not just using it as some sort of anonymous pile of data that helped in search optimization.  They were using it to identify each and every one of us.  They would then develop a profile of each of us which they would sell to advertisers.  The idea was this would allow advertisers to narrowly target their marketing to just the people most likely to be interested in the product.

As an example of how this worked I once searched Amazon for shredders. I didn't need one but my mother, who did not have a computer, needed one.  I soon started to notice that wherever I went on the web an ad for an Amazon shredder would soon follow.   This behavior persisted even after I went back and bought a shredder from Amazon for my mother.  That was modestly entertaining and little or no harm was done to me or anybody else.  So this kind of behavior didn't seem all that bad either to me or pretty much anybody else.  And that's the kind of mental model people had about what was going on.

Okay, so the NSA was sweeping up all this data.  That was the government and they shouldn't be doing that sort of thing.  At least so went the argument by myself and many others.  (There were, of course, lots of people who were okay with this and other intrusive behavior by the government.)  But my point is that if we put this sort of thing on a scale the NSA was generally considered closer to the "bad" end than people like Google.  And a large number of people were of the opinion that it was all fine.

Then the 2016 election happened.  And over time we have learned a lot more about what tech companies in general and Facebook in particular have been up to.  And more and more people have become very angry.  There is something in the business called an EULA, an End User License Agreement.  We have all had to deal with them.  They are long documents full of impenetrable legalese that even expert lawyers can often not make sense out of.  In the backs of our minds we are all pretty sure that there is stuff in them that we would not like if we understood what it was and what it meant.  But you can't get around EULAs.  Everybody uses them so you can't just go to the next company.

And they are all bad to one extent or another so it is impossible from a practical point of view to go with the company that has the least onerous EULA.  They are what is called a "coercive contract".  At least one party (us) is effectively powerless in the negotiation.  So we don't read them.  We just click the "Agree" button and move on.  We have all made a deal with the devil.  If we are going to have access to these compelling tools and gadgets we are going to have to put up with a certain amount of stuff we would rather not have to.  But if we sign up for Facebook, for instance, we expect the bad behavior to be confined to the relationship between us and Facebook.  And we did after all "Agree" to Facebook's EULA.

But we have found out that it is far worse than we thought.  We expect Facebook to use what it has learned about us to try to get us to sign up for more Facebook stuff.  And we expect Facebook to sell profile information to advertisers so that Amazon can pester me with ads for shredders.  But we expect it to stop at that.  But it turns out it didn't.

Facebook has a program that allows companies to build and run applications within the Facebook environment.  Those applications can harvest information.  And the information is not limited to what we tell the application.  A popular type of application is a cute quiz.  "How much do you really know about Star Wars?", or about pop stars, or fashion, or cars, or whatever.  Certainly these quizzes can be constructed to collect information that advertisers would find valuable and, therefore, pay money for.  That sort of thing seems fair.  But these cute applications (they are designed to be cute so that they will be popular so that lots of people will install them) are not limited to harvesting the data you provide while answering the quiz.  It turns out that they get access to all the information Facebook has on you.

That's bad.  I'm pretty sure it's legal because they would be idiots to not put the necessary language into their EULA.  But this "they get all the that Facebook has on you" degree of badness is just the first and least bad level of badness.  It turns out they also get access to what Facebook knows about your friends.  That's the second level of badness.  And this behavior is probably legal because of the Facebook EULA.  There is probably some language in there saying this sort of thing is legal.  But it turns out there is a third level of badness.

Remember the bit about how I was getting those Amazon shredder ads everywhere.  I did my search not in Google or Facebook but on Amazon's web site.  So only Amazon knows I did the search.  What's going on is that advertisers and the companies they do business with like Amazon and Facebook share data in networks.  Amazon shared the information that I had done a search on shredders to its network partners and they placed "Amazon shredder" ads on web sites that I later visited.

It turns out that Facebook does the same thing.  They are part of these information sharing networks so they have access to what's happening on sites that are far away from anything Facebook owns or operates.  So Facebook has a profile on people like me who have NEVER had a Facebook account.  And people like me have never signed an EULA with Facebook or any of the application providers Facebook hosts on their platform.

We have slowly found this out as revelations have trickled out as people have looked at how the 2016 election actually played out.  Facebook has a "commercial" interface so that people who want to make a buck can build and run an application to run on the Facebook platform.  But they also have an "educational" interface so that people doing research can also have access to the Facebook platform and Facebook data.  This latter interface is given wider latitude due to it's presumably non-commercial and beneficial intent.

A Cambridge University Don (professor - Cambridge is in the United Kingdom) took advantage, and as we now know, allowed a company called Cambridge Analytica to harvest vast amounts of data about Americans from Facebook.  First it was data on 50 million people.  Then it was data on 87 million people.  The actual number is and probably will never be known.  And we know they sucked a vast amount of data out of Facebook.  And we know we don't know where it all ended up.  Facebook at one point asked for it all back.  Fat chance.

And that's just Cambridge Analytica.  There is certainly no technical reason dozens or hundreds of others could not have done the same thing.  And we know that Cambridge Analytica was able to harvest data on users who signed up for one of the applications they put together.  They all signed Cambridge's EULA.  But we also know that this group numbers less than a million.  We get to 50 and later 87 million because they were able to collect data on "friends" then "friends of friends" and so on.  All these people at least signed the Facebook EULA.  But were they also able to collect data on people like me, people who have never signed up for Facebook?  The answer is unclear.

So it turns out that Facebook knows a lot about each of its users.  The NSA would probably like to know as much about people as Facebook knows.  So Facebook can positively identify its users.  The positive identification is relative.  They can't tie a specific user to a specific birth certificate.  But they know so much about that person it doesn't matter.  They can more positively identify a person that a bureaucrat at the bureau that issues driver's licenses, or voter registration cards, or passports.  They can do a better job of positively identifying a person that the government can.

And Facebook has taken all the heat.  But the same is true of Google.  Remember they have all that search history (and lots more).  It is probably also true to a lesser extent of Apple and Microsoft and a number of other companies.  (So far the spotlight has shown brightly on Facebook and left the others in the shadows.)  We now have a completely new method of personal identification, one that I did not imagine as recently as two years ago.  You can now be positively identified by your online profile.

Most of us now live on our smartphones.  (Again, I am an exception.)  Back in the stone age of personal computers Intel was going to put a serial number in their 80486 chip.  The privacy advocates of the day talked them out of it and treated this as a big victory for privacy, the ability to use computers and remain anonymous.  But while they won this tiny skirmish they lost the war.  There are hundreds of numbers on smartphones that can easily be accessed that provide a unique identification for a specific device.

Microsoft even pioneered a process for creating a GUID, a Globally Unique IDentification.  The process guaranteed that it would never generate the same number twice.  Microsoft uses GUIDs all over the place in their software.  If you can get access, and it is easy to do, to any of these GUIDs you can uniquely identify a device like a PC (I do have and use those) or a tablet running a Microsoft application or a smartphone running a Microsoft application.  And creating GUIDs is not that hard to do.  So you can't avoid the problem by avoiding Microsoft.

You might just as well try to have a presence on the Internet without ever "Agree"ing to an EULA.  Other vendors have figured out how to generate their equivalent of a GUID.  Then there are all those numbers that behave like a serial number.  Every network interface has a MAC address.  It is effectively a serial number.  Lots of software uses a "License key" or an "Activation key".  They are both effectively serial numbers.  IP addresses often behave like a serial number.  The list goes on and on.

Companies like Facebook can harvest GIUDs and MAC Addresses and license/activation keys and tie a specific profile to a specific device or small list of devices and build up a positive identification.  And they can and have done it better than the government.

I am pro-privacy.  But I am also realistic.  I have argued in a number of posts that when it comes to privacy the horse left the barn long ago and that there is no effective way to get the horse back.in the barn.  And even if you did the horse is likely to escape no matter how much effort you put into horse-proofing the barn.

I think we need to accept the fact that privacy is not possible any more.  That means we need laws, regulations, and social norms to constrain how we and our institutions and our businesses behave in a world where the technology exists that permits the powerful to peer pretty much anywhere they want.  There is technology like encryption that can close some of the doors that let the powerful or just the technologically sophisticated in.  This sort of thing is helpful and should be encouraged.  But it doesn't protect us from those who have access to the inside like Facebook and whoever they license or enable.

This means that we must outlaw behavior that is technologically possible and often easy to do.  We must also demand a high level of tolerance when it comes to what people are permitted to do.  No matter who you are there are some behaviors other people engage in that you don't like, in some cases you don't like it a lot.  As a society we must find ways to constrain the actions you are allowed to take in response to the behaviors you dislike.  If you violate those constraints you need to be punished.

This at first seems like a new and unnatural way for people to behave.  But a thousand or so years ago we all lived in small villages.  It was then easy to look into a doorway as doorways frequently had no doors.  But social morays constrained people from looking into doorways or acknowledging that certain behaviors were taking place in some public place.  This was all enforced by shunning and other social actions.  Society is now a global enterprise encompassing billions of people, millions of companies, and thousands of governments.  Social norms alone are not going to work for us now.

At first blush it might sound like I am advancing a Libertarian agenda.  And that is half right.  Libertarians believe that legal prohibitions on behavior should be kept to a minimum.  That's the part where my position coincides with the libertarian point of view.  Where I differ is that there also needs to be legal prohibitions that outlaw violating the new norms.  The government must step in, sometimes in a heavy handed way, to stop people, organizations, and institutions from doing things they want to do, namely going after people they disagree with and prohibiting behavior they don't like.

Facebook is a rich and powerful corporation with many fans.  It will take a very powerful institution to be able to force them to change their behavior.  They won't do it on their own.  The only institution capable of doing that is a large powerful government doing intrusive things.  And that kind of government is one Libertarians vehemently oppose.

We are a very divided society right now.  And at its most fundamental level what divides us is our vision of how things are and how things should be.  Until we come to a common vision the sort of things I am talking about are impossible.  Even if a common vision were possible the issues I am talking about will be very hard to resolve.  The most likely result of this current division is gridlock with no progress in any direction.

Maybe that is for the best.  It gives all of us time to think about these issues and decide what we think about them and where we stand.  But if recent events tells us anything they tell us that instead of thinking about these hard problems we will chase the next shiny object and the one after that.  Then we will wonder how we got into another fine mess.

Liberal Democracy

This is a subject that I have been planning on getting around to for some time.  In the notes I had assembled on the subject I listed "Democracy 101" and "Government 101" as possible titles.  The post was always supposed to feature a discussion of the phrase "liberal democracy".  In the end I decided that this phrase is central to the discussion.  So I went with it for the title.

I also labeled this post as a "culture" post rather than the more obvious label of "politics".  I decided to do that because of the importance of the subject to our culture as a whole.  It is too important to be pigeon holed as merely politics.

And the immediate event that caused me to transform this subject from a potential to an actual blog post was the recent widespread coverage of Sinclair Communications.  Sinclair owns close to 200 local TV stations and would like to own more.  They are working on a plan to acquire the chain of TV stations owned by a company that grew out of the Chicago Tribune newspaper.  Newspapers have long been considered a dying business so lots of owners have been busily diversifying.  That's what the owners of the "Trib" did.  They bought a bunch of TV stations.  Sinclair wants to buy them.  If the acquisition goes through Sinclair would end up owning over 200 TV stations whose combined reach would extend to cover 80% of the population of the US.

That level of ownership concentration sounds bad.  But that's not what Sinclair got in trouble for.  The people who control Sinclair are, like many wealthy people today, very conservative.  They've got it and they want to keep it.  They are adamantly opposed to anything that might possibly threaten their position.  And in the case of Sinclair their ownership of so many TV stations puts them in a position to actively do something about it.

A lot of people get their news from their local TV station.  The news is supposed to be sacrosanct.  There is an unwritten (or, in the modern era, unenforced) rule that says management does not interfere with news content.  Sinclair has instead been influencing what local news on their stations has to say about some issues.  They have long since moved beyond just making hiring and firing decisions, a power that gives them considerable control over news content.  They also have been sending edicts out for years now saying "this must appear on the air".

Some stations have been burying these segments in low viewership parts of their schedule.  Other stations have been happy to air them in prime time.  But all of them have been forced to respond in some way.  A few stations have managed to avoid airing a few segments.  But for the most part the segments got aired.  The segments Sinclair was sending out were always presented as if the local staff decided they should be aired.  This was not the case.  But the segments were obviously not locally produced.  That, at least, gave the local staff a fig leaf of cover.

Having gotten away with everything so far Sinclair decided it was time to up their game.  They sent out a script and instructed one or more of the local news staff to read it on the air as if it represented that person's own opinion.  The local news staff was forced to go along with this charade.  Like all the other material Sinclair provided the latest statement had a decided conservative slant.  In general all of the pieces were designed to support and enhance the ability of Sinclair's owners to maintain and expand their power and wealth.

In the news business it is fine to have an opinion.  But there is supposed to be a separation between news, fact based information, and commentary, opinion which may or may not be solidly grounded in fact.  In a newspaper there is the "news hole" and the "opinion page".  Broadcast news operations are supposed to maintain a similar separation.  There are two problems with the Sinclair material.  First, it has been presented as news when it is actually opinion.  Secondly, they disguised the source of the information.  If Sinclair had said "the following represents the opinion of the management of Sinclair Communications" everything would have been fine.  But they didn't.

This Sinclair behavior was well known within the news business.  In theory competitive pressures should have caused Sinclair's competitors to make a big deal of the story.  They didn't even though claiming "we are the truly independent people you can relay on and they are a bunch of lying corporate toadies" should have been good for business.  It is important to remember that the news business is a business.  And it is owned almost completely by wealthy and powerful individuals and corporations.  Calling out Sinclair's shortcomings might have caused one's own shortcomings to come under scrutiny.  And that scrutiny might have turned up embarrassing information.  For this or other reasons Sinclair's competitors kept mum.  So the public in general and Sinclair viewers in particular stayed blissfully ignorant.

That is until a few days ago.  Sinclair might have gotten away with this latest stunt.  Initially it drew no coverage.  But then a web site called "Mashable" put out a short video featuring video of anchor after anchor on station after station robotically repeating exactly the same words.  They mashed up the video so that we could see many anchors on many stations simultaneously saying exactly the same thing.  The video went viral and the whole sorry business could no longer be ignored.

Or could it?  My local paper, the Seattle Times, at first didn't cover it at all.  Then they did a story not about Sinclair but about Trump's tweet about Sinclair (and a bit about the pending acquisition).  Finally the story blew up to the point that it couldn't be ignored.  So they did a long in depth piece about the whole thing.  That was yesterday.  Today, the follow up consisted entirely of some nice letters to the editor.  I expect the story to be gone by tomorrow and never reappear.  Okay, that was way too long an introduction.  But I feel better.

The form of government in this country is called a "liberal democracy".  Both words are important.  People generally understand "democracy" to be "by the people, of the people, and for the people", in the words of Abraham Lincoln.  But what's this whole "liberal" thing about?

The first thing to understand is that an authoritarian form of government is the natural state of things.  To cite just one example, classical China was of the opinion that the best form of government was one headed by a "philosopher king".  A king is an authoritarian figure.  That was taken as a given.  What one hoped for was that he was also a philosopher.  The Chinese saw that term quite differently than we do today.  They did not think of an academic pontificating from an ivory tower in a completely impractical way about things that seemed completely disconnected from the real world.  To the Chinese a philosopher was a well educated and intelligent person who was thoughtful and careful and was very familiar with the way the real world worked and how actual people behaved.

If you ignore the formal title a philosopher king is what most people in most places and at most times have hoped for in a leader.  They expect a king, an authoritarian head of government who has absolute or near absolute power.  But they hope he also has the same attributes the Chinese looked for:  intelligent, well educated, thoughtful, practical, and careful.  Such a person would be well versed in the issues of the day.  He would come to a decision about what the right way to proceed was and then he would have the power and authority to "make it so".  The whole point of a liberal democracy was to not have a king, philosopher or otherwise.  But why?

The problem with kings is that while a good king can do a lot of good a bad king can do even more bad.  The big problem with kings in particular and authoritarian systems in general is the succession problem.  Who takes over from the old guy?  Ideally you want to replace a good guy with another good guy.  But all too often the guy you were replacing was a bad guy and he ended up getting replaced by another bad guy.

And there was another problem.  Succession fights could get really messy.  A lot of people could get killed.  A lot of raping and pillaging could happen and that would be bad for everybody.  Say you had a good guy.  In the standard authoritarian scheme he sticks around until he dies.  But late in life he could have all kind of medical problems that turned him from a good guy to a bad guy.  Or he could get dementia or some other mental condition that again turned him from a good guy to a bad guy but left him physically healthy and, therefore, long lived.  And say you eventually replaced him with another good guy but there was a nasty fight over the succession.  A country could go a long time between stretches of "good guy" rule.

The long term effect of this experience was succession rules.  Lots of countries in lots of places and in lots of historical eras went with some kind of blood line thing.  The eldest son of the old king would succeed him to become new king.  The idea was to make the succession as smooth as possible.  The odds that the son of a good guy will also be a good guy were better than average.  So there is that to be said for that particular system.  But what if there is no son?.  Or what if the son is an infant?

Nobody ever came up with a really good succession rule.  Modern corporations are authoritarian.  The usual succession rule for them is "promote the pest qualified person from the ranks of senior management".  But authoritarians are jealous of their power.  So they often surround themselves with yes-men.  Yes-men are by their nature second rate.  And in public companies the ultimate arbiter is the board of directors operating on behalf of shareholders.  Oh no!  Liberal democracy.  So while some kind of "best man for the job" rule is possible in principle it often doesn't work that well in actuality.

The idea behind the liberal democratic form of government came out of a European movement called The Enlightenment.  Steven Pinker just published an excellent book called "Enlightenment Now" that goes into a lot of detail about what The Enlightenment was about and what the key ideas that drove the movement were.

The Enlightenment is associated with the liberal movement.  But it is important to understand that the liberal movement was a reaction to the status quo, the way things are, have always ben, and naturally should be.  The thing that drove the liberal movement and produced The Enlightenment was the simple idea that "there has to be a better way".  Liberalism and The Enlightenment started out as an "anti" movement.  It was only after they convinced themselves that certain ideas were superior to the old way of doing thins that they became "for" anything in particular.

Liberalism and The Enlightenment marched in lock step with science.  The old ways of  finding truth were revelation and authority.  God would reveal to us what was right and true.  And the down-to-earth version of God was an authority of one kind or another.  It might be someone with a lot of military or political power.  It might be an oracle speaking for God.  Or it might be a revered ancient.  Again, these sources of truth work better than some completely random process.  But people started asking themselves a key question:  "is there some method that works better for finding truth?"  Listening to some random dude was not the answer but was that the only alternative?

Eventually they found their answer not in a religious belief or a person but in a process.  We now call that process "scientific investigation".  Don't listen to some random dude.  But do listen to the dude who has scientifically investigated the question or issue.  Galileo looked through a telescope and saw amazing things.  And the cool thing about Galileo's approach was that anyone could look through a telescope and see the same thing.  You didn't have to depend on Galileo.  You could confirm it for yourself.  And this whole idea of confirmable observations leading to new ideas and theories that, in turn, could be buttressed by additional observations revolutionized everything.  And this approach quickly ran up an amazing track record.  New ideas that were produced by this process actually worked.

The Enlightenment took this approach to culture and government.  It was apparent that a good king was amazing.  But even good kings screwed up a lot and bad kings and messy successions made for a lot of bad, and often really bad.  So the first thing members of the enlightenment did was try to figure out how to fix the succession problem.  They failed. They also figured out that nobody gets it right all the time.  So maybe vesting all power in one person was a bad idea.  Experience with various forms of councils and ministries, shared authority and responsibility, often led to a better outcome.

So the idea of replacing monarchies with democracies came out of The Enlightenment and liberal thinking.  And it is important to understand that in this context "liberal" means no more than "not bound by past ways".  "Liberal" now means something else.  But that is because The Enlightenment is now a historical movement.  As such liberalism is now tied to a set of ideas that were developed then and are still around.  Similarly the concept of conservatism has also evolved.  It is no longer just "they way things are and have been".  It is now associated with ideas that may or may not represent how things are now.  Instead, they can be more clearly seen as "in opposition to the standard list of 'liberal' ideas".

This can be quickly be demonstrated by the way we talk about economic systems.  Economic systems are now often lumped into just one of two systems: "capitalist" or "socialist".  But historically speaking, at the dawn of The Enlightenment "the way things are and have always been" would be closer to socialism.  Then government had a close connection to markets and the economy.  And capitalism, the dependence on markets rather than government action, arose out of Enlightenment (i.e. liberal) thinking.  But capitalism is now associated with conservatives and socialism with liberals.  A complete role reversal has taken place.

There are a lot of problems with democracy.  One hopes for "the wisdom of the crowds" and one is often disappointed.  It often seems like an "enlightened despot", the modern equivalent of the Chinese "philosophe king" would be a definite improvement.  And at first blush that's right.  Liberal democracies are hamstrung by process and can't react quickly and effectively to problems.  In theory a benevolent despot could.

The succession problem usually takes a while to become apparent.  And we always assume our benevolent despot of choice will be in the "good king" mold.  Certainly the Italians in the '20s thought that's what they were getting when they put Mussolini in power.  And in fact he worked out well for the Italians for a while.  The same is true for Hitler in Germany in the '30s.  He got the economy out of the ditch and made most Germans feel better about themselves.  It was only after a while that the bad outweighed the good.

So there is always a strong undercurrent pulling us toward an authoritarian system here in the US and elsewhere.  The only time and place you find liberal democracy having true popular support is in places where they have been subject to terrible authoritarian governance for a prolonged period of time and then have, one way or another, found a path to democracy.  This usually doesn't last long as the democrats start screwing up to a greater or lesser extent and people start forgetting how bad the past really was.

It turns out that continuing to have a liberal democratic political system requires buy in from powerful people.  Our founding fathers strongly believed in liberal democracy.  This was especially true of George Washington.  He was very popular when he was elected President.  He saw clearly that he could easily have become King George the First of America.  But he carefully chose another course.  The congress squabbled incessantly, as democratic bodies do.  But he patiently waited them out and established a reputation for restraint and deference.

None other than Richard Nixon, one of my least favorite Presidents, continued that tradition.  When he was convinced he was shortly going to find his Impeachment by the House ratified by the Senate he resigned instead of trying to fight it.  At some level he believed in liberal democratic values.  Many powerful people, some famous and some not, have relinquished power in an orderly manner as required to maintain our form of government.  Our system critically depends on this kind of behavior.

Hitler was democratically elected in Germany but he chose to take a different route.  He had wide public support when he did.  The public had become fed up with democracy.  In that case the problem was not with democracy but with the draconian economic provisions imposed on Germany by Democratic Britain and Democratic France.  We have seen the same scenario play out in Russia.  The problems there were entirely home grown but they were also real.  Putin leveraged them to a position where Russia is no longer a democracy.  And there are dozens of examples of the Hitler/Putin model playing out.

Generally speaking, the response in the US is "it can't happen here".  But it can if people in power abandon liberal democratic values for those that advance their own personal interests.  In short, they behave like the Sinclair Communications people.

The bulwark against this is supposed to be the general public.  What's good for Sinclair Communications is bad for the general public.  But for this to work the general public must recognize what's going on and respond accordingly.  At this pint I am pessimistic about the chances of that happening.  People are supposed to have learned about liberal democracy in civics classes in school.  But civics classes are mostly a thing of the past.  And I am skeptical that even when they were a part of the curriculum  most students ever bought into the idea that this sort of things was important.  Instead I see most people as being pretty apathetic on the subject.  They are not much interesting in voting rights and access to the ballot, for instance.

Contrast that with how deeply they are invested in the fate of local sports teams.  There they pour money, interest, and enthusiasm in in almost limitless quantities.  ESPN is a much bigger cash cow than CNN because it has much higher ratings than CNN.

I am not optimistic that our current form of government will endure much longer.  But we have been through a period like this before.  We had the "Gilded Age".  This was the era of the millionaire robber-baron, people like Rockefeller, Vanderbilt, Carnegie, etc.  That age spawned Teddy Roosevelt and the "progressive era" complete with trust busting and other efforts to reign in the power and wealth of the super-rich.  But history only gives us a template.  It does not force us to follow it.