Privacy

I have been about to write about about this subject for weeks.  But then a new revelation comes out and gives me an excuse to delay.  The latest (new Snowden revelations now seem like a continuous part of the background) is an opinion from a Federal Judge that activities of the NSA violate the Fourth Amendment of the Constitution.  Here is the entire text of the Fourth Amendment:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

It's short and unlike, for instance, the "commerce" clause, this language has been interpreted by the courts in a manner that pretty much aligns with how the average person would understand the meaning of the Amendment.  We are all familiar with the "search warrant" from innumerable cop shows.  A cop has to go before a judge and swear to the judge that he or she has "probable cause" to believe that if he or she searches a certain place (house, car, etc.) he or she will find a specific person (e.g. escaped prisoner) or thing (e.g. illegal drugs).  If the cop doesn't have a search warrant or searches the wrong place (i.e. car instead of house) or the cop finds the wrong thing (i.e. escaped prisoner instead of drugs) the cop can't seize whatever he or she has found.  And if he or she goes ahead anyhow the evidence obtained as a result of the impropriety is "tainted" and can't be used in a court of law.

Now there are some complexities, but let's ignore them for the moment and look at the broader picture.  We all expect a reasonable degree of privacy.  In U.S. law that flows from the Fourth Amendment.  But an expectation of privacy is found everywhere and every when.  So in the most broadest context what's our expectation when it comes to privacy?

For most of history and in most of the world people lived in villages.  The village consisted of a grouping of huts.  A hut has walls, a roof, and a door.  In some cases it has one or more windows.  In other cases there were no windows.  You couldn't see or, in most cases hear, what was going on within the hut without entering it.  Universally a set of manners grew up that said you couldn't enter the hut unless you lived there or you were invited.  And it was a breach of manners to peer in through the door or windows.  So if you were outside the hut you were in public.  If you were inside the hut you were in private.  But it came to extend beyond that.

Sex is usually considered a private act since time immemorial.  Sex usually happens inside the hut.  There were no interior walls or other barriers inside huts so generally speaking the other occupants of the hut were aware of sex, when it was happening.  But taboos developed where this activity was not discussed outside the hut and frequently the other occupants of the hut would ignore the fact that sex was going on while it was going on.  Other activities that happened inside the hut also became off limits to talk about.  They were private.

Many cultures use a variation of an "I see you" as a greeting.  The idea is that you are generally not there until you are acknowledged.  That transitions your activities from effectively being private (people more or less ignore what you are saying or doing) to public (what you are saying or doing goes "on the record").  So the differentiation between "private" and "public" became more sophisticated.  Various cultures developed different sets of rules but all cultures developed rules for separating activities into private and public activities.  And in many situations the boundary between private and public could become very complex.  The same act could be "private" to some observers while being "public" to others.

And technology marched on.  Huts were replaced by houses and buildings.  But the privacy rules evolved out of the "hut" model.  And oral communication became supplemented by written communication.  The adhesives that are required to construct an envelope are a modern development.  In the past a letter was folded in an elaborate way and then "sealed".  Wax was dripped in such a way as to glue the folded paper closed.  A "seal" or "signet" or "chop" was impressed into the wax.  This identified who had sealed the message and was supposed to make tampering evident.  A universal convention quickly developed that what was on the outside, plainly visible without breaking the seal, was public.  What couldn't be read without breaking the seal was private.

These conventions form the basis of the common understanding of privacy.  What's inside the building is private.  What's outside the building is public.  What's on the inside of the letter is private.  What's on the outside of the letter is public.  Elaborations and exceptions were developed.  As cultures get more complex the rules get more complex.  But this is the basic, the "natural expectation" of people when it comes to privacy.  And it comports closely with the plain language of the Fourth Amendment.  The Amendment sets out the rules.  And it specifies the proper procedure for governmental authority to use to override the rule.

Now let's be honest here.  People have been breaking the rules as long as there have been rules.  People listen at doorways or windows.  People speak about what is supposed to remain unspoken with respect to goings on within the hut.  "Black Chamber" departments of governments developed techniques hundreds of years ago for opening letters, copying the contents, and reclosing them in such a way that no (or very little) evidence of tampering remained.  Codes and cyphers were developed so that diplomats could communicate securely in spite of the fact that their sealed correspondence was being read.

But there was a very practical reason why only special people like diplomats worried about their privacy.  It was very expensive to violate privacy conventions.  So Black Chamber departments snooped in the mail of diplomats but they did not have the resources to go after more than a few people.  So the ordinary person was secure from invasions of privacy by the government.  And any non-governmental person or group was breaking the law if they snooped.

At about the time that the Fourth Amendment was written Ben Franklin was investigating electricity.  He was the first to figure out that it came in two kinds.  He posited that it normally flowed from what he called "positive" to what he called "negative".  He was right that it flows but he got the usual direction wrong.  In round numbers, the telegraph was invented about fifty years later.  In round numbers the telephone was invented another fifty years after that.  In round numbers the computer was invented still another fifty years after that.  And finally, in round numbers the Internet was invented still another fifty years later.  Combined, these inventions ultimately revolutionized the privacy game.

The "letter" model was extended to cover the telegraph.  Physically the telegram and the means used to transmit, receive, and process it are public.  There is no practical method to seal the telegram up.  It must be clearly visible to telegraph operators.  The signal can be "tapped" from the telegraph wire.  But the telegram was wrapped in a virtual envelope of privacy.  Employees of telegraph companies were to treat the contents of all telegrams as confidential.  It is illegal to tap a telegraph wire.  In a practical sense, users adopted the "diplomatic letter" model.  They used a variety of codes and ciphers to conceal the contents of their telegraph messages.  A similar approach was adopted in the case of telephone calls.  Telephone company employees were expected to treat all telephone calls as confidential.  And the "get a search warrant" procedure was extended to cover telegrams and telephone calls.  With a warrant the contents of a telegram could be disclosed to a government official.  With a warrant a telephone line could be tapped.

For a long time, computers made no difference.  They were not much used for communication.  And, to the extent they were, they were treated like another piece of communications equipment.  And in its early days the Internet made little difference either.  The Internet is designed to be an open environment.  It is the equivalent of people standing in a public square and talking to each other.  It is bad manners to listen in on the conversation of others but it is certainly possible.  If people want to keep their conversation private they should "get a room" (use something other than the Internet for their communication).  But then a revolution gradually happened.  Computers (and data storage) became unbelievably cheap.  And the speed, ease of use, and ubiquity of the Internet made it the communications channel of choice.  So now more and more communication is done over the Internet.  The cheapness of computers (both the cost of computation and the cost of storing data) and the convenience of the Internet have created a revolution that has only recently been noticed.

As I indicated above, the practical reason that guaranteed our privacy was the difficulty (both in cost and in effort) of violating out privacy.  The FBI under J. Edgar Hoover was famous for decades for tapping phones, opening mail, and doing "black bag jobs" (breaking into homes and businesses to violate the "houses, papers, and effects" of individuals and organizations).  This was completely illegal.  But Hoover's FBI concentrated their efforts on mobsters, spies, and politicians.  (The latter was targeted to secure blackmail material that as used to maintain Hoover's position of power until he died.)  The vast majority of us were safe because we did not fall into one of the above categories and Hoover did not have enough agents to broaden his list of targets.

But in the last few years there has been a profound change.  It is no longer to difficult and expensive as a practical matter to violate people's privacy wholesale.  Computers have evolved (down in cost, up in capability) enough to change the game.  The new capability is usually referred to as "big data".  Vast amounts of data can now be mined relatively inexpensively.  There is a database concept called a "join".  If you have a phone number it can be joined to a name.  The name and phone number can be joined to an address.  A name and address can be joined to a driver's license number or a credit card number or even a social security number.  From there the joins can go on and on into nearly any direction you can think of.

And as recently as a few years ago data was spread all over the place.  Your drivers license information was separate from your credit card information.  And both of them were separated from your tax information (keyed to your social security number) and your medical information (again social security number but in a separate location).  The glue that is now available is the Internet.  Making data Internet accessible is almost a necessity in the modern era.  So now joins can be used to link data from databases that heretofore could not be linked up.  But they can now because they are all accessible via the Internet.

And there is now another class of player.  I used to work for a bank.  And I'd watch movies where the good guy or the bad guy (depending on the movie) would instantly access all this banking data.  It was hard not to laugh.  I knew that our bank did not have this data instantly available.  It was hard to keep it available to the computers located in our data center, let alone anyone or anything located anywhere else.  And I dealt with other banks enough to know that the same was true in their case.  But that was before cash machines and ubiquitous cred card and debit card readers, all hooked up to the bank's databases so that transactions could be validated instantly and cleared instantly.  Now various methods are being worked out so you can buy stuff with your smartphone.  There are currently various incompatible schemes for doing this.  But people all over the world will be able to buy pretty much anything with their smartphone within a decade.  There's just too much money to be made once it's working for anything to get in the way.

Traditional players like banks are in the game and that's an obvious development.  But we also have new players like social media in the guise of Facebook and search purveyors in the guise of Google who are gathering vast amounts of personal data.  The old business model was "we invade your privacy but only to the extent necessary to do business with you".  So banks held personal financial data about you.  And doctors and hospitals held personal medical data about you.  And until recently this personal data was not very useful for anything other than its original purpose.  Doctors in particular have been slow to move from paper records to computerized records.  And if it's not in the computer, it is prohibitively expensive to search.

Consciously or unconsciously we knew that these old line businesses could invade out privacy.  But they had been around a while and standards and practices (and a certain amount of law) had grown up around them.  But Google is less than fifteen years old and Facebook less than ten.  And both companies and their new line brethren have been evolving rapidly.  And these new line companies have substantial big data expertise.  As a group, they invented big data.  It is the old line businesses and government agencies that are playing catch up.

In summary, we have the old line players (government, old line businesses, snoopy neighbors or business competitors) with vast new "big data" capabilities.  They are combined with new line players like search and social media players who have vast amounts of data and the capability to mine it.  The practical barriers to vast and systematic privacy invasions are gone.  They have been done in by cheap computers and Internet connectivity.  And don't forget the fact that as a practical matter we all live on the Internet now.  Illiterate rice farmers in rural villages in India are getting Internet connected via cheap low end phones whose capabilities are not to be found in sci-fi movies of 30 years ago.  We can expect the number of people who are not on the 'net to drop below a billion people within a few years and to virtually zero within a few years after that.  So what should we do?

Well, one thing not to do is to try to put the genie back in the bottle.  The Internet, cheap computers, and big data are not going to go away.  In fact, things are going to go the other way.  More stuff will get connected to the Internet and the Internet will get faster and cheaper.  Computers will also continue to get faster and cheaper.  And big data is still in its infancy.  Techniques for exploiting big data will continue to improve.  This will result in more and better ways for your privacy to get invaded.  The other thing not to try is to give up, to decide we live in an open world where more and more people have more and more capability to learn more and more about you and there's nothing that can or should be done about it.

Specifically with respect to the NSA, it looks like the worm is finally turning.  9/11 was used to scare all of us into turning the NSA loose to do whatever it could figure out how to do.  (They were also given vast amounts of money so cost was not an impediment.)  I don't know why but people were convinced that the NSA would only go after the bad guys.  Pretty much everyone can quote the old saw "power corrupts and absolute power corrupts absolutely" but people somehow believed that giving the NSA vast amounts of money and vast amounts of legal authority would not lead to incompetence, stupidity, and abuses.  It is in the nature of bureaucracies to build empires.  And when a bureaucracy can empire build in an environment where everything they do is secret and their mission is deemed critical then empires will be built.

My reading of the 9/11 commission report is that the government had all the information it needed to stop the attack.  But the information was "siloed", each bureaucracy kept its information in a "silo".  People in their organization got access to the information but people in other organizations did not.  Between the FBI, the CIA, and the NSA all the information to connect the dots and foil the plot was there.  But no one was able to put it all together because no one had access to all the information.  (The FBI is an egregious example here.  Two different supervisors shut down two different field office investigations that would have exposed the whole plot.  And here there was no siloing.  The FBI had everything it needed.)  In spite of this, additional "authorities" were given to various government agencies, but especially to the NSA.  They were authorized to vacuum up everything.  And they did.  There are supposed to be "checks and balances".  But they are pathetically weak.  And everyone who has been able to see what has really been going on has concluded that even these weak checks and balances were ignored routinely.  Read for instance, the opinion of Judge Leon in the NSA case I referred to above ("Klayman et al. v. Obama et al. -- U.S. District Court for the District of Columbia Civil Action No. 13-0851").

The biggest problem with the NSA is that they have collected vast amounts of data into a single NSA managed repository.  The obvious fix here is to take the repository out of the control of the NSA.  A White House study has apparently recommended this.  (The study is classified but this item has leaked and the leak is presumed accurate.)  I think this is a good idea.  I think a quasi-governmental entity should be set up.  This entity should own and maintain the repository.  It should be responsible for making sure that access rules are in place and that those rules are followed.  Telcos and others who currently contribute the data to the NSA don't want the job.  Who knows where the data will come from in the future so I think a separate entity is the way to go.  This means that the government has your data but at least the possibility would exist that it was secured and administered properly.  I think the budget for this new entity should come from the current NSA budget.

I think we should also look at how much of and what kind of data is being collected.  Various claims have been made as to how valuable the data has been so far.  But the data to justify the claim is highly classified.  I have been a student of intelligence matters for a long time.  The intelligence community is fond of saying "we have had many successes but we can't talk about them".  But the record shows that in fact they have had many failures and have used the classification system to hide or minimize them.  There is usually a political advantage for information on the successes to leak out and it does.  For instance, I know of no great cold war success that did not leak out within a few years, perhaps a decade.  The cold war ended more than a quarter of a century ago.  And in that time information on a number of failures and bungled jobs has dribbled out.  See also the Leon opinion I cited above for how valuable this database has been.  He says "the Government does not cite a single instance in which analysis of the NSA's bulk metadata collection actually stopped an imminent attack, or otherwise aided the Government" (page 61 - emphasis in the original).

As a rule of thumb I recommend this:  Of everything that is classified about one percent really needs to be secret for more than a short period of time, say a year (and often much less).  About ten percent is classified to avoid oversight leading to possible embarrassment and about 90% is classified as the result of sheer bureaucratic inertia.  Bottom line:  The NSA is collecting far more data than it needs or can even make use of.  (Although "big data" techniques continue to make it easier to make use of large amounts of data.)

But that's the NSA.  And, of course, similar restrictions could be put on other government agencies.  But, as I indicated above, there are now other non-governmental players:  the new line players and the old line players that are catching up.  Laws, rules, etc. can be used to reign in the government (at least in theory -- let's see what congress actually does).  But these non-governmental players are a whole different kettle of fish.  What should be done?

I have heard a number of proposals made over the years by pro-privacy groups.  I consider most of them misguided.  A couple of lifetimes ago (here I am using Internet years) Intel was going to put an easily accessible serial number in their "486" chip.  The pro-privacy people raised such a ruckus ("they'll be able to know who I am") that Intel gave up on the idea.  So what happened?  Software designers used another of the hundreds of unique numbers found on all computers.  So developers had to write a couple of hundred extra lines of code to find something that would behave like a serial number and went ahead with their plans.  So most of the pro-privacy suggestions are pretty useless.  Okay . . .

I think that a fundamental legal principle needs to be adopted:  If the data is about you then the owner of the data is you.  Right now whatever data Google collects about your searches is owned by Google.  They can promise you anything they want about what they will or won't do with that data but at the end of the day it's their data and they can do whatever they want with it. So they do.  Facebook has gotten into trouble multiple times because they keep tweaking the privacy settings so that it becomes harder and harder for you to keep anything private.  Why?  Because they want to sell as much data about you to as many companies as they can for as much money as possible.  The more data and the higher quality the data the more valuable it is to Facebook.  I don't have a Facebook account but it seems to me that Facebook recently got in trouble for selling pictures from your "wall" to whoever wanted to buy them.

If all this data about you (i.e. the pictures on your wall) was owned by you then the legal situation would be quite different.  Now the collector of the data (Facebook, your bank, your doctor, etc.) would be given a "use" right to your data.  They would be permitted to use it in whatever means were necessary to provide the service you signed up for.  But that would be it.  They would no longer have the right to sell the data (or trade it or lend it out) simply in order to make a buck.

This may seem off point but let's talk about world trade for a minute.  Why are Nike shoes made in China?  There are many components to the answer but I am going to single out only one.  It's because it is cheap to ship raw materials from all over the world to China and it is cheap to ship the finished shoe from China to the U.S.  Shipping is cheap.  And there are two components to this cheapness.  There is the actual cost of shipping the goods.  Then there are the tariff and non-tariff barriers or lack thereof.  As a response to the Great Depression the U.S. passed the Smoot-Hawley Tariff Act.  It succeeded in its design to increase the cost of imported goods. Another trick is to introduce "non-tariff barriers" to trade.  Japan was particularly good at this. They both have the same effect.  They increase the cost of imported goods or, in some cases make importing goods completely impossible.  It should come as no surprise to learn that many countries follow the lead of the U.S. and later Japan,  raising tariff (and later non-tariff) barriers that protected their domestic markets from foreign competition.  And these barriers were very successful.  Trade in the '30s when the tariff wars were at their most fierce fell to very low levels.  As barriers have slowly been removed, especially in the last few decades, world trade has grown by leaps and bounds.

I hope it is now clear why I chose to talk about trade.  Trade barriers worked as a method to reduce trade.  Data interchange used to be effectively impossible.  Think cave men on rafts.  Then it was very difficult.  Think trade around the Mediterranean a thousand years ago.  Then it was only difficult.  Think trade during the '30s.  Now it is quick and easy.  Think the current era of trade.  I suggest that the same techniques that depressed trade would depress wholesale data interchange.  Tariff and non-tariff barriers could be put into place to inhibit the current "wild west" trading of data between companies.  This would ease the pressure to invade people's privacy.

Ultimately norms of privacy protection need to be established and, where there is something to build on, enhanced.  Laws and regulations need to be put in place to protect data.  Companies need to be fined and executives need to be jailed for privacy failures.  It was recently reported that 40 million Target customers had their credit card information stolen.  Given that this kind of security failure is now common, Target will suffer some embarrassment, but very little.  I don't think they will be fined, not even by the card issuers that will actually suffer most of the loss.  And certainly no one at Target will go to jail unless it turns out to be an inside job.  The people that perpetrated this fraud are likely to get away with it.  Even if they don't, they are balancing vast potential gains against a small likelihood of getting caught and, in the worse case, a few years of jail time.  That looks like a pretty sweet proposition to me.

So things need to change.  It seems likely that things at the NSA will change.  But even that is not a certainty.  Change elsewhere is much needed but seems less likely.  Theoretically, libertarians and "defenders of the Constitution" (e.g. groups like the Tea Party) should be leading the charge.  But there is very little activity going on there.  There is some pro-privacy fire on the left.  But it is only some and powerful interests think they can be safely ignored.  Ultimately, the public needs to believe that the issue is important and that something can be done about it.  I think most of the public thinks that it is at least moderately important.  But I think most people are convinced that nothing can be done. 

Unless, of course, you are talking about any kind of information relating to guns.  Then everything can. should, and probably will be done to make sure that no one (except the NRA and gun manufacturers) has access to any kind of data about gun ownership.  If it's gun related then anyone, including foreigners and terrorists should be able to purchase any kind of gun they want.  And they should be able to purchase as many guns and as much ammunition as they want too.  Being on the terrorist watch list is not sufficient to deny you the rights and privileges of gun ownership.

    

            

  

Spies - Part 2

This is a continuation of an earlier post (http://sigma5.blogspot.com/2013/05/spies.html).  It is a result of the now long series of revelations credited to Edward Snowden.  There are more revelations today and it looks like they will continue for some time.  I am not interested in getting stuck in "revelation of the day" mode like the media.  Instead, since the revelations have been going on for some time, I think there are some "big picture" observations to make.

Recently Snowden has been revealing that the NSA has been spying on pretty much everybody, friends and enemies alike.  This is "shocking news" to the media and our allies.  There are two things going on here.  In the case of the news its the result of a lack of journalistic competence and the usual "fixate on the shiny object" mentality that governs news decisions.  In the case of our allies its a matter of gaining a tactical advantage in the "power" game.  I'll be back with more on both these subjects but first let me step back.

We have had two giant leakers in the last few years, Edward Snowden and, before him, Bradley Manning.  At the personal level they are completely different.  But at a higher level they are both the same.  Ben Franklin in the 1700's famously opined "three people can keep a secret if two of them are dead".  It's a cute way at getting to the fact that a secret shared among a lot of people is bound to leak out.  Franklin's idea of "a lot" was three.  So how many people have access to classified data?  The Atlantic Magazine, among many other sources, reports that the answer is staggeringly high.  This post (http://www.theatlanticwire.com/national/2013/06/contract-security-clearance-charts/66059/) is chock full of numbers.  But let's just focus on "Top Secret" (supposedly the "all access" highest level but actually not).  The Atlantic reports that nearly half a million people (including at one time both Manning and Snowden) have this level of access.

If half a million people have access to information what's the chance it will stay secret?  The answer is known to extremely high precision.  There is exactly zero chance that the information will stay secret for any length of time.  We know about Manning and Snowden.  But how about the people we don't know about because they haven't gone public with a big splash and gotten a lot of media coverage?  Madden and Snowden were both Americans that chose to take what they learned to the press.  What about the possibility of people taking this kind of information to Al Qaeda or the Taliban?

Actually, I think this is unlikely.  Both of these organizations smell bad and the intelligence agencies track them closely.  So, if they managed to get an inside connection it is likely the intelligence community would get a whiff.  But it is not impossible.  In the '30s the Russians managed to infiltrate a number of people into British Intelligence.  They ultimately rose to occupy positions at the highest levels of MI6.  They were not exposed until decades later.  If you want details, check out the Wikipedia page on "Cambridge 5".  I will have more to say on who is more likely to have gotten their hands on this kind of material later but first . . .

Manning and Snowden share another attribute.  Both were very low level employees with some technical expertise.  Manning's technical skill was sufficient to land him a job as a "developer" at a software company before he enlisted in the military.  Snowden did "IT Security" work and was a Systems Administrator working on classified facilities during his tenure within various components of the intelligence community.  Neither of them made any kind of mark as some kind of super computer guy.  As far as I can tell they were both journeyman IT types.  I am very familiar with this type because it's the kind of work I spent my professional life at.  We are smarter than average but there are lots of us.

Manning and Snowden are also similar in that there are red flags associated with both of them.  Manning had a long history of personal problems.  He now claims to have "gender identity disorder".  He had no business having access to classified data and a competent background check would have shown this.  But he got his "Top Secret" ticket punched anyhow, apparently without difficulty.  (This is another thread that has received little coverage.)  As far as I know there is nothing in Snowden's background or makeup that would have alerted anyone.  But there is a fundamental question, mostly unasked, as to why either of them had access to the information they had access to.

Manning was in the Army and was doing intelligence work related to the Iraq war.  He was a Private and his duties had nothing to do with anything diplomatic?  So why did he have access to hundreds of thousands of supposedly classified diplomatic documents?  Neither his rank (as low as it goes) nor his duties (supporting the Army mission in Iraq) justify access to these documents?  They didn't but he obviously obtained access anyhow.  As far as I can tell the security on the server that held these documents was so lax that someone with a modest amount of computer expertise and an interest in finding them was able to bypass security.  And as far as I can tell, Manning's access did not set off any alarms.  After the story broke and people traced his activities (probably with Manning's help), only then did anyone know what had gone wrong.  If an Army Private in Iraq could gain access to these documents, then who else could?  The answer is hundreds of thousands of people.

The Snowden case is similar.  I think Snowden's IT expertise was greater.  And, since Snowden came later, he was in a position to learn from Manning's experience.  But Snowden was not interested in keeping his identity secret for any length of time as Manning had been.  Snowden was also more intelligent (in my opinion) and did not have the many personal problems Manning had.  He had, as far as I can tell a very nice and very pretty (the pictures prove this) girlfriend.  And before everything blew up they seemed to be having a good relationship.  But at bottom the Snowden case is like the Manning case in the sense that it is important to ask the question "why did he have access to those documents?"

By the nature of the job, System Administrators have a lot of access.  Among their duties is that of setting and maintaining security levels for all the parts of whatever systems they administer.  They know the security controls better than anyone else.  It's their job.  And its a very unsexy job.  So it's the type of job that attracts "slow track" management types to the supervisory positions above them.  These management types also frequently get yelled at for doing their job too well.  It is common for one of them to get called on the carpet because some well connected hotdog can't get access to something he wants to get access to but is not supposed to get access to.  These management types tend to be happy when no one is yelling at them rather than being happy when their subordinates are doing their jobs well.  So the quality of work frequently depends more than it should on the personal skill and moral fortitude of low level people like Snowden.

That's inside baseball.  Its something I know because I've been there.  But there is a huge red flag that Snowden raised and the media has completely ignored.  Snowden said he got his last job at Booz Allen specifically because it would permit him broader access to the kind of material he was interested in.  There has been no follow up by the media on this.  Booz Allen has been around a long time.  They have been successful at getting and keeping defense and intelligence contracts for a long time.

This depends on getting and keeping profitable contracts.  It is more important to keep the government people who award contracts happy than it is to do good work.  In short, they have mastered the office politics of keeping in the good graces of the military and intelligence community brass.  What they look for from someone like Snowden is an ability to do adequate quality work quickly and an ability to not embarrass the brass.  A good way to do this is to take shortcuts on the assumption you will not get caught.  Shortcuts result in security loopholes that can be easily exploited by someone with the right mix of skill and knowledge.

Snowden was working in an obscure corner of the intelligence establishment in Hawaii.  Like Manning in Iraq, it is hard to imagine a place further from the District of Columbia, home of the State Department and the NSA.  But in this modern era networking makes distance unimportant.  Connectivity is what is important.  It is now obvious that connectivity permitted Manning to access political documents presumably housed on State Department servers.  It is now also obvious that connectivity permitted Snowden to access political/management documents presumably housed on NSA servers.  The NSA prides itself on being the best damn security shop in the world by a mile.  What we now know Snowden successfully accessed (and again as far as we know without raising alarms) should be profoundly embarrassing to the NSA.  I note that so far I haven't heard of any NSA brass or contracting companies getting fired.

Now let me circle back to the "who else could get access" question.  Any "secret" held by hundreds of thousands of people is not a secret.  What is surprising about the revelations credited to Manning and Snowden is not what they revealed.  It's that these secrets and others like them haven't been revealed tens, hundreds, even thousands of other times.  And this begs the question of whether they in fact haven't.

And this brings us to the dumb show and corresponding media coverage of the "shocking revelations" that the NSA has been spying on "friendly" foreign diplomats.  The best single reference on intelligence matters up to and through World War II is "The Code Breakers" by David Kahn.  People have between concealing messages going back at least to the Egyptians and the Romans.  And during this entire period other people have been trying to "crack" these concealed messages, and they have often been remarkably successful.

The modern era of this sort of thing goes back hundreds of years in Europe.  And it's not just Europe.  The ancient Chinese were also noted for this sort of thing.  Anyhow, in the European era it was common for governments to create "black chambers".  Mail would be intercepted, opened, copied, and put back into circulation in such a manner that the tampering was nearly impossible to detect.  The copied messages were decoded and the contents used to political and military advantage.  And the old dictate "keep your enemies close and your friends closer" was the order of the day.

Certainly all kinds of means were used in time of war.  But efforts continued during peacetime.  The Russians famously spied on everyone including their erstwhile friends the British, French, and Americans during the thirties.  And it's not just the "bad old Ruskies" or, before you ask, the "bad old Nazis".  Angela Merkel, the current German Chancellor was born and raised in what was then East Germany.  The East Germans ran a large, extensive, and very effective spying operation into what was then West Germany.  Some of it was for their Russian masters but a lot of it was for domestic consumption.

And then there's the French.  They have been running a large industrial/political espionage operation for decades.  A friend of mine worked for Boeing for a while.  Boeing gave employees traveling in France explicit instructions for taking extraordinary security measures while traveling in France.  Why?  France was interested in both of Boeing's lines of business.  They are competitors in civil aviation against Boeing commercial jets and in the military sphere, selling fighter jets and other kinds of equipment that Boeing Defense and Space also sells.

But you say "it was the cold war" in response to East Germany or "we've never been that close" in response to France.  Well, how about an ally that is closer to us than anyone but the U.K., Israel.  There have been not one but several scandals involving Israel spying on us.  And that ignores the time the Israelis sank a U.S. spy ship during the "Six Day War" in the '60s.   The official story is that it was "accidental" but that's an obvious cover story, "a tale told by an idiot, full of sound and fury, signifying nothing".  Why was the ship sunk?  Because the Israeli military had some tricks up its sleeve that they didn't want the U.S. to know about.

Both Manning and Snowden have exposed secrets.  The official line is that these are very damaging to U.S. interests.  And they are.  But they are not damaging because foreigners, especially the Europeans, didn't know these things.  It's because they no longer have "plausible deniability".  They could no longer pretend they didn't know these things.  And that's embarrassing to them.  It makes them look incompetent to the people that elect them.  If you are embarrassed by some one, in this case the U.S., then you want to hit back.  So we are being hounded from all sides.

It's also useful to get and apply whatever leverage comes your way when dealing with a powerful country like the U.S.  There may be some countries like Brazil where these revelations may actually be a surprise.  But here too, the U.S. has been treating Central and South American countries as second class citizens for at least a hundred years.  And we have repeatedly meddled in their internal affairs.  Quick:  How many coup attempts has the U.S. sponsored in South America?  How about Central America?  So if you are a Central or South American country and you get a chance to get a little back from the U.S., it's an opportunity not to be wasted.

All of the above is publically known to people who take the time to become informed.  It would be nice if the media took the time to become informed or assumed the public had the sophistication of a tree squirrel.  But they don't.  And the public time after time vindicates the media's low opinion of their viewers and (small number of remaining) readers.  It's very depressing.

And it's worth while taking a look at how we got here.  A standard bureaucratic tactic is to engage in empire building, gathering more power at the expense of your bureaucratic colleagues.  One of the manifestations of empire building is what are often referred to as "silos".  These are tube-like structures within which information flows up and down the organization.  They are nicknamed silos because information does not flow across to other parallel organizations.  We saw this in the run up to 9/11.  The CIA didn't share with the FBI and the NSA didn't share with anybody.  As a result the pattern wasn't clear because people inside one silo couldn't see the information in the other silos.  After 9/11 sharing was the new thing.  So databases were opened up and linked together.  This was done quickly and sloppily.

And at the same time vast new databases (i.e. all the telephone records the NSA now routinely collects) were put online and connected up so that everyone (as in hundreds of thousands of people) could see everything.  The government went from not enough sharing to too much sharing.  Everyone with a Top Secret clearance can see everything.  No one wants to be the person who stopped some other person from seeing some piece of data that would have stopped a terrorist attack.  And no one wants to be the person who stops some massive collection project that could have collected the piece of information that could have stopped a terrorist attack.  We are seeing this latter effect play itself out in the wake of the Boston Marathon bombing.  "Gee, if we had just collected some more data we could have stopped those guys."  The problem the intelligence community now has is that they are literally drowning in data.  So much stuff is coming in that no one can deal with any of it.

This is one of these "judgment" things.  It requires judgment to decide how much is the right amount.  But if judgment is applied then some of the time someone will get it wrong.  And then someone will take a pot shot.  Then everyone will hunker down and things will get even worse.  And the media is the last to champion judgment and the application of common sense.  It is much more fun and better for ratings to showcase some showboating politician jumping down the throat of someone who applied judgment and was unlucky.  And it is an even better show to feature some loud mouth going after entirely the wrong target.  I frequently despair of the media and the vast part of the audience who will not put the effort in to sorting the wheat from the chaff.  But probably, "'twas ever thus".   

2013 America's Cup

Recently I posted about the America's Cup in general.  That post can be found at:  http://sigma5.blogspot.com/2013/08/the-americas-cup.html.  Well, the current Cup event is over, finally.  This post summarizes my thoughts on how it went.

First, the results:  The defending team, titled "Oracle Team U.S.A." ended up winning beating the challenger, titled "Emirates Team New Zealand" (Emirates money coupled with New Zealand technical and sailing expertise).  They retained the cup in a titanic battle consisting of 19 finished races spread over 16 race days (days on which at least one race was completed) spread over 19 calendar days.  The final score was 9-8 but Oracle had to actually win 11 races because they started out with a 2 race penalty.  Judged either by the number of days from start to finish or the number of completed races (or the number of races including cancelled ones, for that matter), it was the longest cup series ever.  In the early days the event was a "one and done" event.  It morphed into a "best of 3" event and later into a "best of 7" event.  So historically only a few races were necessary to determine the winner.  So what do I think of it all?

Let's start with the observations and predictions I made in my previous post.  I guessed that the Oracle team would have a slight edge.  It turned out to be more complicated.  In the early going the Kiwis had an advantage in all three important categories:  boat speed, crew work, and tactics.  The Oracle boat was slower, but not by much.  There were only a few races in which either boat showed a substantially better turn of speed than the other boat.  The preparation work that the Kiwis did before the regatta started resulted in a boat that was initially slightly faster.  That preparation work, coupled by the "trial by fire" that the Louis Vuitton Cup represented resulted in a boat that was sailed better and had a better understanding of the tactics.  So the Kiwis dominated in the early races.

But the Oracle people learned fast.  Both teams did but the Oracle people were able to make more and better improvements.  By the middle of the regatta both teams were performing at such a high level in terms of both crew work and tactics that neither team chad a significant advantage in either of these categories.  But the Oracle boat kept getting faster throughout the entire regatta.  The Kiwis were probably improving their boat speed too.  But day by day the boat speed of the Oracle boat kept improving relative to the Kiwi boat.  By the end the Kiwis were no longer competitive.  The speed difference was small but it was enough.  No one wins 8 races in a row against a skilled competitor without having a faster boat.

I made (but did not include in the post) a number of other predictions.  I did not expect the series to go the distance.  I expected one boat or the other to be substantially faster.  So I expected the two race penalty to not be a factor.  And, in a perverse way I was right.  I expected that either the Kiwi boat would be faster and they would wrap it up long before the Oracle boat got close to the nine "official" wins the Oracle boat would need.  Or I expected the Oracle boat to be so fast that having to win a couple of extra races would not be enough to put the Kiwis into the hunt.  It turned out that the Kiwis were dominant in the early going.  The official score was 8-1 in favor of the Kiwis at one point.  But then the Oracle boat ran off a string of 8 wins in a row (not counting races that were abandoned either after they started or while they were within seconds of starting - those all went to the Kiwis but didn't count).  So my prediction that it would not go the distance turned out to be completely wrong.

Another unpublished prediction I made was that it would be a dull event.  The races would turn out to be parades where one boat quickly took control and the other boat was forced to follow at a suitable distance.  This too turned out to be wildly wrong.  Most of the races were extremely close.  In a lot of cases the behind boat was not able to make a pass but they were able to stick with the ahead boat.  And there were a surprising number of passes, more so than in any other Cup.  We even had a couple of races where one boat passed then the other boat passed back later in the same race.  The races themselves were quite exciting.  A major component was the blinding speed of the boats.  These boats are three times as fast as a traditional "competitive racing" boat of only a few years ago.  But the fact that the boats were never far from disaster (the "Nascar" effect) was always present too.  And finally, there was the "he's up -- no he's down" aspect.  It seemed like an actual contest where either boat had a serious chance of winning in a lot of the races.  This aspect has not been present often in Cup competitions.  

A final unpublished prediction was that the event, and here I mean the actual America's Cup, would be well administered.  Here too I turned out to be totally wrong.  The plan was to do two races per day and roughly a two days "on" and one day "off".  This should have resulted in 4 completed races every 3 days.  The actual result was 19 completed races in 19 days.  The organizers were only able to complete two races in one day on 6 occasions.  This must be balanced against the two different days on which no scheduled races were completed at all.  The smallest contributor to this delay were the teams.  Each team was issued a "get out of jail free" card that allowed them to cancel the second race of the day without penalty.  But only one of these cards was exercised (by Oracle).  Races were cancelled due to too much wind (frequently), not enough wind (two races one day, one race another day, and one race after it had already started - technically for "time limit expired"), and (shockingly) wind from the wrong direction.

Some of this can be attributed to a combination of mother nature and safety concerns.  The high wind limit was put in after the Artemis fatality.  But the low wind was a result of trying to schedule two races (expected duration - about 30 minutes each) into a 2 hour "TV" time window.  TV constraints also stopped the organizers from shifting  race starts to earlier in the day, which would have avoided the "wind builds late in the day" problem that caused several races to not go off.  And the organizers had opted to have a single course, whose layout could not be changed.  Normally Cup races take place in a circular area out in the ocean.  So organizers lay out the direction of the course based on the wind direction on race day.  Races only need to be cancelled or delayed when the wind direction is changing wildly and rapidly.  This was done to enhance the "fan appeal".  The course could be seen from many parts of downtown San Francisco.  And they built stands along the shore to accommodate the crowds.  But crowds never materialized.  A planned expansion of the stands was cancelled and the stands that were built were mostly empty.  So losing the flexibility of being able to change the direction of the course to accommodate the wind direction was lost and this translated directly into delays.

Finally, there is one problem that is completely inexplicable to me.  Between the Louis Vuitton Cup and the America's Cup, races were broadcast in four different places.  Races could be seen on You Tube and ESPN3 (both internet only) and on both NBC and NBCSN (both broadcast).  I can somewhat understand the NBC/MBCSN split.  The event was not expected to be a ratings powerhouse.  So pushing races off to NBCSN kept it in the family.  And You Tube is neutral.  It's owned by Google.  So pushing a race that is not expected to draw NBCSN sized ratings on to You Tube makes sense.  You Tube ran an "America's Cup" channel where you could view a lot of video of cup related content.  But ESPN3 is owned by ABC, arch competitor of NBC.  What's going on there?  The only theory I can think of is that it was a clever plot top depress ESPN3's ratings.  But, since the same broadcast crew was used for all the races, the cost to ABC of this move would be modest, and that's on a good day.

And frankly, this was not a big problem for the actual America's Cup itself.  The first two races were on NBC and all the other races were on NBCSN.  Once you got used to that, and the fact that TV coverage went from 1 PM to 3 PM (on the West Coast -- add 3 hours for the East Coast times) the races were not that hard to find.  But Oracle is a technology company.  Their core product is their Oracle database but they pride themselves on being full service.  You can buy hardware, applications, and, most germane to our discussion, internet expertise from Oracle.  There is an America's Cup web site at americascup.com.  There is a section on the site where you can look up details of TV coverage for anywhere in the world.  The schedule kept having to be changed and extended as the racing went on and on and on.  But after we got through the first few days when things were on schedule the TV information on the web site was no longer correct.  On many days the TV schedule for that day was completely missing, even if you checked a few minutes before the first race was scheduled to start.

To update the information would have required adding or changing text in an amount equal to a single Twitter tweet.  And a giant, highly capable (to her them tell it), company that prides itself on being able to provide solutions to company web sites that need to handle millions, perhaps even billions, of transactions per day, should be up to the challenge of making a small single update per day to the Cup website.  And many of these companies Oracle is selling to need the information on their web site to be accurate on a time scale of seconds or less.  But apparently the event organizers could not manage to handle a "1 update a day" traffic load.  That is truly pathetic.

Looking forward, as I indicated in my previous post, I expect that the next cup will be raced in catamarans with solid sails.  And apparently my suggestion of downsizing to a boat that is 55-60 feel long has also occurred to a lot of other people.  I outlined the pros and cons of changing boat sizes in my last post so I won't repeat it here.  Since the Larry Ellison / Oracle people will remain in charge I do not expect the course to move.  If they can address the safety concerns the wind limit can be removed.  The low wind problem can be addressed by expanding the period during which races can go off from 2 to 2 1/2 hours.  With appropriate lead time I think TV people can accommodate this.  If nothing else they can do a delayed broadcast.  As someone who lives on the west coast I am way too familiar with this.  Many "live" shows are delayed three hours routinely for west coast broadcast.

And it is important to give them credit for some things they got right.  Affiliated events like "Americas Cup World Series" and "Youth Americas Cup" (both sailed in the smaller 45' boats) give more people access to and a reason to develop an interest in the main event.  The decision to go with wing sail catamarans is also a good one.  Organizers did not envision "foiling" (technically:  hydro foiling) but they have embraced it and I expect it to stay.

On the negative side, safety needs to be addressed.  But, as I indicated previously, I think this can be done between cups.  I expect the next cup in 3 years, plenty of time to figure out what changes need to be made and give participants ample time to implement them.  The big problem is cost.  Rumor has it that an AC72 boat costs between 8 and 10 million dollars to build.  If you include other costs, (paying the crew, a large and sophisticated development effort, on shore support, perhaps a second boat) the cost of an entire team is estimated to run $100 million or more.  Ellison is supposed to have spent $250 million to win the cup last time.  The fact that his boat kept getting faster and faster as this Cup event continued indicates to me that he did not stint on his "support" operation this time.  So I would guess he spent between $150 and $250 million this time.

The Kiwis came with a well run and presumably amply funded operation.  And they got slowly ground down by the Oracle operation.  Potential participants usually go with one of two approaches.  There are the "we just want to participate" bunch.  They don't want to be embarrassingly bad but mostly what they are looking for is exposure.  That can probably be done at the $100 million level, perhaps for a little less.  That's a lot of money to shell out for exposure, effectively an ad campaign.  If you want to be one of the "in it to win it" bunch, then it looks like you will need at least $150 million and perhaps a lot more.  You have to ask how many people there are out there that are willing to plunk down that kind of money.  That's what happened this time.  We had one defender and three challengers.  Of the three challengers the only one that had a chance was the Kiwi team.  So did Artemis and Prada get their money's worth?  And how much more will it cost to put together a Kiwi class effort.  And, remember the Kiwis failed, so you better plan on spending more money than they did.

The America's Cup has always been the kind of event where it is considered ok to pour lots of money in, to try to buy a victory.  So it goes against the grain to try to stop the "money fire hose" approach.  But it has gotten so expensive to play that it may be necessary to do something.  This has happened before.  The Cup went to "12 meter" boats in the '50s because the "J" boats had just become too expensive.  The whole "foiling" thing was come up with by the Kiwis a challenger.  Putting wings on keels was also an idea that a challenger came up with.  Dennis Connor put a "wing" sail on a catamaran first when he was desperate to successfully defend the cup.  Larry Ellison put one on a trimaran when he successfully challenged a few years ago.  No one, including Larry, thought the "trimaran" idea was a keeper.  So I don't think we will be seeing that idea come back, even though it worked in that race.  We should see how the 35th America's Cup will shape up within the next six months to a year.

The America's Cup

The America's Cup is the oldest continuously defended trophy in sport.  It dates back to 1851, before the Civil War.  The thirty-fourth defense of the trophy is happening in September of this year (2013).  I got seriously interested in the defense of the Cup in 1982.  1982 was also the first year it was broadcast live on TV.  Initially the races were broadcast on a local TV station in Newport, Rhode Island.  But by the time of the seventh race in the series interest was so great that the race that decided who won the cup was broadcast on national TV.  All of the Cup defenses, more or less, have been broadcast since.  Defense of the Cup involves the clash of big money and even bigger egos.  The result is a lot of drama and some very interesting stories.  The first of these stories involves the very first race for the Cup.

By the 1850's Americans, at least some of them, were full of beans and figured they were second to none.  The premier world naval power of the time was the British.  At the time nearly all boats were powered by sails as steam power was just starting to come into its own.  Sailboats of any size at the time were nearly always used for fishing, cargo hauling, or military purposes.  There were a large number of sailboats in use for recreational purposes but almost all of them were small (under 30' in length).  The rich of the period did sometimes own pleasure boats, which they typically used for entertainment or for transportation.  In the latter situation it would be akin to a movie star or big executive owning a private Jet today.

One of the groups of men (it was exclusively men at the time) who owned large private yachts were organized as the Royal Yacht Squadron.  All of them were wealthy and well connected to the British monarchy.  The Royal Yacht Squadron periodically organized races.  In most cases this was more of a social event than anything else but some of the members took the competition seriously.  And of course, being British, it was taken as a foregone conclusion that whoever had the fastest boat in the Royal Yacht Squadron fleet had the fastest boat in the world.  In 1851 a group of Americans, organized as the New York Yacht Club had enough beans to beg to differ.

So the Commodore (in modern terms - CEO) of the Yacht Club, one John Cox Stevens, commissioned the construction of a fast schooner of roughly a hundred foot length that was eventually christened "America" with the intention of challenging the British.  The boat was sailed across the Atlantic to France where it was reconfigured for racing and then sailed across the English Channel with the intention of taking on the best of the Royal Yacht Squadron.  The British knew the boat was coming and a couple of their fastest boats under the guise of escorting the America into harbor did some informal speed testing.  The results were shocking.  The British then spent the summer dodging a contest.  Finally they relented and invited the America to participate in a race around the Isle of Wight.  At stake was a trophy referred to in the early days as the "100 Guinea Cup", a typical trophy of the time for a RYS event.  Of course, as the British feared, the America won the race.

Fearing additional humiliation the British dithered rather than holding additional races and eventually the Americans took the Cup back to New York.  They then offered to host "friendly challenges" for the Cup in local waters and the Cup (and the race for ownership of the cup) eventually took on its modern name, "The America's Cup".  In the early years challenges were erratic.  And at some point the Americans started worrying that the British would find a way to win the cup back so they started to resort to dirty tricks (my characterization) to keep the Cup.  And, while early races were sailed in Long Island Sound, the closest appropriate body of water to New York City, the race was eventually moved to Newport, Rhode Island.  This was because the races were held in the Summer and everyone who was anyone went to Newport "for the season".

Among the most interesting of the early challengers was Sir Thomas Lipton.  He challenged four times and came close.  And, although he lost in the end every time, he succeeded in selling a lot of Tea and establishing the Lipton brand in the U.S. market.  So in the end his challenges were seen as a terrific success in terms of marketing.  He also comported himself as a perfect gentleman at all times (one reason for the success of the tea).  He sprinkled "Lipton Cup" trophies all over the U.S.  The Seattle Yacht Club in my neck of the woods has one of them on display to this day.

The initial race was organized as a Yacht Club to Yacht Club challenge and the contest continues that tradition to this day, at least nominally.  So the contest was open to boats owned by members of each yacht club and the rule was "first boat over the finish line wins".  Gradually it evolved into a race with lots of rules including a "class rule" defining the characteristics of an eligible boat.  A lot of sailboat races involve the use of a "handicapping" role that adjusts the times of the boats based on an estimate of their theoretical speed.  For many years the America's Cup has been a "boat for boat" race on the theory that the detailed class rule makes the theoretical speed of all boats identical.

Up to about the turn of the twentieth century boats of wildly differing design were raced.  Starting in about 1900 "class" rules were introduced and subsequently tightened in stages to the point where the boats looked roughly the same to the untrained eye.  By the 1930's the "J Class" rule was in use.  This resulted in the use of spectacularly gorgeous boats of about 130 foot length.  But the boats were also spectacularly expensive to build and maintain.  As the Great Depression continued, challenges tailed off, eventually to nothing by the late '30s.  After World War II it was felt that a cheaper boat would be needed if anyone was going to be able to afford to participate.  The "12 meter" rule was adopted resulted in very pretty boats of about 60' length coming into use.  The change was a success.  Challenges appeared and the race was held about every three years.  And the Americans kept winning.

There was some grumbling among challengers that the rules were tilted in favor of the defenders, all of whom had to be members of the New York Yacht Club.  But the rules were seen as fair enough that challengers appeared regularly.  And under the pressure of their fear of losing we did see some loosening up of things on the American side.  Ted Turner won the cup one year.  He was from Atlanta.  His nickname was "the mouth from the South".  Among his many achievements was the founding of CNN, the cable news channel.  Another member of the NYYC was Dennis Connor, a San Diego sailor.  He was admitted to the hallowed halls of the NYYC because he was generally considered the most fierce and successful competitor of his time.  All this set things up for the 1982 contest.

As I indicated, there was some resentment of the NYYC in international sailing circles over how the America's Cup events were staged.  The Australian (at some point the challenge, which started as strictly a British versus the Americans competition, was opened up to all comers) showed up with a boat that featured a "winged keel".  This was widely seen as a violation of the "12 meter" class rule.  But after all the activities of the NYYC over the years the international sailing committee that had jurisdiction of these matters decided that the winged keel was legal.  A boat that was otherwise considered on a par with the American boat ended up with a slight speed advantage.  And, to make a long and very interesting story short, the Australians won the competition four races to three.

The result of this shocking development was a dramatic increase in interest in the contest.  The Australians put on a truly excellent event in 1987.  By this time all the boats had winged keels and, true to his reputation, Dennis Connor won the event.  But now the stranglehold of the NYYC was broken so he took the race back to San Diego.  (He was at some point thinking of holding the event in Hawaii, which I thought would have spectacular.  But the San Diego people, who had put the money up to keep him competitive, demanded that the race be held in San Diego.)

Then, as has happened repeatedly in the America's Cup, weirdness set in.  Dennis had been very successful campaigning in 12 meter designs.  So it was his intention to continue to do so.  But remember how the NYYC ran things for decades.  Well, as a result of this there is a document called "the dead of gift" that is the America's Cup equivalent of the U.S. Constitution.  And the official interpreters of the dead of gift is the New York State Supreme Court.  Don't be fooled by the title.  It is equivalent to a "Superior Court" in most states.  A New Zealander named Sir Michael Fey read the deed carefully and found a loophole.  The New York Supreme Court agreed with him.  As a result Fey showed up in San Diego with a 135 foot monster and said "wanna race"?  After being blind sided with this piece of legal legerdemain, Dennis read the dead and said "sure but I am going to defend in a catamaran".  He too was successful in the New York Supreme Court so in 1989 we were treated to a race between a 135 foot sailboat of fairly traditional design and a 60' catamaran.

You know those south seas boats with outriggers.  They are catamarans.  A catamaran has two hulls.  The boat Dennis sailed has two similar hulls rather than a hull and an outrigger but that is a technical detail.  Dennis did something else very dramatic.  He built two boats.  One had traditional cloth sails.  (By now the "cloth" was actually some kind of space age fabric out of a chemistry lab but to the uninitiated it looked pretty much like standard fabric.)  But on the other boat he replaced the mast, the tall pole the sails hang from, and the mainsail, the big triangular sail attached to the back of the mast, with what amounted to an airplane wing that stuck straight up in the air.  Sailors had speculated that a wing design would work better than sails but no one had had the guts and money to try it out.  Well, Dennis had the money and desperation necessary to try it out.  And it turned out that an airplane wing works a lot better than a traditional mast and sail.  It is likely that Dennis's cloth sail boat would have beat the Kiwis.  But the "wing" boat was so fast that Dennis's team had to sail the boat badly in order to keep the victory margin down to only 20 or 30 minutes.

So the Cup stayed in San Diego.  But everyone wanted to avoid the possibility of another "Frankenstein" event.  This doomed the 12 meter rule.  The rule was just too constraining.  It resulted in a very safe, relatively pretty, but also relatively slow boat.  The result was the IACC rule.  It produced pretty boats that were relatively safe, although one boat did manage to break in half and sink in the middle of one race.  But the rule loosened things up enough for people to try a lot of things and the boats were significantly faster then the 12 meter boats.  The IACC boats were about 70' long and looked like people expected a sailboat to look.

The IACC rule was very successful for abut 15 years.  In the first couple of challenges people tried out all kinds of new things like two rudders and no keel.  But the wackier ideas did not pan out.  The Americans lost the second Cup to be sailed in IACC boats.  Dennis Connor is the only skipper to lose the Cup twice (in Newport in 1982 and later in San Diego in 1995).  By this time the race had become truly internationalized.  The winning boat was from New Zealand.  After a successful defense the Kiwis eventually lost to a Swiss boat in 2003.  Since Switzerland lacked a salt water coastline, and the dead effectively mandated the races take place in salt water, the contest was moved to Spain.

The calm of the IACC rule era came to an abrupt end in 2010 when Larry Ellison of Oracle software fame was successful in another series of legal wrangling's that made the Michael Fey 1988 legal shenanigans look like patty cake.  The result was that he secured the right to be the sole challenger that year.  The Swiss said "if you want to be that way we are going to defend in a 115' catamaran".  Larry responded "well, if you're going to be that way we will challenge in a 115' trimaran (three hulls) and we are going to use a wing sail".  Frankly the Swiss didn't think Ellison could put his boat together fast enough.  But Ellison dumped 250 million dollars into the campaign, got the boat built, and won the Cup.

Now I am going to change the subject slightly for a minute.  As I indicated above, the 1982 race was the first one to be shown on TV.  It turned out that the Australian challenge was a rating smash.  Boats from all over the world showed up.  And everyone had to tune in to cheer on their own national boat.  And the TV technology was now getting good enough to do a good job of showing the race.  A large international audience developed for the event.  And enough interesting things happened to maintain audience enthusiasm for subsequent Cups.

That is right up until 2007.  Going in it looked like all the ingredients for an exciting event were present.  And by this time the challenger series had become a very big deal.  The Louis Vuitton fashion house had been signed up as a continuing sponsor of the "Louis Vuitton Cup".  The actual America's Cup race was always between two boats, a challenger and a defender.  In 2007 11 challengers showed up.  The racing necessary to winnow this fleet down to a single boat was exciting and interesting.  So the Louis Vuitton Cup races were a TV success.  But then came the actual America's Cup races.  The Louis Vuitton Cup was decided on June 6.  Racing for the America's Cup was supposed to take place at the rate of a race a day with about one day in 3 thrown in as rest days.  But the last of the 7 races it took to decide the Cup did not take place until July 3.  The Swiss were very unsure of the speed of their boat.  They claimed that they did not interfere with the race committee but there were many delays and outright postponements.  This played bloody hell with the TV coverage.  Watching sailboats bob up and down waiting for something to happen does not make for compelling television.

2010 turned into an even bigger fiasco than 2007, from a TV perspective.  The boats were big and fast.  This should have made for good TV.  But as a result of the court room maneuvers there were only two America's Cup races and no Louis Vuitton Cup races at all.  To make a really bad situation even worse, the court machinations were so fierce that it was unclear when the races would go off until a week before the first race.  So most of the world had no rooting interest in the outcome.  Only the Swiss and the Americans had a dog in the hunt.  And the races were scheduled to go off literally in the middle of the night.  The start time translated to 2 AM in the Pacific Time Zone.  As a result of all this there was no TV coverage in the U.S.

And then there was the problem of the wind.  You can't have a sailboat race without wind.  Normally the Cup races were scheduled for the Summer.  The part of Spain (Valencia) that was the venue has pretty reliable winds in the Summer.  But, again due to the legal back and forth, the races ended up taking place in February.  Wind is much harder to come by in Valencia in February.  So the races were delayed until about 6 AM my time.  So each race day I was treated to about 4 hours of watching boats bob up and down before the race actually went off.  This is a shall we say less than optimal set of circumstances in which to build and keep a big TV audience.

So that brings us to the current America's Cup challenge, the thirty-fourth.  So, how's it going?  The organizers are very aware of the things have gone on the last two times around and have tried to fix them.  The first thing they did was to decide to hold the races in San Francisco, literally in the harbor.  This makes for wonderful backdrops.  Typically the boats have had to be towed miles out into the open ocean to the race course.  Then the background consists primarily of open water with a few spectator boats bobbing around.  The "Bay" backdrop is much more convenient for both competitors and spectators.  If you have a view of the right part of San Francisco Bay, say from an office building, then you can watch the boats and you can do it for free.  And you have not just the races to watch but all the training etc. that goes into preparing for the races.  This is a giant improvement.

The second thing they did was to try and make the races themselves more exciting to watch.  They did this two ways.  First then invested in a large amount of technology.  There are microphones and  TV cameras all over the boats.  So you can see a number of "crew's eye" views of what is going on.  It's the same idea as the "in car" cameras in NASCAR cars.  But the boats are much bigger (72' long), much more complex, and have a crew of about 10.  So you get a bunch of views of how the boat is behaving and what the crew is doing.  Then they have added a number of high tech overlays to the TV pictures.  You know how they will digitally add the "first down" line to a TV picture.  Will this is done to the N'th degree.

They add virtual 100 meter grid lines so you can see where the boats are on the course.  The boats have super-GPS so they know where the boats are down to the inch.  They also know how fast the boats are going and in what direction, also to very high precision.  So in the long shots they can tag each boat with a flag indicating the identity of the boat.  They can also show how fast and in what direction the boat is going.  They can even calculate which boat is ahead and by how much.  (This is difficult to do because you have to figure in the current wind direction and do some trigonometry to get the correct answer.)  They also put virtual tracks on the water so you can see exactly where each boat has gone.  All this makes it much easier to follow the action.

They have also amped up the action.  The boats they are using are not traditional pretty sailboats.  They are high tech catamarans.  The hulls are 72' long.  They have "wing" sails that are about 130' in size.  And they are freaking fast.  A typical 12 meter would go 6-8 knots (roughly miles per hour).  A typical IACC boat would go 10-12 knots.  These boats go 25-40 knots.  The commentators are frequently pointing out that the boats are going faster than the speed limit on the nearby Golden Gate bridge.  One boat actually went faster than 50 miles per hour for a few seconds.  Then there is the part that wasn't planned.  It just happened.

 The basic design of the wet part of the boat consists of two skinny hulls that are about 72' long.  Each hull has a long skinny rudder at the back that is in round numbers 10 feet deep and 2 feet wide.  Each hull is also equipped with a dagger board.  This is roughly 15' deep and perhaps 3' wide.  The dagger board on one hull, the "windward" hull is raised to cut down on drag and the other one is left down to reduce the sideways motion of the boat.  The rest of the boat is supposed to be dry above the water.  That's the idea anyhow.

But about a year ago the New Zealand team (officially "Emirates Team New Zealand", colloquially the "Kiwis") figured out how to hydrofoil the boat.  Dragging a wetted surface through the water, especially at high speed, takes a lot of energy.  If you dreg less wetted surface through the water you can go a lot faster if your sails, etc. provide the same amount of energy.  A hydrofoil is an airplane wing that is in the water.  It takes a big wing to lift something in the air.  It takes a much smaller wing to lift something in the water.  The Kiwis figured out how to get the whole boat out of the water with the exception of the rudder and the dagger board.

They did this by using a bent dagger board.  Most of it goes straight down as in the old configuration.  But they added a bend and an additional piece about 5' long that sticks out sideways.  They also put a small wing on the bottom of the rudder.  One of these boats weighs about 14,000 lbs. (very heavy, if you are lifting it yourself, very light for something that size).  Hydro foiling is very hard to pull off.  But if you do pull it off the small amount of surface area represented by the small wing on the bottom of the rudder and the bent part of the dagger board is enough to hold the rest of the boat out of the water.  And getting both 72' long hulls out of the water reduces the wetted surface dramatically.  And with both hulls out of the water the boat goes unbelievably fast, if you can avoid crashing it.  (A couple of boats have crashed and the result has been a number of parts floating in the water.)

So these boats are unbelievably fast.  And the scenery is great.  And the high tech toys really help viewers follow what's going on.  So the whole thing should be a roaring success.  What could possibly go wrong?  Well, a couple of things.  The first thing is that these boats are fantastically expensive.

Remember when 11 teams showed up to challenge in 2007.  Well, this time around the organizers did things to try to get the same kind of turnout.  They sponsored a bunch of races all over the world that were billed as "America's Cup runup races".  To make it easier for teams to participate in these runup races they had a bunch of essentially identical 45' catamarans with wing sails built.  With lots of events spread around the world sponsor money should have been easier to come by.  It worked.  They had about 8 teams participate.  Several teams campaigned two boats,  And the events were pretty popular. So the "runup" events were considered a success.

But when it came to be time to build the boats for the real event things changed.  The boats for the real event are 72' long and each team is on its own to build their boat.  This means that each team has much more freedom to come up with whatever they think will make their boat go fast.  But also drives the expense up a lot.  The boats are longer.  But they are also wider and taller.  So they don't cost roughly twice what a 45' boat will cost, they cost about 8 times (2x2x2).  This turned out to be too rich for most of the teams.  There are only three challengers, teams from Sweden (Artemis Challenge), Italy (officially "Luna Rosa" but essentially Prada), and the aforementioned Kiwis (oil money from the United Arab Emirates, boat building and sailing skills from the New Zealanders).

That's bad enough but then tragedy happened.  The Artemis boat flipped and a crew member got caught underneath.  It took 10 minutes to cut him free.  He had emergency air but only 3 minutes worth.  He drowned.  This caused everyone to become extremely safety conscious.  One of the complaints about the 2007 event was that a narrow wind range caused too many delays.  So the wind range for the 34th Cup was initially set to be very broad.  But after the heightened concern for safety the top end of the wind range was cut.  This has resulted in several races being cancelled due to too much wind.

As I write this we have finished the preliminary rounds and are into the finals of the Louis Vuitton Cup.  So far we have had four scheduled race days in this series.  Another change from previous Cups is to schedule two races per day.  So far at least one race has gone off each race day.  But on three of the four days scheduled so far the second race has had to be cancelled due to too much wind.  I would characterize this as "not good".  The good news is that we have gotten off at least one race each day.  This is not a disaster.  But having an event, in this case the second race, not go off at its scheduled time is not how you make TV executives happy.

And there has also been another problem.  The races have been parades.  If neither boat breaks down, then the boat that is the first one to the first mark has always wins the race, at least so far.  But the problem is even worse that that.  If neither boat breaks the Kiwis have always beat the Italians.  And if neither boat breaks the Italians always beat the Artemis boat.  In fact, at this point the Artemis boat has been officially eliminated.  This leaves only the Kiwis and the Italians.  That's why it is the "finals".  We are down to just two boats.  But so far the Italians have not been able to figure out how to beat the Kiwis.

Like the problem with the small field this problem may not be as bad as it looks. It has been true many times that there are fast boats and slow boats and it hasn't taken long to be able to sort out which is which.  In spite of this, TV audiences have been large and enthusiastic in the past.  The Louis Vuitton Cup contest is not over yet.  So it is possible that the Italians will find some way to start winning.  But they don't have much time.

And we really don't know who is faster between the (presumably) Kiwis and the defenders, the Oracle team sponsored by Larry Ellison.  There have been no "head to head" matches under race conditions.  On paper Oracle looks to be favored.  Ellison has very deep pockets and has assembled a first rate team.  But the Kiwis have pulled a rabbit out of a hat before.  Remember they have won the Cup once and defended it successfully once.  So they know how to play effectively at this level.  They have lots of oil money behind them.  And finally, they were the first team to pull off "foiling", the short hand name for this hydro foiling business.  It turns out to be very tricky to do.  And if you handle one of these boats badly, and by this I mean "slightly wrong", you can lose 100 meters of position relative to the other boat in a few seconds.  If you actually handle the boat badly then you can end up with a boat that is upside down and in pieces, also in a few seconds.

Let me wrap things up with a quick peak toward the future.  I don't know who is going to win.  I would have to give a slight edge to Oracle at this point.  But I may change my mind after the first race between the Oracle boat and the Louis Vuitton winner.  Regardless of how it comes out though, another interesting question is what will the 35th defense look like?  There are a lot of people including my mother that would like things to go back to when the boats looked like traditional sailboats.  I don't think that is going to happen.  I also think all this high tech TV stuff is here to stay.  I think everyone feels it is a good addition.  I hope that if the venue moves because Oracle loses, the new venue will be as audience friendly as the San Francisco Bay venue is.

Larry Ellison has opined that he is concerned with cost and safety issues relating to the current 72' boats.  He said he is thinking of maybe switching to the 45' "runup" boats for the Cup.  (This assumes he wins the Cup allowing him to maintain control.)  I am not sure it is possible to put that genie back in the bottle.  The current boats are way cool so I can't see going away from a "catamaran with a wing sail" design.  One easy fix to the safety issue is to jack up the weight requirements.  The current boats weigh about 14,000 lbs.  You can make the boats much stronger and do some additional changes by changing the rule to make the boats, say 1,000 lbs. heavier.  By requiring various parts to be a little heavier, a hundred pounds here, two hundred pounds there, you can eliminate any reason to make parts just strong enough.  If you are required to make a part heavier you may as well make it stronger.

You can mandate that the hulls be stronger by requiring them to be heavier. You can also change the rule to require them to contain more volume.  The increase in volume, if properly done, will make the boats harder to go nose over.  These and other rule changes can make the boats safer.  This will perhaps make them a little slower.  But I don't think that designers, builders, and sailors have run out of tricks to make the boats to go faster.  So I think you can make relatively small tweaks that make the boats a lot safer with little sacrifice in speed given an otherwise identical design.  But at the same time designers, builders, and sailors will simultaneously figure out how to make the boats faster.  The design will not stay "otherwise identical".  I think the result of combining pro-safety changes with "go faster" changes will be new boats that are both faster and safer than the current boats.

The biggest factor affecting the cost of the boats is their overall size.  The 45' boats are definitely cheaper.  But I am not sure that you can go to a 45' boat and have people associate that boat with the "biggest and baddest thing out there", which is what people expect of a modern America's Cup boat.  If I had to bet, I would predict that they will stick with the current 72 footer.  But if they do decide to change the size of the boat I would recommend that they not go all the way to 45'.  They could perhaps go to 60', the size of the Dennis Conner boat in '89.  Perhaps you can go a little smaller, say 55'.  But I don't think you want to go any smaller than that.  Downsizing to 60' or 55' would reduce cost and increase safety but it would also eliminate the "used boat" option.

One money saving option that has been available for most past Cups is to buy a boat from one of the teams after the race.  It is many years since an older boat has been competitive.  But it represents a starting point for a new team and a trial horse for an established team.  So even a non-competitive boat is helpful.  In the current Cup no one could get a used boat because no one had ever build a boat like the current ones before.  If the Cup sticks with the 72' boats there will be old boats available after the Cup has been decided.  Assuming none of the boats is destroyed, there will be an Artemis, one or two (I think 2 but I am not sure) Kiwi boats, one or two (again I am not sure) Italian boats, and two Oracle boats that represent possible "used boat" opportunities.  It's not a lot of boats but it is more than none.

One thing I am sure of is that the technology will continue to evolve at warp speed between the current defense and the next one.  It (the technology race) should be fun to watch regardless of how interesting (or not) the sailboat races also turn out to be.

Home Networking

The other day I saw an add in the paper.  I know.  I'm old.  I still read newspapers.  Anyhow, it was a full page add on the back page of the sports section.  This made it pretty high profile.  The headline in the add was "Public gets Free TV with no monthly bills".  Talk about a blast from the past.  The add was for a TV antenna, something you could hook up to your TV so that you could get channels "over the air for free" instead of paying for expensive cable or satellite service.  Then it hit me.  This "new" service is the way everyone used to get their TV back in the olden days.  But this company felt it needed to put an expensive add in the paper to tell people that they could still get TV the way everyone got TV a half century ago.  And you can.  Over the air TV is still there.  Its just that so few people now get their TV this way that lots of people need to be reminded that the option is still available.  And, of course, this company is still in the business of selling you the gadget you need to make it work, a new fangled version of the "rabbit ears" antenna of my youth.

This story is an example of technology on the move.  But the march of technology is a messy process.  And right now we are in a pretty messy place.  The "way of the present" (think "way of the future" and the phrase will make sense) is always a combination of the now possible and the way we have always done it.  I consider myself pretty tech savvy and my house is a mess.  I have all kinds of gadgets that are unnecessarily difficult to use and don't play together well.  I am annoyed by this but I believe I can figure all this out if I just take the time and energy to dig sufficiently deeply into the manuals for my various gadgets.  But I know lots of people who are completely baffled.  And I sincerely believe that these otherwise bright people don't stand a chance of figuring it out even if they did put an appropriate amount of effort into it.  They just don't have the "tech" gene I was blessed with so they are doomed to be permanently baffled.  It shouldn't be that way.  You shouldn't have to be the proverbial "rocket scientist" to get all these gadgets to play nice together.

The first thing I am going to do is to go over a little "how we got here" history.  Then I am going to peer a little way into the future to show where I think things are going.  I will finish up with a short remark on what I think the chances of my predicted future of actually coming into being.  And away we go.

A classic saw has it that "form follows function".  And there is a lot of truth to this saw.  And recently (e.g. the last 50 years) it has become apparent that what we now call technologists have a much greater impact on history than kings and generals.  For most of history historians have worked for kings and generals so it should be no surprise who ended up with top billing.  Roads and aqueducts made the Roman Empire possible but I don't know the name of a single designer or builder of either.  In the modern era technologists like Bill Gates and Steve Jobs have deployed massive public relations operations so everyone knows their names.  And they and their brethren will loom much larger than any political or military figure in the shaping of our times.  So lets talk about some functions.

I grew up in Seattle.  TV rolled out in the U.S. in the late '40s.  At the time sending a broadcast quality signal across country was fabulously expensive and terribly difficult.  So a single cable was built connecting the east coast (New York) to the west coast (Los Angeles).  There was a primitive connection that ran up and down parts of the east coast through New York.  There was a similar hookup on the west coast through Los Angeles.  So network TV could reach a chunk of the east coast and a chunk of the west coast.  But Seattle was too small a market.  So it wasn't on the national hookup.  So there was a "kinescope" gadget.  It consisted on one end of essentially a 16 mm movie camera hooked up to a TV set.  Film was shot of a particular TV show, then developed, printed, and thrown on an airplane and flown up to Seattle.  The other end of the kinescope consisted of a 16 mmm film projector hooked up to a TV camera.  This lash up was used to get network shows on the air on the one station then operating in Seattle.  Needless to say there was a lot of what is now called "local origination" going on at the time.  And these old kinescope movies ended up stored away in various Seattle garages where they eventually ended up providing most of the material used to study the early days of television broadcasting.

So how was what is now called "the last mile problem" solved in those days?  Well TV stations had great big radio transmitters that beamed out a signal that could be picked up in homes for miles around.  That was what the technology was capable of doing at the time so that's how it worked. More cables were installed and more TV stations got on the air through the '50s and '60s.  We ended up with the "big three" networks (ABC, CBS, and NBC) plus a part of a network (PBS) and "independent" stations.  Anyone who lived in a medium to large market typically had access to 5 or 6 stations:  the local affiliates of the big three, a PBS station, and one or two independents.  That was it.  You tuned in to whatever station you wanted to at a particular time and watched what they were broadcasting at that time or you turned your set off.  And most homes had only one TV as they were expensive.  A TV in the '50s cost about as much as a decent used car.

But time and technology marches on.  Besides accumulating all those kinescopes of early TV broadcasts that later turned out so valuable to historians, Seattle was responsible for another innovation.  Seattle has a lot of hills and these are serious hills.  TV signals go in roughly a straight line.  TV engineers know this so they built high towers to put their transmitting antennas on.  But still it turned out that a lot of Seattle was shaded.  There was some hill between you and the station's antenna so you couldn't pick up their signal.  The first TV station in Seattle was started as a gimmick to sell TV sets.   People wouldn't buy TV sets if they couldn't pick up TV stations.  One answer was high antennas on houses.  When my did bought our first TV set he had to get a 40' tall antenna put on top of the house to get a picture.  A neighbor had to put up a 60' antenna.  This is expensive and ugly.  There had to be a better way.

So the first TV station started putting a TV antenna on the top of a hill.  This could pick up the TV signal just fine.  Then they strung a bunch of wire to all the houses in the shadow of the local hill.  For a small price (they wanted lots of people to sign up so the made the service cheap) you could hook up to your neighborhood Community Antenna TV (CATV).  The name of the service got shortened to "cable TV" and spread like wildfire.  And someone decided you could make money with "cable TV" and started raising prices.  And that's where the modern "cable" industry came from.

You may have heard of another cable pioneer.  His name is Ted Turner.  In the '70s people figured out how to put satellites up that could be used to relay TV signals.  Originally this was fantastically expensive.  But over time the price dropped.  Renting a "transponder" is now so cheap that local TV stations do it so they can broadcast a minor sporting event involving a local team.  Ted figured out early on while prices were still pretty steep that transponders represented a business opportunity.  He owned a not very successful independent station in Atlanta.  He rented a transponder and put the signal on it.  He then offered it to cable companies for free.  At that time cable companies could handle more channels than there were broadcast channels in any specific market.  So for the price of a satellite receiver (then too expensive for an individual but cheap enough so a cable company could afford it) they could add a "bonus channel".  It didn't matter that the programming was pretty cheesy, it was free and it was at least a little different than what the local stations were offering.  Ted could raise his advertising rates for adds on his little Atlanta station enough to recover his costs and make a boatload of money besides.  Within a few years Ted was offering a number of channels including TBS and CNN.  It made him a billionaire several times over and paved the way for ESPN, C-SPAN, HBO, the 700 club, shopping channels, the hundreds of offerings served up by your local cable company.

This was all the result of the evolution of broadcast TV.  Broadcast TV begat cable TV.  The combination of cable TV and satellite transponders begat the modern TV landscape.  A lot of evolution has taken place in the technical details.  Color has replaced black & white.  Digital has replaced analog.  Hi-Def is replacing standard def.  But it's all just improved technology applied to the same basic idea.  But there has been a parallel evolution that is starting to be the dominant influence in TV ecology.  That is the internet.

The internet started out as a lab project in a corner of the Defense Department called (at the time) ARPA.  It was obvious that computers were becoming more and more important to the military and that hooking computers into a network, if it could be pulled off, had the possibility of becoming a game changer.  So the DOD funded something called ARPANET.  And it worked.  A bunch of computer science and electrical engineering types figured out how to network together a bunch of computers.  And they did it at the then unheard of speed of 50,000 bits per second.  At the time a "fast" modem was capable of communicating at 300 bits per second so this was way cool.  The technology kept improving but for a while it seemed like a solution in search of a problem.  Various military facilities and research universities were hooked up but it just kind of puttered along under the radar until a bunch of humanities types discovered email.

Email was the first killer application on the net.  It turned out to be fantastically valuable to humanities types because they could quickly and easily collaborate on research and publications.  Email was much faster and better in every way than snail mail.  All of a sudden network usage skyrocketed and the traffic was not driven by tech types.  Fortunately, the technology started improving and network speeds started going up, just in time to handle the rapid increase in traffic.  And every college and university in the country started clamoring to get hooked up.  And then people started clamoring to be able to hook up at home.  And the PC came out and it just kept spiraling.  ARPANET went public in the '90s.  Also in the '90s Tim Burners-Lee invented the concept of a web.  You put "web pages" up on a server that can be viewed by a "browser".  This made it possible to organize and present fantastic amounts of data to people through an intuitive interface that regular people would figure out.

The design of web pages was very flexible.  They could be used to present text.  But they could also be used to present other things like still pictures, audio, and video.  This opened fantastic opportunities.  The most obvious one for the purposes of our discussion is that the internet can be used to deliver video and TV is just video.  Technology limited us to a few broadcast video channels in the early days.  With cable interties and satellite transponders it became easier and easier to move video around the country.  Cable TV wiring enabled hundreds of channels to be fed into a home.  But there were restrictions.  Some of them were technical.  There is only one over the air radio spectrum.  There are hundreds of possible uses so it gets sliced and diced up.  Only a small amount of the radio spectrum can be dedicated to sending video over the air.  It will never be possible to do video "on demand" over the air.

With the cable companies the restrictions are more of a business nature.  The days of content being given away to cable companies for free are long gone.  Content is a precious commodity and who pays how much for what are now determined by painfully negotiated contracts between the players. And the "players" are the cable companies and content providers.  Us subscribers don't get a seat at the negotiating table.  As I write this there is a fight going on between CBS and Time Warner Cable about who will pay how much for CBS content on Time Warner Cable channels.  And cable subscribers get essentially no say in the eventual outcome.  All parties agree that it would be a bad thing if the suckers, excuse me, I mean the subscribers knew the details of the deal.  So we all get a choice between various cable "package" options bloated with channels we don't want.  The idea is to provide us with an option that is just barely perceived as a good deal.  The content of each offering is designed to maximize amount of money flowing into the coffers of the cable company.

Enough history.  Let's look at the present.  I will take my house as an example.  I have a "land line" phone.  It is more expensive than other options but I want something that works.  The regulatory authorities demand that land line phone companies but a tremendous amount of infrastructure in place so that lots of things can be broken and the phone still works.  This is not so true of cell phones,  It is definitely not so true of cable or internet phones.  I also have a cheapie prepaid cell phone.  It costs me a hundred bucks a year.  It is way cheaper than my land line.  But this is because I don't use it much. But there are a number of situations when it is nice to have.  So I have phone redundancy.

I also have a cable.  But it's not just cable.  It is also internet.  Both come in over the same wire.  But due to the magic of electronics they behave pretty much like they are separate services.  Except when something happens to the physical wire.  Then they both go out.  And the TV part of my cable service is much more reliable than the internet part.  The internet part has gone down when the TV part hasn't a number of times.  I don't think the TV part has ever gone down without the internet part going down at the same time.  Of course over the air TV is available but I haven's owned a TV antenna in decades so it might as well not be there.

Inside my house I have a "wired" network and a wireless network.  All of the wired part of my network is in the same room so I have been able to get away with just running wires along the floor up next to the wall where people don't notice them.  The wired part of my network runs at 1 gigabit.  That's way faster than most devices need.  But it is solid and trouble free.  There are three standards for wireless, "b", "g", and "n".  "b" is the oldest and slowest, "g" is in the middle, both by age and speed.  "n" is the newest and fastest.  I run a "g" network.  It is best to run only one speed.  At work we were having wireless network problems.  We finally traced the problem to devices hopping between "b" and "g" for reasons we were never able to figure out.  Every time the device hopped it dropped out for a couple of seconds.  We fixed the problem by running all the network hubs at only one speed.  With only one speed to pick from the hopping stopped and the problem went away.  I disabled the "b" capability of my home hub and have never regretted it.  I only have one device on my wireless network, a laptop belonging to my renter.  Even so, I occasionally have a problem with my wireless network.  It can usually be fixed by rebooting something but it is still annoying.

And have I mentioned that I have some wireless handsets for my land line phone.  These would stop working if the power went out.  But I do have an old fashioned phone that would keep working.

Now let's look at my TV setup.  I only have one TV.  I have a direct feed from my cable into the TV.  This used to work just fine but a component in my TV went out a couple of years ago and it doesn't work any more.  But that's ok.  I actually feed my cable signal into three devices.  One of them is an old VCR.  I haven't fired it up in ages and I have reason to believe it doesn't work any more.  It is an "analog" VCR so it probably doesn't know what to do with a digital cable signal.  When digital came in the FCC required TV and cable companies to maintain backward capability to the old NTSC standard.  But I think that has pretty much been phased out now.  The third feed is to my TiVo DVR box.  The TiVo and the VCR feed into my TV via auxiliary inputs.  So I generally watch TV through the TiVo.  That's why I don't much care that the direct feed to my TV doesn't work.

As an aside let me note that I have no "cable box"es, and, therefore, no cable remote controls.  This is because both my TV and my TiVo have something called a "cable card".  (The part of my TV that is broken is the part that interfaces with the cable card).  The cable card comes from the cable company and provides the programming that allows the TiVo and TV to accept and decode the cable signal properly.  It even unscrambles the scrambled signal for the scrambled channels that I pay to see.  So aside from the fact that the direct cable connection to my TV doesn't work, and my VCR doesn't work, and I have no way to "tape" something over the air on my VCR (assuming it is working) from my TiVo, everything is hunky dory.  (I note that I could rewire thing so that my VCR could tape from my TiVo but then I couldn't tape from over the air.)  I also have an old laserdisc player (think DVD player but a previous generation of technology) and a Blu-Ray player.  Both are wired into auxiliary ports of my TV but neither is wired so that I can record things from them to my VCR.  (Both have VCR ports but again I would have to disconnect and rewire things to get that to work.)

The latest thing is called "on demand".  I don't do on demand but I know others who do.  One friend does it using her Comcast remote on her TV that is hooked to Comcast cable through a Comcast cable box.  There are various ways to get on demand stuff to play on my PC.  My Blu-Ray also has various services I can sign up for.  It is hooked up to my wired internet so it has direct internet access.  The menu system on the Blu-Ray lets me sign up for and use these services.  I have not.  My TV is old enough (about 10 years) so that it does not have on demand type features built in.

This is a complicated mess.  I have 5 remotes.  I use the TiVo one 90% of the time.  It is programmed so that it can turn the TV off and on, change the volume, and change the input source.  The "channel change" function is actually handled directly by the TiVo.  But I occasionally have to use one of the other four remotes for one thing and another.  I have noted the lack of interconnectedness.  Another thing is feature redundancy.  The remotes control the devices they came with but they also want to control something else, typically the TV.  I can get on demand stuff a couple of different ways.  I have two cable cards because the TiVo needs to decode the cable signal sometimes and the TV other times.  Theoretically it would be nice if the VCR was cable card compatible.  (I also note that new digital TVs don't seem to come with cable card capability -- why not?)  And all the remotes work a little differently so there is a learning curve for each.  Why all the complexity?

Part of it is history.  I detailed the history of the TV path and the internet path.  If you want to drill down there is a history of VCRs, laserdisc/DVD/Blu-Ray devices, of DVRs, of "on demand", etc.  In the context of this history the current configuration of each device makes sense if you stick to only the history of that device and ignore the ecosystem the device operates in.  And you have a bunch of companies and industries fighting each other to get more than their fair share of the pie.  If you don't provide a certain feature or capability in your device then someone else is going to include your base capability as a feature or "add on" to their device.  That's how the current mess came to be.  Now let's imagine how it could be.

As I have noted above, over the air TV is small potatoes.  If it went away completely most of us would not notice.  I think there is still a place for local TV stations or more specifically local content.  What I am talking about most immediately is the method of delivery.  If I can get that local content over a wire into my house then I am getting the valuable part.  Historically local TV stations were the vehicle for delivering network content.  I don't think this is adding value any more.  There are any number of cable channels like USA and A&E that deliver network-like content.  I see no reason why I couldn't get the ABC or CBS or NBC "channel" without the content having gone through my local network affiliate.  The current model still prevails for what were technological reasons and are now political and financial reasons mixed in with a lot of inertia.

Boiling all this down, I want a "cable TV"-like capability.  I want what is local content (local news,  sports, "what's happening", etc.)  I also want all my cable channels (well at least some of them).  And I want a "watch TV"-like experience where I do my couch potato thing and it just comes at me.  But I don't need the current delivery system.  I certainly don't need the over the air delivery system.  But I also don't really need the "cable TV" delivery system either.  The internet can do all that without all the cable specific hardware, software, specifications, etc. of the current system.  Give me a screen that gives me the same options that my current cable menu gives me and I'm a happy camper.  In short, give me the internet part of what the wire that is connected to my house delivers, and a "cable TV" web site, and I don't need the "cable TV" part of what comes in on the wire now.  That would save a lot of hardware up and down the line.  It would also simplify things tremendously.

On demand is a mess.  You have You Tube, Netflix, HuLu, Comcast on Demand, and dozens and dozens of other on demand service providers out there.  They all have their own system with its pricing model, passwords, etc.  If I know what I want it is very hard to tell where I can get it from.  Ideally you would have a small number of providers.  Popular content would be available from more than one of them.  You could then sign up with the one that provided the best service and the pricing model you liked the best, confident that you could get what you wanted from that provider.

I am even ok with some categorization.  Let's say TCM-on-demand had every movie that had ever been made that was more than 10 years old and Hulu-on-demand had every TV show that was more than a year old and Universal-on-demand had every Universal movie that had been released to the theaters between 1 and 10 years ago, and, well you get the idea.  But now Hulu has some TV shows but not all.  And some are available for free and others require you sign up for and pay for the premium service.  And Netflix has a lot of movies but not all movies, not even all old movies.  It's just too confusing.  Then there are new Netflix shows like "House of Cards".  It's the wild west out there.  So on demand needs to settle and there need to be a few major players that have much broader and deeper libraries.  But let's say that in the next few years there is a big shakeout and we get a couple of on demand services that have broad enough scale that you can confidently sign up for one or a few of them and be confident you will get what you want.  Moving on.

What's the defining characteristic of a TV?  Historically there was one but not any more.  I remember the year the number of computer screens manufactured that year exceeded the number of TV picture tubes.  It turns out that a computer screen was just a high quality TV screen (in the early days) and a TV screen was just a low quality computer screen (later).  This high quality/low quality differentiation continues to this day.  A flat screen TV screen is just a low quality computer screen with different electronics delivering the image.  It all boils down to size, resolution, and picture quality.  A TV is now just a limited function computer screen.  And that's the best way to think of it.  What we have are just screens.  We have big screens (formerly TVs) on the wall.  We have medium size screens on computers.  We have small screens on portable devices like smart phones.  So what if the thing on the wall is just a big network accessible computer monitor.  It might be a good idea to have a network accessible audio system on the floor underneath it.  But we can make placement flexible.

The foundational idea is that we have a computer network in our home with devices connected to it.  Then we have boxes with capabilities connected to the network.  We can use wired or wireless connections as appropriate.  And the current network standards (wired LAN or Wi-Fi) are perfectly capable of provided all the functionality we need.  Some of the boxes can be of the traditional kinds like PCs, laptops, smart phones, etc.  We add some capabilities and we put those capabilities wherever it is most convenient.  What is currently a TV would be a limited capability device.  It would be a display.  It could also have a low end sound system built in if we wish.  If we want a better sound system we have one that consists of one or more network components.  Perhaps we have one woofer on the floor beneath the large display.  We have two medium/high frequency capable sound boxes to the left and right on the side of the room opposite the display..  We have three similar boxes to the left, center, and right of the display (or perhaps built in).  Together they constitute a 5.1 sound system.  Each unit would receive a separate digital signal across the network, decode it, amplify it and turn it into sound.

We need some more boxes.  A critical box is the internet interface.  It would connect to wire coming into the house on one side the home LAN on the other side.  Qualcomm CEO Paul Jacobs suggested that it is practical for this to run at a speed of 1 gigabit.  That would be more than fast enough to handle current and future internet loads, on demand loads, and, in my opinion, "cable TV emulation" loads.  1 GB wired LAN is mainstream now.  Speeds are rising on Wi-Fi but I think more improvement (e.g. beyond "n") is needed.  But there is every reason to believe that evolution in this area will continue.

That leaves us with the need for boxes with "TV world" capabilities.  The first requirement is the most obvious.  A box (and one box should be enough - multitask) needs to interface with the cable company.  It needs to know what is available (e.g. what feeds has the customer signed up for). On the customer side it would field requests (e.g. I want to see this show or this channel now) and serve up the response (e.g. here's the feed for channel "X" or show "Y").  The other box would be a DVR like repository, essentially a smart disk that would archive streams when they became available and serve them up when their display was requested.  Since the processing necessary to feed the video to the wall display (or smaller display or laptop or phone) is modest by modern standards it could be done by both the able box and the DVR box.  The same is true of the audio.

Another annoying feature of the modern world is the proliferation of remotes.  There is no reason these couldn't all be replaced by software.  The software could run on a PC, table, smartphone, whatever.  Microsoft standardized driver software with Windows 95.  For each video card they wrote a master driver module.  This was served by a parameter file that instructed the module about the particular quirks of a specific network card.  There is no reason the same thing could not be done with the various boxes I have discussed.  A "display" driver would be configured with the characteristics of the particular display in the family room.  It would be also configured for the smaller and less capable display in the master bedroom, etc.  This would enable true "universal remote" software to be developed.  You would load the configuration files for each of the components that needed to be controlled.  Then one piece of software on one device could control them all.  It could schedule which programs would come down from the "cable provider" at what time, which "on demand" programs would come down from which on demand provider, what should go up on the rec room screen (and be fed to the rec room sound system) now, etc.  Actually, since there are multiple members in most families and likely multiple screens, etc. multiple pieces of master control software could be in use on multiple devices.  But that is easy to sort out.

All the current "TV specification" devices would be replaced by "network specification" devices.  There would no doubt be a very messy transition period.  But in the end home networks hopefully will follow the car model.  In the olden days (before about 1965) cars were pretty simple to work on and there were a lot of backyard mechanics out there.  Then when catalytic converters started showing up cars started to get very complicated.  After several decades things have gotten simpler again.  Cars are far more complex.  But a lot of that complexity can be ignored.  All modern cars have a "diagnostic plug".  You get a special cable.  One end hooks up to your laptop.  The other end goes into the diagnostic plug.  In this way you can read out the car computer (actually multiple computers).  A modern car is just a bunch of FRUs hooked together.  A FRU is a Field Replaceable Unit.   The computer tells you which FRU is misbehaving.  You replace it and you are good to go.  And a lot of tune up stuff is now software.  You can tweak the performance by updating some table.  What this means is that it is now possible to be a backyard mechanic again.  You just need a little computer expertise.

Hopefully home networking will follow the same model.  In the old days things were simple and if you had a little talent you could do a lot of things yourself.  Everything now is a lot more complicated and only delivers a little more functionality.  But if we can make it out the other side then things will get simpler again and the capabilities that current setups theoretically deliver will be actually delivered.

I see no technical impediments to anything I have described.  All the impediments I see are political/financial.  There will be winners and losers.  No one wants to be a loser.  Corporations are now expert at working the lobbying game in DC.  It is easy to put regulatory impediments in front of the necessary changes.  Since these impediments would be buried in the fine print of thousand page bills no one will find them until after they have done their work.  There is no reason why much of the reconfigured equipment (e.g. "TV monitors" that only accept streaming signals across the network, audio equipment that accepts a network delivered audio signal) aren't already on the market.  Universal remote software exists.  But the device makers make it hard.  They expect the device to receive the same old infrared signals used by current remotes.  The hide the specs of those remote signals.

At some point I expect techies to start building the devices I have described themselves using the open source software model.  These devices are not so complex that the required expertise is beyond everyone.  It's just beyond most of us.  The expertise required to build a Unix from scratch was also beyond the expertise of most of us.  But it was done.  And now Linux is so successful that it has almost completely killed traditional Unix.