Windows 8.1 - try 1

A few days ago I took my first stab at Windows 8.1.  I have a lot of experience doing installs and upgrades.  In particular, I have done a lot of installs and upgrades of various Microsoft products.  My Microsoft OS experience goes way back to the DOS era that preceded Windows.  So I should know what I am doing.  But it certainly didn't show.

I know better.  When you are doing an install or upgrade, particularly for the first time, you should be organized and be careful.  I was neither.  Fortunately, I got out with my skin intact.  But it was a close run thing.  Windows 8.x has been quite controversial so I have no excuse.  I should have known better.  I will certainly behave differently next time.  And there will be a next time.  So what was I trying to do?

I was trying to upgrade my home PC.  I am currently running Windows 7 "Pro".  That's the version that is designed for work use.  The big difference between this version and the "home" version is that it supports Microsoft "NT" networking.  Most home users have a very loose network.  They don't share files and other resources between the various computers they have around the house.  They share networking infrastructure at the wiring (or Wi-Fi) level.  But this is primarily used so that all computers can access the internet.  In a work environment there are servers that house corporate hardware, data, and (sometimes) software.  Home users generally keep it simple.  But as a former system administrator I know how to set up and maintain an "NT" domain.  I have a box running Windows Server and I do the same kind of sharing in a small way that companies do in a large way.  Is there any good reason for me to do this?  No!  I do it because I can and because I want to.

So I was trying to upgrade my Windows 7 "Pro" desktop to Windows 8.1.  I made sure I got the "Pro" version of Windows 8.1 so that all my "NT Domain" stuff would continue to work.  And that part of the upgrade worked fine.  All the "NT Domain" stuff was working just fine after the upgrade.  And, as far as I know, this "Pro" and "NT Domain" stuff made absolutely no difference to the upgrade process.

I will outline how I should have proceeded below.  But let me start with how I actually proceeded.  Fortunately, I run the Microsoft provided "Backup" process daily and automatically.  That's what bailed me out.  I was able to "full restore" my computer back to its old Windows 7 self after I had given up on Windows 8.1.   That's one of the few things I got right.  So first step:  Full Backup.  And make sure you create a "Recovery disk".  This is a CD you can boot from that lets you run the restore software and access your backup files.  Remember you are restoring what amounts to a completely broken system so you need something that boots and runs independently of any software on your hard drive.  If you are not familiar with burning CDs and booting from them practice and test.  You don't have to actually run the restore.  But make sure you can get everything to work up to the stage where the restore wants to start running.  (If you don't think you can pull this off then you want to either hire the upgrade done or get a new computer and do a "computer to computer" transfer -- see below.)

My second step was to run the Microsoft "Upgrade Assistant".  This is available at //windows.com/upgrade.  This page has a lot of general requirements for upgrading and lots of links.  The page has two main sections:  Windows 8.1 and Windows 8.  At the bottom of the Windows 8.1 section is a link marked "Upgrade Assistant".  Click on that to run the assistant.  It will check your computer and issue a report.  This was my first disappointment.  It will tell you if any of your hardware is problematic.  That's good.  The closest to a problem I got was that my hardware did not support a new feature called "Secure Boot".  This just meant that this new feature would not be available to me.  This information is not exactly a show stopper.  So the report in summary said there were no show stopper issues.  If you run the report and show stopper issues show up STOP.  You must get these issues fixed before trying to go to 8.1.

So what was the disappointment?  Well the assistant gave me a list of software but the list was not complete.  I can't make hide nor hair out of what is in the list versus what is not listed.  Microsoft Silverlight was listed (along with a lot of other Microsoft software and plenty of non-Microsoft software like my Adobe Reader) but my Office 365 was not listed.  So the first lesson learned is:  You need to make a complete inventory of all the software on your PC.  Then divide it into software you care about (i.e. needs to work when you are done) and software you don't care about.  If the software is not a Windows component it will  have to be reinstalled later.

There should have been a bunch of additional steps (see below) but I just went from here to doing the install.  You need to decide whether you want to do a "clean" install or an "upgrade" install.  With a clean install you end up with a bare bones windows installation with none of your files and applications.  From there you go on to install whatever applications you want and load whatever files you want.  This gives you a "clean vanilla" box.  This will fix many problems where things have gotten messed up on your old box.  But you lose whatever you have of value from the old box.  Most people want to do an upgrade.  In this case all your files are carried over and eventually all (or at least the ones you care about) of your applications are carried over.  It turns out that the 8.1 upgrade process gives you a third, middle option.  You can have the upgrade carry over your personal files.  Microsoft calls this a "data only" upgrade.  My plan was to do an upgrade that would preserve both data and application settings but it didn't work out that way.

To do the upgrade I just inserted the proper CD.  Microsoft provides "32 bit" and "64 bit" CDs.  If your PC has more than 4 GB of RAM you will be forced to the 64 bit version.  Older versions provided more backward compatibility in the 32 bit version but the days of the 32 bit version are definitely numbered.  My PC has 8 GB of RAM so I was already running the 64 bit version of Windows 7 so I loaded the 64 bit 8.1 CD and let it "autorun".  The upgrade process itself takes roughly an hour (at least on my PC) and ran without problem (except see below) the first time.  It asks you some questions and then grinds away.  Your PC will be rebooted several times (in my case 3 times).  Another small annoyance was there was never a "you're done" screen.  I just noticed that I had a screen that looked like it might be the Windows 8.1 screen and nothing seemed to be happening.  Eventually I poked around and got a "Login" screen.  (Many home users set their PCs up so that the login screen gets bypassed.  This is bad security.  But, in this case, it might have been more obvious that the process was complete.)

Before going on let me point out the major problem I had that I didn't even know I had until later.  One of the install screens should have asked me to select one of three options:  (1)  "Keep Windows Settings, Personal Files, And Apps";  (2)  "Keep Personal Files Only"; and (3)  "Nothing".  These options represent "upgrade", "keep your personal files" and "clean install".  The option I wanted was the first one.  But the screen only showed options 2 and 3.  I selected option 2 and it worked just fine as far as it went.  But it caused my Outlook Address book, for instance, to be lost.  Since I had never worked with 8.1 and had certainly not tried an upgrade to 8.1 before I did not know that option 1 should have been listed.  I just chugged along after selecting option 2 as the option closest to what I wanted.  I plan to stop at this screen and ask Microsoft what's going if I hit this screen and option 1 is not shown.  I did ask Microsoft about this after the fact but they were not helpful.  I also have thought of a work around (see below) if I do not get a satisfactory answer.       

Once I logged in I was totally confused.  I was able to flounder around to some extent.  For instance, I was able to establish that all the "NT Domain" stuff had connected up ok.  I was also able to confirm that my personal files had been carried over successfully.  (Note:  Being an old timer I put my personal files in new separate directories.  I do NOT put them in "My Files" or "My Pictures" or any of the other "My whatever" locations Microsoft recommends you use.  I don't know if things would have carried over if I had used the "My whatever" location.  I am pretty confident they would but can not speak from personal knowledge on this issue.)  I was able to flounder about and find some things.  I was able to get my printer installed, for instance.  But I was not able to figure out how to print a web page.  And there were lots of other things I was not able to figure out.  When people say that Windows 8.x is a complete redesign, believe them.

After a certain amount of flailing around I gave up.  I used the "Repair disk" CD I had built earlier to boot into software that enabled me to do a full restore.  Since I have a backup scheduled to run daily the backup was very current.  After the restore was complete I was able to boot in the normal way back to my Windows 7 configuration, which had all my stuff and worked perfectly.  Based on my experience I strongly make the following recommendation:

Do not try to use Windows 8.x without getting and reading a book about how to use Windows 8.x.  You will not be able to figure it out on your own.

After putting my computer back together I went out and bought "Windows 8.1 Inside Out" by Tony Northrup.  It was the right book for me.  I have had good luck with Microsoft Press books in the past and I am happy with this one.  I deliberately selected a "techie" oriented book because I am a techie.  But it might not be right for you.  There is a "for dummies" book (and many others).  One of them might be a better fit for you.  But I repeat:  You will not be able to figure out Windows 8.x on your own.  Instead you will become very frustrated and come to hate the product.

I have learned through a long and varied and successful carrier as a computer techie that there is a secret to success.  Software systems have a "mind set".  There is a way of thinking behind their design and the way the designers intend them to be used.  If you can figure out the "mind set" then you will be much better off.  It will make it much easier for you to "intuit" how to do things and where to find things.  This allows you to bootstrap some basic knowledge about how to do some things into a more complete ability to drive the product.  It doesn't matter if a product can do something it you can't figure out how to make the product do it.

I have only sampled my "8.1" book.  But even this quick look has given my a lot of insight into the mind set of 8.x.  Windows 8.1 has a drastically different mind set in comparison to older versions of Windows.  However much you have absorbed of this older mind set will actually make it harder for you to figure out Windows 8.x.  The next time I dive into that particular deep end I expect to be able to swim much more effectively than I could the last time.  This is because I have been given a look at the 8.x mind set as a result of reading this book.  Before moving on let me give you a quick look at the new mind set.

I have been in this business for a long time.  In the old days most systems had a "Command Line Interface" (CLI).  You typed in obscure commands at the prompt.  I am still most comfortable in this old CLI world.  But the world has moved on.  Windows up through Windows 7 is designed to use a "Graphical User Interface" (GUI).  More specifically, it is designed for use with a keyboard and a mouse.  And as time has gone by this has meant more mouse, less keyboard.  We are now given "radio buttons" (little circles - click on one and it goes out on the other ones) or "check boxes" (click on the  box to check or uncheck a particular option).  These are both easily done with a mouse.  Then there is the "drop down list".  You click and you are shown a list.  You can click on any line in the list to select a particular item.  A drop down list is a substitute for typing.  You select the item from the list instead of typing its name in.  The more frequent use of drop down lists over time is an example of an effort to make it easier to drive the system using just the mouse.  The need for keyboard entry is minimized.  I have adapted to the replacement of the CLI with the GUI over time.  8.x is the next generation in this CLI to GUI evolution.

8.x is designed to work without there being any keyboard or mouse.  Instead you have a "touch screen" that you can tap or touch.  This is a kind of substitute for the mouse.  But a mouse is more precise than your finger and it has buttons and a scroll wheel on it.  8.x is designed to be driven literally by hand (i.e. using the touch screen).  You can tap or swipe or use two finger moves like opening or closing your fingers.  That's the natural way 8.x is designed to be operated.  So Microsoft has completely redesigned Windows to work well with this "by hand" way of operation.  They do provide "backward compatible" ways of doing keyboard or mouse things instead of "by hand" things but these are substitutes for the real thing.  To be comfortable with 8.x you need to think of "by hand" operations instead of keyboard or mouse things.  If, as is the case with me, you are doing the mouse/keyboard thing you need to think in terms of the mouse/keyboard substitute for the "by hand" operation.  I now understand that so I expect to be far less frustrated next time around.  I also intend to read much more of the book so I will know (or know where to find in the book) how to do various things.

A final note before I go into summarization mode.  Windows 8.x is not a good fit for a work environment.  There are multiple reasons for this.  Let me highlight a couple.  (1) Work environments are much more likely to be "heads down - data entry" environments.  8.x is poorly designed for this.  Remember that a keyboard is no longer a natural input device.  (2) In 8.x you are supposed to get your software from the Microsoft App store.  This is good in the sense that Microsoft will test Apps before putting them into the store.  This means they are much more likely to play nice.  But companies run lots of applications that are unlikely to be found in the MS App store.  Microsoft was provide a work around.  If you get the "Pro" (i.e. "for work") version, you can install the Apps in the traditional manner.  But still this is NOT the way you are supposed to do business.  (3)  It turns out that companies use a lot of old, in some cases very old, applications.

At my last job we bought a piece of lab equipment in about 2010.  The lab equipment company provided a PC to drive the equipment.  We were explicitly told "don't hook the PC up to the network".  Why?  (Start "tech talk" section - skip it if you don't care and you trust me)  The software that connected to the PC to the lab equipment was so old it did not use a buffered serial interface.  The "freeze out" time required by network cards caused the serial interface to miss some data bytes.  This made the instrument effectively unusable.  No network meant that the network card was not used so nothing interfered with collecting the data from the instrument and everything worked like it was supposed to.  Of course, we had to come up with a method to get the data from the PC to our other systems, which we did.   (End "tech talk" section)  Now at this time buffered serial interfaces were at least ten years old and software to use them was available for free on the internet.  But the company had been too cheap to upgrade their software to use buffering.

I do regular business in my personal life with at least two different companies that to this day use old "3270 emulation" software.  (It has a distinctive look that is easily recognizable once you've had it pointed out to you.)   "3270 emulation" software emulates a device called an IBM 3270 terminal that was popular in the '70s.  The company I worked for (not exactly on the cutting edge in many ways) retired the last system that needed 3270 emulation at the end of 1999 (remember Y2K - if you don't, that's ok, just focus on the "1999" part).  Depending on software that goes back to the '90s, or even earlier, or has other weird characteristics is all too common in corporate environments.  I'm sure that a lot of this can be overcome.  But it will take a lot of work and that sounds like a lot of money to corporate bean counters.  So I expect 8.x to have a tiny rate of penetration in the corporate world for at least the next several years.

So let me summarize my lessons learned so far:

1. Don't trust the software list that is generated by the "Upgrade Assistant" to be complete.
2. Get a book and read it before you start.  Otherwise, you will not be able to figure out how to drive 8.x
3. Make sure you have a way to put everything back (full restore) in case things go badly wrong.
4. Have a plan for each piece of software you are currently running.  Your plan may be as simple as "dump it - don't need it any more".
5. For those pieces of software you will be carrying forward make sure you know how to reinstall it and where to get the install software.  Note:  If you are using the "home" version of 8.x the only place you may be able to get it from is the Microsoft Store.
6. If it's a corporate PC - don't bother unless you have to.
7. Read and understand the "Upgrade plan" shown below.

I really think Microsoft has dome something interesting with Windows 8.1.  But it will take a lot of work to get there and it will take a lot of getting used to.  If that sounds daunting then stick with Windows 7.  If you are not going to go to 8.x then the changes between say XP and "7" are pretty modest and pretty easy to get used to.  So, if you are still on an older version of Windows and have decided not to go to 8.1 then I recommend you go to "7", if you can.  If your hardware is old enough it probably doesn't have enough gas to run "7".  In that case I recommend you get a new PC.  Low end PCs are pretty cheap and have plenty of gas to run "7" well.  Microsoft has a utility called "Easy Transfer".  It works wonderfully to move your old stuff from XP or Vista to "7" (or 8.x - see below).  If you are running anything older than XP you are asking for trouble and, unfortunately, your migration options are poor.  If you are running Vista I feel sorry for you.  But the good news is that almost all Vista boxes will run "7" just fine.

So, if you are still with me, here's my revised upgrade plan:

1. Document all the installed apps.  For each app:
   Come up with a plan (drop, upgrade, keep as is).
2. Back up your current configuration.  Run a full backup.  This will require a big chunk of space.  A thumb drive will probably not be big enough.  The simplest method is to use a USB connected external disk.  If you search Amazon for "usb portable hard drive" you will find a number of devices listed that are plenty big (1 TB or more) for less than $100.  The Best Buy web site also lists several models.  Most of them are USB 3.0.  Your older computer may only have USB 2.0.  As far as I can tell USB 3.0 devices should still work.  They will just be slower than if plugged into a USB 3.0 port.  Warning: I have not personally run any tests to verify this.
3. Install (if necessary) the Microsoft "Easy Transfer" utility on your current box.  I believe it comes pre-installed on Windows 7.  On "7" try "Start", "All Programs", "Accessories", "System Tools".  If you see "Windows Easy Transfer", it's already installed.  If not do a "Bing" search (a Google search gives confusing results).  You should be able to locate a Microsoft page you can download it from.
4. Run Easy Transfer.  (The following information is from the "7" version.)  Select:
   "Next" on the "Welcome" screen.
   "An external hard disk . . ." on the "What do you want to use . . ." screen
   "This is my old computer" on the "Which Computer . . ." screen.
   The "Choose what you want to transfer" screen will take a minute to fill in.  You can then uncheck accounts you do NOT want to carry over.  Do NOT uncheck "Shared".  Note that it will tell you how much data is going to be transferred at the bottom of the screen.  Click "Next" when you are done fiddling.
   You will now be prompted for a password.  I would suggest using one.  Don't forget to make a note of what you choose.
   Select the device that will house the MIG file by clicking on it.  Click "Open".
   Click "Save".  The MIG file build process will now run.
   You will be given an individual status on each component selected for saving.
   The process takes a while.  In my case about 20 GB gets saved.
5. Run the upgrade to 8.1.  Be sure to select the "Keep Windows Settings" option if you can.  If the setting is not there you might as well select "Nothing" as your files will be put back by easy transfer.
6. If you were able to keep windows settings then you don't have to import your MIG file.  If you need to process the MIG file just navigate around to it with Windows Explorer and double click on it.  Windows will automatically run the easy transfer import process based on the fact that the file name ends with "MIG".  This should result in all your settings now being present on your new system.  Note:  You will not be able to see a lot of them because the application that recognizes them hasn't been installed yet.
7. Install your applications.  Your settings should magically come back at this point.
8. Have fun navigating around in your new environment.

A variation on this plan can be used if you are moving on to new hardware.  You don't need to do a full backup because you will still have your old machine around.  You can also do a "machine to machine" easy transfer if both of your PCs are on your home LAN at the same time.  Select the appropriate answer on the "what do you want to use" screen.  You need to be running easy transfer on both boxes at the same time.  Fire it up on your old PC.  When it is ready to transfer it will give you a password (a number, as I recall).  Fire Easy Transfer up on the new box.  I can't tell you how to do this from my own knowledge but my book says "open the start screen, type transfer, and select Windows Easy Transfer".  All the files and setting will be copied over as above.  Then go ahead and do the application install, etc. as above.

I promise an update when I try this again.
    

Privacy

I have been about to write about about this subject for weeks.  But then a new revelation comes out and gives me an excuse to delay.  The latest (new Snowden revelations now seem like a continuous part of the background) is an opinion from a Federal Judge that activities of the NSA violate the Fourth Amendment of the Constitution.  Here is the entire text of the Fourth Amendment:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

It's short and unlike, for instance, the "commerce" clause, this language has been interpreted by the courts in a manner that pretty much aligns with how the average person would understand the meaning of the Amendment.  We are all familiar with the "search warrant" from innumerable cop shows.  A cop has to go before a judge and swear to the judge that he or she has "probable cause" to believe that if he or she searches a certain place (house, car, etc.) he or she will find a specific person (e.g. escaped prisoner) or thing (e.g. illegal drugs).  If the cop doesn't have a search warrant or searches the wrong place (i.e. car instead of house) or the cop finds the wrong thing (i.e. escaped prisoner instead of drugs) the cop can't seize whatever he or she has found.  And if he or she goes ahead anyhow the evidence obtained as a result of the impropriety is "tainted" and can't be used in a court of law.

Now there are some complexities, but let's ignore them for the moment and look at the broader picture.  We all expect a reasonable degree of privacy.  In U.S. law that flows from the Fourth Amendment.  But an expectation of privacy is found everywhere and every when.  So in the most broadest context what's our expectation when it comes to privacy?

For most of history and in most of the world people lived in villages.  The village consisted of a grouping of huts.  A hut has walls, a roof, and a door.  In some cases it has one or more windows.  In other cases there were no windows.  You couldn't see or, in most cases hear, what was going on within the hut without entering it.  Universally a set of manners grew up that said you couldn't enter the hut unless you lived there or you were invited.  And it was a breach of manners to peer in through the door or windows.  So if you were outside the hut you were in public.  If you were inside the hut you were in private.  But it came to extend beyond that.

Sex is usually considered a private act since time immemorial.  Sex usually happens inside the hut.  There were no interior walls or other barriers inside huts so generally speaking the other occupants of the hut were aware of sex, when it was happening.  But taboos developed where this activity was not discussed outside the hut and frequently the other occupants of the hut would ignore the fact that sex was going on while it was going on.  Other activities that happened inside the hut also became off limits to talk about.  They were private.

Many cultures use a variation of an "I see you" as a greeting.  The idea is that you are generally not there until you are acknowledged.  That transitions your activities from effectively being private (people more or less ignore what you are saying or doing) to public (what you are saying or doing goes "on the record").  So the differentiation between "private" and "public" became more sophisticated.  Various cultures developed different sets of rules but all cultures developed rules for separating activities into private and public activities.  And in many situations the boundary between private and public could become very complex.  The same act could be "private" to some observers while being "public" to others.

And technology marched on.  Huts were replaced by houses and buildings.  But the privacy rules evolved out of the "hut" model.  And oral communication became supplemented by written communication.  The adhesives that are required to construct an envelope are a modern development.  In the past a letter was folded in an elaborate way and then "sealed".  Wax was dripped in such a way as to glue the folded paper closed.  A "seal" or "signet" or "chop" was impressed into the wax.  This identified who had sealed the message and was supposed to make tampering evident.  A universal convention quickly developed that what was on the outside, plainly visible without breaking the seal, was public.  What couldn't be read without breaking the seal was private.

These conventions form the basis of the common understanding of privacy.  What's inside the building is private.  What's outside the building is public.  What's on the inside of the letter is private.  What's on the outside of the letter is public.  Elaborations and exceptions were developed.  As cultures get more complex the rules get more complex.  But this is the basic, the "natural expectation" of people when it comes to privacy.  And it comports closely with the plain language of the Fourth Amendment.  The Amendment sets out the rules.  And it specifies the proper procedure for governmental authority to use to override the rule.

Now let's be honest here.  People have been breaking the rules as long as there have been rules.  People listen at doorways or windows.  People speak about what is supposed to remain unspoken with respect to goings on within the hut.  "Black Chamber" departments of governments developed techniques hundreds of years ago for opening letters, copying the contents, and reclosing them in such a way that no (or very little) evidence of tampering remained.  Codes and cyphers were developed so that diplomats could communicate securely in spite of the fact that their sealed correspondence was being read.

But there was a very practical reason why only special people like diplomats worried about their privacy.  It was very expensive to violate privacy conventions.  So Black Chamber departments snooped in the mail of diplomats but they did not have the resources to go after more than a few people.  So the ordinary person was secure from invasions of privacy by the government.  And any non-governmental person or group was breaking the law if they snooped.

At about the time that the Fourth Amendment was written Ben Franklin was investigating electricity.  He was the first to figure out that it came in two kinds.  He posited that it normally flowed from what he called "positive" to what he called "negative".  He was right that it flows but he got the usual direction wrong.  In round numbers, the telegraph was invented about fifty years later.  In round numbers the telephone was invented another fifty years after that.  In round numbers the computer was invented still another fifty years after that.  And finally, in round numbers the Internet was invented still another fifty years later.  Combined, these inventions ultimately revolutionized the privacy game.

The "letter" model was extended to cover the telegraph.  Physically the telegram and the means used to transmit, receive, and process it are public.  There is no practical method to seal the telegram up.  It must be clearly visible to telegraph operators.  The signal can be "tapped" from the telegraph wire.  But the telegram was wrapped in a virtual envelope of privacy.  Employees of telegraph companies were to treat the contents of all telegrams as confidential.  It is illegal to tap a telegraph wire.  In a practical sense, users adopted the "diplomatic letter" model.  They used a variety of codes and ciphers to conceal the contents of their telegraph messages.  A similar approach was adopted in the case of telephone calls.  Telephone company employees were expected to treat all telephone calls as confidential.  And the "get a search warrant" procedure was extended to cover telegrams and telephone calls.  With a warrant the contents of a telegram could be disclosed to a government official.  With a warrant a telephone line could be tapped.

For a long time, computers made no difference.  They were not much used for communication.  And, to the extent they were, they were treated like another piece of communications equipment.  And in its early days the Internet made little difference either.  The Internet is designed to be an open environment.  It is the equivalent of people standing in a public square and talking to each other.  It is bad manners to listen in on the conversation of others but it is certainly possible.  If people want to keep their conversation private they should "get a room" (use something other than the Internet for their communication).  But then a revolution gradually happened.  Computers (and data storage) became unbelievably cheap.  And the speed, ease of use, and ubiquity of the Internet made it the communications channel of choice.  So now more and more communication is done over the Internet.  The cheapness of computers (both the cost of computation and the cost of storing data) and the convenience of the Internet have created a revolution that has only recently been noticed.

As I indicated above, the practical reason that guaranteed our privacy was the difficulty (both in cost and in effort) of violating out privacy.  The FBI under J. Edgar Hoover was famous for decades for tapping phones, opening mail, and doing "black bag jobs" (breaking into homes and businesses to violate the "houses, papers, and effects" of individuals and organizations).  This was completely illegal.  But Hoover's FBI concentrated their efforts on mobsters, spies, and politicians.  (The latter was targeted to secure blackmail material that as used to maintain Hoover's position of power until he died.)  The vast majority of us were safe because we did not fall into one of the above categories and Hoover did not have enough agents to broaden his list of targets.

But in the last few years there has been a profound change.  It is no longer to difficult and expensive as a practical matter to violate people's privacy wholesale.  Computers have evolved (down in cost, up in capability) enough to change the game.  The new capability is usually referred to as "big data".  Vast amounts of data can now be mined relatively inexpensively.  There is a database concept called a "join".  If you have a phone number it can be joined to a name.  The name and phone number can be joined to an address.  A name and address can be joined to a driver's license number or a credit card number or even a social security number.  From there the joins can go on and on into nearly any direction you can think of.

And as recently as a few years ago data was spread all over the place.  Your drivers license information was separate from your credit card information.  And both of them were separated from your tax information (keyed to your social security number) and your medical information (again social security number but in a separate location).  The glue that is now available is the Internet.  Making data Internet accessible is almost a necessity in the modern era.  So now joins can be used to link data from databases that heretofore could not be linked up.  But they can now because they are all accessible via the Internet.

And there is now another class of player.  I used to work for a bank.  And I'd watch movies where the good guy or the bad guy (depending on the movie) would instantly access all this banking data.  It was hard not to laugh.  I knew that our bank did not have this data instantly available.  It was hard to keep it available to the computers located in our data center, let alone anyone or anything located anywhere else.  And I dealt with other banks enough to know that the same was true in their case.  But that was before cash machines and ubiquitous cred card and debit card readers, all hooked up to the bank's databases so that transactions could be validated instantly and cleared instantly.  Now various methods are being worked out so you can buy stuff with your smartphone.  There are currently various incompatible schemes for doing this.  But people all over the world will be able to buy pretty much anything with their smartphone within a decade.  There's just too much money to be made once it's working for anything to get in the way.

Traditional players like banks are in the game and that's an obvious development.  But we also have new players like social media in the guise of Facebook and search purveyors in the guise of Google who are gathering vast amounts of personal data.  The old business model was "we invade your privacy but only to the extent necessary to do business with you".  So banks held personal financial data about you.  And doctors and hospitals held personal medical data about you.  And until recently this personal data was not very useful for anything other than its original purpose.  Doctors in particular have been slow to move from paper records to computerized records.  And if it's not in the computer, it is prohibitively expensive to search.

Consciously or unconsciously we knew that these old line businesses could invade out privacy.  But they had been around a while and standards and practices (and a certain amount of law) had grown up around them.  But Google is less than fifteen years old and Facebook less than ten.  And both companies and their new line brethren have been evolving rapidly.  And these new line companies have substantial big data expertise.  As a group, they invented big data.  It is the old line businesses and government agencies that are playing catch up.

In summary, we have the old line players (government, old line businesses, snoopy neighbors or business competitors) with vast new "big data" capabilities.  They are combined with new line players like search and social media players who have vast amounts of data and the capability to mine it.  The practical barriers to vast and systematic privacy invasions are gone.  They have been done in by cheap computers and Internet connectivity.  And don't forget the fact that as a practical matter we all live on the Internet now.  Illiterate rice farmers in rural villages in India are getting Internet connected via cheap low end phones whose capabilities are not to be found in sci-fi movies of 30 years ago.  We can expect the number of people who are not on the 'net to drop below a billion people within a few years and to virtually zero within a few years after that.  So what should we do?

Well, one thing not to do is to try to put the genie back in the bottle.  The Internet, cheap computers, and big data are not going to go away.  In fact, things are going to go the other way.  More stuff will get connected to the Internet and the Internet will get faster and cheaper.  Computers will also continue to get faster and cheaper.  And big data is still in its infancy.  Techniques for exploiting big data will continue to improve.  This will result in more and better ways for your privacy to get invaded.  The other thing not to try is to give up, to decide we live in an open world where more and more people have more and more capability to learn more and more about you and there's nothing that can or should be done about it.

Specifically with respect to the NSA, it looks like the worm is finally turning.  9/11 was used to scare all of us into turning the NSA loose to do whatever it could figure out how to do.  (They were also given vast amounts of money so cost was not an impediment.)  I don't know why but people were convinced that the NSA would only go after the bad guys.  Pretty much everyone can quote the old saw "power corrupts and absolute power corrupts absolutely" but people somehow believed that giving the NSA vast amounts of money and vast amounts of legal authority would not lead to incompetence, stupidity, and abuses.  It is in the nature of bureaucracies to build empires.  And when a bureaucracy can empire build in an environment where everything they do is secret and their mission is deemed critical then empires will be built.

My reading of the 9/11 commission report is that the government had all the information it needed to stop the attack.  But the information was "siloed", each bureaucracy kept its information in a "silo".  People in their organization got access to the information but people in other organizations did not.  Between the FBI, the CIA, and the NSA all the information to connect the dots and foil the plot was there.  But no one was able to put it all together because no one had access to all the information.  (The FBI is an egregious example here.  Two different supervisors shut down two different field office investigations that would have exposed the whole plot.  And here there was no siloing.  The FBI had everything it needed.)  In spite of this, additional "authorities" were given to various government agencies, but especially to the NSA.  They were authorized to vacuum up everything.  And they did.  There are supposed to be "checks and balances".  But they are pathetically weak.  And everyone who has been able to see what has really been going on has concluded that even these weak checks and balances were ignored routinely.  Read for instance, the opinion of Judge Leon in the NSA case I referred to above ("Klayman et al. v. Obama et al. -- U.S. District Court for the District of Columbia Civil Action No. 13-0851").

The biggest problem with the NSA is that they have collected vast amounts of data into a single NSA managed repository.  The obvious fix here is to take the repository out of the control of the NSA.  A White House study has apparently recommended this.  (The study is classified but this item has leaked and the leak is presumed accurate.)  I think this is a good idea.  I think a quasi-governmental entity should be set up.  This entity should own and maintain the repository.  It should be responsible for making sure that access rules are in place and that those rules are followed.  Telcos and others who currently contribute the data to the NSA don't want the job.  Who knows where the data will come from in the future so I think a separate entity is the way to go.  This means that the government has your data but at least the possibility would exist that it was secured and administered properly.  I think the budget for this new entity should come from the current NSA budget.

I think we should also look at how much of and what kind of data is being collected.  Various claims have been made as to how valuable the data has been so far.  But the data to justify the claim is highly classified.  I have been a student of intelligence matters for a long time.  The intelligence community is fond of saying "we have had many successes but we can't talk about them".  But the record shows that in fact they have had many failures and have used the classification system to hide or minimize them.  There is usually a political advantage for information on the successes to leak out and it does.  For instance, I know of no great cold war success that did not leak out within a few years, perhaps a decade.  The cold war ended more than a quarter of a century ago.  And in that time information on a number of failures and bungled jobs has dribbled out.  See also the Leon opinion I cited above for how valuable this database has been.  He says "the Government does not cite a single instance in which analysis of the NSA's bulk metadata collection actually stopped an imminent attack, or otherwise aided the Government" (page 61 - emphasis in the original).

As a rule of thumb I recommend this:  Of everything that is classified about one percent really needs to be secret for more than a short period of time, say a year (and often much less).  About ten percent is classified to avoid oversight leading to possible embarrassment and about 90% is classified as the result of sheer bureaucratic inertia.  Bottom line:  The NSA is collecting far more data than it needs or can even make use of.  (Although "big data" techniques continue to make it easier to make use of large amounts of data.)

But that's the NSA.  And, of course, similar restrictions could be put on other government agencies.  But, as I indicated above, there are now other non-governmental players:  the new line players and the old line players that are catching up.  Laws, rules, etc. can be used to reign in the government (at least in theory -- let's see what congress actually does).  But these non-governmental players are a whole different kettle of fish.  What should be done?

I have heard a number of proposals made over the years by pro-privacy groups.  I consider most of them misguided.  A couple of lifetimes ago (here I am using Internet years) Intel was going to put an easily accessible serial number in their "486" chip.  The pro-privacy people raised such a ruckus ("they'll be able to know who I am") that Intel gave up on the idea.  So what happened?  Software designers used another of the hundreds of unique numbers found on all computers.  So developers had to write a couple of hundred extra lines of code to find something that would behave like a serial number and went ahead with their plans.  So most of the pro-privacy suggestions are pretty useless.  Okay . . .

I think that a fundamental legal principle needs to be adopted:  If the data is about you then the owner of the data is you.  Right now whatever data Google collects about your searches is owned by Google.  They can promise you anything they want about what they will or won't do with that data but at the end of the day it's their data and they can do whatever they want with it. So they do.  Facebook has gotten into trouble multiple times because they keep tweaking the privacy settings so that it becomes harder and harder for you to keep anything private.  Why?  Because they want to sell as much data about you to as many companies as they can for as much money as possible.  The more data and the higher quality the data the more valuable it is to Facebook.  I don't have a Facebook account but it seems to me that Facebook recently got in trouble for selling pictures from your "wall" to whoever wanted to buy them.

If all this data about you (i.e. the pictures on your wall) was owned by you then the legal situation would be quite different.  Now the collector of the data (Facebook, your bank, your doctor, etc.) would be given a "use" right to your data.  They would be permitted to use it in whatever means were necessary to provide the service you signed up for.  But that would be it.  They would no longer have the right to sell the data (or trade it or lend it out) simply in order to make a buck.

This may seem off point but let's talk about world trade for a minute.  Why are Nike shoes made in China?  There are many components to the answer but I am going to single out only one.  It's because it is cheap to ship raw materials from all over the world to China and it is cheap to ship the finished shoe from China to the U.S.  Shipping is cheap.  And there are two components to this cheapness.  There is the actual cost of shipping the goods.  Then there are the tariff and non-tariff barriers or lack thereof.  As a response to the Great Depression the U.S. passed the Smoot-Hawley Tariff Act.  It succeeded in its design to increase the cost of imported goods. Another trick is to introduce "non-tariff barriers" to trade.  Japan was particularly good at this. They both have the same effect.  They increase the cost of imported goods or, in some cases make importing goods completely impossible.  It should come as no surprise to learn that many countries follow the lead of the U.S. and later Japan,  raising tariff (and later non-tariff) barriers that protected their domestic markets from foreign competition.  And these barriers were very successful.  Trade in the '30s when the tariff wars were at their most fierce fell to very low levels.  As barriers have slowly been removed, especially in the last few decades, world trade has grown by leaps and bounds.

I hope it is now clear why I chose to talk about trade.  Trade barriers worked as a method to reduce trade.  Data interchange used to be effectively impossible.  Think cave men on rafts.  Then it was very difficult.  Think trade around the Mediterranean a thousand years ago.  Then it was only difficult.  Think trade during the '30s.  Now it is quick and easy.  Think the current era of trade.  I suggest that the same techniques that depressed trade would depress wholesale data interchange.  Tariff and non-tariff barriers could be put into place to inhibit the current "wild west" trading of data between companies.  This would ease the pressure to invade people's privacy.

Ultimately norms of privacy protection need to be established and, where there is something to build on, enhanced.  Laws and regulations need to be put in place to protect data.  Companies need to be fined and executives need to be jailed for privacy failures.  It was recently reported that 40 million Target customers had their credit card information stolen.  Given that this kind of security failure is now common, Target will suffer some embarrassment, but very little.  I don't think they will be fined, not even by the card issuers that will actually suffer most of the loss.  And certainly no one at Target will go to jail unless it turns out to be an inside job.  The people that perpetrated this fraud are likely to get away with it.  Even if they don't, they are balancing vast potential gains against a small likelihood of getting caught and, in the worse case, a few years of jail time.  That looks like a pretty sweet proposition to me.

So things need to change.  It seems likely that things at the NSA will change.  But even that is not a certainty.  Change elsewhere is much needed but seems less likely.  Theoretically, libertarians and "defenders of the Constitution" (e.g. groups like the Tea Party) should be leading the charge.  But there is very little activity going on there.  There is some pro-privacy fire on the left.  But it is only some and powerful interests think they can be safely ignored.  Ultimately, the public needs to believe that the issue is important and that something can be done about it.  I think most of the public thinks that it is at least moderately important.  But I think most people are convinced that nothing can be done. 

Unless, of course, you are talking about any kind of information relating to guns.  Then everything can. should, and probably will be done to make sure that no one (except the NRA and gun manufacturers) has access to any kind of data about gun ownership.  If it's gun related then anyone, including foreigners and terrorists should be able to purchase any kind of gun they want.  And they should be able to purchase as many guns and as much ammunition as they want too.  Being on the terrorist watch list is not sufficient to deny you the rights and privileges of gun ownership.

    

            

  

Spies - Part 2

This is a continuation of an earlier post (http://sigma5.blogspot.com/2013/05/spies.html).  It is a result of the now long series of revelations credited to Edward Snowden.  There are more revelations today and it looks like they will continue for some time.  I am not interested in getting stuck in "revelation of the day" mode like the media.  Instead, since the revelations have been going on for some time, I think there are some "big picture" observations to make.

Recently Snowden has been revealing that the NSA has been spying on pretty much everybody, friends and enemies alike.  This is "shocking news" to the media and our allies.  There are two things going on here.  In the case of the news its the result of a lack of journalistic competence and the usual "fixate on the shiny object" mentality that governs news decisions.  In the case of our allies its a matter of gaining a tactical advantage in the "power" game.  I'll be back with more on both these subjects but first let me step back.

We have had two giant leakers in the last few years, Edward Snowden and, before him, Bradley Manning.  At the personal level they are completely different.  But at a higher level they are both the same.  Ben Franklin in the 1700's famously opined "three people can keep a secret if two of them are dead".  It's a cute way at getting to the fact that a secret shared among a lot of people is bound to leak out.  Franklin's idea of "a lot" was three.  So how many people have access to classified data?  The Atlantic Magazine, among many other sources, reports that the answer is staggeringly high.  This post (http://www.theatlanticwire.com/national/2013/06/contract-security-clearance-charts/66059/) is chock full of numbers.  But let's just focus on "Top Secret" (supposedly the "all access" highest level but actually not).  The Atlantic reports that nearly half a million people (including at one time both Manning and Snowden) have this level of access.

If half a million people have access to information what's the chance it will stay secret?  The answer is known to extremely high precision.  There is exactly zero chance that the information will stay secret for any length of time.  We know about Manning and Snowden.  But how about the people we don't know about because they haven't gone public with a big splash and gotten a lot of media coverage?  Madden and Snowden were both Americans that chose to take what they learned to the press.  What about the possibility of people taking this kind of information to Al Qaeda or the Taliban?

Actually, I think this is unlikely.  Both of these organizations smell bad and the intelligence agencies track them closely.  So, if they managed to get an inside connection it is likely the intelligence community would get a whiff.  But it is not impossible.  In the '30s the Russians managed to infiltrate a number of people into British Intelligence.  They ultimately rose to occupy positions at the highest levels of MI6.  They were not exposed until decades later.  If you want details, check out the Wikipedia page on "Cambridge 5".  I will have more to say on who is more likely to have gotten their hands on this kind of material later but first . . .

Manning and Snowden share another attribute.  Both were very low level employees with some technical expertise.  Manning's technical skill was sufficient to land him a job as a "developer" at a software company before he enlisted in the military.  Snowden did "IT Security" work and was a Systems Administrator working on classified facilities during his tenure within various components of the intelligence community.  Neither of them made any kind of mark as some kind of super computer guy.  As far as I can tell they were both journeyman IT types.  I am very familiar with this type because it's the kind of work I spent my professional life at.  We are smarter than average but there are lots of us.

Manning and Snowden are also similar in that there are red flags associated with both of them.  Manning had a long history of personal problems.  He now claims to have "gender identity disorder".  He had no business having access to classified data and a competent background check would have shown this.  But he got his "Top Secret" ticket punched anyhow, apparently without difficulty.  (This is another thread that has received little coverage.)  As far as I know there is nothing in Snowden's background or makeup that would have alerted anyone.  But there is a fundamental question, mostly unasked, as to why either of them had access to the information they had access to.

Manning was in the Army and was doing intelligence work related to the Iraq war.  He was a Private and his duties had nothing to do with anything diplomatic?  So why did he have access to hundreds of thousands of supposedly classified diplomatic documents?  Neither his rank (as low as it goes) nor his duties (supporting the Army mission in Iraq) justify access to these documents?  They didn't but he obviously obtained access anyhow.  As far as I can tell the security on the server that held these documents was so lax that someone with a modest amount of computer expertise and an interest in finding them was able to bypass security.  And as far as I can tell, Manning's access did not set off any alarms.  After the story broke and people traced his activities (probably with Manning's help), only then did anyone know what had gone wrong.  If an Army Private in Iraq could gain access to these documents, then who else could?  The answer is hundreds of thousands of people.

The Snowden case is similar.  I think Snowden's IT expertise was greater.  And, since Snowden came later, he was in a position to learn from Manning's experience.  But Snowden was not interested in keeping his identity secret for any length of time as Manning had been.  Snowden was also more intelligent (in my opinion) and did not have the many personal problems Manning had.  He had, as far as I can tell a very nice and very pretty (the pictures prove this) girlfriend.  And before everything blew up they seemed to be having a good relationship.  But at bottom the Snowden case is like the Manning case in the sense that it is important to ask the question "why did he have access to those documents?"

By the nature of the job, System Administrators have a lot of access.  Among their duties is that of setting and maintaining security levels for all the parts of whatever systems they administer.  They know the security controls better than anyone else.  It's their job.  And its a very unsexy job.  So it's the type of job that attracts "slow track" management types to the supervisory positions above them.  These management types also frequently get yelled at for doing their job too well.  It is common for one of them to get called on the carpet because some well connected hotdog can't get access to something he wants to get access to but is not supposed to get access to.  These management types tend to be happy when no one is yelling at them rather than being happy when their subordinates are doing their jobs well.  So the quality of work frequently depends more than it should on the personal skill and moral fortitude of low level people like Snowden.

That's inside baseball.  Its something I know because I've been there.  But there is a huge red flag that Snowden raised and the media has completely ignored.  Snowden said he got his last job at Booz Allen specifically because it would permit him broader access to the kind of material he was interested in.  There has been no follow up by the media on this.  Booz Allen has been around a long time.  They have been successful at getting and keeping defense and intelligence contracts for a long time.

This depends on getting and keeping profitable contracts.  It is more important to keep the government people who award contracts happy than it is to do good work.  In short, they have mastered the office politics of keeping in the good graces of the military and intelligence community brass.  What they look for from someone like Snowden is an ability to do adequate quality work quickly and an ability to not embarrass the brass.  A good way to do this is to take shortcuts on the assumption you will not get caught.  Shortcuts result in security loopholes that can be easily exploited by someone with the right mix of skill and knowledge.

Snowden was working in an obscure corner of the intelligence establishment in Hawaii.  Like Manning in Iraq, it is hard to imagine a place further from the District of Columbia, home of the State Department and the NSA.  But in this modern era networking makes distance unimportant.  Connectivity is what is important.  It is now obvious that connectivity permitted Manning to access political documents presumably housed on State Department servers.  It is now also obvious that connectivity permitted Snowden to access political/management documents presumably housed on NSA servers.  The NSA prides itself on being the best damn security shop in the world by a mile.  What we now know Snowden successfully accessed (and again as far as we know without raising alarms) should be profoundly embarrassing to the NSA.  I note that so far I haven't heard of any NSA brass or contracting companies getting fired.

Now let me circle back to the "who else could get access" question.  Any "secret" held by hundreds of thousands of people is not a secret.  What is surprising about the revelations credited to Manning and Snowden is not what they revealed.  It's that these secrets and others like them haven't been revealed tens, hundreds, even thousands of other times.  And this begs the question of whether they in fact haven't.

And this brings us to the dumb show and corresponding media coverage of the "shocking revelations" that the NSA has been spying on "friendly" foreign diplomats.  The best single reference on intelligence matters up to and through World War II is "The Code Breakers" by David Kahn.  People have between concealing messages going back at least to the Egyptians and the Romans.  And during this entire period other people have been trying to "crack" these concealed messages, and they have often been remarkably successful.

The modern era of this sort of thing goes back hundreds of years in Europe.  And it's not just Europe.  The ancient Chinese were also noted for this sort of thing.  Anyhow, in the European era it was common for governments to create "black chambers".  Mail would be intercepted, opened, copied, and put back into circulation in such a manner that the tampering was nearly impossible to detect.  The copied messages were decoded and the contents used to political and military advantage.  And the old dictate "keep your enemies close and your friends closer" was the order of the day.

Certainly all kinds of means were used in time of war.  But efforts continued during peacetime.  The Russians famously spied on everyone including their erstwhile friends the British, French, and Americans during the thirties.  And it's not just the "bad old Ruskies" or, before you ask, the "bad old Nazis".  Angela Merkel, the current German Chancellor was born and raised in what was then East Germany.  The East Germans ran a large, extensive, and very effective spying operation into what was then West Germany.  Some of it was for their Russian masters but a lot of it was for domestic consumption.

And then there's the French.  They have been running a large industrial/political espionage operation for decades.  A friend of mine worked for Boeing for a while.  Boeing gave employees traveling in France explicit instructions for taking extraordinary security measures while traveling in France.  Why?  France was interested in both of Boeing's lines of business.  They are competitors in civil aviation against Boeing commercial jets and in the military sphere, selling fighter jets and other kinds of equipment that Boeing Defense and Space also sells.

But you say "it was the cold war" in response to East Germany or "we've never been that close" in response to France.  Well, how about an ally that is closer to us than anyone but the U.K., Israel.  There have been not one but several scandals involving Israel spying on us.  And that ignores the time the Israelis sank a U.S. spy ship during the "Six Day War" in the '60s.   The official story is that it was "accidental" but that's an obvious cover story, "a tale told by an idiot, full of sound and fury, signifying nothing".  Why was the ship sunk?  Because the Israeli military had some tricks up its sleeve that they didn't want the U.S. to know about.

Both Manning and Snowden have exposed secrets.  The official line is that these are very damaging to U.S. interests.  And they are.  But they are not damaging because foreigners, especially the Europeans, didn't know these things.  It's because they no longer have "plausible deniability".  They could no longer pretend they didn't know these things.  And that's embarrassing to them.  It makes them look incompetent to the people that elect them.  If you are embarrassed by some one, in this case the U.S., then you want to hit back.  So we are being hounded from all sides.

It's also useful to get and apply whatever leverage comes your way when dealing with a powerful country like the U.S.  There may be some countries like Brazil where these revelations may actually be a surprise.  But here too, the U.S. has been treating Central and South American countries as second class citizens for at least a hundred years.  And we have repeatedly meddled in their internal affairs.  Quick:  How many coup attempts has the U.S. sponsored in South America?  How about Central America?  So if you are a Central or South American country and you get a chance to get a little back from the U.S., it's an opportunity not to be wasted.

All of the above is publically known to people who take the time to become informed.  It would be nice if the media took the time to become informed or assumed the public had the sophistication of a tree squirrel.  But they don't.  And the public time after time vindicates the media's low opinion of their viewers and (small number of remaining) readers.  It's very depressing.

And it's worth while taking a look at how we got here.  A standard bureaucratic tactic is to engage in empire building, gathering more power at the expense of your bureaucratic colleagues.  One of the manifestations of empire building is what are often referred to as "silos".  These are tube-like structures within which information flows up and down the organization.  They are nicknamed silos because information does not flow across to other parallel organizations.  We saw this in the run up to 9/11.  The CIA didn't share with the FBI and the NSA didn't share with anybody.  As a result the pattern wasn't clear because people inside one silo couldn't see the information in the other silos.  After 9/11 sharing was the new thing.  So databases were opened up and linked together.  This was done quickly and sloppily.

And at the same time vast new databases (i.e. all the telephone records the NSA now routinely collects) were put online and connected up so that everyone (as in hundreds of thousands of people) could see everything.  The government went from not enough sharing to too much sharing.  Everyone with a Top Secret clearance can see everything.  No one wants to be the person who stopped some other person from seeing some piece of data that would have stopped a terrorist attack.  And no one wants to be the person who stops some massive collection project that could have collected the piece of information that could have stopped a terrorist attack.  We are seeing this latter effect play itself out in the wake of the Boston Marathon bombing.  "Gee, if we had just collected some more data we could have stopped those guys."  The problem the intelligence community now has is that they are literally drowning in data.  So much stuff is coming in that no one can deal with any of it.

This is one of these "judgment" things.  It requires judgment to decide how much is the right amount.  But if judgment is applied then some of the time someone will get it wrong.  And then someone will take a pot shot.  Then everyone will hunker down and things will get even worse.  And the media is the last to champion judgment and the application of common sense.  It is much more fun and better for ratings to showcase some showboating politician jumping down the throat of someone who applied judgment and was unlucky.  And it is an even better show to feature some loud mouth going after entirely the wrong target.  I frequently despair of the media and the vast part of the audience who will not put the effort in to sorting the wheat from the chaff.  But probably, "'twas ever thus".   

2013 America's Cup

Recently I posted about the America's Cup in general.  That post can be found at:  http://sigma5.blogspot.com/2013/08/the-americas-cup.html.  Well, the current Cup event is over, finally.  This post summarizes my thoughts on how it went.

First, the results:  The defending team, titled "Oracle Team U.S.A." ended up winning beating the challenger, titled "Emirates Team New Zealand" (Emirates money coupled with New Zealand technical and sailing expertise).  They retained the cup in a titanic battle consisting of 19 finished races spread over 16 race days (days on which at least one race was completed) spread over 19 calendar days.  The final score was 9-8 but Oracle had to actually win 11 races because they started out with a 2 race penalty.  Judged either by the number of days from start to finish or the number of completed races (or the number of races including cancelled ones, for that matter), it was the longest cup series ever.  In the early days the event was a "one and done" event.  It morphed into a "best of 3" event and later into a "best of 7" event.  So historically only a few races were necessary to determine the winner.  So what do I think of it all?

Let's start with the observations and predictions I made in my previous post.  I guessed that the Oracle team would have a slight edge.  It turned out to be more complicated.  In the early going the Kiwis had an advantage in all three important categories:  boat speed, crew work, and tactics.  The Oracle boat was slower, but not by much.  There were only a few races in which either boat showed a substantially better turn of speed than the other boat.  The preparation work that the Kiwis did before the regatta started resulted in a boat that was initially slightly faster.  That preparation work, coupled by the "trial by fire" that the Louis Vuitton Cup represented resulted in a boat that was sailed better and had a better understanding of the tactics.  So the Kiwis dominated in the early races.

But the Oracle people learned fast.  Both teams did but the Oracle people were able to make more and better improvements.  By the middle of the regatta both teams were performing at such a high level in terms of both crew work and tactics that neither team chad a significant advantage in either of these categories.  But the Oracle boat kept getting faster throughout the entire regatta.  The Kiwis were probably improving their boat speed too.  But day by day the boat speed of the Oracle boat kept improving relative to the Kiwi boat.  By the end the Kiwis were no longer competitive.  The speed difference was small but it was enough.  No one wins 8 races in a row against a skilled competitor without having a faster boat.

I made (but did not include in the post) a number of other predictions.  I did not expect the series to go the distance.  I expected one boat or the other to be substantially faster.  So I expected the two race penalty to not be a factor.  And, in a perverse way I was right.  I expected that either the Kiwi boat would be faster and they would wrap it up long before the Oracle boat got close to the nine "official" wins the Oracle boat would need.  Or I expected the Oracle boat to be so fast that having to win a couple of extra races would not be enough to put the Kiwis into the hunt.  It turned out that the Kiwis were dominant in the early going.  The official score was 8-1 in favor of the Kiwis at one point.  But then the Oracle boat ran off a string of 8 wins in a row (not counting races that were abandoned either after they started or while they were within seconds of starting - those all went to the Kiwis but didn't count).  So my prediction that it would not go the distance turned out to be completely wrong.

Another unpublished prediction I made was that it would be a dull event.  The races would turn out to be parades where one boat quickly took control and the other boat was forced to follow at a suitable distance.  This too turned out to be wildly wrong.  Most of the races were extremely close.  In a lot of cases the behind boat was not able to make a pass but they were able to stick with the ahead boat.  And there were a surprising number of passes, more so than in any other Cup.  We even had a couple of races where one boat passed then the other boat passed back later in the same race.  The races themselves were quite exciting.  A major component was the blinding speed of the boats.  These boats are three times as fast as a traditional "competitive racing" boat of only a few years ago.  But the fact that the boats were never far from disaster (the "Nascar" effect) was always present too.  And finally, there was the "he's up -- no he's down" aspect.  It seemed like an actual contest where either boat had a serious chance of winning in a lot of the races.  This aspect has not been present often in Cup competitions.  

A final unpublished prediction was that the event, and here I mean the actual America's Cup, would be well administered.  Here too I turned out to be totally wrong.  The plan was to do two races per day and roughly a two days "on" and one day "off".  This should have resulted in 4 completed races every 3 days.  The actual result was 19 completed races in 19 days.  The organizers were only able to complete two races in one day on 6 occasions.  This must be balanced against the two different days on which no scheduled races were completed at all.  The smallest contributor to this delay were the teams.  Each team was issued a "get out of jail free" card that allowed them to cancel the second race of the day without penalty.  But only one of these cards was exercised (by Oracle).  Races were cancelled due to too much wind (frequently), not enough wind (two races one day, one race another day, and one race after it had already started - technically for "time limit expired"), and (shockingly) wind from the wrong direction.

Some of this can be attributed to a combination of mother nature and safety concerns.  The high wind limit was put in after the Artemis fatality.  But the low wind was a result of trying to schedule two races (expected duration - about 30 minutes each) into a 2 hour "TV" time window.  TV constraints also stopped the organizers from shifting  race starts to earlier in the day, which would have avoided the "wind builds late in the day" problem that caused several races to not go off.  And the organizers had opted to have a single course, whose layout could not be changed.  Normally Cup races take place in a circular area out in the ocean.  So organizers lay out the direction of the course based on the wind direction on race day.  Races only need to be cancelled or delayed when the wind direction is changing wildly and rapidly.  This was done to enhance the "fan appeal".  The course could be seen from many parts of downtown San Francisco.  And they built stands along the shore to accommodate the crowds.  But crowds never materialized.  A planned expansion of the stands was cancelled and the stands that were built were mostly empty.  So losing the flexibility of being able to change the direction of the course to accommodate the wind direction was lost and this translated directly into delays.

Finally, there is one problem that is completely inexplicable to me.  Between the Louis Vuitton Cup and the America's Cup, races were broadcast in four different places.  Races could be seen on You Tube and ESPN3 (both internet only) and on both NBC and NBCSN (both broadcast).  I can somewhat understand the NBC/MBCSN split.  The event was not expected to be a ratings powerhouse.  So pushing races off to NBCSN kept it in the family.  And You Tube is neutral.  It's owned by Google.  So pushing a race that is not expected to draw NBCSN sized ratings on to You Tube makes sense.  You Tube ran an "America's Cup" channel where you could view a lot of video of cup related content.  But ESPN3 is owned by ABC, arch competitor of NBC.  What's going on there?  The only theory I can think of is that it was a clever plot top depress ESPN3's ratings.  But, since the same broadcast crew was used for all the races, the cost to ABC of this move would be modest, and that's on a good day.

And frankly, this was not a big problem for the actual America's Cup itself.  The first two races were on NBC and all the other races were on NBCSN.  Once you got used to that, and the fact that TV coverage went from 1 PM to 3 PM (on the West Coast -- add 3 hours for the East Coast times) the races were not that hard to find.  But Oracle is a technology company.  Their core product is their Oracle database but they pride themselves on being full service.  You can buy hardware, applications, and, most germane to our discussion, internet expertise from Oracle.  There is an America's Cup web site at americascup.com.  There is a section on the site where you can look up details of TV coverage for anywhere in the world.  The schedule kept having to be changed and extended as the racing went on and on and on.  But after we got through the first few days when things were on schedule the TV information on the web site was no longer correct.  On many days the TV schedule for that day was completely missing, even if you checked a few minutes before the first race was scheduled to start.

To update the information would have required adding or changing text in an amount equal to a single Twitter tweet.  And a giant, highly capable (to her them tell it), company that prides itself on being able to provide solutions to company web sites that need to handle millions, perhaps even billions, of transactions per day, should be up to the challenge of making a small single update per day to the Cup website.  And many of these companies Oracle is selling to need the information on their web site to be accurate on a time scale of seconds or less.  But apparently the event organizers could not manage to handle a "1 update a day" traffic load.  That is truly pathetic.

Looking forward, as I indicated in my previous post, I expect that the next cup will be raced in catamarans with solid sails.  And apparently my suggestion of downsizing to a boat that is 55-60 feel long has also occurred to a lot of other people.  I outlined the pros and cons of changing boat sizes in my last post so I won't repeat it here.  Since the Larry Ellison / Oracle people will remain in charge I do not expect the course to move.  If they can address the safety concerns the wind limit can be removed.  The low wind problem can be addressed by expanding the period during which races can go off from 2 to 2 1/2 hours.  With appropriate lead time I think TV people can accommodate this.  If nothing else they can do a delayed broadcast.  As someone who lives on the west coast I am way too familiar with this.  Many "live" shows are delayed three hours routinely for west coast broadcast.

And it is important to give them credit for some things they got right.  Affiliated events like "Americas Cup World Series" and "Youth Americas Cup" (both sailed in the smaller 45' boats) give more people access to and a reason to develop an interest in the main event.  The decision to go with wing sail catamarans is also a good one.  Organizers did not envision "foiling" (technically:  hydro foiling) but they have embraced it and I expect it to stay.

On the negative side, safety needs to be addressed.  But, as I indicated previously, I think this can be done between cups.  I expect the next cup in 3 years, plenty of time to figure out what changes need to be made and give participants ample time to implement them.  The big problem is cost.  Rumor has it that an AC72 boat costs between 8 and 10 million dollars to build.  If you include other costs, (paying the crew, a large and sophisticated development effort, on shore support, perhaps a second boat) the cost of an entire team is estimated to run $100 million or more.  Ellison is supposed to have spent $250 million to win the cup last time.  The fact that his boat kept getting faster and faster as this Cup event continued indicates to me that he did not stint on his "support" operation this time.  So I would guess he spent between $150 and $250 million this time.

The Kiwis came with a well run and presumably amply funded operation.  And they got slowly ground down by the Oracle operation.  Potential participants usually go with one of two approaches.  There are the "we just want to participate" bunch.  They don't want to be embarrassingly bad but mostly what they are looking for is exposure.  That can probably be done at the $100 million level, perhaps for a little less.  That's a lot of money to shell out for exposure, effectively an ad campaign.  If you want to be one of the "in it to win it" bunch, then it looks like you will need at least $150 million and perhaps a lot more.  You have to ask how many people there are out there that are willing to plunk down that kind of money.  That's what happened this time.  We had one defender and three challengers.  Of the three challengers the only one that had a chance was the Kiwi team.  So did Artemis and Prada get their money's worth?  And how much more will it cost to put together a Kiwi class effort.  And, remember the Kiwis failed, so you better plan on spending more money than they did.

The America's Cup has always been the kind of event where it is considered ok to pour lots of money in, to try to buy a victory.  So it goes against the grain to try to stop the "money fire hose" approach.  But it has gotten so expensive to play that it may be necessary to do something.  This has happened before.  The Cup went to "12 meter" boats in the '50s because the "J" boats had just become too expensive.  The whole "foiling" thing was come up with by the Kiwis a challenger.  Putting wings on keels was also an idea that a challenger came up with.  Dennis Connor put a "wing" sail on a catamaran first when he was desperate to successfully defend the cup.  Larry Ellison put one on a trimaran when he successfully challenged a few years ago.  No one, including Larry, thought the "trimaran" idea was a keeper.  So I don't think we will be seeing that idea come back, even though it worked in that race.  We should see how the 35th America's Cup will shape up within the next six months to a year.

The America's Cup

The America's Cup is the oldest continuously defended trophy in sport.  It dates back to 1851, before the Civil War.  The thirty-fourth defense of the trophy is happening in September of this year (2013).  I got seriously interested in the defense of the Cup in 1982.  1982 was also the first year it was broadcast live on TV.  Initially the races were broadcast on a local TV station in Newport, Rhode Island.  But by the time of the seventh race in the series interest was so great that the race that decided who won the cup was broadcast on national TV.  All of the Cup defenses, more or less, have been broadcast since.  Defense of the Cup involves the clash of big money and even bigger egos.  The result is a lot of drama and some very interesting stories.  The first of these stories involves the very first race for the Cup.

By the 1850's Americans, at least some of them, were full of beans and figured they were second to none.  The premier world naval power of the time was the British.  At the time nearly all boats were powered by sails as steam power was just starting to come into its own.  Sailboats of any size at the time were nearly always used for fishing, cargo hauling, or military purposes.  There were a large number of sailboats in use for recreational purposes but almost all of them were small (under 30' in length).  The rich of the period did sometimes own pleasure boats, which they typically used for entertainment or for transportation.  In the latter situation it would be akin to a movie star or big executive owning a private Jet today.

One of the groups of men (it was exclusively men at the time) who owned large private yachts were organized as the Royal Yacht Squadron.  All of them were wealthy and well connected to the British monarchy.  The Royal Yacht Squadron periodically organized races.  In most cases this was more of a social event than anything else but some of the members took the competition seriously.  And of course, being British, it was taken as a foregone conclusion that whoever had the fastest boat in the Royal Yacht Squadron fleet had the fastest boat in the world.  In 1851 a group of Americans, organized as the New York Yacht Club had enough beans to beg to differ.

So the Commodore (in modern terms - CEO) of the Yacht Club, one John Cox Stevens, commissioned the construction of a fast schooner of roughly a hundred foot length that was eventually christened "America" with the intention of challenging the British.  The boat was sailed across the Atlantic to France where it was reconfigured for racing and then sailed across the English Channel with the intention of taking on the best of the Royal Yacht Squadron.  The British knew the boat was coming and a couple of their fastest boats under the guise of escorting the America into harbor did some informal speed testing.  The results were shocking.  The British then spent the summer dodging a contest.  Finally they relented and invited the America to participate in a race around the Isle of Wight.  At stake was a trophy referred to in the early days as the "100 Guinea Cup", a typical trophy of the time for a RYS event.  Of course, as the British feared, the America won the race.

Fearing additional humiliation the British dithered rather than holding additional races and eventually the Americans took the Cup back to New York.  They then offered to host "friendly challenges" for the Cup in local waters and the Cup (and the race for ownership of the cup) eventually took on its modern name, "The America's Cup".  In the early years challenges were erratic.  And at some point the Americans started worrying that the British would find a way to win the cup back so they started to resort to dirty tricks (my characterization) to keep the Cup.  And, while early races were sailed in Long Island Sound, the closest appropriate body of water to New York City, the race was eventually moved to Newport, Rhode Island.  This was because the races were held in the Summer and everyone who was anyone went to Newport "for the season".

Among the most interesting of the early challengers was Sir Thomas Lipton.  He challenged four times and came close.  And, although he lost in the end every time, he succeeded in selling a lot of Tea and establishing the Lipton brand in the U.S. market.  So in the end his challenges were seen as a terrific success in terms of marketing.  He also comported himself as a perfect gentleman at all times (one reason for the success of the tea).  He sprinkled "Lipton Cup" trophies all over the U.S.  The Seattle Yacht Club in my neck of the woods has one of them on display to this day.

The initial race was organized as a Yacht Club to Yacht Club challenge and the contest continues that tradition to this day, at least nominally.  So the contest was open to boats owned by members of each yacht club and the rule was "first boat over the finish line wins".  Gradually it evolved into a race with lots of rules including a "class rule" defining the characteristics of an eligible boat.  A lot of sailboat races involve the use of a "handicapping" role that adjusts the times of the boats based on an estimate of their theoretical speed.  For many years the America's Cup has been a "boat for boat" race on the theory that the detailed class rule makes the theoretical speed of all boats identical.

Up to about the turn of the twentieth century boats of wildly differing design were raced.  Starting in about 1900 "class" rules were introduced and subsequently tightened in stages to the point where the boats looked roughly the same to the untrained eye.  By the 1930's the "J Class" rule was in use.  This resulted in the use of spectacularly gorgeous boats of about 130 foot length.  But the boats were also spectacularly expensive to build and maintain.  As the Great Depression continued, challenges tailed off, eventually to nothing by the late '30s.  After World War II it was felt that a cheaper boat would be needed if anyone was going to be able to afford to participate.  The "12 meter" rule was adopted resulted in very pretty boats of about 60' length coming into use.  The change was a success.  Challenges appeared and the race was held about every three years.  And the Americans kept winning.

There was some grumbling among challengers that the rules were tilted in favor of the defenders, all of whom had to be members of the New York Yacht Club.  But the rules were seen as fair enough that challengers appeared regularly.  And under the pressure of their fear of losing we did see some loosening up of things on the American side.  Ted Turner won the cup one year.  He was from Atlanta.  His nickname was "the mouth from the South".  Among his many achievements was the founding of CNN, the cable news channel.  Another member of the NYYC was Dennis Connor, a San Diego sailor.  He was admitted to the hallowed halls of the NYYC because he was generally considered the most fierce and successful competitor of his time.  All this set things up for the 1982 contest.

As I indicated, there was some resentment of the NYYC in international sailing circles over how the America's Cup events were staged.  The Australian (at some point the challenge, which started as strictly a British versus the Americans competition, was opened up to all comers) showed up with a boat that featured a "winged keel".  This was widely seen as a violation of the "12 meter" class rule.  But after all the activities of the NYYC over the years the international sailing committee that had jurisdiction of these matters decided that the winged keel was legal.  A boat that was otherwise considered on a par with the American boat ended up with a slight speed advantage.  And, to make a long and very interesting story short, the Australians won the competition four races to three.

The result of this shocking development was a dramatic increase in interest in the contest.  The Australians put on a truly excellent event in 1987.  By this time all the boats had winged keels and, true to his reputation, Dennis Connor won the event.  But now the stranglehold of the NYYC was broken so he took the race back to San Diego.  (He was at some point thinking of holding the event in Hawaii, which I thought would have spectacular.  But the San Diego people, who had put the money up to keep him competitive, demanded that the race be held in San Diego.)

Then, as has happened repeatedly in the America's Cup, weirdness set in.  Dennis had been very successful campaigning in 12 meter designs.  So it was his intention to continue to do so.  But remember how the NYYC ran things for decades.  Well, as a result of this there is a document called "the dead of gift" that is the America's Cup equivalent of the U.S. Constitution.  And the official interpreters of the dead of gift is the New York State Supreme Court.  Don't be fooled by the title.  It is equivalent to a "Superior Court" in most states.  A New Zealander named Sir Michael Fey read the deed carefully and found a loophole.  The New York Supreme Court agreed with him.  As a result Fey showed up in San Diego with a 135 foot monster and said "wanna race"?  After being blind sided with this piece of legal legerdemain, Dennis read the dead and said "sure but I am going to defend in a catamaran".  He too was successful in the New York Supreme Court so in 1989 we were treated to a race between a 135 foot sailboat of fairly traditional design and a 60' catamaran.

You know those south seas boats with outriggers.  They are catamarans.  A catamaran has two hulls.  The boat Dennis sailed has two similar hulls rather than a hull and an outrigger but that is a technical detail.  Dennis did something else very dramatic.  He built two boats.  One had traditional cloth sails.  (By now the "cloth" was actually some kind of space age fabric out of a chemistry lab but to the uninitiated it looked pretty much like standard fabric.)  But on the other boat he replaced the mast, the tall pole the sails hang from, and the mainsail, the big triangular sail attached to the back of the mast, with what amounted to an airplane wing that stuck straight up in the air.  Sailors had speculated that a wing design would work better than sails but no one had had the guts and money to try it out.  Well, Dennis had the money and desperation necessary to try it out.  And it turned out that an airplane wing works a lot better than a traditional mast and sail.  It is likely that Dennis's cloth sail boat would have beat the Kiwis.  But the "wing" boat was so fast that Dennis's team had to sail the boat badly in order to keep the victory margin down to only 20 or 30 minutes.

So the Cup stayed in San Diego.  But everyone wanted to avoid the possibility of another "Frankenstein" event.  This doomed the 12 meter rule.  The rule was just too constraining.  It resulted in a very safe, relatively pretty, but also relatively slow boat.  The result was the IACC rule.  It produced pretty boats that were relatively safe, although one boat did manage to break in half and sink in the middle of one race.  But the rule loosened things up enough for people to try a lot of things and the boats were significantly faster then the 12 meter boats.  The IACC boats were about 70' long and looked like people expected a sailboat to look.

The IACC rule was very successful for abut 15 years.  In the first couple of challenges people tried out all kinds of new things like two rudders and no keel.  But the wackier ideas did not pan out.  The Americans lost the second Cup to be sailed in IACC boats.  Dennis Connor is the only skipper to lose the Cup twice (in Newport in 1982 and later in San Diego in 1995).  By this time the race had become truly internationalized.  The winning boat was from New Zealand.  After a successful defense the Kiwis eventually lost to a Swiss boat in 2003.  Since Switzerland lacked a salt water coastline, and the dead effectively mandated the races take place in salt water, the contest was moved to Spain.

The calm of the IACC rule era came to an abrupt end in 2010 when Larry Ellison of Oracle software fame was successful in another series of legal wrangling's that made the Michael Fey 1988 legal shenanigans look like patty cake.  The result was that he secured the right to be the sole challenger that year.  The Swiss said "if you want to be that way we are going to defend in a 115' catamaran".  Larry responded "well, if you're going to be that way we will challenge in a 115' trimaran (three hulls) and we are going to use a wing sail".  Frankly the Swiss didn't think Ellison could put his boat together fast enough.  But Ellison dumped 250 million dollars into the campaign, got the boat built, and won the Cup.

Now I am going to change the subject slightly for a minute.  As I indicated above, the 1982 race was the first one to be shown on TV.  It turned out that the Australian challenge was a rating smash.  Boats from all over the world showed up.  And everyone had to tune in to cheer on their own national boat.  And the TV technology was now getting good enough to do a good job of showing the race.  A large international audience developed for the event.  And enough interesting things happened to maintain audience enthusiasm for subsequent Cups.

That is right up until 2007.  Going in it looked like all the ingredients for an exciting event were present.  And by this time the challenger series had become a very big deal.  The Louis Vuitton fashion house had been signed up as a continuing sponsor of the "Louis Vuitton Cup".  The actual America's Cup race was always between two boats, a challenger and a defender.  In 2007 11 challengers showed up.  The racing necessary to winnow this fleet down to a single boat was exciting and interesting.  So the Louis Vuitton Cup races were a TV success.  But then came the actual America's Cup races.  The Louis Vuitton Cup was decided on June 6.  Racing for the America's Cup was supposed to take place at the rate of a race a day with about one day in 3 thrown in as rest days.  But the last of the 7 races it took to decide the Cup did not take place until July 3.  The Swiss were very unsure of the speed of their boat.  They claimed that they did not interfere with the race committee but there were many delays and outright postponements.  This played bloody hell with the TV coverage.  Watching sailboats bob up and down waiting for something to happen does not make for compelling television.

2010 turned into an even bigger fiasco than 2007, from a TV perspective.  The boats were big and fast.  This should have made for good TV.  But as a result of the court room maneuvers there were only two America's Cup races and no Louis Vuitton Cup races at all.  To make a really bad situation even worse, the court machinations were so fierce that it was unclear when the races would go off until a week before the first race.  So most of the world had no rooting interest in the outcome.  Only the Swiss and the Americans had a dog in the hunt.  And the races were scheduled to go off literally in the middle of the night.  The start time translated to 2 AM in the Pacific Time Zone.  As a result of all this there was no TV coverage in the U.S.

And then there was the problem of the wind.  You can't have a sailboat race without wind.  Normally the Cup races were scheduled for the Summer.  The part of Spain (Valencia) that was the venue has pretty reliable winds in the Summer.  But, again due to the legal back and forth, the races ended up taking place in February.  Wind is much harder to come by in Valencia in February.  So the races were delayed until about 6 AM my time.  So each race day I was treated to about 4 hours of watching boats bob up and down before the race actually went off.  This is a shall we say less than optimal set of circumstances in which to build and keep a big TV audience.

So that brings us to the current America's Cup challenge, the thirty-fourth.  So, how's it going?  The organizers are very aware of the things have gone on the last two times around and have tried to fix them.  The first thing they did was to decide to hold the races in San Francisco, literally in the harbor.  This makes for wonderful backdrops.  Typically the boats have had to be towed miles out into the open ocean to the race course.  Then the background consists primarily of open water with a few spectator boats bobbing around.  The "Bay" backdrop is much more convenient for both competitors and spectators.  If you have a view of the right part of San Francisco Bay, say from an office building, then you can watch the boats and you can do it for free.  And you have not just the races to watch but all the training etc. that goes into preparing for the races.  This is a giant improvement.

The second thing they did was to try and make the races themselves more exciting to watch.  They did this two ways.  First then invested in a large amount of technology.  There are microphones and  TV cameras all over the boats.  So you can see a number of "crew's eye" views of what is going on.  It's the same idea as the "in car" cameras in NASCAR cars.  But the boats are much bigger (72' long), much more complex, and have a crew of about 10.  So you get a bunch of views of how the boat is behaving and what the crew is doing.  Then they have added a number of high tech overlays to the TV pictures.  You know how they will digitally add the "first down" line to a TV picture.  Will this is done to the N'th degree.

They add virtual 100 meter grid lines so you can see where the boats are on the course.  The boats have super-GPS so they know where the boats are down to the inch.  They also know how fast the boats are going and in what direction, also to very high precision.  So in the long shots they can tag each boat with a flag indicating the identity of the boat.  They can also show how fast and in what direction the boat is going.  They can even calculate which boat is ahead and by how much.  (This is difficult to do because you have to figure in the current wind direction and do some trigonometry to get the correct answer.)  They also put virtual tracks on the water so you can see exactly where each boat has gone.  All this makes it much easier to follow the action.

They have also amped up the action.  The boats they are using are not traditional pretty sailboats.  They are high tech catamarans.  The hulls are 72' long.  They have "wing" sails that are about 130' in size.  And they are freaking fast.  A typical 12 meter would go 6-8 knots (roughly miles per hour).  A typical IACC boat would go 10-12 knots.  These boats go 25-40 knots.  The commentators are frequently pointing out that the boats are going faster than the speed limit on the nearby Golden Gate bridge.  One boat actually went faster than 50 miles per hour for a few seconds.  Then there is the part that wasn't planned.  It just happened.

 The basic design of the wet part of the boat consists of two skinny hulls that are about 72' long.  Each hull has a long skinny rudder at the back that is in round numbers 10 feet deep and 2 feet wide.  Each hull is also equipped with a dagger board.  This is roughly 15' deep and perhaps 3' wide.  The dagger board on one hull, the "windward" hull is raised to cut down on drag and the other one is left down to reduce the sideways motion of the boat.  The rest of the boat is supposed to be dry above the water.  That's the idea anyhow.

But about a year ago the New Zealand team (officially "Emirates Team New Zealand", colloquially the "Kiwis") figured out how to hydrofoil the boat.  Dragging a wetted surface through the water, especially at high speed, takes a lot of energy.  If you dreg less wetted surface through the water you can go a lot faster if your sails, etc. provide the same amount of energy.  A hydrofoil is an airplane wing that is in the water.  It takes a big wing to lift something in the air.  It takes a much smaller wing to lift something in the water.  The Kiwis figured out how to get the whole boat out of the water with the exception of the rudder and the dagger board.

They did this by using a bent dagger board.  Most of it goes straight down as in the old configuration.  But they added a bend and an additional piece about 5' long that sticks out sideways.  They also put a small wing on the bottom of the rudder.  One of these boats weighs about 14,000 lbs. (very heavy, if you are lifting it yourself, very light for something that size).  Hydro foiling is very hard to pull off.  But if you do pull it off the small amount of surface area represented by the small wing on the bottom of the rudder and the bent part of the dagger board is enough to hold the rest of the boat out of the water.  And getting both 72' long hulls out of the water reduces the wetted surface dramatically.  And with both hulls out of the water the boat goes unbelievably fast, if you can avoid crashing it.  (A couple of boats have crashed and the result has been a number of parts floating in the water.)

So these boats are unbelievably fast.  And the scenery is great.  And the high tech toys really help viewers follow what's going on.  So the whole thing should be a roaring success.  What could possibly go wrong?  Well, a couple of things.  The first thing is that these boats are fantastically expensive.

Remember when 11 teams showed up to challenge in 2007.  Well, this time around the organizers did things to try to get the same kind of turnout.  They sponsored a bunch of races all over the world that were billed as "America's Cup runup races".  To make it easier for teams to participate in these runup races they had a bunch of essentially identical 45' catamarans with wing sails built.  With lots of events spread around the world sponsor money should have been easier to come by.  It worked.  They had about 8 teams participate.  Several teams campaigned two boats,  And the events were pretty popular. So the "runup" events were considered a success.

But when it came to be time to build the boats for the real event things changed.  The boats for the real event are 72' long and each team is on its own to build their boat.  This means that each team has much more freedom to come up with whatever they think will make their boat go fast.  But also drives the expense up a lot.  The boats are longer.  But they are also wider and taller.  So they don't cost roughly twice what a 45' boat will cost, they cost about 8 times (2x2x2).  This turned out to be too rich for most of the teams.  There are only three challengers, teams from Sweden (Artemis Challenge), Italy (officially "Luna Rosa" but essentially Prada), and the aforementioned Kiwis (oil money from the United Arab Emirates, boat building and sailing skills from the New Zealanders).

That's bad enough but then tragedy happened.  The Artemis boat flipped and a crew member got caught underneath.  It took 10 minutes to cut him free.  He had emergency air but only 3 minutes worth.  He drowned.  This caused everyone to become extremely safety conscious.  One of the complaints about the 2007 event was that a narrow wind range caused too many delays.  So the wind range for the 34th Cup was initially set to be very broad.  But after the heightened concern for safety the top end of the wind range was cut.  This has resulted in several races being cancelled due to too much wind.

As I write this we have finished the preliminary rounds and are into the finals of the Louis Vuitton Cup.  So far we have had four scheduled race days in this series.  Another change from previous Cups is to schedule two races per day.  So far at least one race has gone off each race day.  But on three of the four days scheduled so far the second race has had to be cancelled due to too much wind.  I would characterize this as "not good".  The good news is that we have gotten off at least one race each day.  This is not a disaster.  But having an event, in this case the second race, not go off at its scheduled time is not how you make TV executives happy.

And there has also been another problem.  The races have been parades.  If neither boat breaks down, then the boat that is the first one to the first mark has always wins the race, at least so far.  But the problem is even worse that that.  If neither boat breaks the Kiwis have always beat the Italians.  And if neither boat breaks the Italians always beat the Artemis boat.  In fact, at this point the Artemis boat has been officially eliminated.  This leaves only the Kiwis and the Italians.  That's why it is the "finals".  We are down to just two boats.  But so far the Italians have not been able to figure out how to beat the Kiwis.

Like the problem with the small field this problem may not be as bad as it looks. It has been true many times that there are fast boats and slow boats and it hasn't taken long to be able to sort out which is which.  In spite of this, TV audiences have been large and enthusiastic in the past.  The Louis Vuitton Cup contest is not over yet.  So it is possible that the Italians will find some way to start winning.  But they don't have much time.

And we really don't know who is faster between the (presumably) Kiwis and the defenders, the Oracle team sponsored by Larry Ellison.  There have been no "head to head" matches under race conditions.  On paper Oracle looks to be favored.  Ellison has very deep pockets and has assembled a first rate team.  But the Kiwis have pulled a rabbit out of a hat before.  Remember they have won the Cup once and defended it successfully once.  So they know how to play effectively at this level.  They have lots of oil money behind them.  And finally, they were the first team to pull off "foiling", the short hand name for this hydro foiling business.  It turns out to be very tricky to do.  And if you handle one of these boats badly, and by this I mean "slightly wrong", you can lose 100 meters of position relative to the other boat in a few seconds.  If you actually handle the boat badly then you can end up with a boat that is upside down and in pieces, also in a few seconds.

Let me wrap things up with a quick peak toward the future.  I don't know who is going to win.  I would have to give a slight edge to Oracle at this point.  But I may change my mind after the first race between the Oracle boat and the Louis Vuitton winner.  Regardless of how it comes out though, another interesting question is what will the 35th defense look like?  There are a lot of people including my mother that would like things to go back to when the boats looked like traditional sailboats.  I don't think that is going to happen.  I also think all this high tech TV stuff is here to stay.  I think everyone feels it is a good addition.  I hope that if the venue moves because Oracle loses, the new venue will be as audience friendly as the San Francisco Bay venue is.

Larry Ellison has opined that he is concerned with cost and safety issues relating to the current 72' boats.  He said he is thinking of maybe switching to the 45' "runup" boats for the Cup.  (This assumes he wins the Cup allowing him to maintain control.)  I am not sure it is possible to put that genie back in the bottle.  The current boats are way cool so I can't see going away from a "catamaran with a wing sail" design.  One easy fix to the safety issue is to jack up the weight requirements.  The current boats weigh about 14,000 lbs.  You can make the boats much stronger and do some additional changes by changing the rule to make the boats, say 1,000 lbs. heavier.  By requiring various parts to be a little heavier, a hundred pounds here, two hundred pounds there, you can eliminate any reason to make parts just strong enough.  If you are required to make a part heavier you may as well make it stronger.

You can mandate that the hulls be stronger by requiring them to be heavier. You can also change the rule to require them to contain more volume.  The increase in volume, if properly done, will make the boats harder to go nose over.  These and other rule changes can make the boats safer.  This will perhaps make them a little slower.  But I don't think that designers, builders, and sailors have run out of tricks to make the boats to go faster.  So I think you can make relatively small tweaks that make the boats a lot safer with little sacrifice in speed given an otherwise identical design.  But at the same time designers, builders, and sailors will simultaneously figure out how to make the boats faster.  The design will not stay "otherwise identical".  I think the result of combining pro-safety changes with "go faster" changes will be new boats that are both faster and safer than the current boats.

The biggest factor affecting the cost of the boats is their overall size.  The 45' boats are definitely cheaper.  But I am not sure that you can go to a 45' boat and have people associate that boat with the "biggest and baddest thing out there", which is what people expect of a modern America's Cup boat.  If I had to bet, I would predict that they will stick with the current 72 footer.  But if they do decide to change the size of the boat I would recommend that they not go all the way to 45'.  They could perhaps go to 60', the size of the Dennis Conner boat in '89.  Perhaps you can go a little smaller, say 55'.  But I don't think you want to go any smaller than that.  Downsizing to 60' or 55' would reduce cost and increase safety but it would also eliminate the "used boat" option.

One money saving option that has been available for most past Cups is to buy a boat from one of the teams after the race.  It is many years since an older boat has been competitive.  But it represents a starting point for a new team and a trial horse for an established team.  So even a non-competitive boat is helpful.  In the current Cup no one could get a used boat because no one had ever build a boat like the current ones before.  If the Cup sticks with the 72' boats there will be old boats available after the Cup has been decided.  Assuming none of the boats is destroyed, there will be an Artemis, one or two (I think 2 but I am not sure) Kiwi boats, one or two (again I am not sure) Italian boats, and two Oracle boats that represent possible "used boat" opportunities.  It's not a lot of boats but it is more than none.

One thing I am sure of is that the technology will continue to evolve at warp speed between the current defense and the next one.  It (the technology race) should be fun to watch regardless of how interesting (or not) the sailboat races also turn out to be.