Magic Johnson has surfaced in the news recently. His name has come up in the context of the "Donald Sterling" affair. As I write this it is unclear how things there are ultimately going to turn out. But that will have no effect on what I want to talk about.
On the theory that the Sterling affair has dropped from consciousness, let me briefly review it. Sterling has been the owner of the Los Angeles Clippers NBA franchise for 30 years. It has been an open secret for many years that he was a raciest. But things only blew up when he and his girlfriend/mistress V. Stiviano had a fight. (BTW, Sterling also has an "estranged" wife, Rochelle.) The fight was recorded and the media got hold of the recording. This brought everything out into the open. As of now Sterling has been fined 2 1/2 million dollars and banned for life from the NBA. He has also been told to sell the franchise. At this point it is unclear whether any of this will stick as Sterling says he is going to fight. I am going to leave it at that because the Sterling affair is just background.
Johnson is a famous basketball star who has stayed active and maintained a high public profile. As such he has been one of the people who the media has contacted for comment. So we have seen Johnson commenting on the Sterling affair. But Johnson is now also well known as a business man. So the other thing that people have weighed in on has to do with Johnson's prospects for being part of a syndicate that would eventually buy the team. And that's my segue into what I want to actually talk about. But first another digression.
The civil rights movement dates back to before the U.S. existed as a country. It has consistently been a fiercely contested battle. I want to review some of the history of this battle but I don't want to write volumes on the subject so I am going to start with the end of World War II and I am going to massively simplify things. A lot of black men ended up in the military during WWII. WWII was a "fight for freedom". They were not treated as free men while in the service and went home to an environment where a lot of people wanted to go "back to the good old days" of high levels of oppression. But, a line of thinking goes, this group of ex-soldiers decided they "weren't going to take it any more". However much or little truth there is to this idea, it is definitely true that the civil rights movement was pursued with a greatly increased amount of vigor after the war and this generated a great deal of pushback.
One thing that had changed was the advent of TV and with it TV news. Now the technology was very primitive in the '50s. Pictures were captured using hand held 16 millimeter film cameras. But even so, the ability to capture moving pictures far surpassed what had been possible previously. And movies of black people and their white supporters being set upon by vicious dogs, fire hoses, truncheons, and other forms of violence, made for compelling TV. So the national TV news programs devoted a lot of coverage to the struggle and many second string TV reporters became stars as a result. This resulted in even more coverage. And the coverage gave black people hope that real change was possible. So they raised their expectations.
The path of anything as big as the civil rights movement is never smooth. The combination of raised expectations and laggard delivery (and the heightened TV coverage didn't hurt) resulted in a series of race riots. Black people rioted and went on a rampage of destruction. This happened not once but several times so that a pattern was established. Rioters moved through the business districts of their communities. There they burned businesses and looted stores. One of the biggest of these happened in the Watts district of Los Angeles in 1965. The "Watts Riot" was perhaps the biggest but it was by no means the only one. There was a round of rioting in 1968 associated with the assassination of Dr. Martin Luther King, for instance.
I thought these riots were a bad idea because they did more harm than good. People doing business in black communities were only the tip of the iceberg. And that's if you assumed that they were bad people. Most of the people who "were part of the problem" lived and worked elsewhere. And a lot of the business people in black communities were not bad people. They were just people trying to get along. In fact many of them were Koreans who were subject to the same kinds of discrimination as blacks. But the result of this extended cycle of destruction was that a lot of legitimate business people decided that setting up in black communities was too risky. So they stayed away. And the people who stayed often felt that they had to jack prices up or reduce service in order to cover the "risk premium" of doing business in the black community. This resulted in what is what business school classes call an "underserved market".
So that's the general background. Now let me spend some time on the specific background of Magic Johnson. His legal name is Earvin Johnson Jr. With a mouthful like that it's no surprise that he acquired a nickname. Magic was born in 1959 and first came to prominence as a basketball player. He was a standout in college and went on to become a standout in the NBA playing for the Los Angeles Lakers. He was in his early '30s and at the height of his carrier when he was diagnosed with HIV in 9991. When he announced that he was HIV positive in November of 1991 he was one of the first high profile people to voluntarily make that kind of announcement.
At that time the AIDS epidemic had gone through several phases. The first phase was "what's this mystery ailment that is killing a lot of people in strange ways?" The second phase was "it's AIDS and it is inevitably fatal." The third phase was "well there are some drugs that seem to help." The fourth phase was "we actually have treatments that can keep you alive for long periods of time". AIDS has gone through a number of additional phases. It is now considered treatable and there are techniques that work well to limit its spread. Johnson made his announcement when things were transitioning from phase three to phase four. AIDS was then considered a very scary disease that had a high mortality rate and was easily transmitted by incidental contact. Besides the reduced mortality rate we now know a lot about how it is transmitted. And we know specifically that if an HIV positive person is taking his medications then he doesn't infect people through incidental contact.
AIDS at the time was also considered a "gay disease". It's not. It was just easiest to detect in the early days in the gay community. And it eventually came out that Johnson was not gay. He had contracted the disease by engaging in casual sex. Then and now, high profile athletes have a lot of opportunity to engage in casual sex. So Johnson was not unusual in this respect. He was unusual in his willingness to disclose his condition and in how he comported himself in the aftermath.
The "fear factor" then surrounding HIV drove Johnson to cut his NBA carrier short and to move on with the rest of his life sooner than he otherwise would have. And it is his business activities that I want to focus on. Professional athletes like Johnson make a lot of money. They are paid a lot. And on top of that they can earn even more in endorsements and appearance fees. Michael Jordon is a spectacular example of this. His "Air Jordon" shoe deal with Nike has made Nike and Jordon spectacular amounts of money. But athletes usually turn out to be poor business men. A certain percentage of them end up moderately successful as front men for car dealerships and the like. But more often than not they are broke a decade or so after their carrier ends. Johnson set out to go down a different path and he has succeeded.
He tried a number of ventures with more or less success before he got to the first one I want to talk about in more detail. What I am talking about is the Magic Johnson Theater chain. Starting out in 1994 Johnson approached the Lowe's chain of movie theaters. He wanted to open a movie theater in the Crenshaw district of Los Angeles. Crenshaw is a predominantly black neighborhood. And this is where Johnson and the civil rights movement converge. After the riots of the '60s businesses like Lowe's (after a series of mergers they are now part of the AMC movie theater chain) frankly did not want to do business in black neighborhoods. Even if the theater itself was not burned down they were concerned about vandalism and about getting and keeping good employees. The movie theater business is a "thin margin" business. Most of the ticket price goes back to the studios. If your facility required a high amount of maintenance (e.g. to fix vandalism) or has high insurance costs (e.g. a "riot" premium) or has high employee turnover (high labor costs) it is hard to turn a profit. All of these are legitimate concerns.
From the customer perspective, they were concerned about being treated poorly, about being ripped off (higher ticket prices to cover higher expenses), or getting a low quality product (letting the facility deteriorate due to high overhead costs). This made them reluctant to patronize such a facility. Thus fear and distrust on both sides resulted in a reluctance on the one side to do business in black neighborhoods and a reluctance to patronize non-black owned businesses on the other side.
This was the opportunity Johnson saw. At its most basic level Johnson could "front" for a white owned business. But Johnson wanted to put more on the table. He wanted to actually manage the business. He found the location. He wanted to put up what was an essentially standard multiplex movie theater. But he would provide the management and hire the staff. He also wanted to put his stamp on things. He did this in a small way. He changed the menu in the concession stand. He added some "black" offerings. This allowed him to go to the black community and say "this is being run by one of your own". I think it was probably a hard sell to Lowe's management. Could a basketball player put together a team to run a multiplex? I don't know if it was part of the original pitch but Johnson may have indicated that he wanted to franchise the idea if he could make the initial theater successful. In any case, Lowe's eventually bit and the theater opened and was a big success.
Johnson eventually did franchise the operation. He went on to open a number of Magic Johnson Theaters around the country. They, along with the rest of the multiplex business, have been hit hard by the crash of 2008 and by technology changes like Netflix. But Johnson succeeded in what he set out to do. Patrons treated the facilities with respect because they were treated with respect. The company was able to do business successfully in black communities along lines essentially similar to the way they did business elsewhere. It was a classic case of "win - win".
Johnson then leveraged his initial success. He started talking to Starbucks. He was able to convince Starbucks that there was a market for the high end coffee experience that Starbucks represents in minority communities in spite of the fact that the average income is lower there than it is in the areas Starbucks usually sets up shop in. The partnership has turned out to be quite successful on both sides. Starbucks just wants to sell high end coffee everywhere. Johnson turned out to be right that there was a profitable market in black and minority communities for what Starbucks was selling. It turns out that if black people believe they will be treated respectfully, as people "whose money spends just like every one else's money", then they are very happy to pony up the cash to participate in the Starbucks experience.
Johnson has since moved onward and upward to even more business success. And he is to be congratulated. But what I most want to celebrate is the fact that more than anyone else Johnson has been responsible for repairing the rift between mainstream businesses and the black community that was the result of the riots of the '60s. Black people now know that there are a large number of businesses that are happy to treat them just like everyone else. It turns out that's all they every wanted. And businesses now know that that they can be given a fair chance to win the loyalty of black consumers. And this is a good thing all around.
Is everything fine now? No! One of the things that has come out in the coverage of the Sterling affair is how racist the business practices he has long been engaged in are. But Sterling has benefited from them to such a great extent that he is now a billionaire. And, to make things even worse, he stands to profit to the tune of an additional half a billion dollars should he ultimately be forced to sell the Clippers. Sterling is not alone in profiting in a major way from bigotry. He is just the poster child of the moment for such behavior. And we don't know how long his moment in the spotlight will last.
And this brings me to my final observation. Magic Johnson is now a very wealthy man. He is now solidly ensconced on the rolls of the "one percenters". But I do not resent his wealth and hope he continues to be successful. The media likes to divide everyone into opposite extreme camps. In this case the camps would be the "good 99%-ers" and the "evil 1%-ers". But the situation here, like in most cases, is more complex. I think something needs to be done about the concentration of wealth and the fact that currently the U.S. resembles an oligarchy more than it does a democracy. But I wish Magic Johnson nothing but continued success. And I wish the same to Bill Gates and Warren Buffett, the two richest men in the country and, therefore, presumably the most oligarchic of the oligarchs. Why? Because all three of them have demonstrated a consistent commitment to the civic good, to working toward the goal of ensuring that everyone does well, not just the fortunate few. Unfortunately, many, perhaps most of the wealthiest among us are content with a situation where they and a few of their friends do well. As for the rest, they have nothing but contempt. They fall into the Sterling camp rather than the Johnson camp.
Wednesday, April 30, 2014
Wednesday, April 9, 2014
Vietnam - Lessons Learned
The most intense U. S. involvement with Vietnam spanned the period from 1966 to 1972. That roughly coincides with my time in college. So the big windup to Vietnam happened while I was in high school and the wind down happened while I was in the early stages of my career. The whole Vietnam experience had a profound impact on me even though I never had to serve. The U. S. involvement in the "Vietnam War" is generally characterized as a failure. This opinion set in quickly after it ended and was broadly held. I certainly saw it as such. So in the years following I spent some time trying to figure out how it went so horribly wrong and what should have been done instead. Society in general wanted desperately to move on so trying to figure out what lessons should be learned quickly faded from the public sphere of discourse. But I, and some academics and a number of political types, stuck with it.
Initially the impact was stealth. The U. S. developed an aversion to foreign military involvement. The subject surfaced briefly when the U. S. invaded Granada in 1983. But this was such a small affair that the discussion quickly died down again. It resurfaced again for the Gulf War (1990/91) and has been a subject bubbling quietly in the background for the current Iraq (officially over) and Afghanistan (still going on at the time of this writing) wars. In all these cases "Vietnam" has, at least in the public discourse, been mostly used by the political right to bash the left and vice versa. As is the norm with political discourse the discussion has been shallow and not very helpful. I propose to dig into things more deeply and to better effect.
The pop culture reason to do this is summarized by a quotation attributed to George Santayana. What he actually said ("The Life of Reason", volume 1 - 1905-06) was "Those who cannot remember the past are condemned to repeat it." Either the original or a slight distortion of the original surfaces regularly. Vietnam was a bad experience. It seems reasonable try to study Vietnam in the hope of finding ways to avoid future bad experiences. That seems like sufficient motivation to explore the topic. It certainly was my justification for putting some effort into the subject in the years following the wind down of the war. But this whole process is harder than it looks. Granada turned out fine. But it was so small it could hardly go bad in any major way. The Gulf War was supposed to help reduce the need for future wars in the area. It was manifestly a failure in that we were back in a hot war with the same country (Iraq) about a decade later. And neither the Iraq nor the Afghanistan wars have gone well. So whatever "learning" has happened has not been sufficient to get the "avoid bad outcomes" job done. With that as a preamble, on to the task at hand.
The Vietnamese have a name for the events in question. They call it "The 10,000 Day War". That's about 30 years. But the event we (Americans) see only lasted about 6 years. What's going on? The Vietnamese think the event started in 1945 and ended in 1975. What? Well, I think they are right. It is certainly easy to justify the 1975 end date. That's when what was then called Saigon and is now called Ho Chi Minh City fell. Saigon was the seat of the U. S. supported government in the South and with the fall of the city all resistance to the North ceased. But why the 1945 start date? That is a less well known (in the U. S.) part of the story.
World War II was fought on behalf of "freedom". At least that's the story the U. S. and its allies told. The other guys were Nazi Germany and the Empire of Japan. Both used aggressive military tactics to create and expand empires. To keep things focused I am going to ignore Germany (and other Axis allies) and concentrate on Japan. In the early part of the 20th Century Japan went on an empire building spree. Japan invaded and occupied Korea in 1910. They invaded Manchuria in 1931 and China in 1937. The Pearl Harbor attack on the U. S. in December of 1941 was a continuation of their efforts at imperial expansion. And, since France was an ally of the U. S., the Japanese went ahead and occupied what was then called French Indo-China in 1942. Later French Indo-China was broken up into several countries. One of them is Vietnam.
In an effort to avoid refighting World War II let me cut to the chase. In 1945 when Japan was defeated French Indo-China was being occupied by the Japanese. Having heard the siren call of "freedom" from the allies the Vietnamese (although still part of French Indo-China at this point) expected that the next step would be to allow Vietnam its freedom. They were sadly disappointed. The first thing that happened is that the U. S. decided it was stretched too thin. So they let the Japanese continue to occupy and control Vietnam. Ok. That's maybe not the best first step but its understandable. A couple of years later French Indo-China was partitioned into Vietnam and other countries. This was probably a good idea but beside the point in terms of the fight for freedom. However, the next thing that happened was that all the pieces were returned to French control. This is definitely an anti-freedom move when viewed from the Vietnamese perspective. By this time the Cold War was getting under way and the U. S. decided it needed a strong France as an ally. France wanted its colonial empire back so France got its colonial empire back. And that meant that France got control of its old Indo-China territory back. And that meant that Vietnam went back to French control.
The Vietnamese were understandably very unhappy about this. They had been sold out. So they back dated the point where they should have achieved their freedom to 1945, a not unreasonable move, and started the "freedom clock". They stopped the clock in 1975 and rounded the duration of the process off to 10,000 days. (Everybody likes round numbers.) And here be the first lesson: "If you are going to promise freedom people are going to expect you to deliver on your promise". George W. Bush did a lot of freedom promising. That concerned me because I did not see any indication that he wanted to actually deliver on the promise. Unfortunately, my concern has turned out to be justified. And there are now a lot of people around the world who dislike us because we have not followed through on that promise. I am mystified why so many U. S. commentators are "mystified" about why these people are mad at us.
This whole "1945" business is something that a lot of otherwise well informed people are not familiar with. So let me cover some more historical ground in an effort to shed more light. It should now be no surprise that the Vietnamese were unhappy. The center of effective action to remedy this unhappiness was Hanoi, a city in the north of Vietnam. And the political focus was a man named Ho Chi Minh. He had spent a lot of time exiled in Paris. He returned to Vietnam after the war and got active politically. While in Paris he had hung around in Communist circles. When he started trying to organize effective action he knew he needed powerful allies. France, the U. S., etc. were out of the question. So he turned to the other side, the then U. S. S. R., frequently short handed as "The Soviets". With his "Communist" credentials, and with the fact that he was well positioned to do U. S. interests harm, he was quickly embraced. Arms, money, training, etc. started flowing. Minh quickly recruited a man later generally referred to as General Giap to run the military side of things.
Minh started ramping up anti-imperialist (an accurate characterization as the French definitely saw Vietnam as part of the French "colonial empire") activity. Several years later this culminated in The Battle of Dien Bien Phu with Giap running the military operation for the Hanoi faction. The French were defeated, surrendering on May 7, 1954. (I note that the U. S. secretly propped the French up to the tune of a billion dollars while publically staying above the fray.) This led to a conference in Geneva, which in turn led to a temporary partitioning of the country into North and South, and a promised election in 1956. Rather than let the democratic process play out, which would have resulted in a win everywhere for the Hanoi faction, the U. S. got publically involved. They effectively sabotaged the election in the South and installed a puppet government in Saigon to run what was now called South Vietnam. They then started pumping a lot of money (publically and secretly) and other support into the Saigon government in an effort to create a "bastion of anti-communism". The Cold War was the big priority that drove tactics with respect to places like Vietnam.
The Saigon government was corrupt and inept. At this point most of the population of Vietnam, North and South, just wanted everyone to go away so that average people could just get back to their lives. But by this time the U. S. was heavily invested in South Vietnam and the Soviets were invested (but not so heavily) in North Vietnam so disengagement was not an option. The Hanoi government remained generally popular in the North and had a significant following in the South. What resulted was the first heavy involvement since the U. S. Revolutionary War in what was then referred to as a Guerilla War. (I noted the parallels at the time. But associating the "good" Revolutionary War patriots with the "bad" Communist guerillas was too much for most people and a serious effort was made to discourage this line of thought.)
By this point the U. S. had considerable familiarity with and success with "conventional" war. We had relatively recently fought WW I, WW II, and Korea, all conventional wars. Conventional wars involve the clash of large armies deploying the most powerful military weapons and tactics of the times. The conflict is decided by battle ships, bombers, artillery, tanks, and large numbers of infantry soldiers using "conventional" tactics. The core strategy is to destroy the opponent's army. Once that's done then everything else follows. In modern parlance this is sometimes referred to as a "symmetrical" war. In guerrilla wars, now popularly referred to as "asymmetrical" wars, an entirely different approach is required. Guerilla wars are called asymmetrical because the military capabilities of the two armed forces are wildly different. One is strong and the other is weak. In a conventional war the strong army would wipe up the floor with the weak army and coast quickly to victory. If anything is now obvious, it is that there is no "coast to victory" going on in modern guerrilla/asymmetrical warfare.
But Vietnam was the first real experience the U. S. had with this sort of thing. Theoretically, this was recognized early and appropriate tactics were used. President Lynden Johnson frequently opined that Vietnam was about "winning the hearts and minds of the Vietnamese people". The problem was that the on the ground tactics did not support this characterization of the overall strategy. The South Vietnamese were incapable of dealing with the North on their own. The South had more population and more industrial capability. But it also had more incompetence and corruption in its government. By the mid-'60s this became readily apparent. As a result U. S. conventional military forces were inserted into the middle of things. At the peak over 500,000 U. S. troops were engaged. And by the end over 58,000 were killed and many hundreds of thousands wounded. The Vietnamese casualties on both the North and the South side ran much higher. But ultimately the North prevailed. Why?
The main strategy used by U. S. troops was "search and destroy". The idea was to search out North Vietnamese forces and destroy them. The calculation was that there were a finite number of them. As soon as you have destroyed enough of them they will admit defeat and you win. That's a very conventional calculation and it ultimately failed as a strategy. North Vietnamese casualties were always higher, usually far higher, than U. S. casualties. But the North (both the government and the civilian population) always felt that the casualties were an acceptable cost to pay for the eventual attainment of the goal, namely "freedom". It doesn't matter if you think they were wrong and they did not end up with "freedom". That's how they saw it and their perception is the critical one.
It is instructive to examine a single event, "The Tet Offensive". Seen in classical military terms is was a massive defeat for the North. Simultaneous attacks were launched across the length and breadth of South Vietnam on the "Tet" holiday (a big deal in Vietnam). Ultimately every single one was beaten back, frequently with heavy losses on the Northern side. But the fact that Tet was a massive military success for the South and failure for the North turned out to not matter to either side. The north decided that "we hurt them badly and if we keep coming at them we will eventually win". The South, particularly the American public, concluded that "this thing is just going to go on and on and the benefit just doesn't seem to outweigh the cost".
One reason for this American attitude was the obvious poor performance of the South Vietnamese armed forces. In case after case the U. S. forces had to come in and save their butts. Why were the North Vietnamese forces so much better than the South Vietnamese forces? Both sides shared the same culture and history but the South had the benefit of all that U. S. training and equipment. Tet turned out to be a turning point for both sides. The North decided that they really could win if they just kept at it long enough so, if anything, they doubled down. The U. S. decided it was time to seriously start looking for an exit strategy. The U. S. came up with "Vietnamization" and exited. The South Vietnamese government continued to be corrupt and incompetent. The South Vietnamese military held on for a few more years after the U. S. military left but eventually fell apart in 1975.
And after that it was "lessons learned" time. The main lesson the U. S. military learned was "don't trust politicians". Early on the military was concerned that Vietnam was unwinnable. And there was a lot of tactical interference by the Johnson White House. The U. S. military was never "unleashed" to fight a high intensity conventional war. Everything was negotiated with the politicians and ended up getting watered down. Where to bomb. What areas were "free fire zones" where everything that moved could be shot versus areas where fire had to be withheld for fear of killing or injuring "innocent civilians". And so on. A big problem in Vietnam was telling the combatants from the noncombatants. In a conventional war the combatants (soldiers) wear readily identifiable uniforms and the noncombatants (civilians) don't. In Vietnam everyone dressed alike and gorillas would frequently hide their weapons and blend in with the civilian population. This was very frustrating.
At the same time, the military had no interest in nor any clue how to "win the hearts and minds" of the general Vietnamese population. In World War II a smile and a stick of gum or a cigarette seemed to get the job done. Similar tactics were a complete and utter failure in Vietnam. As a result the military never figured out how to fight the "hearts and minds" battle successfully and bitterly resented being asked to try. So the other main take away of the military was "stay out of anything that looks like it could turn into a guerilla war". A corollary was "don't try to figure out how to successfully prosecute a guerrilla war". Since the military was ultimately unsuccessful in pursuing their strategy of avoiding guerrilla wars the corollary turned out to be extremely expensive.
The take away of the political right was just as simplistic. The U. S. military had been "hamstrung" by invidious left wing politicians (Democrats). If the military had just been unleashed we could have won. I have heard the following piece of idiocy from a generally left leaning acquaintance: "we gave them guns but we withheld the bullets". But over a half a million U. S. troops were on the ground in Vietnam at one point. And fantastic amounts of money were spent on supplies and equipment. A returning vet I talked to while the war was still on said that he routinely carried over a thousand rounds of ammunition while on patrol. WW II Infantry soldiers carried around a tenth of that. The U. S. is famous for very generously supplying material to its army. In Vietnam besides a vary generous ammunition allowance (as much as you can carry), large numbers of helicopters, gun ships, armored vehicles, fighters and bombers, etc. were deployed. The North used soldiers equipped with AK-47s, some machine guns, and a very small number of artillery pieces. They had essentially no air force, armor, or other "heavy" equipment. It's "men" that the North was generous with and that we tend to be stingy with.
If Vietnam is an example of the U. S. being stingy with the number of troops deployed, then consider that both Afghanistan and Iraq saw far fewer U. S. soldiers deployed than Vietnam did. Yet there has only been a modest hew and cry from the right about the force levels of any of these wars. If the U. S. was too stingy in Vietnam then perhaps 750,000 soldiers should have been deployed there. That means that the troop levels should have been tripled in Iraq and raised by a factor of 8 or maybe even 10 in Afghanistan. But in all cases the problem was not with U. S. forces. It was with local forces. I can't see how more U. S. forces fixes the "local forces" problem. Once you have enough U. S. forces to do any and all necessary training (a level exceeded in all three conflicts) any additional troops don't contribute to fixing the real problem.
More generally, the right contends that "if we had only done this (or more of it)" or "if we had only done that (or more of it)" we would have turned things around and won the war. In order to decide if they are right I look at the morale of the North. If there are dips then maybe something could have been done to turn the dips into a real interest in giving up. But throughout the war they took much higher casualties than the South. They suffered many military defeats. Yet morale stayed high even during the period when we had the most troops in country, when Tet happened, when "pick whatever you want" happened. There is no time in the entire 30 year period of the war when there was a serious dip in morale in the North. That's an extremely long and therefore an extremely difficult period of time to maintain morale. But the North did it. This leads me to believe that we were never close to getting them to seriously think about giving up. So no tweak was likely to transform defeat into victory.
I also note that in the early most critical years of both Iraq and Afghanistan the Republicans were in control of the House, the Senate, and the White House. They had ample opportunity to demonstrate then that they had learned the correct lessons from Vietnam. But that's exactly the time period when things went the worst. The "hearts and minds"-esque Arab Awakening happened later. It is one of the few "light at the end of the tunnel" moments that happened during the entire Iraq war. And, as I noted above, "hearts and minds" has never been part of the right's play book for how to do things. I can think of no "light at the end of the tunnel" moment in Afghanistan. "Mission Accomplished" turned out to be a classic "headlight of an oncoming train" moment.
The left has always been leery of military endeavors. War is seen as more of a right wing thing. But the champion of the left, FDR, had been very successful in prosecuting a just war, namely WW II. So at the start of Vietnam the left had to some extent made peace with the concept of a "just war". As Vietnam went on the left shifted into a more anti-war posture because the "just war" argument when applied to Vietnam kept looking weaker and weaker.
There is another thing going on that has been lost in the mists of time. The Johnson administration decided that state "National Guard" units and the "Reserves" ("weekend warrior" forces associated with each service branch) should not be called up. Instead the draft should be used to increase the size of the regular army. The idea was that the Guard and Reserve were full of middle class types who, as a group, had a lot of political clout. Draftees, on the other hand, came from groups with less clout. This calculation turned out to be wrong. It didn't take long for a lot of activity to spring up on college campuses. You could temporarily avoid the draft by qualifying for a "college deferment". In spite of this "dodge" ("draft dodger" was a common slur during this period) anti-war sentiment hardened on college campuses and political power against the war was effectively marshaled. The left then made a bad tactical decision. They decided to blame the soldier. So a soldier in uniform became an unpopular figure. (The right has exaggerated the amount of hostility toward ordinary soldiers for political reasons. But the fact is that most soldiers found it smart to avoid being seen in public in uniform whenever possible.)
All this ended up having large consequences. The modern cycle of wars have involved depending heavily on guard and reserves. Shortly after the Vietnam War ended the draft was effectively abolished. We now have (at least on paper) an "all volunteer army". That was a lesson the politicians learned. Meanwhile, the left decided their "anti-solder" tactic was a bad one. They no longer blame the soldier. In fact, all parts of the political spectrum now celebrate our "heroes". Wearing a uniform now is a good way to get a free beer and a pat on the back. The difference now between the left and the right is that the left follows through on this by trying to improve veterans benefits and create jobs programs targeted at vets while the right seems only interested in cutting the budget for anything other than weapons procurement.
Beyond this I believe there are a number of further lessons that need to be learned. I have characterized the Saigon government as corrupt and incompetent. Unfortunately, the U. S. policy on this was the same before Vietnam. It did not change during Vietnam. And it continues unchanged to this day. It can be summarized by the oft quoted (but never reliably sourced) aphorism that "he may be a son of a bitch but he's our son of a bitch". We picked various people to run the Saigon government based primarily on their political reliability. We then engineered a number of coups. But we focused more on deposing someone that we felt had failed us than in getting someone who was reasonably honest and reasonably competent in to replace him. As a result there was a lot of "churn" in the holder of the top spot in Saigon without a significant improvement in the results.
The U. S. (both the chattering class and the general public) seemed to be surprised over and over that "furriners" notice our meddling and are reluctant to get whole heartedly behind out designated flavor of the month. I note too that the "Commies" seem to be much better at this than we are. I think that a secret of their success is that they pay more attention to competence and honesty (or at least willingness to hew to the party line and not embarrass the boss by the level of thievery on display). The leaders the Soviets put in place in Eastern Europe in the late '40s seemed to have had a good measure of success as indicated by their longevity on the job. And the longest serving head of state is Fidel Castro, even if you assume that his brother Raoul has been running things for the past couple of years. I also note that Fidel is still very popular with the Cuban public.
The lesson learned that seems to be the most problematic is the whole Counterinsurgency thing. Counterinsurgency is the way you successfully fight a guerrilla/asymmetric battle. I blogged about it in more depth in http://sigma5.blogspot.com/2012/03/counterinsurgency.html. Rather than rehash what I said there let me make a few supplemental observations. (1) "hearts and minds" is the core of the problem. You must look at things from the perspective of the locals and adjust your tactics accordingly. (2) It's labor intensive. I believe a "high tech" "modern warfare" approach is doomed to failure. What is actually required is a lot of boots on the ground. (3) Those boots on the ground must understand the local culture. This means speaking the language and having some knowledge of the history and culture of the area and how the local factions work. That takes a lot of work. (4) This sort of thing is not something the military is good at. Their primary mission is generally "kill people and blow things up". Ok, so it's more complicated than that but to a great extent conventional warfare is about killing people (the soldiers in the other army) and blowing things up (bridges, equipment depots, tanks, airplanes, factories, etc.).
If you change the mission of the military so radically that it is not into killing people and blowing things up then you have something that is no longer an army. This kind of thing is best done by non-military assets. The State Department comes immediately to mind. Diplomats are supposed to be good at "hearts and minds" kind of stuff. It's what they do. David Petraeus makes some mention of the State Department role in the "Counterinsurgency Field Manual" authored under his direction. (There is more about this in my "Iraq" blog post, which can be found at http://sigma5.blogspot.com/2012/03/iraq.html.) But the State Department is always much smaller than the Defense Department. It has a budget perhaps a tenth the size and it's political clout within the beltway is also much smaller. Petraeus' thinking, as laid out in the field manual is that the military and the civilians (i.e. State Department) should coordinate to achieve results. He spent little time (it wasn't his responsibility and his target audience was the uniformed military) on how you get enough non-military resources to hold up their end nor what they should be doing. But at least he was all for a substantial civilian role.
A movie called "The Monuments Men" came out recently. It tells the story of a group of WW II U. S. soldiers who were tasked to preserve European cultural artifacts. The government and the armed forces of the period also spent a lot of time and effort on the "civilian administration" problem. Whenever the military won territory it had administrative teams ready to move in and see to the civilian administration of the people who lived and worked in the conquered areas. I have already discussed U. S. efforts along these lines in Vietnam. Contrast that with our efforts in Iraq and Afghanistan. In both cases the idea was that we could pop in and pop out quickly. As with Vietnam, we were just smart enough to go with locals. But, again as with Vietnam, we didn't do very well with the whole "honest and competent" thing.
In Afghanistan we basically backed a faction called the Northern Alliance. The guy running that was named Hamid Karzai. So we let him roll over into running the whole of Afghanistan rather than the parts that the Northern Alliance has traditionally controlled. He has turned out to be modestly competent and wildly corrupt. He has been competent enough to continue to control the Northern Alliance. But he is disliked and distrusted in large parts of the country and corruption out where everybody can see it has been rampant. As I write this Afghanistan is embarking on national elections and Karzai is not running. The early word is that the guy Karzai is supporting is doing badly. I have no idea how this is going to play out.
With Iraq we did an even worse job. Other than getting out of there almost immediately after Sadam was toppled it is hard to figure out what the plan was. Rachel Maddow has done a documentary arguing persuasively that the real reason we went into Iraq was the Oil. (She had previously done another documentary thoroughly documenting why all the official reasons for going in were bunk.) But so far no one has been able to put anything together explaining what the governance plan was for a post-Sadam Iraq. Here's my theory. An Iraqi by the name of Ahmed Chalabi had been very successful knocking around D. C. He was the head of one of the more successful "Iraqis in exile" groups and was liked and trusted in U. S. conservative political circles. So I think the plan was to knock Sadam off and install Chalabi in his place. The problem is that Chalabi was not well liked in Iraq. He never built a substantial political base within the country after the fall of Bagdad. And once Chalbi failed there was no plan B. So things immediately fell apart. Iraq is currently limping along and the U. S. is out (except for the piles of money we shovel in each year).
Now let me circle back to this "Monuments Men" thing. Maddow in her recent documentary points out that the Iraqi Oil Ministry was carefully guarded as soon as we could get soldiers to it. You know what wasn't guarded? All the museums and other culturally important locations. Within days of the fall of Bagdad pretty much every one of them had been ransacked. And it was not like no one thought of this possibility beforehand. Iraq is one of the earliest cradles of human civilization in the world. Scientists were keenly aware of this and the vast number of precious (from both a cultural perspective and from a pure cash money perspective) items these culturally important locations housed. They made a number of approaches to the Bush Administration in the run up to the war. They offered to provide maps, do ride alongs, use local contacts, whatever, in order to assist in securing, preserving, and protecting these locations. All approaches were rebuffed and Iraq's cultural heritage was trashed on a scale that makes the sack of Rome look like a low key frat party. And Iraq was old when Rome was young.
Certainly Iraqis bear responsibility for this. But people remember. Europeans were grateful for the work of the Monuments Men and other efforts to preserve and protect European cultural heritage. In a similar sense Iraqis are ungrateful that we thought so little of their cultural heritage that it ended up way below the Oil Ministry on our priority list. I find it hard to fault the Iraqis for their attitude on this subject.
And this leads me to my final lesson learned. I always look for the level of interest and commitment of the locals in whatever endeavor is contemplated. In Afghanistan we were invading them because we believed they had done a bad thing (hide and shelter al Qaeda). It's not surprising that under those circumstances they failed to greet us as liberators. And this response should have been factored into our calculations. We should have expected to need a large occupying force to straighten things out after the shooting stopped. Instead we went straight to "Mission Accomplished" and on to Iraq.
The story we were sold to justify the Iraq War was that Sadam was such a bad guy that the locals would be happy to see us. They would "greet us as liberators". I never believed that. But it is at least an internally consistent argument. But we needed to allow for the possibility that we were wrong and plan for it. We didn't.
Unfortunately, with Vietnam, Afghanistan, and Iraq things stretched out enough so that a long term perspective became necessary. Here too a key factor for ultimate success is investment by the locals. They have to step up to the plate. This factor was always missing in Vietnam. I did not know then but I know now how important it is. This problem is why I am leery of raising the cry of "democracy". Democracy requires buy in. It requires powerful people to take actions that are detrimental to their short term interests. They don't do that if they don't have a long term commitment to democracy.
It would be nice if we had done a better job of learning the lessons of Vietnam. And at a minimum it would be nice if we had totted up all the costs of not learning the lessons of Vietnam. This would logically lead to a determination to mend our ways. Alas, it looks like we are bound and determined to continue repeating the same old mistakes. A "lessons learned" perspective should make us very careful in our approach to Syria. It should cause us to be less surprised about how the Egyptian situation is playing out. But perversely it should give us optimism about the eventual fate of Egypt.
The bad guys won in Vietnam. But the Soviets were not in a position to exploit their victory for more than a few years. We now have a very cordial relationship with Vietnam. And they seem to be firmly and comfortably in the clutches of the west. But this took a number of stops and starts and many years to achieve. Egypt could easily have to find its way through several crises on a path that eventually leads to a more open and west facing country. I see signs in Egypt that a commitment to democracy is forming among the locals. It's not there yet in sufficient quantities to be effective. Maybe I will turn out to be wrong. In the mean time, however, I see reason for hope.
Initially the impact was stealth. The U. S. developed an aversion to foreign military involvement. The subject surfaced briefly when the U. S. invaded Granada in 1983. But this was such a small affair that the discussion quickly died down again. It resurfaced again for the Gulf War (1990/91) and has been a subject bubbling quietly in the background for the current Iraq (officially over) and Afghanistan (still going on at the time of this writing) wars. In all these cases "Vietnam" has, at least in the public discourse, been mostly used by the political right to bash the left and vice versa. As is the norm with political discourse the discussion has been shallow and not very helpful. I propose to dig into things more deeply and to better effect.
The pop culture reason to do this is summarized by a quotation attributed to George Santayana. What he actually said ("The Life of Reason", volume 1 - 1905-06) was "Those who cannot remember the past are condemned to repeat it." Either the original or a slight distortion of the original surfaces regularly. Vietnam was a bad experience. It seems reasonable try to study Vietnam in the hope of finding ways to avoid future bad experiences. That seems like sufficient motivation to explore the topic. It certainly was my justification for putting some effort into the subject in the years following the wind down of the war. But this whole process is harder than it looks. Granada turned out fine. But it was so small it could hardly go bad in any major way. The Gulf War was supposed to help reduce the need for future wars in the area. It was manifestly a failure in that we were back in a hot war with the same country (Iraq) about a decade later. And neither the Iraq nor the Afghanistan wars have gone well. So whatever "learning" has happened has not been sufficient to get the "avoid bad outcomes" job done. With that as a preamble, on to the task at hand.
The Vietnamese have a name for the events in question. They call it "The 10,000 Day War". That's about 30 years. But the event we (Americans) see only lasted about 6 years. What's going on? The Vietnamese think the event started in 1945 and ended in 1975. What? Well, I think they are right. It is certainly easy to justify the 1975 end date. That's when what was then called Saigon and is now called Ho Chi Minh City fell. Saigon was the seat of the U. S. supported government in the South and with the fall of the city all resistance to the North ceased. But why the 1945 start date? That is a less well known (in the U. S.) part of the story.
World War II was fought on behalf of "freedom". At least that's the story the U. S. and its allies told. The other guys were Nazi Germany and the Empire of Japan. Both used aggressive military tactics to create and expand empires. To keep things focused I am going to ignore Germany (and other Axis allies) and concentrate on Japan. In the early part of the 20th Century Japan went on an empire building spree. Japan invaded and occupied Korea in 1910. They invaded Manchuria in 1931 and China in 1937. The Pearl Harbor attack on the U. S. in December of 1941 was a continuation of their efforts at imperial expansion. And, since France was an ally of the U. S., the Japanese went ahead and occupied what was then called French Indo-China in 1942. Later French Indo-China was broken up into several countries. One of them is Vietnam.
In an effort to avoid refighting World War II let me cut to the chase. In 1945 when Japan was defeated French Indo-China was being occupied by the Japanese. Having heard the siren call of "freedom" from the allies the Vietnamese (although still part of French Indo-China at this point) expected that the next step would be to allow Vietnam its freedom. They were sadly disappointed. The first thing that happened is that the U. S. decided it was stretched too thin. So they let the Japanese continue to occupy and control Vietnam. Ok. That's maybe not the best first step but its understandable. A couple of years later French Indo-China was partitioned into Vietnam and other countries. This was probably a good idea but beside the point in terms of the fight for freedom. However, the next thing that happened was that all the pieces were returned to French control. This is definitely an anti-freedom move when viewed from the Vietnamese perspective. By this time the Cold War was getting under way and the U. S. decided it needed a strong France as an ally. France wanted its colonial empire back so France got its colonial empire back. And that meant that France got control of its old Indo-China territory back. And that meant that Vietnam went back to French control.
The Vietnamese were understandably very unhappy about this. They had been sold out. So they back dated the point where they should have achieved their freedom to 1945, a not unreasonable move, and started the "freedom clock". They stopped the clock in 1975 and rounded the duration of the process off to 10,000 days. (Everybody likes round numbers.) And here be the first lesson: "If you are going to promise freedom people are going to expect you to deliver on your promise". George W. Bush did a lot of freedom promising. That concerned me because I did not see any indication that he wanted to actually deliver on the promise. Unfortunately, my concern has turned out to be justified. And there are now a lot of people around the world who dislike us because we have not followed through on that promise. I am mystified why so many U. S. commentators are "mystified" about why these people are mad at us.
This whole "1945" business is something that a lot of otherwise well informed people are not familiar with. So let me cover some more historical ground in an effort to shed more light. It should now be no surprise that the Vietnamese were unhappy. The center of effective action to remedy this unhappiness was Hanoi, a city in the north of Vietnam. And the political focus was a man named Ho Chi Minh. He had spent a lot of time exiled in Paris. He returned to Vietnam after the war and got active politically. While in Paris he had hung around in Communist circles. When he started trying to organize effective action he knew he needed powerful allies. France, the U. S., etc. were out of the question. So he turned to the other side, the then U. S. S. R., frequently short handed as "The Soviets". With his "Communist" credentials, and with the fact that he was well positioned to do U. S. interests harm, he was quickly embraced. Arms, money, training, etc. started flowing. Minh quickly recruited a man later generally referred to as General Giap to run the military side of things.
Minh started ramping up anti-imperialist (an accurate characterization as the French definitely saw Vietnam as part of the French "colonial empire") activity. Several years later this culminated in The Battle of Dien Bien Phu with Giap running the military operation for the Hanoi faction. The French were defeated, surrendering on May 7, 1954. (I note that the U. S. secretly propped the French up to the tune of a billion dollars while publically staying above the fray.) This led to a conference in Geneva, which in turn led to a temporary partitioning of the country into North and South, and a promised election in 1956. Rather than let the democratic process play out, which would have resulted in a win everywhere for the Hanoi faction, the U. S. got publically involved. They effectively sabotaged the election in the South and installed a puppet government in Saigon to run what was now called South Vietnam. They then started pumping a lot of money (publically and secretly) and other support into the Saigon government in an effort to create a "bastion of anti-communism". The Cold War was the big priority that drove tactics with respect to places like Vietnam.
The Saigon government was corrupt and inept. At this point most of the population of Vietnam, North and South, just wanted everyone to go away so that average people could just get back to their lives. But by this time the U. S. was heavily invested in South Vietnam and the Soviets were invested (but not so heavily) in North Vietnam so disengagement was not an option. The Hanoi government remained generally popular in the North and had a significant following in the South. What resulted was the first heavy involvement since the U. S. Revolutionary War in what was then referred to as a Guerilla War. (I noted the parallels at the time. But associating the "good" Revolutionary War patriots with the "bad" Communist guerillas was too much for most people and a serious effort was made to discourage this line of thought.)
By this point the U. S. had considerable familiarity with and success with "conventional" war. We had relatively recently fought WW I, WW II, and Korea, all conventional wars. Conventional wars involve the clash of large armies deploying the most powerful military weapons and tactics of the times. The conflict is decided by battle ships, bombers, artillery, tanks, and large numbers of infantry soldiers using "conventional" tactics. The core strategy is to destroy the opponent's army. Once that's done then everything else follows. In modern parlance this is sometimes referred to as a "symmetrical" war. In guerrilla wars, now popularly referred to as "asymmetrical" wars, an entirely different approach is required. Guerilla wars are called asymmetrical because the military capabilities of the two armed forces are wildly different. One is strong and the other is weak. In a conventional war the strong army would wipe up the floor with the weak army and coast quickly to victory. If anything is now obvious, it is that there is no "coast to victory" going on in modern guerrilla/asymmetrical warfare.
But Vietnam was the first real experience the U. S. had with this sort of thing. Theoretically, this was recognized early and appropriate tactics were used. President Lynden Johnson frequently opined that Vietnam was about "winning the hearts and minds of the Vietnamese people". The problem was that the on the ground tactics did not support this characterization of the overall strategy. The South Vietnamese were incapable of dealing with the North on their own. The South had more population and more industrial capability. But it also had more incompetence and corruption in its government. By the mid-'60s this became readily apparent. As a result U. S. conventional military forces were inserted into the middle of things. At the peak over 500,000 U. S. troops were engaged. And by the end over 58,000 were killed and many hundreds of thousands wounded. The Vietnamese casualties on both the North and the South side ran much higher. But ultimately the North prevailed. Why?
The main strategy used by U. S. troops was "search and destroy". The idea was to search out North Vietnamese forces and destroy them. The calculation was that there were a finite number of them. As soon as you have destroyed enough of them they will admit defeat and you win. That's a very conventional calculation and it ultimately failed as a strategy. North Vietnamese casualties were always higher, usually far higher, than U. S. casualties. But the North (both the government and the civilian population) always felt that the casualties were an acceptable cost to pay for the eventual attainment of the goal, namely "freedom". It doesn't matter if you think they were wrong and they did not end up with "freedom". That's how they saw it and their perception is the critical one.
It is instructive to examine a single event, "The Tet Offensive". Seen in classical military terms is was a massive defeat for the North. Simultaneous attacks were launched across the length and breadth of South Vietnam on the "Tet" holiday (a big deal in Vietnam). Ultimately every single one was beaten back, frequently with heavy losses on the Northern side. But the fact that Tet was a massive military success for the South and failure for the North turned out to not matter to either side. The north decided that "we hurt them badly and if we keep coming at them we will eventually win". The South, particularly the American public, concluded that "this thing is just going to go on and on and the benefit just doesn't seem to outweigh the cost".
One reason for this American attitude was the obvious poor performance of the South Vietnamese armed forces. In case after case the U. S. forces had to come in and save their butts. Why were the North Vietnamese forces so much better than the South Vietnamese forces? Both sides shared the same culture and history but the South had the benefit of all that U. S. training and equipment. Tet turned out to be a turning point for both sides. The North decided that they really could win if they just kept at it long enough so, if anything, they doubled down. The U. S. decided it was time to seriously start looking for an exit strategy. The U. S. came up with "Vietnamization" and exited. The South Vietnamese government continued to be corrupt and incompetent. The South Vietnamese military held on for a few more years after the U. S. military left but eventually fell apart in 1975.
And after that it was "lessons learned" time. The main lesson the U. S. military learned was "don't trust politicians". Early on the military was concerned that Vietnam was unwinnable. And there was a lot of tactical interference by the Johnson White House. The U. S. military was never "unleashed" to fight a high intensity conventional war. Everything was negotiated with the politicians and ended up getting watered down. Where to bomb. What areas were "free fire zones" where everything that moved could be shot versus areas where fire had to be withheld for fear of killing or injuring "innocent civilians". And so on. A big problem in Vietnam was telling the combatants from the noncombatants. In a conventional war the combatants (soldiers) wear readily identifiable uniforms and the noncombatants (civilians) don't. In Vietnam everyone dressed alike and gorillas would frequently hide their weapons and blend in with the civilian population. This was very frustrating.
At the same time, the military had no interest in nor any clue how to "win the hearts and minds" of the general Vietnamese population. In World War II a smile and a stick of gum or a cigarette seemed to get the job done. Similar tactics were a complete and utter failure in Vietnam. As a result the military never figured out how to fight the "hearts and minds" battle successfully and bitterly resented being asked to try. So the other main take away of the military was "stay out of anything that looks like it could turn into a guerilla war". A corollary was "don't try to figure out how to successfully prosecute a guerrilla war". Since the military was ultimately unsuccessful in pursuing their strategy of avoiding guerrilla wars the corollary turned out to be extremely expensive.
The take away of the political right was just as simplistic. The U. S. military had been "hamstrung" by invidious left wing politicians (Democrats). If the military had just been unleashed we could have won. I have heard the following piece of idiocy from a generally left leaning acquaintance: "we gave them guns but we withheld the bullets". But over a half a million U. S. troops were on the ground in Vietnam at one point. And fantastic amounts of money were spent on supplies and equipment. A returning vet I talked to while the war was still on said that he routinely carried over a thousand rounds of ammunition while on patrol. WW II Infantry soldiers carried around a tenth of that. The U. S. is famous for very generously supplying material to its army. In Vietnam besides a vary generous ammunition allowance (as much as you can carry), large numbers of helicopters, gun ships, armored vehicles, fighters and bombers, etc. were deployed. The North used soldiers equipped with AK-47s, some machine guns, and a very small number of artillery pieces. They had essentially no air force, armor, or other "heavy" equipment. It's "men" that the North was generous with and that we tend to be stingy with.
If Vietnam is an example of the U. S. being stingy with the number of troops deployed, then consider that both Afghanistan and Iraq saw far fewer U. S. soldiers deployed than Vietnam did. Yet there has only been a modest hew and cry from the right about the force levels of any of these wars. If the U. S. was too stingy in Vietnam then perhaps 750,000 soldiers should have been deployed there. That means that the troop levels should have been tripled in Iraq and raised by a factor of 8 or maybe even 10 in Afghanistan. But in all cases the problem was not with U. S. forces. It was with local forces. I can't see how more U. S. forces fixes the "local forces" problem. Once you have enough U. S. forces to do any and all necessary training (a level exceeded in all three conflicts) any additional troops don't contribute to fixing the real problem.
More generally, the right contends that "if we had only done this (or more of it)" or "if we had only done that (or more of it)" we would have turned things around and won the war. In order to decide if they are right I look at the morale of the North. If there are dips then maybe something could have been done to turn the dips into a real interest in giving up. But throughout the war they took much higher casualties than the South. They suffered many military defeats. Yet morale stayed high even during the period when we had the most troops in country, when Tet happened, when "pick whatever you want" happened. There is no time in the entire 30 year period of the war when there was a serious dip in morale in the North. That's an extremely long and therefore an extremely difficult period of time to maintain morale. But the North did it. This leads me to believe that we were never close to getting them to seriously think about giving up. So no tweak was likely to transform defeat into victory.
I also note that in the early most critical years of both Iraq and Afghanistan the Republicans were in control of the House, the Senate, and the White House. They had ample opportunity to demonstrate then that they had learned the correct lessons from Vietnam. But that's exactly the time period when things went the worst. The "hearts and minds"-esque Arab Awakening happened later. It is one of the few "light at the end of the tunnel" moments that happened during the entire Iraq war. And, as I noted above, "hearts and minds" has never been part of the right's play book for how to do things. I can think of no "light at the end of the tunnel" moment in Afghanistan. "Mission Accomplished" turned out to be a classic "headlight of an oncoming train" moment.
The left has always been leery of military endeavors. War is seen as more of a right wing thing. But the champion of the left, FDR, had been very successful in prosecuting a just war, namely WW II. So at the start of Vietnam the left had to some extent made peace with the concept of a "just war". As Vietnam went on the left shifted into a more anti-war posture because the "just war" argument when applied to Vietnam kept looking weaker and weaker.
There is another thing going on that has been lost in the mists of time. The Johnson administration decided that state "National Guard" units and the "Reserves" ("weekend warrior" forces associated with each service branch) should not be called up. Instead the draft should be used to increase the size of the regular army. The idea was that the Guard and Reserve were full of middle class types who, as a group, had a lot of political clout. Draftees, on the other hand, came from groups with less clout. This calculation turned out to be wrong. It didn't take long for a lot of activity to spring up on college campuses. You could temporarily avoid the draft by qualifying for a "college deferment". In spite of this "dodge" ("draft dodger" was a common slur during this period) anti-war sentiment hardened on college campuses and political power against the war was effectively marshaled. The left then made a bad tactical decision. They decided to blame the soldier. So a soldier in uniform became an unpopular figure. (The right has exaggerated the amount of hostility toward ordinary soldiers for political reasons. But the fact is that most soldiers found it smart to avoid being seen in public in uniform whenever possible.)
All this ended up having large consequences. The modern cycle of wars have involved depending heavily on guard and reserves. Shortly after the Vietnam War ended the draft was effectively abolished. We now have (at least on paper) an "all volunteer army". That was a lesson the politicians learned. Meanwhile, the left decided their "anti-solder" tactic was a bad one. They no longer blame the soldier. In fact, all parts of the political spectrum now celebrate our "heroes". Wearing a uniform now is a good way to get a free beer and a pat on the back. The difference now between the left and the right is that the left follows through on this by trying to improve veterans benefits and create jobs programs targeted at vets while the right seems only interested in cutting the budget for anything other than weapons procurement.
Beyond this I believe there are a number of further lessons that need to be learned. I have characterized the Saigon government as corrupt and incompetent. Unfortunately, the U. S. policy on this was the same before Vietnam. It did not change during Vietnam. And it continues unchanged to this day. It can be summarized by the oft quoted (but never reliably sourced) aphorism that "he may be a son of a bitch but he's our son of a bitch". We picked various people to run the Saigon government based primarily on their political reliability. We then engineered a number of coups. But we focused more on deposing someone that we felt had failed us than in getting someone who was reasonably honest and reasonably competent in to replace him. As a result there was a lot of "churn" in the holder of the top spot in Saigon without a significant improvement in the results.
The U. S. (both the chattering class and the general public) seemed to be surprised over and over that "furriners" notice our meddling and are reluctant to get whole heartedly behind out designated flavor of the month. I note too that the "Commies" seem to be much better at this than we are. I think that a secret of their success is that they pay more attention to competence and honesty (or at least willingness to hew to the party line and not embarrass the boss by the level of thievery on display). The leaders the Soviets put in place in Eastern Europe in the late '40s seemed to have had a good measure of success as indicated by their longevity on the job. And the longest serving head of state is Fidel Castro, even if you assume that his brother Raoul has been running things for the past couple of years. I also note that Fidel is still very popular with the Cuban public.
The lesson learned that seems to be the most problematic is the whole Counterinsurgency thing. Counterinsurgency is the way you successfully fight a guerrilla/asymmetric battle. I blogged about it in more depth in http://sigma5.blogspot.com/2012/03/counterinsurgency.html. Rather than rehash what I said there let me make a few supplemental observations. (1) "hearts and minds" is the core of the problem. You must look at things from the perspective of the locals and adjust your tactics accordingly. (2) It's labor intensive. I believe a "high tech" "modern warfare" approach is doomed to failure. What is actually required is a lot of boots on the ground. (3) Those boots on the ground must understand the local culture. This means speaking the language and having some knowledge of the history and culture of the area and how the local factions work. That takes a lot of work. (4) This sort of thing is not something the military is good at. Their primary mission is generally "kill people and blow things up". Ok, so it's more complicated than that but to a great extent conventional warfare is about killing people (the soldiers in the other army) and blowing things up (bridges, equipment depots, tanks, airplanes, factories, etc.).
If you change the mission of the military so radically that it is not into killing people and blowing things up then you have something that is no longer an army. This kind of thing is best done by non-military assets. The State Department comes immediately to mind. Diplomats are supposed to be good at "hearts and minds" kind of stuff. It's what they do. David Petraeus makes some mention of the State Department role in the "Counterinsurgency Field Manual" authored under his direction. (There is more about this in my "Iraq" blog post, which can be found at http://sigma5.blogspot.com/2012/03/iraq.html.) But the State Department is always much smaller than the Defense Department. It has a budget perhaps a tenth the size and it's political clout within the beltway is also much smaller. Petraeus' thinking, as laid out in the field manual is that the military and the civilians (i.e. State Department) should coordinate to achieve results. He spent little time (it wasn't his responsibility and his target audience was the uniformed military) on how you get enough non-military resources to hold up their end nor what they should be doing. But at least he was all for a substantial civilian role.
A movie called "The Monuments Men" came out recently. It tells the story of a group of WW II U. S. soldiers who were tasked to preserve European cultural artifacts. The government and the armed forces of the period also spent a lot of time and effort on the "civilian administration" problem. Whenever the military won territory it had administrative teams ready to move in and see to the civilian administration of the people who lived and worked in the conquered areas. I have already discussed U. S. efforts along these lines in Vietnam. Contrast that with our efforts in Iraq and Afghanistan. In both cases the idea was that we could pop in and pop out quickly. As with Vietnam, we were just smart enough to go with locals. But, again as with Vietnam, we didn't do very well with the whole "honest and competent" thing.
In Afghanistan we basically backed a faction called the Northern Alliance. The guy running that was named Hamid Karzai. So we let him roll over into running the whole of Afghanistan rather than the parts that the Northern Alliance has traditionally controlled. He has turned out to be modestly competent and wildly corrupt. He has been competent enough to continue to control the Northern Alliance. But he is disliked and distrusted in large parts of the country and corruption out where everybody can see it has been rampant. As I write this Afghanistan is embarking on national elections and Karzai is not running. The early word is that the guy Karzai is supporting is doing badly. I have no idea how this is going to play out.
With Iraq we did an even worse job. Other than getting out of there almost immediately after Sadam was toppled it is hard to figure out what the plan was. Rachel Maddow has done a documentary arguing persuasively that the real reason we went into Iraq was the Oil. (She had previously done another documentary thoroughly documenting why all the official reasons for going in were bunk.) But so far no one has been able to put anything together explaining what the governance plan was for a post-Sadam Iraq. Here's my theory. An Iraqi by the name of Ahmed Chalabi had been very successful knocking around D. C. He was the head of one of the more successful "Iraqis in exile" groups and was liked and trusted in U. S. conservative political circles. So I think the plan was to knock Sadam off and install Chalabi in his place. The problem is that Chalabi was not well liked in Iraq. He never built a substantial political base within the country after the fall of Bagdad. And once Chalbi failed there was no plan B. So things immediately fell apart. Iraq is currently limping along and the U. S. is out (except for the piles of money we shovel in each year).
Now let me circle back to this "Monuments Men" thing. Maddow in her recent documentary points out that the Iraqi Oil Ministry was carefully guarded as soon as we could get soldiers to it. You know what wasn't guarded? All the museums and other culturally important locations. Within days of the fall of Bagdad pretty much every one of them had been ransacked. And it was not like no one thought of this possibility beforehand. Iraq is one of the earliest cradles of human civilization in the world. Scientists were keenly aware of this and the vast number of precious (from both a cultural perspective and from a pure cash money perspective) items these culturally important locations housed. They made a number of approaches to the Bush Administration in the run up to the war. They offered to provide maps, do ride alongs, use local contacts, whatever, in order to assist in securing, preserving, and protecting these locations. All approaches were rebuffed and Iraq's cultural heritage was trashed on a scale that makes the sack of Rome look like a low key frat party. And Iraq was old when Rome was young.
Certainly Iraqis bear responsibility for this. But people remember. Europeans were grateful for the work of the Monuments Men and other efforts to preserve and protect European cultural heritage. In a similar sense Iraqis are ungrateful that we thought so little of their cultural heritage that it ended up way below the Oil Ministry on our priority list. I find it hard to fault the Iraqis for their attitude on this subject.
And this leads me to my final lesson learned. I always look for the level of interest and commitment of the locals in whatever endeavor is contemplated. In Afghanistan we were invading them because we believed they had done a bad thing (hide and shelter al Qaeda). It's not surprising that under those circumstances they failed to greet us as liberators. And this response should have been factored into our calculations. We should have expected to need a large occupying force to straighten things out after the shooting stopped. Instead we went straight to "Mission Accomplished" and on to Iraq.
The story we were sold to justify the Iraq War was that Sadam was such a bad guy that the locals would be happy to see us. They would "greet us as liberators". I never believed that. But it is at least an internally consistent argument. But we needed to allow for the possibility that we were wrong and plan for it. We didn't.
Unfortunately, with Vietnam, Afghanistan, and Iraq things stretched out enough so that a long term perspective became necessary. Here too a key factor for ultimate success is investment by the locals. They have to step up to the plate. This factor was always missing in Vietnam. I did not know then but I know now how important it is. This problem is why I am leery of raising the cry of "democracy". Democracy requires buy in. It requires powerful people to take actions that are detrimental to their short term interests. They don't do that if they don't have a long term commitment to democracy.
It would be nice if we had done a better job of learning the lessons of Vietnam. And at a minimum it would be nice if we had totted up all the costs of not learning the lessons of Vietnam. This would logically lead to a determination to mend our ways. Alas, it looks like we are bound and determined to continue repeating the same old mistakes. A "lessons learned" perspective should make us very careful in our approach to Syria. It should cause us to be less surprised about how the Egyptian situation is playing out. But perversely it should give us optimism about the eventual fate of Egypt.
The bad guys won in Vietnam. But the Soviets were not in a position to exploit their victory for more than a few years. We now have a very cordial relationship with Vietnam. And they seem to be firmly and comfortably in the clutches of the west. But this took a number of stops and starts and many years to achieve. Egypt could easily have to find its way through several crises on a path that eventually leads to a more open and west facing country. I see signs in Egypt that a commitment to democracy is forming among the locals. It's not there yet in sufficient quantities to be effective. Maybe I will turn out to be wrong. In the mean time, however, I see reason for hope.
Thursday, March 27, 2014
Foundations of Science
I have written a couple of posts on the most basic questions that surround Science. This is a continuation of the series after a long delay. The last post was http://sigma5.blogspot.com/2011/12/what-is-science.html, a post I made back in 2011. In these previous posts I have argued that you can't prove Science is true using scientific methods. In fact, in the post I link to above I argue that Science is a popularity contest that is scored using peculiar rules. In this post I want to get into areas that are not generally spoken of. These ideas are so foundational that they are accepted without even acknowledging their presence. That makes them critically important. Frequently in these posts I jump into a digression. The digression serves the purpose of setting up what follows. This time around I am going to do the opposite. I am going to state the key items right up front. Then I will spend some time discussing them.
There is a single real world that is external to each of us but shared by all of us.
There are rules that govern how the real world works. At least some of them are discoverable.
The rules that govern the real world all interact with each other in a consistent manner.
Whatever works is right.
Whatever doesn't work is wrong.
There are degrees of wrongness.
The first belief is akin to the "there is only one God" idea in religion.
Science is grounded in observation. If some of us experience a different world (i. e. a world with different rules) then what is observed depends on which group the observer is a member of. Yet Science depends heavily on the concept of "repeatability". Two people, any two people, each of whom does the same experiment are expected to get the same result. In fact, if other scientists are convinced that both people did the experiment correctly but got different results they decide there is a serious problem that needs to be addressed. Now it is sometimes found that one or both did the experiment poorly. Then the next thing to do is do the experiment yet again and try to get it right this time.
If scientists decide that both did the experiment correctly then they start looking for something else to explain the situation. They have found in many cases that "all other things being equal" is not true. There is some influence that has not been properly accounted for. Early experiments to determine "G", the Gravitational constant, ran into this. People would make very precise measurements of how fast things fell. Good experimenters got differing results. This led to small adjustments for altitude. But there were still problems. This led to the idea that the density of the earth differed from place to place. This density difference has since been turned inside out. Geologists now use sensitive gravitational measurements to determine some of the attributes of rock that may be buried way below the surface of the ground. This is one of the tricks in the bag of people who look for oil for a living.
Now this is definitely a belief. It is not a provable proposition. If you find a thousand or a million or a billion examples of everyone seeing the same thing that doesn't mean that there is not some situation just around the corner that will be the exception that disproves the rule. And to some extent this is a self fulfilling prophecy.
In quantum mechanics there is a theory called the "many worlds" theory. (Do not confuse this with the "many worlds" theory in cosmology. It's a different theory that I do not have time to get into.) In quantum mechanics we have all these situations where (to keep things simple) there are two possible outcomes. The "many worlds" theory says that both outcomes actually occur. But the world splits into two copies. In one world the first thing happens. In the other world the second thing happens. We (or at least the copy of "we" that we are following) exist in only one copy of these many worlds. As we look back we see, for instance, the first outcome of the most recent event. And we also see a specific outcome of the previous event. And so on. There is a world in which each possible outcome of each event happened. We see only the world as it is in our "world line" but there are zillions of other world lines in which events resulted in different outcomes.
In this conception of quantum mechanics there are in fact many worlds with a copy of each of us occupying many of them (remember, there may be events that caused us to not come into existence in some of these other world lines). But we can only see the world and the world line we are in. Scientists (and science fiction writers) have spent a lot of time working out detailed "many worlds" theories. No one has come up with proof that the theories are wrong. How could you? But most scientists go about their daily business as if the theories are wrong. And so far they have not come across any evidence that they are right.
Another line of thinking is the "Matrix" (from the popular movie) idea. Maybe the real world looks quite different but we live inside a giant sophisticated computer simulation of the world we see. Again, if the simulation is sophisticated enough and well enough done, there is no way to determine that it is a simulation. Here the proper response is "who cares?" The problem for Science just becomes "what are the rules in the simulated universe?" We can't beat the simulation. Therefore we have no access to the "real" real world. So it does us no good to try to study it. So we might as well stick to trying to figure out the simulated rules in our simulated world. Those are the rules we can use to advantage.
The second belief is one that philosophers have spent a lot of time on. There are systems of philosophy and religion that say "you can't so don't". There are also religious systems that say "the anointed get into heaven no matter what they do and the rest don't get in no matter what they do so why should anyone try to figure things out?" And, if your objective is to get into heaven, then it makes sense to not try if you subscribe to the "anointed" school of religion. But here we are talking about your efforts to achieve salvation in the next world. Most of us spend a lot of time trying to navigate around in the current world. For those science says "Science can be at least of some help".
And Science has a pretty good track record. Science has found out a lot about how the world works. Some scientists believe that all of the rules are discoverable. Some don't. But all scientists believe that there are undiscovered rules that can be discovered. So it doesn't really matter if they can all be discovered or not. As long as there are more rules that are currently undiscovered but haven't yet been discovered there is reason to continue the endeavor.
No scientist believes that all the rules that can be discovered have already been discovered. Nor do they believe that knowing the rules of science allows you to find the correct solution to all problems. There is even a shorthand for this. "Science is about 'what'. Religion/philosophy is about 'why'". Put another way, Science tells you a lot about what is possible but it doesn't tell you which choice among the possible choices is the "right" choice.
The third belief is actually quite a subtle one. Scientists believe that all the rules apply all the time. Now each rule can have a specific range of operation. So most rules don't apply in most situations. They can be effectively ignored. But part of a proper scientific rule is to define what situations it applies to. If a rule is applicable to a particular situation then it must be applied. Put another way, you must look at all the data when trying to create a rule. If you can't say why the rule applies here but doesn't apply there then there is a problem with your rule. This trips nonscientists up all the time. A rule may look perfectly good if you apply it to only a limited number of situations. But if it doesn't apply universally you better have a "whereas" that says in effect "but it doesn't apply in these other situations". Let me give you an example of this sort of thing done right.
Newton said "f=ma". As far as he knew this was a universal rule. But then Einstein came along and said in effect "f=ma + a relativistic correction". And he showed that the relativistic correction was such a tiny number in a lot of "normal world" situations that it could be ignored. Scientists and engineers know where the boundary between "normal world" and "relativistic world" is so they can keep it simple most of the time but fall back to the more complex relativistic methods when they need to.
A more common situation is where someone makes a "bold new" discovery. But no one else can get the method they claim to have used to work. This is common in the world psychic phenomenon. They say essentially "I did this and here's what happened". For instance they find certain people that can identify a card selected at random "at a rate above chance". But others make what appear to be the same test but don't get the "above chance" result. This situation has come up repeatedly. And the same explanation has been made repeatedly: shyness. Apparently the phenomenon is shy. If there are too many "negative vibrations" around the phenomenon goes into hiding. Many scientists have a different explanation: bad experimental procedure.
The same thing happens in more traditionally scientific situations. Check the Wikipedia articles on "N rays" or "cold fusion" for two examples that happened more than 80 years apart. The "shyness" explanation was trotted out in both of these examples. Eventually scientists concluded that the original discoverers just got it wrong.
Now let me move on to my corollaries.
The first one ("whatever works is right") seems obvious and sensible. But scientists have gotten really good at coming up with really weird results. The classic example is photons. People have been studying light for a long time. As I pointed out in my previous post, Newton did some wonderful work and in 1704 published his results in a book whose title rendered in English is "Optics". A lot of good work was done before this and a whole lot was done afterwards. In spite of this no one could decide whether light was composed of waves or particles. That is until Einstein came along in 1905. His answer was "neither". Instead light was composed of things called photons. Photons behave like waves in some situations and particles in other situations.
To a lot of people this was a completely unsatisfactory solution. An answer of "particles" or an answer of "waves" would have been natural and satisfactory. But the fact is that the extremely weird theory of "quantum electrodynamics" (the modern successor to Einstein's original work) does an excellent job of telling us how light will behave in all kinds of situations. QED (the standard shorthand for quantum electrodynamics) works. "Waves" doesn't work. "Particles" doesn't work. And science has now built up a long list of truly weird theories that work. Evolution (which is not actually that weird) works. Quantum Mechanics (actually its modern descendent "The Standard Model") works. Relativity works. The list goes on.
The second corollary ("whatever doesn't work is wrong") also seems obvious and sensible. But in practice it is not. As I just got done pointing out, in the case of light, "waves" doesn't work and "particles" doesn't work. But they are both connected to our day to day experience. Similarly, the Bible as an accurate historical reference doesn't work. Pre-QM atomic theories don't work. What is now called "Newtonian Mechanics" (e.g. "f=ma" and other non-relativistic theories) doesn't work. In some cases the old theory that doesn't work is readily discarded. If it strikes people as weird or unnatural they are happy to let it go.
Most people had no idea what "N rays" were about. They didn't much care when they were "discovered". And they didn't much care when they were debunked. So it was pretty easy to discard the whole "N Ray" theory. And the whole "N Ray" battle was scientists versus scientists. The general public felt it didn't have a dog in the fight. But in the case of the other now discarded theories I mentioned, people had a lot more invested in the old theories. There are large, well funded groups invested in the whole Evolution "debate" so leaving the discredited alternatives behind has turned out to be much harder. But that's why this corollary is so important. Except . . .
There are degrees of wrongness. All scientific theories have problems. The gold standard of scientific theories is "The Standard Model". It is the governing model for particle physics and is generally characterized as "spectacularly successful". It has problems. The biggest and most obvious problem is that no one knows how to reconcile it with Relativity. So it fails the test for perfect scientific theory. In fact there are no known perfect scientific theories. So how do scientists deal with this? The short answer is they fudge. That's why scientific theories are always referred to as "theories". It is always possible that the theory will be modified or even completely replaced by a quite different theory.
Scientists have now had hundreds of years of experience dealing with this issue. And part of that experience is with dealing with theories that are known to be wrong but have proved to be useful anyhow. They try to balance the utility of each theory with the problems its inadequacies can cause. This has resulted in an informal "wrongness" scale:
Best - These are well established theories. Scientists believe that the core of the theory is correct but that some modifications may be needed in the future around the edges. Also, any theory that is in contravention to the core of the theory is easily demonstrated to be wrong. A classic example of this is Evolution. Evolution grew out of problems with all bible based theories. So all bible based theories are known to be wrong. The core of evolution is known to be solid. Some tweaking (the relationship of two species, evolutionary mechanisms, details of the evolutionary record) may need tweaking but the core is solid.
Very good - These are theories that have had a great deal of success. But there are known problems for which no known solution currently exists. A classic example of this is The Standard Model. The relationship of this theory to Relativity is currently a big problem. There are other issues where scientists are not sure they have everything right. Note: Finding the Higgs Boson substantially reduced the problem areas seen with the Standard Model.
Good - The theory works well in a lot of situations but has a number of known problems. Like Very Good theories the core looks solid but here there are a lot more problems around the edges. A classic example of this is Plate Tectonics. This theory solved a lot of problems. It gives us a good way to understand a lot of geology. But there are a number of open problems having to do with how some plate boundaries work. There are a lot of areas where current understanding can best be described as "incomplete". But there are a number of areas where the theory works very well and there is currently no better theory on the horizon.
Ok - Science has had a lot of success with an incremental approach. A bad theory is better than no theory at all. A lot of theories don't work very well. But they work some of the time and the fact that we know where they work and where they have problems is helpful in determining what additional experiments might yield useful information or what problems a new theory might solve. A classis example of this kind of theory is the "Cooper pair" explanation for superconductivity. It works pretty well some of the time and not well at all a lot of the time. Nobody currently has any ideas that seem to work better. No one would be surprised if a new theory came along that was quite different to do a better job of explaining Superconductivity.
Poor - These theories are typically found where scientists really don't have much understanding. A good example of this would be planetary formation. There used to be what appeared to be a good theory. Small "rocky" planets formed close to stars where gasses were boiled off. Gas giants formed further away where it was cooler. Then came all the data on exoplanets from the Kepler spacecraft. It turns out there are all kinds of planets of the wrong kind and size in the wrong places. Scientists have some ideas but mostly it's early days.
Not even wrong - This phrase is attributed to Wolfgang Pauli, the great physicists. In describing ok and poor degrees of wrongness I emphasized that these theories all were useful to the extent that they were helpful in pointing to where a better theory might be found. "Not even wrong" theories are so bad that they just add noise to the situation.
So given all this wrongness why do scientists get so exercised about theories or criticisms from outsiders? Scientists are very aware of how problematic some of their theories are. And they dearly love it when a new better theory comes along. But is generally pretty easy to slot a lot of "new" theory into one of the above categories. Scientists get exercised when a new theory slots into a category that is lower than the one their current favored theory occupies. They also get exercised when a theory is advanced or a criticism is leveled that they have seen before (frequently many times) and have already thoroughly investigated.
The evolution "debate" falls into this category. All of the early scientists that worked on evolution were practicing Christians. They were very familiar with the bible. Where the bible appeared to get it right, those ideas were incorporated in evolution. But in many areas the bible got it wrong. Darwin's "Origin of Species" was first published in 1859. It was immediately criticized. Several new editions were quickly put out that included additional material addressing these criticisms. The sixth edition, the one commonly available now, was published in 1872. It should be shocking how many "new" arguments against evolution are in fact old arguments that Darwin addressed in great detail over a hundred years ago but apparently it is not. But most people who oppose evolution have never bothered to read Darwin or familiarize themselves with subsequent developments. I recently published a post on "Ken Ham Creationism" (http://sigma5.blogspot.com/2014/02/ken-ham-creationism.html). Many of Hams "modern" criticisms of evolution find well supported refutations in Origin, a book that is now over a hundred years old.
Three key unspoken beliefs that all Scientists share
There is a single real world that is external to each of us but shared by all of us.
There are rules that govern how the real world works. At least some of them are discoverable.
The rules that govern the real world all interact with each other in a consistent manner.
Three corollaries to the three beliefs
Whatever works is right.
Whatever doesn't work is wrong.
There are degrees of wrongness.
Discussion
The first belief is akin to the "there is only one God" idea in religion.
Science is grounded in observation. If some of us experience a different world (i. e. a world with different rules) then what is observed depends on which group the observer is a member of. Yet Science depends heavily on the concept of "repeatability". Two people, any two people, each of whom does the same experiment are expected to get the same result. In fact, if other scientists are convinced that both people did the experiment correctly but got different results they decide there is a serious problem that needs to be addressed. Now it is sometimes found that one or both did the experiment poorly. Then the next thing to do is do the experiment yet again and try to get it right this time.
If scientists decide that both did the experiment correctly then they start looking for something else to explain the situation. They have found in many cases that "all other things being equal" is not true. There is some influence that has not been properly accounted for. Early experiments to determine "G", the Gravitational constant, ran into this. People would make very precise measurements of how fast things fell. Good experimenters got differing results. This led to small adjustments for altitude. But there were still problems. This led to the idea that the density of the earth differed from place to place. This density difference has since been turned inside out. Geologists now use sensitive gravitational measurements to determine some of the attributes of rock that may be buried way below the surface of the ground. This is one of the tricks in the bag of people who look for oil for a living.
Now this is definitely a belief. It is not a provable proposition. If you find a thousand or a million or a billion examples of everyone seeing the same thing that doesn't mean that there is not some situation just around the corner that will be the exception that disproves the rule. And to some extent this is a self fulfilling prophecy.
In quantum mechanics there is a theory called the "many worlds" theory. (Do not confuse this with the "many worlds" theory in cosmology. It's a different theory that I do not have time to get into.) In quantum mechanics we have all these situations where (to keep things simple) there are two possible outcomes. The "many worlds" theory says that both outcomes actually occur. But the world splits into two copies. In one world the first thing happens. In the other world the second thing happens. We (or at least the copy of "we" that we are following) exist in only one copy of these many worlds. As we look back we see, for instance, the first outcome of the most recent event. And we also see a specific outcome of the previous event. And so on. There is a world in which each possible outcome of each event happened. We see only the world as it is in our "world line" but there are zillions of other world lines in which events resulted in different outcomes.
In this conception of quantum mechanics there are in fact many worlds with a copy of each of us occupying many of them (remember, there may be events that caused us to not come into existence in some of these other world lines). But we can only see the world and the world line we are in. Scientists (and science fiction writers) have spent a lot of time working out detailed "many worlds" theories. No one has come up with proof that the theories are wrong. How could you? But most scientists go about their daily business as if the theories are wrong. And so far they have not come across any evidence that they are right.
Another line of thinking is the "Matrix" (from the popular movie) idea. Maybe the real world looks quite different but we live inside a giant sophisticated computer simulation of the world we see. Again, if the simulation is sophisticated enough and well enough done, there is no way to determine that it is a simulation. Here the proper response is "who cares?" The problem for Science just becomes "what are the rules in the simulated universe?" We can't beat the simulation. Therefore we have no access to the "real" real world. So it does us no good to try to study it. So we might as well stick to trying to figure out the simulated rules in our simulated world. Those are the rules we can use to advantage.
The second belief is one that philosophers have spent a lot of time on. There are systems of philosophy and religion that say "you can't so don't". There are also religious systems that say "the anointed get into heaven no matter what they do and the rest don't get in no matter what they do so why should anyone try to figure things out?" And, if your objective is to get into heaven, then it makes sense to not try if you subscribe to the "anointed" school of religion. But here we are talking about your efforts to achieve salvation in the next world. Most of us spend a lot of time trying to navigate around in the current world. For those science says "Science can be at least of some help".
And Science has a pretty good track record. Science has found out a lot about how the world works. Some scientists believe that all of the rules are discoverable. Some don't. But all scientists believe that there are undiscovered rules that can be discovered. So it doesn't really matter if they can all be discovered or not. As long as there are more rules that are currently undiscovered but haven't yet been discovered there is reason to continue the endeavor.
No scientist believes that all the rules that can be discovered have already been discovered. Nor do they believe that knowing the rules of science allows you to find the correct solution to all problems. There is even a shorthand for this. "Science is about 'what'. Religion/philosophy is about 'why'". Put another way, Science tells you a lot about what is possible but it doesn't tell you which choice among the possible choices is the "right" choice.
The third belief is actually quite a subtle one. Scientists believe that all the rules apply all the time. Now each rule can have a specific range of operation. So most rules don't apply in most situations. They can be effectively ignored. But part of a proper scientific rule is to define what situations it applies to. If a rule is applicable to a particular situation then it must be applied. Put another way, you must look at all the data when trying to create a rule. If you can't say why the rule applies here but doesn't apply there then there is a problem with your rule. This trips nonscientists up all the time. A rule may look perfectly good if you apply it to only a limited number of situations. But if it doesn't apply universally you better have a "whereas" that says in effect "but it doesn't apply in these other situations". Let me give you an example of this sort of thing done right.
Newton said "f=ma". As far as he knew this was a universal rule. But then Einstein came along and said in effect "f=ma + a relativistic correction". And he showed that the relativistic correction was such a tiny number in a lot of "normal world" situations that it could be ignored. Scientists and engineers know where the boundary between "normal world" and "relativistic world" is so they can keep it simple most of the time but fall back to the more complex relativistic methods when they need to.
A more common situation is where someone makes a "bold new" discovery. But no one else can get the method they claim to have used to work. This is common in the world psychic phenomenon. They say essentially "I did this and here's what happened". For instance they find certain people that can identify a card selected at random "at a rate above chance". But others make what appear to be the same test but don't get the "above chance" result. This situation has come up repeatedly. And the same explanation has been made repeatedly: shyness. Apparently the phenomenon is shy. If there are too many "negative vibrations" around the phenomenon goes into hiding. Many scientists have a different explanation: bad experimental procedure.
The same thing happens in more traditionally scientific situations. Check the Wikipedia articles on "N rays" or "cold fusion" for two examples that happened more than 80 years apart. The "shyness" explanation was trotted out in both of these examples. Eventually scientists concluded that the original discoverers just got it wrong.
Now let me move on to my corollaries.
The first one ("whatever works is right") seems obvious and sensible. But scientists have gotten really good at coming up with really weird results. The classic example is photons. People have been studying light for a long time. As I pointed out in my previous post, Newton did some wonderful work and in 1704 published his results in a book whose title rendered in English is "Optics". A lot of good work was done before this and a whole lot was done afterwards. In spite of this no one could decide whether light was composed of waves or particles. That is until Einstein came along in 1905. His answer was "neither". Instead light was composed of things called photons. Photons behave like waves in some situations and particles in other situations.
To a lot of people this was a completely unsatisfactory solution. An answer of "particles" or an answer of "waves" would have been natural and satisfactory. But the fact is that the extremely weird theory of "quantum electrodynamics" (the modern successor to Einstein's original work) does an excellent job of telling us how light will behave in all kinds of situations. QED (the standard shorthand for quantum electrodynamics) works. "Waves" doesn't work. "Particles" doesn't work. And science has now built up a long list of truly weird theories that work. Evolution (which is not actually that weird) works. Quantum Mechanics (actually its modern descendent "The Standard Model") works. Relativity works. The list goes on.
The second corollary ("whatever doesn't work is wrong") also seems obvious and sensible. But in practice it is not. As I just got done pointing out, in the case of light, "waves" doesn't work and "particles" doesn't work. But they are both connected to our day to day experience. Similarly, the Bible as an accurate historical reference doesn't work. Pre-QM atomic theories don't work. What is now called "Newtonian Mechanics" (e.g. "f=ma" and other non-relativistic theories) doesn't work. In some cases the old theory that doesn't work is readily discarded. If it strikes people as weird or unnatural they are happy to let it go.
Most people had no idea what "N rays" were about. They didn't much care when they were "discovered". And they didn't much care when they were debunked. So it was pretty easy to discard the whole "N Ray" theory. And the whole "N Ray" battle was scientists versus scientists. The general public felt it didn't have a dog in the fight. But in the case of the other now discarded theories I mentioned, people had a lot more invested in the old theories. There are large, well funded groups invested in the whole Evolution "debate" so leaving the discredited alternatives behind has turned out to be much harder. But that's why this corollary is so important. Except . . .
There are degrees of wrongness. All scientific theories have problems. The gold standard of scientific theories is "The Standard Model". It is the governing model for particle physics and is generally characterized as "spectacularly successful". It has problems. The biggest and most obvious problem is that no one knows how to reconcile it with Relativity. So it fails the test for perfect scientific theory. In fact there are no known perfect scientific theories. So how do scientists deal with this? The short answer is they fudge. That's why scientific theories are always referred to as "theories". It is always possible that the theory will be modified or even completely replaced by a quite different theory.
Scientists have now had hundreds of years of experience dealing with this issue. And part of that experience is with dealing with theories that are known to be wrong but have proved to be useful anyhow. They try to balance the utility of each theory with the problems its inadequacies can cause. This has resulted in an informal "wrongness" scale:
Degrees of wrongness
Best - These are well established theories. Scientists believe that the core of the theory is correct but that some modifications may be needed in the future around the edges. Also, any theory that is in contravention to the core of the theory is easily demonstrated to be wrong. A classic example of this is Evolution. Evolution grew out of problems with all bible based theories. So all bible based theories are known to be wrong. The core of evolution is known to be solid. Some tweaking (the relationship of two species, evolutionary mechanisms, details of the evolutionary record) may need tweaking but the core is solid.
Very good - These are theories that have had a great deal of success. But there are known problems for which no known solution currently exists. A classic example of this is The Standard Model. The relationship of this theory to Relativity is currently a big problem. There are other issues where scientists are not sure they have everything right. Note: Finding the Higgs Boson substantially reduced the problem areas seen with the Standard Model.
Good - The theory works well in a lot of situations but has a number of known problems. Like Very Good theories the core looks solid but here there are a lot more problems around the edges. A classic example of this is Plate Tectonics. This theory solved a lot of problems. It gives us a good way to understand a lot of geology. But there are a number of open problems having to do with how some plate boundaries work. There are a lot of areas where current understanding can best be described as "incomplete". But there are a number of areas where the theory works very well and there is currently no better theory on the horizon.
Ok - Science has had a lot of success with an incremental approach. A bad theory is better than no theory at all. A lot of theories don't work very well. But they work some of the time and the fact that we know where they work and where they have problems is helpful in determining what additional experiments might yield useful information or what problems a new theory might solve. A classis example of this kind of theory is the "Cooper pair" explanation for superconductivity. It works pretty well some of the time and not well at all a lot of the time. Nobody currently has any ideas that seem to work better. No one would be surprised if a new theory came along that was quite different to do a better job of explaining Superconductivity.
Poor - These theories are typically found where scientists really don't have much understanding. A good example of this would be planetary formation. There used to be what appeared to be a good theory. Small "rocky" planets formed close to stars where gasses were boiled off. Gas giants formed further away where it was cooler. Then came all the data on exoplanets from the Kepler spacecraft. It turns out there are all kinds of planets of the wrong kind and size in the wrong places. Scientists have some ideas but mostly it's early days.
Not even wrong - This phrase is attributed to Wolfgang Pauli, the great physicists. In describing ok and poor degrees of wrongness I emphasized that these theories all were useful to the extent that they were helpful in pointing to where a better theory might be found. "Not even wrong" theories are so bad that they just add noise to the situation.
So given all this wrongness why do scientists get so exercised about theories or criticisms from outsiders? Scientists are very aware of how problematic some of their theories are. And they dearly love it when a new better theory comes along. But is generally pretty easy to slot a lot of "new" theory into one of the above categories. Scientists get exercised when a new theory slots into a category that is lower than the one their current favored theory occupies. They also get exercised when a theory is advanced or a criticism is leveled that they have seen before (frequently many times) and have already thoroughly investigated.
The evolution "debate" falls into this category. All of the early scientists that worked on evolution were practicing Christians. They were very familiar with the bible. Where the bible appeared to get it right, those ideas were incorporated in evolution. But in many areas the bible got it wrong. Darwin's "Origin of Species" was first published in 1859. It was immediately criticized. Several new editions were quickly put out that included additional material addressing these criticisms. The sixth edition, the one commonly available now, was published in 1872. It should be shocking how many "new" arguments against evolution are in fact old arguments that Darwin addressed in great detail over a hundred years ago but apparently it is not. But most people who oppose evolution have never bothered to read Darwin or familiarize themselves with subsequent developments. I recently published a post on "Ken Ham Creationism" (http://sigma5.blogspot.com/2014/02/ken-ham-creationism.html). Many of Hams "modern" criticisms of evolution find well supported refutations in Origin, a book that is now over a hundred years old.
Friday, March 21, 2014
Fixing healthcare.gov
This is the second post I have done that is in response to material published by Steven Brill. My first post is at http://sigma5.blogspot.com/2013/02/medical-costs.html. Both were published in Time magazine. His more recent article is in the March 10, 2014 issue and is titled "Code Red". Brill is an astute observer of the Health Care industry. But he is not a techie. This did not get in the way of his previous article. With his substantial business background he was able to dissect the hospital business to devastating effect. I recommend both his original story and my analysis of it. And, unfortunately, all of the defects he found then are still with us.
His lack of computer expertise shows in his recent outing. But he still has a lot of useful things to say. So it is a worthwhile endeavor on his part. And don't forget the news environment. Problems with healthcare.gov were wretchedly excessively over covered by all parts of the media. The "it's fixed" story has been covered, although perhaps less than it should have been given the overkill that preceded it. But Brill's story is unique in my experience in that he addresses the "how it got fixed" story. If you are tempted to say "that's not news" then check out any sports section from any paper on any day. Every one is jammed full of "how we won/lost" stories. On to Brill's piece.
The obamacare.gov saga can be broken down into 4 eras. Before October 1 the story was "It will work great from the get go on October 1. Trust me." Era two ran from October 1 through October 17. This was the "It's broken. Nothing serious. We'll have it fixed any minute now." era. The third era (October 17 through December 23) is the one Brill focuses most of his attention on. This is the period when the site transitioned from seriously broken to working well enough for the moment. The fourth era (after December 24) is the modern "It's working in the main. There's more to fix but we have things under control." era.
Brill throws in some "tech talk" tidbits to spice things up but does not get into that much nitty gritty. Instead he puts more focus into a management level perspective. This turns out to be very useful. At a low level all tech projects are different. But at the management level there is a lot of commonality and a lot that can be usefully applied to the next tech project. So let's take a look at what Brill found.
And let me start by picking something out of the middle. I believe it's the most important thing Brill reports. Three rules for behavior were quickly adopted. 1. Meeting are for solving problems. Blame games can be played somewhere else. 2. The ones who should be talking are the ones who know the most, not the ones with the highest rank. 3. We need to stay focused on the most urgent issues.
If you take the opposite of these rules you will have the rules politics are usually played by. But these rules are used over and over to pull big technical projects off successfully. Politics has a big effect in the short run. Technical changes have an even bigger effect in the long run. Political effects tend to cancel each other out fairly quickly. Technical changes tend to change everything and endure. As an aside, it would be nice if the media, particularly the beltway media, stopped reporting everything through the lens of political rules and instead reported things through the lens of technical rules. This change would help fix the mess our national politics are currently in.
The bottom line is that by adopting the above behavior (and some other things) a small team was able to fix the web site in six weeks or less. Brill focuses on the contributions of the small team of techie superstars that the Obama Administration assembled, once they figured out they had a big problem. And, as Brill points out, the Obama people had another big problem that happened to exactly coincide with era number two. This is the time when the government shutdown was in effect. Imagine the hew and cry if the administration had diverted its efforts away from getting the government shutdown ended and had instead focused on the "minor" (certainly how it would have been characterized) problem represented by the web site. I don't think that it is a coincidence that the administration took major action to fix the site immediately after the shutdown crisis was resolved.
If you want lots more details of how the site was fixed, read the article. But let me cherry pick some more. A lot of time has been spent on the "blame the contractors" game. This game has been played all over the place, not just with healthcare.gov. But the first thing Brill reports is that the technical people employed by the site's contractors were not "defensive or hostile". In fact the small team of fixers and the bigger teams of contractor people quickly developed a good working relationship and worked together well. And once the fixers took over management of the project the contractor management also quickly fell into line. In fact one of the contracting companies solved a big administrative problem by putting the fixers on their payroll. This made them kosher with respect to various government rules and regulations. The same turned out to be true of the Obama administration types. No one started playing politics or putting up unnecessary roadblocks. This broad cooperation up and down the line was one of the most important keys to getting the site fixed quickly.
The only administration side management flaw Brill found was that there seemed to be no one individual that was in overall charge. Responsibility was diffuse and it was unclear who was responsible for what. On the other hand, Obama himself comes off very well. Early and often he was asking everyone if things were on track. He was repeatedly reassured that they were. Once the site started failing he held daily meetings to try to get a handle on the situation. And as soon as the shutdown was over he tasked a single individual, the one person in his inner circle who actually had technical expertise, to get 'er fixed. Things immediately started moving quickly from there.
The fixer team consisted of the best. And they had deep and broad expertise in big technology projects. Several were silicone valley heavyweights. The fixer team immediately set out to fix things. Some of this involved hands on work by members of the team like fixing code. But they also applied their hard earned knowledge of how you do this kind of thing. The first thing they did was put together something called a "dashboard". It performs for software what the dashboard in your car does for your car. It tells you what's working and what isn't (e.g. "check engine"). It also tells you things like how fast you are going or, in the case of the web site, how many people are accessing the site. Then their are general health indicators (gas gage, engine temp). In the case of the site things like the access times of the several databases the site depended on. This allowed the team to translate "its broken" into "these specific things are not working right". The more detailed information from the dashboard allowed engineers to investigate specific components looking for specific problems. And that in turn led to specific fixes that made the site work better. To boil it all down, six weeks worth of the right specific fixes resulted in a site that worked pretty well and did all the "needs to be working right now" things correctly.
Now let me back off and apply my technical expertise to fill in some blanks. I'm sure the fix team fixed a lot of stuff. But I suspect that most of the fixes were actually done by the engineers employed by the contractors. One of the fix team observes "these guys want to fix things". I suspect that the problem was not a lack of talent on the part of the contractors. I suspect that the main problem was poor project management. I have been involved in enough IT projects to know from personal experience that any IT project can be screwed up by managing it badly enough. And it doesn't matter what anyone else does. The best management has a good understanding of the technology (in this case web sites and databases) and a good understanding of the business (in this case health insurance). Given enough good will (and time and money) shortcoming in one or both areas can be overcome. In this case the basic technology side did not look that hard to me. But, as I will explain below, the business side was actually a lot harder than it would appear.
Plenty of money was spent on the site. Brill reports the amount was $300 million. That should have been more than enough. One problem that was identified early was that the hardware was inadequate. But that problem was identified almost immediately. Hardware was added in early October, well before the 17th. And more hardware kept being added well into December. It may be that more hardware is still being added.
Now let me provide some broader perspective, perspective that is beyond the scope of the subject matter of Brill's article. I am going to take a quick look at two state sites, Washington and Oregon. Both of these state sites had a simpler task. They only had to deal with the idiosyncrasies of one state. The federal site had to deal with the idiosyncrasies of more than 20 states. Both states are blue states that are in decent shape from a budget perspective. So there presumably was enough money in each case and there were no political roadblocks. But the experience of the two states couldn't be more different.
The Washington site went up on time. It has had a few glitches. But for the most part it has worked from the beginning and worked well. Oregon's site has been a disaster. And it has been a far bigger disaster than the federal site. It never worked at all during the entire October - November time period. The last time I saw any news coverage about the site (a couple of months ago now) reports were that it was still completely broken. Oregon gave up and declared some time in October that they were going to an all manual system. So why the differing outcomes? Well, there were apparently some differences between how each state behaved that turned out to be critical.
Washington decided a couple of years ago that the state systems and databases needed to support the state web site were not up to the task. So they rolled out new modern state systems before October 1. This meant that the interfaces between the state web site and the state systems could be simple and that the state systems would be able to deliver exactly what the state site needed. The web site would not have to "crutch around" inadequacies in the state system. Washington also bid out the web site in the usual manner but was able to provide substantial state supervision "in house". The result, as I said, was a site that worked pretty well pretty much from day one.
As you might expect, Oregon did things differently. I don't know whether Oregon did a major upgrade of their state systems like Washington did. I certainly haven't heard that they did. The other thing Oregon did was outsource pretty much the whole thing including project management to Oracle Corporation. I have a lot of personal experience dealing with Oracle. They have good people and they have bad people. The company I worked for was not big enough to justify always getting the good people so we frequently got the bad people. Oracle missed out on a big business opportunity by so mismanaging a project (an outsourcing as it happens) that we cancelled the project and kept that functionality in house. I don't know anything about the specifics of the Oregon project. I definitely know nothing about who was or was not assigned to the project by Oracle. But I do note that things failed. And they failed spectacularly and in a very public manner.
So how does this relate to the federal project? First, as I noted above, the federal project has to interface with a large number of state systems. That is much more complicated than what Oregon had to pull off. And the federal system always has worked better than the Oregon system, even in those bad old days of early October. The only thing you can say in favor of the Oregon experience is that the site worked so badly that it was easy to decide early on to just scrap it. So the Oregon experience is obviously apt. The task was harder to pull off and easier to screw up that it would appear from the outside. But what does the Washington experience tell us?
Remember the part where I pointed out that the state decided to do a major upgrade to the state systems the web site would depend on. Many of the states that depend on the federal site are southern states. These are states that are generally technology averse and poor. What kind of shape do you think the state systems are in that have to tie into the federal site? My guess is that they are in very poor shape. This means that the federal site would have to have a lot of "crutch around" capability.
Add to this the fact that many of the states depend on the federal site are red states. The GOP has been adamantly opposed to Obamacare. I think the anti-Obamacare vote count in the U.S. House is now up to 50. I pointed out above that it is easy for bad management to mess up IT projects. It is also easy for hostile state governments to put all kinds of road blocks in the way of a successful web site implementation. So the federal site has to deal with the complexity of servicing multiple states. I don't know for sure but it is a good guess that it has to deal with state systems that are more or less inadequate. And it has to deal with state administrations that are hostile to Obamacare, and therefore the web site, for political reasons. Finally, I will note that some states made the decision to not do their own site very late in the game. This meant that the federal site had to be reconfigured to handle these states very late in the game. In short, the federal site had a number of things to contend with that neither Washington (success) nor Oregon (failure) did.
The site should have worked on October 1. It seems apparent in retrospect that the biggest cause was that the project was poorly managed by the Obama administration. One specific cause was that no single individual was in charge of getting the web site right. But, if the site could be fixed in 6 weeks, it wasn't really all that broken. And there were a number of contributing factors. The government shutdown and the long run up to the shutdown were very disruptive. And the knee jerk hostility of Republicans were definitely contributed. As just one example, how much time was spent by senior administration officials trying to move other senior officials through the Senate confirmation process. The time they spent in this necessary but essentially useless task could perhaps been spent doing a better job of managing the web site rollout instead. And it is important to remember that the Oregon experience shows us that it was easier to bungle this process than it appeared.
His lack of computer expertise shows in his recent outing. But he still has a lot of useful things to say. So it is a worthwhile endeavor on his part. And don't forget the news environment. Problems with healthcare.gov were wretchedly excessively over covered by all parts of the media. The "it's fixed" story has been covered, although perhaps less than it should have been given the overkill that preceded it. But Brill's story is unique in my experience in that he addresses the "how it got fixed" story. If you are tempted to say "that's not news" then check out any sports section from any paper on any day. Every one is jammed full of "how we won/lost" stories. On to Brill's piece.
The obamacare.gov saga can be broken down into 4 eras. Before October 1 the story was "It will work great from the get go on October 1. Trust me." Era two ran from October 1 through October 17. This was the "It's broken. Nothing serious. We'll have it fixed any minute now." era. The third era (October 17 through December 23) is the one Brill focuses most of his attention on. This is the period when the site transitioned from seriously broken to working well enough for the moment. The fourth era (after December 24) is the modern "It's working in the main. There's more to fix but we have things under control." era.
Brill throws in some "tech talk" tidbits to spice things up but does not get into that much nitty gritty. Instead he puts more focus into a management level perspective. This turns out to be very useful. At a low level all tech projects are different. But at the management level there is a lot of commonality and a lot that can be usefully applied to the next tech project. So let's take a look at what Brill found.
And let me start by picking something out of the middle. I believe it's the most important thing Brill reports. Three rules for behavior were quickly adopted. 1. Meeting are for solving problems. Blame games can be played somewhere else. 2. The ones who should be talking are the ones who know the most, not the ones with the highest rank. 3. We need to stay focused on the most urgent issues.
If you take the opposite of these rules you will have the rules politics are usually played by. But these rules are used over and over to pull big technical projects off successfully. Politics has a big effect in the short run. Technical changes have an even bigger effect in the long run. Political effects tend to cancel each other out fairly quickly. Technical changes tend to change everything and endure. As an aside, it would be nice if the media, particularly the beltway media, stopped reporting everything through the lens of political rules and instead reported things through the lens of technical rules. This change would help fix the mess our national politics are currently in.
The bottom line is that by adopting the above behavior (and some other things) a small team was able to fix the web site in six weeks or less. Brill focuses on the contributions of the small team of techie superstars that the Obama Administration assembled, once they figured out they had a big problem. And, as Brill points out, the Obama people had another big problem that happened to exactly coincide with era number two. This is the time when the government shutdown was in effect. Imagine the hew and cry if the administration had diverted its efforts away from getting the government shutdown ended and had instead focused on the "minor" (certainly how it would have been characterized) problem represented by the web site. I don't think that it is a coincidence that the administration took major action to fix the site immediately after the shutdown crisis was resolved.
If you want lots more details of how the site was fixed, read the article. But let me cherry pick some more. A lot of time has been spent on the "blame the contractors" game. This game has been played all over the place, not just with healthcare.gov. But the first thing Brill reports is that the technical people employed by the site's contractors were not "defensive or hostile". In fact the small team of fixers and the bigger teams of contractor people quickly developed a good working relationship and worked together well. And once the fixers took over management of the project the contractor management also quickly fell into line. In fact one of the contracting companies solved a big administrative problem by putting the fixers on their payroll. This made them kosher with respect to various government rules and regulations. The same turned out to be true of the Obama administration types. No one started playing politics or putting up unnecessary roadblocks. This broad cooperation up and down the line was one of the most important keys to getting the site fixed quickly.
The only administration side management flaw Brill found was that there seemed to be no one individual that was in overall charge. Responsibility was diffuse and it was unclear who was responsible for what. On the other hand, Obama himself comes off very well. Early and often he was asking everyone if things were on track. He was repeatedly reassured that they were. Once the site started failing he held daily meetings to try to get a handle on the situation. And as soon as the shutdown was over he tasked a single individual, the one person in his inner circle who actually had technical expertise, to get 'er fixed. Things immediately started moving quickly from there.
The fixer team consisted of the best. And they had deep and broad expertise in big technology projects. Several were silicone valley heavyweights. The fixer team immediately set out to fix things. Some of this involved hands on work by members of the team like fixing code. But they also applied their hard earned knowledge of how you do this kind of thing. The first thing they did was put together something called a "dashboard". It performs for software what the dashboard in your car does for your car. It tells you what's working and what isn't (e.g. "check engine"). It also tells you things like how fast you are going or, in the case of the web site, how many people are accessing the site. Then their are general health indicators (gas gage, engine temp). In the case of the site things like the access times of the several databases the site depended on. This allowed the team to translate "its broken" into "these specific things are not working right". The more detailed information from the dashboard allowed engineers to investigate specific components looking for specific problems. And that in turn led to specific fixes that made the site work better. To boil it all down, six weeks worth of the right specific fixes resulted in a site that worked pretty well and did all the "needs to be working right now" things correctly.
Now let me back off and apply my technical expertise to fill in some blanks. I'm sure the fix team fixed a lot of stuff. But I suspect that most of the fixes were actually done by the engineers employed by the contractors. One of the fix team observes "these guys want to fix things". I suspect that the problem was not a lack of talent on the part of the contractors. I suspect that the main problem was poor project management. I have been involved in enough IT projects to know from personal experience that any IT project can be screwed up by managing it badly enough. And it doesn't matter what anyone else does. The best management has a good understanding of the technology (in this case web sites and databases) and a good understanding of the business (in this case health insurance). Given enough good will (and time and money) shortcoming in one or both areas can be overcome. In this case the basic technology side did not look that hard to me. But, as I will explain below, the business side was actually a lot harder than it would appear.
Plenty of money was spent on the site. Brill reports the amount was $300 million. That should have been more than enough. One problem that was identified early was that the hardware was inadequate. But that problem was identified almost immediately. Hardware was added in early October, well before the 17th. And more hardware kept being added well into December. It may be that more hardware is still being added.
Now let me provide some broader perspective, perspective that is beyond the scope of the subject matter of Brill's article. I am going to take a quick look at two state sites, Washington and Oregon. Both of these state sites had a simpler task. They only had to deal with the idiosyncrasies of one state. The federal site had to deal with the idiosyncrasies of more than 20 states. Both states are blue states that are in decent shape from a budget perspective. So there presumably was enough money in each case and there were no political roadblocks. But the experience of the two states couldn't be more different.
The Washington site went up on time. It has had a few glitches. But for the most part it has worked from the beginning and worked well. Oregon's site has been a disaster. And it has been a far bigger disaster than the federal site. It never worked at all during the entire October - November time period. The last time I saw any news coverage about the site (a couple of months ago now) reports were that it was still completely broken. Oregon gave up and declared some time in October that they were going to an all manual system. So why the differing outcomes? Well, there were apparently some differences between how each state behaved that turned out to be critical.
Washington decided a couple of years ago that the state systems and databases needed to support the state web site were not up to the task. So they rolled out new modern state systems before October 1. This meant that the interfaces between the state web site and the state systems could be simple and that the state systems would be able to deliver exactly what the state site needed. The web site would not have to "crutch around" inadequacies in the state system. Washington also bid out the web site in the usual manner but was able to provide substantial state supervision "in house". The result, as I said, was a site that worked pretty well pretty much from day one.
As you might expect, Oregon did things differently. I don't know whether Oregon did a major upgrade of their state systems like Washington did. I certainly haven't heard that they did. The other thing Oregon did was outsource pretty much the whole thing including project management to Oracle Corporation. I have a lot of personal experience dealing with Oracle. They have good people and they have bad people. The company I worked for was not big enough to justify always getting the good people so we frequently got the bad people. Oracle missed out on a big business opportunity by so mismanaging a project (an outsourcing as it happens) that we cancelled the project and kept that functionality in house. I don't know anything about the specifics of the Oregon project. I definitely know nothing about who was or was not assigned to the project by Oracle. But I do note that things failed. And they failed spectacularly and in a very public manner.
So how does this relate to the federal project? First, as I noted above, the federal project has to interface with a large number of state systems. That is much more complicated than what Oregon had to pull off. And the federal system always has worked better than the Oregon system, even in those bad old days of early October. The only thing you can say in favor of the Oregon experience is that the site worked so badly that it was easy to decide early on to just scrap it. So the Oregon experience is obviously apt. The task was harder to pull off and easier to screw up that it would appear from the outside. But what does the Washington experience tell us?
Remember the part where I pointed out that the state decided to do a major upgrade to the state systems the web site would depend on. Many of the states that depend on the federal site are southern states. These are states that are generally technology averse and poor. What kind of shape do you think the state systems are in that have to tie into the federal site? My guess is that they are in very poor shape. This means that the federal site would have to have a lot of "crutch around" capability.
Add to this the fact that many of the states depend on the federal site are red states. The GOP has been adamantly opposed to Obamacare. I think the anti-Obamacare vote count in the U.S. House is now up to 50. I pointed out above that it is easy for bad management to mess up IT projects. It is also easy for hostile state governments to put all kinds of road blocks in the way of a successful web site implementation. So the federal site has to deal with the complexity of servicing multiple states. I don't know for sure but it is a good guess that it has to deal with state systems that are more or less inadequate. And it has to deal with state administrations that are hostile to Obamacare, and therefore the web site, for political reasons. Finally, I will note that some states made the decision to not do their own site very late in the game. This meant that the federal site had to be reconfigured to handle these states very late in the game. In short, the federal site had a number of things to contend with that neither Washington (success) nor Oregon (failure) did.
The site should have worked on October 1. It seems apparent in retrospect that the biggest cause was that the project was poorly managed by the Obama administration. One specific cause was that no single individual was in charge of getting the web site right. But, if the site could be fixed in 6 weeks, it wasn't really all that broken. And there were a number of contributing factors. The government shutdown and the long run up to the shutdown were very disruptive. And the knee jerk hostility of Republicans were definitely contributed. As just one example, how much time was spent by senior administration officials trying to move other senior officials through the Senate confirmation process. The time they spent in this necessary but essentially useless task could perhaps been spent doing a better job of managing the web site rollout instead. And it is important to remember that the Oregon experience shows us that it was easier to bungle this process than it appeared.
Wednesday, March 5, 2014
Caller ID is sick and getting sciker
Caller ID is a relatively new technology. It only dates back to the '90s. Caller ID was not possible until telephone switching equipment became computerized. If you look at old black and white crime movies you will see what it used to take to "trace" a call. You literally had to have people standing by at telephone "central offices", buildings full of complex electromechanical equipment. When he got the signal the technician would literally look at the equipment to follow the path of the telephone call. According to dialog in these old movies it typically took a minimum of two minutes to trace a call. And tracing was only possible if it was a local call.
The advent of computerized telephone equipment changed all this. First the industry had to change over to computer control of the telephone equipment. Then standardized methodology for doing a trace had to be developed. Finally a method of passing the trace information to a customer's phone had to be developed. But all that eventually happened. And when caller ID first rolled out it was very cool. A little display on your phone would show you the name and phone number of the caller. Back not so long ago this little trick seemed totally awesome. Now in the era of smart phones and Skype it doesn't seem like a big deal.
And it turns out that caller ID depends on a technology that is rapidly disappearing, the phone book. It took me a while to figure out that this connection existed but it does. I have a phone book but I believe it is now about three years old. (The phone company has stopped dropping a new one off every year.) Phone books used to be a cash cow. You got a set of "white pages", which cost the phone company a bundle of money to produce. But in exchange you also got a set of "yellow pages". And not so long ago yellow pages were a very effective method of advertising. So businesses signed up and paid through the nose to be prominently featured. And they frequently had to buy multiple ads to cover all of what we now call "search terms". You would want your pizza joint listed under "Pizza" and "Restaurant", for instance.
But it is now easier and more effective to do a search, on your computer (if you are an old fart like me), or on your smart phone (if you are not). So the value of putting an ad in the yellow pages has plummeted. And most businesses have figured that out. You are far better off putting your money into Google "adwords" instead. And with the plunge in revenue from yellow pages has come a plunge in interest in complete and accurate white pages by the phone companies. And that, in turn has become a problem for caller ID.
I am new to caller ID. First I didn't have a phone that would display caller ID information. Then it didn't seem worth the trouble. Finally I got fed up with all the nuisance phone calls I was getting. So I bit and got caller ID. I am looking at it with fresh eyes compared to someone who has had it for some time. And my conclusion is that it doesn't work nearly as well as it should. And it looks to me like the situation is only going to continue to get worse. So what's the problem?
Well, technically caller ID works just fine. Every single time the phone rings a "caller ID" informational display pops up on my phone. But the actual information is not very informative a lot of the time. The obvious problem is with "blocked" calls. A provision was put into the regulations that allowed a caller to block caller ID. The justification was to protect the privacy of single women living alone. But as far as I can tell anyone can have caller ID blocked.
This is actually a far smaller problem than you would expect. Most women have found that blocking caller ID is more trouble than it is worth so they don't bother. So I only occasionally see a "blocked" message come up on my display. It turns out their are better methods for hiding your identity. And I have trained the one person I know who has caller ID blocked to enter an "unblock caller ID for this call" code before she calls me. So "blocked" calls are not much of a problem. And to the extent this is a problem the impact is going down because fewer and fewer callers have blocking turned on.
There is a technical term for a much bigger problem with caller ID. The term is "spoofing". Spoofing covers any situation where the number shown is not the actual number of the calling phone. You would think that spoofing would be limited to spies and crooks. But it is actually in widespread use by many companies large and small. An obvious example is an "800" number. Free to the caller long distance has been around for a long time now. I even know of a situation where an individual had children scattered across the country. In order to encourage them to call home he got a personal 800 number. It cost little and had the desired result of causing his children to call home more often.
He was happy with the result (this was several years ago) but cost of long distance has dropped so much it would hardly be worth the trouble now. So what's the point? The point is that no phone line is ever actually assigned an 800 number. The 800 number is an "overlay". Some computer somewhere in the phone network replaces the 800 number you dial with an actual phone number. The call is completed to this normal number, which you never see. But many calls I get from companies show an 800 number on the caller ID display. Any time I see an 800 number I know the call has been spoofed.
Now this is not necessarily a problem. In most cases if I call the 800 number I will get the company that called me. So this example of spoofing is harmless. The problem is that the "name" part of the caller ID display often does not show the name of the company. It often dumbly duplicates the same 800 number shown on the "number" line. So I don't know who's calling me without answering the phone. This even happened to me with a call from Bank of America. You'd think they would want me to know it was them calling.
So, to recapitulate, phone numbers are frequently spoofed by businesses. And I have even seen it done by very small businesses. And the number displayed is often an 800 number. That can be justified on the basis that it can make it more convenient for me, the customer, to get back to the company in question. But the point is that spoofing is common. So in many cases you can't trust the number caller ID displays. I know that I am being spoofed when I see an 800 number on the display. So I know a large number of caller ID displays are showing spoofed numbers. Do I know that the other numbers I see displayed are the number of the caller? No. It is possible (and likely in the case of the frequent "marketing" calls I get) that other numbers are being spoofed too. It's just that I can't immediately tell that this is happening.
So there is a big problem with spoofing the caller's number. There is an even bigger problem with the other piece of information caller ID is supposed to give you. There is frequently no name in the "name" field. I indicated above that my call from Bank of America just repeated the 800 number rather than saying "Bank of America". This is a particularly bad idea on their part. Over the years I have received many calls from "Rachel from cardholder services". I even wrote a blog post about it (see http://sigma5.blogspot.com/2012/02/rachel-from-cardholder-services.html). Rachel fronts for a scammer that wants me to switch to a different credit card. You would think that Bank of America would want to do everything they could to discourage this. But their legitimate call to me looked very much like a "scam" call.
So I see a lot of calls where the "name" field just repeats the information in the "number" field. This seems to happen most frequently with calls from businesses. But, since I no longer answer calls when I can't figure out who the caller is I don't really know. So repeating the number in the "name" field is a problem. But there is another problem with the information in the "name" field.
Frequently it contains just a "City ST" combination. I recently was called from "Rochester NY", for instance. I have received a number of calls from "Albuquerque NM" in the past few weeks. Apparently I have a fan base spread all across the country. Many of these calls are apparently from places where I know I don't know anyone. So it is easy to ignore those calls. But sometimes I bite and answer the phone. The "City ST" combination happens to match that of a distant friend or relative. So it might be a call from them. But when I do bite and answer it has often turned out to be a scam/marketing call. But not always.
Cell phones (or "mobile" phones, as most of the rest of the world calls them) routinely pop up with this same "City ST" type of display in the "name" field. In fact, I have never gotten a call from a cell phone where the person's name popped up. As far as I can tell cell providers never register a name for a cell phone number. I recently got my mother a prepaid phone. You can register them. When you do you provide the usual contact information. But in this case I could not change the name from "MOBILE USER" to that of my mother. Now the company involved (it was an AT&T "go phone") certainly knew my mother's address from the contact information I filled in on her behalf. They even would have had an email address except my mother doesn't have an email account. But they literally made it impossible to change the name on the account.
And this is where the death of white pages ties in. It is no longer important to have a phone book to look numbers up in. In fact, it is of decreasing importance to be able to look a person's number up online. So apparently cell phone companies have decided as a matter of policy to not publish name and address information for cell phones. And the world is increasingly going mobile. The number of people who only have a mobile is growing. Kids do it. But I know a number of budget challenged adults who have given up their land line to save money. They all feel that going "mobile only" is feasible but going "landline only" is not. My guess is that caller ID gets the name information from public directories (the online version of white pages). But more and more phone numbers are not listed in any public directories. So the percentage of the time that the "name" field in a caller ID entry has useful information is low and dropping and this makes caller ID less useful. The situation has gotten so bad that it is apparently cutting into the ability of the NSA to spy on us. A recent story (http://gizmodo.com/want-to-avoid-the-nsa-use-a-cell-phone-1518318611) reports that cell phones are giving the NSA fits.
My recent experience getting a prepaid phone for my mother allowed me to answer the burning question "are burner phones for real?" As anyone who has seen a cop show or a thriller recently knows, there is something called a "burner phone". It is a phone that can't be traced back to the user. And, since it is cheap and untraceable, it can be "burned" (thrown away) at any time. Lots of things that show up in entertainment pieces are fiction. It is way harder to crack encryption than the movies and TV lead you to believe. But it turns out that everything they say about burner phones is true. I bought my mother's phone at a drug store for cash. I put money into it using a credit card. So, with a little effort my mother's phone could be tracked back to me because I used my credit card. But I could have bought a prepaid card for cash and loaded money into the phone that way.
All the company wants is their money. If you buy the phone (for cash) and load money into the phone (say with a prepaid card bought with cash) then they get their money. So they don't care who you are or what you use the phone for. And that's a problem for the NSA. And its a problem for people like me who want to screen out nuisance calls using caller ID. The laws and regulations make all this legal. They could be changed. But the phone companies (and other businesses) are doing just fine with things as they are. So their lobbyists will make sure that the laws and regulations stay the way they are.
The advent of computerized telephone equipment changed all this. First the industry had to change over to computer control of the telephone equipment. Then standardized methodology for doing a trace had to be developed. Finally a method of passing the trace information to a customer's phone had to be developed. But all that eventually happened. And when caller ID first rolled out it was very cool. A little display on your phone would show you the name and phone number of the caller. Back not so long ago this little trick seemed totally awesome. Now in the era of smart phones and Skype it doesn't seem like a big deal.
And it turns out that caller ID depends on a technology that is rapidly disappearing, the phone book. It took me a while to figure out that this connection existed but it does. I have a phone book but I believe it is now about three years old. (The phone company has stopped dropping a new one off every year.) Phone books used to be a cash cow. You got a set of "white pages", which cost the phone company a bundle of money to produce. But in exchange you also got a set of "yellow pages". And not so long ago yellow pages were a very effective method of advertising. So businesses signed up and paid through the nose to be prominently featured. And they frequently had to buy multiple ads to cover all of what we now call "search terms". You would want your pizza joint listed under "Pizza" and "Restaurant", for instance.
But it is now easier and more effective to do a search, on your computer (if you are an old fart like me), or on your smart phone (if you are not). So the value of putting an ad in the yellow pages has plummeted. And most businesses have figured that out. You are far better off putting your money into Google "adwords" instead. And with the plunge in revenue from yellow pages has come a plunge in interest in complete and accurate white pages by the phone companies. And that, in turn has become a problem for caller ID.
I am new to caller ID. First I didn't have a phone that would display caller ID information. Then it didn't seem worth the trouble. Finally I got fed up with all the nuisance phone calls I was getting. So I bit and got caller ID. I am looking at it with fresh eyes compared to someone who has had it for some time. And my conclusion is that it doesn't work nearly as well as it should. And it looks to me like the situation is only going to continue to get worse. So what's the problem?
Well, technically caller ID works just fine. Every single time the phone rings a "caller ID" informational display pops up on my phone. But the actual information is not very informative a lot of the time. The obvious problem is with "blocked" calls. A provision was put into the regulations that allowed a caller to block caller ID. The justification was to protect the privacy of single women living alone. But as far as I can tell anyone can have caller ID blocked.
This is actually a far smaller problem than you would expect. Most women have found that blocking caller ID is more trouble than it is worth so they don't bother. So I only occasionally see a "blocked" message come up on my display. It turns out their are better methods for hiding your identity. And I have trained the one person I know who has caller ID blocked to enter an "unblock caller ID for this call" code before she calls me. So "blocked" calls are not much of a problem. And to the extent this is a problem the impact is going down because fewer and fewer callers have blocking turned on.
There is a technical term for a much bigger problem with caller ID. The term is "spoofing". Spoofing covers any situation where the number shown is not the actual number of the calling phone. You would think that spoofing would be limited to spies and crooks. But it is actually in widespread use by many companies large and small. An obvious example is an "800" number. Free to the caller long distance has been around for a long time now. I even know of a situation where an individual had children scattered across the country. In order to encourage them to call home he got a personal 800 number. It cost little and had the desired result of causing his children to call home more often.
He was happy with the result (this was several years ago) but cost of long distance has dropped so much it would hardly be worth the trouble now. So what's the point? The point is that no phone line is ever actually assigned an 800 number. The 800 number is an "overlay". Some computer somewhere in the phone network replaces the 800 number you dial with an actual phone number. The call is completed to this normal number, which you never see. But many calls I get from companies show an 800 number on the caller ID display. Any time I see an 800 number I know the call has been spoofed.
Now this is not necessarily a problem. In most cases if I call the 800 number I will get the company that called me. So this example of spoofing is harmless. The problem is that the "name" part of the caller ID display often does not show the name of the company. It often dumbly duplicates the same 800 number shown on the "number" line. So I don't know who's calling me without answering the phone. This even happened to me with a call from Bank of America. You'd think they would want me to know it was them calling.
So, to recapitulate, phone numbers are frequently spoofed by businesses. And I have even seen it done by very small businesses. And the number displayed is often an 800 number. That can be justified on the basis that it can make it more convenient for me, the customer, to get back to the company in question. But the point is that spoofing is common. So in many cases you can't trust the number caller ID displays. I know that I am being spoofed when I see an 800 number on the display. So I know a large number of caller ID displays are showing spoofed numbers. Do I know that the other numbers I see displayed are the number of the caller? No. It is possible (and likely in the case of the frequent "marketing" calls I get) that other numbers are being spoofed too. It's just that I can't immediately tell that this is happening.
So there is a big problem with spoofing the caller's number. There is an even bigger problem with the other piece of information caller ID is supposed to give you. There is frequently no name in the "name" field. I indicated above that my call from Bank of America just repeated the 800 number rather than saying "Bank of America". This is a particularly bad idea on their part. Over the years I have received many calls from "Rachel from cardholder services". I even wrote a blog post about it (see http://sigma5.blogspot.com/2012/02/rachel-from-cardholder-services.html). Rachel fronts for a scammer that wants me to switch to a different credit card. You would think that Bank of America would want to do everything they could to discourage this. But their legitimate call to me looked very much like a "scam" call.
So I see a lot of calls where the "name" field just repeats the information in the "number" field. This seems to happen most frequently with calls from businesses. But, since I no longer answer calls when I can't figure out who the caller is I don't really know. So repeating the number in the "name" field is a problem. But there is another problem with the information in the "name" field.
Frequently it contains just a "City ST" combination. I recently was called from "Rochester NY", for instance. I have received a number of calls from "Albuquerque NM" in the past few weeks. Apparently I have a fan base spread all across the country. Many of these calls are apparently from places where I know I don't know anyone. So it is easy to ignore those calls. But sometimes I bite and answer the phone. The "City ST" combination happens to match that of a distant friend or relative. So it might be a call from them. But when I do bite and answer it has often turned out to be a scam/marketing call. But not always.
Cell phones (or "mobile" phones, as most of the rest of the world calls them) routinely pop up with this same "City ST" type of display in the "name" field. In fact, I have never gotten a call from a cell phone where the person's name popped up. As far as I can tell cell providers never register a name for a cell phone number. I recently got my mother a prepaid phone. You can register them. When you do you provide the usual contact information. But in this case I could not change the name from "MOBILE USER" to that of my mother. Now the company involved (it was an AT&T "go phone") certainly knew my mother's address from the contact information I filled in on her behalf. They even would have had an email address except my mother doesn't have an email account. But they literally made it impossible to change the name on the account.
And this is where the death of white pages ties in. It is no longer important to have a phone book to look numbers up in. In fact, it is of decreasing importance to be able to look a person's number up online. So apparently cell phone companies have decided as a matter of policy to not publish name and address information for cell phones. And the world is increasingly going mobile. The number of people who only have a mobile is growing. Kids do it. But I know a number of budget challenged adults who have given up their land line to save money. They all feel that going "mobile only" is feasible but going "landline only" is not. My guess is that caller ID gets the name information from public directories (the online version of white pages). But more and more phone numbers are not listed in any public directories. So the percentage of the time that the "name" field in a caller ID entry has useful information is low and dropping and this makes caller ID less useful. The situation has gotten so bad that it is apparently cutting into the ability of the NSA to spy on us. A recent story (http://gizmodo.com/want-to-avoid-the-nsa-use-a-cell-phone-1518318611) reports that cell phones are giving the NSA fits.
My recent experience getting a prepaid phone for my mother allowed me to answer the burning question "are burner phones for real?" As anyone who has seen a cop show or a thriller recently knows, there is something called a "burner phone". It is a phone that can't be traced back to the user. And, since it is cheap and untraceable, it can be "burned" (thrown away) at any time. Lots of things that show up in entertainment pieces are fiction. It is way harder to crack encryption than the movies and TV lead you to believe. But it turns out that everything they say about burner phones is true. I bought my mother's phone at a drug store for cash. I put money into it using a credit card. So, with a little effort my mother's phone could be tracked back to me because I used my credit card. But I could have bought a prepaid card for cash and loaded money into the phone that way.
All the company wants is their money. If you buy the phone (for cash) and load money into the phone (say with a prepaid card bought with cash) then they get their money. So they don't care who you are or what you use the phone for. And that's a problem for the NSA. And its a problem for people like me who want to screen out nuisance calls using caller ID. The laws and regulations make all this legal. They could be changed. But the phone companies (and other businesses) are doing just fine with things as they are. So their lobbyists will make sure that the laws and regulations stay the way they are.
Subscribe to:
Posts (Atom)