The December 2015 edition of Scientific American contains a plug for a new/old idea called Microwave Rocketry. They date the idea to 1924 (the "old" part) but see it is a possible "World Changing Idea" (the "new" part). If you have never heard of this before, don't worry that you have somehow missed something. Nobody has figured out how to make it actually work yet. And the story didn't suggest that success was imminent.
But why would somebody want it to work? Because the current methods of transporting things into space don't work very well. I'll get to why later. Instead I am going to review the history of rockets (the current method that doesn't work very well). And, to make the subject more fun, I am going to look at rockets in the Science Fiction world as well as in the real world. Where to start? At the beginning, of course.
In round numbers the Chinese invented gunpowder, also called black powder, about a thousand years ago. Shortly thereafter they invented what we now call the skyrocket. A skyrocket is a firework. It's an entertainment device. But The Chinese quickly tried to repurpose it as a military weapon. It turns out that "shock and awe" is a thousand year old idea. When one side shot off a bunch of rockets at the other side for the first time they were shocked and awed. But that didn't last very long. The other side quickly figured out that rockets were so inaccurate they did not pose a serious threat to life and limb. So they quickly started ignoring them. The shock and awe thing didn't last very long.
And, in spite of the lack of success on the part of the Chinese, the very same tactic was tried over and over in the intervening centuries. During the Napoleonic Wars (roughly 1800) fusillades of rockets were shot off. The results were the same. A short period of shock and awe was quickly followed by indifference. The Soviets tried the same thing during World War II. They had a device called a Stalin Organ. It consisted of a trailer containing banks of rockets. All the rockets were shot off within a matter of a few seconds. At first the Germans were shocked and awed. But then they noticed that rockets were still wildly inaccurate and ignored them. That's the real world. Let's detour into the world of fiction.
Joules Verne was the first modern Science Fiction writer. He was very familiar with the science and technology of the time. Starting in 1863 he extrapolated from what he knew to form the technological base of a series of adventure novels. One of his first was called (in English, Verne wrote in French) "From the Earth to the Moon". The basic story was about a group of intrepid adventurers traveling from the Earth to the Moon and back again. If such a tale was being written now the method of transportation would have been a rocket ship. But Verne distained that method and instead went with a giant cannon.
Any technologist of his time would have made the same choice. There was a lot of scientific understanding of how to create large artillery pieces (fancy talk for cannons) and what they were capable of. The state of the art had advanced to the point where it was conceivable that a slightly more advanced design would be capable of hitting the moon with a large shell. There were a lot of other problems, dealing with the acceleration, navigation, slowing down when you got to the moon, etc. But in terms of picking the basic technology Vern made the right choice. There was literally no such thing as Rocket Science in the late 1800's.
Verne's scientifically credible style fell out of favor. About 50 years after Verne started writing Edgar Rice Burroughs, the guy who invented Tarzan of the Apes, also invented John Carter of Mars. (Do NOT see the movie of a few years ago. It's terrible!) Burroughs used completely fantastical methods to get Carter back and forth between Earth and Mars. And, for the most part, so did a lot of authors as the new literary style called Science Fiction came into existence in roughly 1920. Science Fiction largely was organized around the fantasies of young males. So it featured a lot of babes, monsters, and ray guns. But this gradually changed. The prime mover behind this change was an editor named John W. Campbell.
Campbell championed the idea of making Science Fiction science based. Sticking purely to what was scientifically known and understood was no fun. So Campbell gave his authors one out. They were allowed to violate one law of science as it was known and understood. This rule was not rigidly adhered to. But it pushed Science Fiction in the direction of being much more science based. The result is what fans now call the first golden age of Science Fiction. So it was ultimately every effective.
There was still no actual rocket science in the early phases of this period. But that soon changed dramatically. And the dramatic change can be attributed to a single individual. His name is Robert H. Goddard. Goddard is now mostly lost to the annals of history. He is memorialized in the name of the Goddard Space Flight Center in Greenbelt, Maryland. But this is a NASA facility and who pays attention to NASA these days? This piece is actually a shout out to Goddard. He operated with a shoe string budget his entire life. He literally invented Rocket Science and yet he was pretty much unsung during his fruitful years and is pretty much unknown now. So who was Goddard and what did he do?
He was born in 1882 and got most of his academic training at a small Massachusetts school called Clark College. He did his research between about 1915 and about 1940. From the beginning until about 1930 he mostly worked at Clark where was a professor. For the first decade he funded his research solely out of his modest salary. In the '20s he got some foundation money and continued his research, still operating out of Clark. In the '30s he got a bigger chunk of money from the Guggenheim foundation. This allowed him to move to Roswell New Mexico where he could conduct bigger and messier experiments without blowing up his neighbors.. His New Mexico work continued until 1941. He died in 1945.
So what did he come up with? The answer is pretty much everything. As I indicated above, before Goddard rockets were pretty much fireworks. The military experimented with them but what they used was pretty much skyrockets on steroids. The design, construction, and effectiveness was skyrocket-quality. Goddard changed all this. His first inquiry was into the rocket part.
Rockets are your classical Newtonian device. Newton said "for every action there is an equal and opposite reaction". You make a rocket go by throwing things out the back. That's the "action". The result is that the rocket speeds away in the opposite direction. That's the "reaction". And the "equal and opposite part" tells you exactly how much reaction you will get for a given amount of action. Throw it twice as fast and the rocket speeds up twice as much. Throw something that is twice as heavy and the rocket speeds up twice as much. It's really as simple as that.
So Goddard first focused on throwing things out the back faster. He designed a nozzle to put on the rocket that meant the gasses flying out the back ended up going a lot faster. This seems obvious but Goddard was the first person to actually take a hard look at what was going on and try to improve things. Another now obvious question Goddard was the first to investigate was "do rockets work in a vacuum?" They do but no one had even thought to ask the question. Goddard asked the question and designed and executed experiments to provide the answer. Goddard then designed and built the first working Ion Drive. More about that below. During this period Goddard was able to procure a massive (just kidding) grant totaling $5,000. Even in those days this was not a lot of money.
Goddard went on to investigate other aspects of Rocket Science. One was getting rockets to go where you wanted them to. He figured out that if he used a design that put most of the weight toward the bottom the rocket would naturally want to fly straight up. He also investigated fins and gyrocompass systems for accurately guiding the rocket in the path you wanted it to follow.
There's another trick to throwing the fuel out the back faster. That's what fuel you use. Goddard investigated various classes (solid, liquid) and types of fuels (chemical composition). When you burn a fuel it generates a specific amount of heat per pound. Chemists know exactly how much heat burning any specific combination of chemicals will generate. This leads to a shorthand trick for figuring out what makes the best fuel. The hotter it burns the better.
If this combination burns hotter than that combination and you can figure out how to keep the rocket motor from burning up then the hotter burning fuel can be made to go out the back faster. It gives you more thrust per pound. Goddard investigated a bunch of fuel combinations and ended up with recommendations that are still used to this day. Hydrogen and Oxygen are one of the best combinations. Kerosene and Oxygen work almost as well. Black powder, the classic rocket fuel, is actually pretty terrible. If you want to go with solid fuel then there are lots of combinations that work much better than black powder but none of them work quite as well as the best liquid combinations.
A rocket essentially consists of two things: the fuel and everything else. It is obviously a good idea to keep the "everything else" part to a minimum. All of it has to be hauled around and none of it makes the rocket go faster. In fact, the more the "everything else" weighs the slower the rocket goes. So Goddard investigated ideas in this area too. And he came up with the multistage rocket. If you have one device then it has to have big engines (heavy) and a big structure to hold all that fuel (more weight). But what if you have a big stage on the bottom? This would have big engines and big fuel tanks. But let's say you stack a much smaller stage on top of it? This can be much lighter because it can have smaller engines and fuel tanks. And so on.
Goddard came up with some stage designs, built them, and tested them. He found he could improve overall efficiency by a few percent. He even figured out where the point of diminishing returns was. If you have a dozen stages then you end up with so much weight invested in all the various engines and fuel tanks then you are worse off than when you started. Depending on the details, 2-4 stages is optimum.
Goddard also did the calculations for how well the much improved rockets he was building could work. He was the first one to say that a rocket could be used to get to space and to have the science with which to back this statement up. For his trouble the New York Times castigated him as some kind of kook in an editorial. And so it went. People with no technical background or who failed to investigate his work carefully wrote him off as a crackpot. This made it hard for him to be taken seriously. It also made it particularly hard for him to raise money. During his best times he was operating on a $100,000 Guggenheim grant. But that had to last 4 years and cover all of his costs. He never was accorded much respect during his lifetime. Well, not in the US.
There was one group of people who respected his work tremendously. That was Werner von Braun and the group of Nazi rocket scientists he assembled. They extended and enhanced his work to develop the V-2 rocket. The V-2 was the first rocket to be used successful as a military weapon. It wrecked a lot of devastation on England and its design became the foundation of all subsequent rocket programs around the world. In the immediate aftermath of World War II there was a rush by both the US and the USSR (as Russia was called at the time) to scoop up what they could of the Nazi rocket program. Roughly speaking the USSR got the bulk of the equipment and the US got the bulk of the people. The V-2 finally convinced everyone that rockets could be for real.
Now back to fiction. Science Fiction quickly abandoned cannons and went with rockets at about the time Campbell started to become influential. I don't know if it was his influence or for some other reason. At that time the specifications of these early fictional rockets were literally fantastic. This can best be illustrated by considering a couple of famous movie serials that were hugely popular in the '30s. One featured Buck Rogers and the other featured Flash Gordon.
They both used the same crude special effects that were the best available at the time. The result was rockets that behaved more like airplanes. They took off and landed on their bellies and flew all over the place without refueling. And on one was killed by the rocket exhaust. When it came to rocket exhaust, pretty much the same was true of print fiction of the period. But all of this can be excused because, with the exception of Goddard's work, actual scientists didn't know what the rules were either.
The V-2 changed all that. After the War fictional rockets started taking off and landing on their tails. Generally some effort was made to safely handle the rocket exhaust. Rockets still had infinite quantities of fuel. But guns in the fiction of the time also had infinite quantities of bullets so it is hard to get mad at the people creating rocket shows. The shape of the rocket also went from being non-specific to being V-2-like.
The V-2 shape was based on the shape of high speed bullets. The V-2 had a sharp point. The tail of the V-2 nipped in unlike a bullet but that was a result of the design of the rocket engine. And fictional rockets of the period emulated that design feature too. The V-2 had control fins at the bottom so fictional rockets also had them. In short, the fictional world of the rocket quickly adapted to more closely match reality now that there was a real world model to copy. Interestingly enough, the Hugo Award, one of the major science fiction awards to this day still closely matches the shape of a slightly sexed up V-2.
Returning to the real world, science marched on fairly quickly after the War using the V-2 design as the foundation. Goddard's work was quickly resurrected and celebrated in this period but by this time he was dead. An obvious thought had occurred to pretty much everyone. If a V-2 can be used to carry an explosive charge from Europe to London can a super-V-2 be used to carry an Atomic bomb from one side of the world to the other? The answer looked like it was at least "possibly". That was close enough to "yes" to spur the military in several countries to immediately become very interested in rocket design. And that meant that the lean Goddard days were over when it came to funding for rocket research.
And it turned out that bomb design influenced rocket design in an unexpected way. The US invented the atomic bomb and was quickly able to move on to small, light, designs. The USSR was not as able to come up with small light designs nearly so quickly. So they decided that it was important to develop big rockets that could handle their heavier bombs. So they were able to build big rockets much sooner than the US. And this resulted in the Russians being the first to launch the first artificial satellite and to put the first man in space. They were able to use this early "throw weight" advantage to rack up many firsts early in the space race. One of the ideas that went into Kennedy's "go to the moon" speech was the knowledge that it would take several years for the US to match the Russians in heavy rockets. The moon shot was picked because it was ten years out. And ten years was long enough for the US to catch up in throw weight.
The period from about 1955 to about 1975 was the golden age for both factual rocketry and fictional rocketry. Both were making rapid advances. We went from being barely being able to put a beach ball sized satellite into low earth orbit (about 100-250 miles up) to putting a man on the moon (250,000 miles away). And there were enough unknowns that the fiction writers had scope enough to write lots of fun and exciting stories that didn't stray too far from the science. And, most obviously, by now there actually was such a thing as Rocket Science. So both did very well. But then a strange thing happened. It all kind of petered out. Why?
My theory is that Goddard actually did too thorough a job. He was too cash constrained to take rockets to their logical limits. But he did an amazing job of mapping out where those logical limits were. Nobody has built a 12 stage rocket. Nobody has come up with an amazing fuel that works much better than any of the ones Goddard investigated. It turns out that there are rules for how good you can make a rocket. One of them is what I call the 90-9-1 rule.
I'm now going to subdivide "other stuff" into "structure" (the boring stuff - rocket engines, tanks, stuff like that) and "payload" (the interesting stuff - people, scientific instruments, stuff like that). A modern rocket consists of 90% fuel, 9% structure, and 1% payload. You can fudge numbers a little with improved designs and materials but you can't fudge them by much.
The Saturn 5 that took men to the moon in 1970 pretty much followed this rule. The Space Shuttle that followed it and was the workhorse of the US space program for several decades pretty much followed this rule. We now have the latest and greatest "privately funded" rocket designs done by people like those at Elon Musk's Space-X. But as far as I can tell the Space-X designs (and those of their competitors) hew very closely to this 90-9-1 rule. If you think NASA is a bureaucratic mess then Musk and his ilk can drive some of the cost out of the system by running a lean and mean operation. But it still looks like if one of these new companies wants to put 1 pound of something into low earth orbit they are going to need to buy about 90 pounds of fuel.
You are going to also have some other expenses (those have probably gone up) but fuel costs haven't changed a lot since the Apollo program. The Falcon 9 (Musk's rocket) uses pretty much the same fuel as the Saturn 5 did. And fuel costs pretty much the same per pound (adjusting for inflation) as it did back then. So the fundamentals haven't changed much in the last 40 or so years. And that means getting to space will always be very expensive.
This stagnation over the past 40 years has been bad for space exploration, or at least manned (or womanned - they are actually a better choice because they tend to be lighter) space exploration. It has also been bad for Science Fiction. Hewing to a stagnant science resulted in telling the same stories over and over. Science Fiction has reacted to this problem by splitting and going down two paths. Down one path you have the traditional space epic. In terms of popularity they are now more popular than ever in the movies and on TV. These tend to have a superficial scientific gloss. But if you look at the propulsion systems, for instance, it's all hand-wavium. You have people briefly waving their hands around and using scientific sounding terms like "warp drive" and "dilithium" and then quickly pivoting away to the exciting stuff.
Fictional space ships are either powered by rockets of unspecified design or use some entirely different method of propulsion. And in many cases the story is standard "wagon train to the stars" where the Science Fiction elements are there mostly to spice things up. This approach has been very popular in visual mediums but has been mostly abandoned in print.
The other trend is to abandon science entirely and go back to fantasy. I read a lot of modern fantasy. In terms of quality it is head and shoulders above the likes of Burroughs' John Carter. Some fantasy is recognizable as Science Fiction but the most fantasy is now vaguely medieval "Swords and Sorcery" stuff. The most obvious example of this is the "Game of Thrones" (TV show title - the books go by "A Song of Ice and Fire") series by George R. R. Martin. There are literally dozens of other examples. The Martin series is the most popular at the moment but a number of the others have large loyal followings.
So why has Science Fiction abandoned science based rocket design for space travel? In my opinion it is because none of the scientific fundamentals of rocket design have changed in several decades. Well, you might ask, what about going with future stuff or fictional stuff, things like the Microwave Rocket I mentioned at the top of this piece? The problem the Microwave Rocket is supposed to solve is the 90-9-1 problem. And, if you could get it to work, it would solve the problem, at least in terms of getting something into low earth orbit. But nobody has figured out how to make it work in the last 90 years. And I don't think anybody will get it to work any time soon.
Let me give you another example of a way to avoid the 90-9-1 problem (again just for trips to low earth orbit). It is called the Space Elevator. The idea was popularized by a book called "The Fountains of Paradise". It was written by Arthur C. Clark, a noted Science Fiction author, and published in 1978. Everybody loves the idea but no one has figured out how to build it. People have dreamed up a lot of ideas in the last 40 years, or the last 90 years, if you count from the introduction of the idea of the Microwave Rocket. Its just that nobody has been able to get any of them to work. So is there nothing out there?
Well, there is the Ion drive. The benefit of the Ion drive is that it throws things out the back at extremely high speed. The speed is way higher than the fastest possible speed a rocket design based on chemical reactions can achieve. This means you get a whole lot more push per pound of fuel. And a number of Ion drive rocket motors have been built and successfully put into operation. But frankly we only know how to make small low power Ion drives. And they only work in the vacuum of space. So they only look like a fit for a few selected deep space missions. Anything else? Surely Science Fiction writers or somebody has come up with something else. Funny you should ask?
There is a web site for that. Here's the link: http://www.projectrho.com/public_html/rocket/index.php. It was created by a guy who does illustrations for fantasy works, a guy named Winchell Chung. He set out to do the math (or find someone who had already done the math) on pretty much every aspect of rocket ships. So the results from all the calculations behind standard "real world" stuff are posted here in user friendly form (frequently as an easy to use graph). But then there are also results posted concerning pretty crazy but maybe possible ideas like a rocket powered by atomic bomb explosions. But he goes even farther. He has analyzed how well the various types of fantastical space drives would work. He has even explored ancillary topics like ray guns, space suits, lasers, phasers, you name it.
This site has now been around for some time and it hasn't changed much in recent years. Why? Because people have not been able to come up with any really new ideas in a while. And Chung's site tells you how well various real, slightly fictional, and totally way out there technologies would actually work. Unfortunately, mostly the answer is not nearly as well as they do in the books and movies. And that's after he assumes that what some author has made up actually works generally the way the author says it would. Usually it wouldn't and the web site tells you why.
This doesn't stop me from enjoying myself anyhow when it comes to my entertainment habits. I'm pretty good at ignoring reality if its entertaining enough. I hope this post doesn't get in the way of you enjoying the best in contemporary entertainment either. There are times when a willing suspension of disbelief is the right and true thing to do.
Sunday, November 22, 2015
Tuesday, November 17, 2015
McCarthyism
I frequently despair over the current state of our politics and our public discourse. Pretty much everyone else does too. And there is a consensus on this across the political divide. Conservatives decry the "liberal media" as doing a bad job. They want them to be more like the conservative media. People outside the world of conservatism decry the same bunch. Only they call them the "mainstream media" and they want them to go after conservatives and the conservative media for their lack of fidelity to and interest in the truth.
I have a dog in this fight. But instead of going straight at it I want to approach things from a different angle. Were things as bad then as they are now? With this ultimate objective in mind I am going to dive into the McCarthy era and see what things looked like then. To do that I need a short, concise example of thinking from that bygone time. I found it in the most unlikely of places.
I read a lot. And a some of what I read, or in this case reread, is the fiction of my youth. That and the fiction my parents read and filed in the bookshelf in the hall. One example is the "Shell Scott" books by Richard S. Prather. Even in the less exalted halls of '50s detective pulp fiction, Shell was definitely considered Junior Varsity. When I read one a few years ago I still found it to be JV level work. But I also found it quite entertaining. So I set out to read a bunch more.
Shell was your standard issue two fisted hunk who beat people up and got beat up in return. And did I mention he screwed a lot of women. And, like many of his contemporaries, he was actually a model of gentlemanly behavior when it came to the ladies. It's just that the babes kept throwing themselves at him. They were all voluptuous and, in the case of the Scott books, frequently made it known to our boy that they were not wearing a brassiere. And this was in spite of the fact that these women's proportions proudly advertised the need for substantial support.
So anyhow, here I am peacefully plowing through what is supposed to be '50s escapist pulp fiction when I come across this tirade. It was unexpected to say the least. Mostly this type of fiction restricts tirades to the subject of the bad guy kicking the crap out of the good guy or vice versa. This tirade was actually on a political subject. Generally speaking these books avoid politics. Why jeopardize a part of your customer base unnecessarily?
Our boy Shell is supposed to be chasing, or being chased by babes, beating people up and getting beaten up, drinking far beyond wretched excess while suffering little or no consequence, and generally providing me with a good time. But in this book he stops to provide a lecture in the form of a "debate". The only thing I can think of is that the author thought that the words he was put into Shell's mouth were so non-controversial that it was unlikely that they would turn off a significant portion of his readership. As such, I think it is fair for me to take them as representative of mainstream thought of the time about the subject at hand. So I will.
The book in question is "Pattern for Panic" published in 1954. It was originally rejected by the publisher so Prather changed the name of the lead character to something else and got it published by someone else. But the Shell Scott books continued to gain in popularity. So Prather re-edited his manuscript to change the name of the lead character back to Shell Scott and republished it in 1961. The version I read was the 1961 version. As far as I can tell, other than a quick edit to change the name of the lead character back and forth, the books are the same.
As I indicated, the book contains a "debate" between a minor character and our boy Shell. The books are first person narrations so we only get Shell's perspective. So let's start by looking at Shell's description of his debating opponent. He is described as:
He is also described as "something of a fathead". He is, in fact, a microbiologist. Egghead attempts to make an argument for peaceful coexistence with communists in general and the USSR (the then name for what is now roughly Russia) in particular. Scott belittles his arguments and interrupts a lot. Here's Scott's argument:
And a little later in a response to Egghead he says:
He completes his thesis with the following statement:
Fidel Castro is still well known as the dictator of Cuba until a few years ago. Cuba is now run by his brother Raoul. Fidel, either directly or indirectly through his brother, is still running Cuba 60 years later. It may be that he has maintained a first class propaganda machine all that time. But he was and still is liked, trusted, and respected by the bulk of Cuban people. Cuba is a poor country. But it has little abject poverty and probably the best medical system in the third world. The rest of the players are no longer with us.
Mao Zedong (as it is now spelled) led an uprising against Chiang Kai-shek, the leader (his title changed regularly) of the government of China. The communist forces eventually routed the Chiang forces on the mainland and Chiang retreated to Formosa, also known as Taiwan, in 1950. At that point the communists took complete control of mainland China. They maintain that control to this day although many argue that their economic system can no longer accurately be described as communist.
Syngman Rhee was the President of South Korea from 1948 to 1960. South Korea was created when Korea was partitioned in to North and South areas at the end of World War II. He was described as a strongman and was only ousted from power after an election dispute.
Rafael Trujillo was President of the Dominican Republic from 1930 until 1961. His nickname was El Jefe, a phrase popularized by movies and TV shows. El Jefe is inevitably a corrupt, sneering, Hispanic bad guy. Trujillo ruled until he was assassinated. And that leaves us with McCarthy.
Joseph McCarthy was a Republican Senator from Minnesota. He took office on January 3, 1947 and served until his death on May 2, 1957. He ended up lending his name to an era, "The McCarthy Era" and to a movement, "McCarthyism". So what's the story?
The Democrats became immensely popular under President Franklin Delano Roosevelt. Most people gave him credit for fighting the Great Depression. Then he was Commander in Chief during World War II, an eventually very popular war. At the end of the war the Republicans were at a loss as to what to do. Then the whole Iron Curtain thing happened. A (very) little background:
During World War I the Germans smuggled some prominent communists, notably Trotsky and Lenin, into Russia with the idea that they would be a disruptive force and weaken one of the countries fighting against the Germans. The plan worked far better than it was supposed to. The communists not only caused disruption. They took over the government in a coup and took Russia out of the war. So far so good.
The problem was that they stuck around and kept running the country in spite of substantial allied efforts to prop up the "White Russians" in the '20s. The German government was taken over by the Nazis in the early '30s and Russia, now known officially as the USSR, came in against the Germans in World War II. That was not so good for the Germans. That was all well and good in the US while the War was still going on, however. But it quickly became a problem starting in the immediate postwar period. And by this time Stalin had been running things in the USSR for some time.
Stalin decided he needed a buffer to prevent the Germans from invading a third time. (The Germans attacked Russia in both World War I and World War II.) So he forcefully took control of Eastern Europe. He used, shall we say, non-democratic means to do this. The Republicans seized on this as an opportunity. They started loudly asking "who lost Eastern Europe?" And, of course, the obvious answer, as far as they were concerned, was "the Democrats". And this led to the obvious follow on question: How was it done? The Republican answer was "subversion from the inside done by the communists but aided and abetted by the Democrats". Why? Because "they are soft on communism". And, of course, "the evidence is all around us". The principal spokesman for this line was Senator McCarthy.
He launched numerous "investigations" in the expectation of finding communists everywhere. Why do I use scare quotes around the work "investigations"? Because McCarthy and his staff did little or no investigating. The FBI and some other agencies actually did some investigating and they actually turned up some spies and sympathizers. They decided it was to their political advantage to reveal these results through McCarthy so they did and McCarthy got the public credit. But what little actual investigation was done was done poorly.
Stalin was running a large spy operation in the US at the time. The most obvious success this spying operation chalked up was the theft of both of the Atomic Bomb designs that the U.S. developed during the War. They were the "implosion" and the "gun" designs. The gun design was subsequently abandoned but the implosion design is the basis of all subsequent A-bomb and H-bomb designs developed since.
There were several spies at Los Alamos. The most important one was Claus Fuchs. He came to the program as part of the British contingent. So he was vouched for by the Brits. But it turns out that British intelligence was riddled with Soviet spies. A group called the Cambridge 5 had been successfully recruited by Soviet Intelligence in the '30s. They became long term assets that were able to continue operating into the '60s. With their fingers everywhere in British Intelligence the Russians were able to use the British pipeline to get their people into all kinds of places both in the UK and in the US.
After the theft of the bomb designs was completed Soviet Intelligence later had a second less well known but equally important success. The CIA attempted to infiltrate agents into Eastern Europe starting in the late '40s. All of these efforts, 100%, were failures. It was obvious that there was a leak somewhere. A man by the name of James Jesus Angleton was put in charge of finding the leak. He had "all access" within the CIA. And it was important that no one be able to track his activities as this might allow the bad guys to counter them. So Angleton had access to pretty much anyone and anything in the CIA but no records of his activities were kept. So if you tried to track down who knew about a certain blown program Angleton's name would not show up even if he actually knew all about it.
And it turned out that Angleton was the source of the leak. Was he a spy? No! He just chose his friends poorly. He was friends with Kim Philby, the most highly placed of the Cambridge 5 Russian spies. Angleton trusted Philby. So when Philby came from London to Washington DC, which he did frequently, the two would get together. They would then have off the record discussions of what the CIA in general and Angleton in particular was up to. Philby was able to suss out information on all the CIA infiltration efforts and pass enough information about them back to Moscow for the Russians to thwart them all.
None of this, the Cambridge 5, the Angleton to Philby hemorrhaging of operational information, nothing was turned up by McCarthy or his operatives. Actual spies were all uncovered by investigations happening elsewhere. And little enough of that happened either. A lot of this was only uncovered after McCarthyism died down and ceased to be a distraction. McCarthy's ineffectiveness was well known to insiders. But this secret was carefully kept from the public by those who benefitted politically from his antics. He was only brought down when ABC decided to broadcast the Army-McCarthy hearings on TV. There the public got to see for themselves how he operated instead of having to rely on the highly sanitized media version of his activities.
Angleton was well liked by the CIA top brass. So in spite of his spectacular blunder and the incredible amount of damage he did he was kept on at the CIA in a senior position until 1975. This was in spite of the fact that Philby had defected in 1963. So if Angleton, who richly deserved it, wasn't dumped, who was?
During World War II it was a US priority to keep Russia in the war and there was good reason for this. Russia suffered tremendous losses and inflicted tremendous losses on the Nazis. As an example the largest tank battle in history was fought between the Nazis and the Russians. This "largest" is also true if you measure the number of soldiers involved, the number of airplanes involved, the number of casualties involved, the amount of land involved. Clashes on the "Eastern Front" were either the largest or one of the largest clashes of the War as measured by any of these criteria. So keeping Russia in the war was sound strategic thinking. And who manned the front lines of diplomatic efforts to achieve this critical strategic objective? State Department officials. After the war they became targets of opportunity for cheap shots, all as a result of them doing their jobs.
There was another group of people who became targets of opportunity. The communist takeover of Russia in roughly 1920 was the first example of communist thinking put into action. Up until then it was just a theoretical concept. As such it had its good points and its bad points. And no one knew whether it was practical or not. So in the '20s communism became fashionable in some circles. In these early days a lot of things happening in the USSR could be dismissed as "teething problems" or working out the kinks. But particularly after Stalin came to power excusing bad behavior became more and more difficult. First, industry was nationalized. This could be excused as being only fair because the rich and powerful (the owners), often abused their powers.
Then Stalin collectivized agriculture. Many small family farms were consolidated into a few large collective farms. This too might have been a good idea. But it quickly became obvious that the whole endeavor was a giant disaster that benefitted no one and definitely did not benefit the former family farmers. Then Stalin started the purges. He killed and imprisoned large numbers of people. It quickly became apparent that the only goal of this was to consolidate power into the hands of Stalin and his close associates. This shed a new light on the collectivization of agriculture. It too could now be seen as part of a plan to consolidate power into the hands of Stalin and his close associates. The purges were so destructive that the Soviet military was unable to operate effectively early in the War because so many senior officers had been purged.
So we have an early period where communism is legitimate. Then we have the time when communism loses its legitimacy under the leadership of Stalin. Then, for pragmatic reasons, the USSR (and by extension Stalin and communism) is rehabilitated so it can contribute to the War effort. A lot of people were swept up in one or the other of these honeymoon periods. In many cases well before the late '40s start of the McCarthy era, and definitely by the late '40s, most of those who had gotten involved with communism at one point or another had rightly abandoned it. But the cry became "are you or have you ever been" a communist or a communist sympathizer. If the answer is yes then you should be hounded out of your job and your place in polite society. Any subtlety was lost on McCarthy and his sympathizers. You can see this in the statements I have quoted above. They are the oft voiced sentiments of the pro-McCarthy side. Let's go back and take another look at them.
In the second quotation we are given a list of "the press, television, and radio, publishing, movies, schools, government agencies" where communist stooges are supposed to running things and brainwashing the masses. Yet many of these same institutions, and specifically "the press, television, radio, publishing, the movies" are all part of large companies run by people who are strongly anticommunist. And if you look at what was actually printed, broadcasted, etc., it is loaded with material that is similar to what I have reproduced above. It is loaded with "don't trust those dirty commies and their dirty propaganda" messages rather than "trust what those fine communist gentlemen are saying" messages. As for the schools, they were run by local groups and those groups by and large followed the anticommunist line. That left government agencies.
The State Department and many of its staff was guilty of saying nice things about the USSR. And many of those nice things were wrong. But those people engaged in misleading the American public were doing their jobs. And their job was "do what it takes to keep the USSR in the war". And unfortunately "what it takes" frequently involved looking the other way when Russian bad behavior was involved and saying nice things about the USSR that were not justified. But diplomats are employed and paid to do what they are told whether it comports with their beliefs (or even the truth) or not. It makes their jobs incredibly difficult in the best of times. So the whole second statement was B.S.
Let's look at the third statement. The first thing the third statement does is accuse the media of playing up Russian successes and playing down American failures. Yet that is exactly what the pro-McCarthy statement does. It plays up Russian successes and plays down American failures. Why? Because the McCarthyites needed a powerful enemy to justify the drastic measures they wanted to enact. If the bad guys are wimps then you don't have to work very hard to beat them. You only need maximal effort if the bad guys are formidable. So the McCarthyites accuse the other side of the very bad behavior they themselves are engaged in. But wait, it gets worse.
Look at the last quotation. We are told that Chiang Kai-shek has been falsely accused of being a 'corrupt dictator' as have Syngman Rhee and Rafael Trujillo. The noted historian Barbara Tuchman has written a wonderful biography of General Joseph Stillwell. Stilwell had extensive dealings with Chiang. He concluded that Chiang and his entire administration was deeply corrupt. The matter is no longer in dispute. You can find plenty of other sources that come to the same conclusion and provide abundant documentation to support their conclusion.
I alluded to Trujillo's nickname. I note that it took decades of effort to turn Taiwan (Chiang), South Korea (Rhee) and the Dominican Republic (Trujillo) into democracies after these three characters were removed from power, typically by death. So all three actually were corrupt dictators. And now that the McCarthy era is long over that characterization is no longer in dispute. So in case of all four individuals cited above as being unfairly characterized as 'corrupt dictators' they actually were fairly characterized as corrupt dictators. So we see McCarthyites being apologists for corrupt dictators while loudly protesting that these same individuals are not corrupt dictators.
And no one referred to Mao as merely an "agrarian reformer". Instead he was seen as either the head of the Chinese government or a powerful player in the Chinese government. His power was diminished by the failure and chaos that marked the Cultural Revolution. But that happened long after the time period we are examining. In spite of the damage the Cultural Revolution did to his reputation and power base he was still considered a powerful figure right up until his death in 1976.
So McCarthyites are wrong on who is not a corrupt dictator. They are wrong on how people characterized Mao's position in China. The only place they come close is where they claim it is inaccurate to characterize Fidel Castro as a "Cuban Robin Hood". But many Cubans see him just that way. So the most you can do is give the McCarthyites credit for being possibly right in this one instance. That's a pretty poor track record.
Finally, let me circle back to the first quotation. Here Scott assassinates the character of his opponent then opines that he is "shooting off at the mouth about subjects he knew nothing about". But, what are Scott's credentials? He is a pulp private detective. But let's put that aside and enter his fictional world. There he is an L.A. private detective who does a lot of work in Hollywood. He is not known for his intellectual prowess nor for his careful research into the issues of the day. Instead he is a guns, fists, and babes, kind of guy. It is possible but unlikely and out of character that Shell has studied up on the subjects of communism and McCarthyism. But we are told by none other than Shell himself that Egghead has also studied up on the subject.
So, at worst, it is studious amateur versus studious amateur. In other words, their credentials are about equal, and that's assuming Scott has done something out of character and studied up on the subject himself. But it is more likely that Shell has not studied up. Instead he has absorbed the McCarthyite ranting that populated much of the, dare I say it, sensational press of the day. I have demonstrated that Scott is singularly uninformed on the subject he claims expertise on. And that's exactly what you expect from the sensational press. So it is more likely that he is the one who is soaked up a bunch of propaganda and is "shooting off at the mouth about subjects he knew nothing about".
In defense of the author, he needs a good conflict to drive the action. To that extent, and only to that extent, his approach is justified. But if he frames his conflict in a way that is at odds with the understanding of a large segment of his reading audience he risks losing a lot of readers. So it behooves him to stick to sentiments he believes are broadly held. I think that's just what the author did.
This book was in the middle of the ascent of the Shell Scott phenomenon. The ascent was not impeded by this book so I judge that Prather's calculation was correct. He in fact put beliefs in the mouth of his main character that were widely held at the time by a large segment of the general public. That's why I think my selection of this source, odd though it may seem, as an accurate and reliable representation of the arguments advanced at the time by McCarthyites to support their position is justified.
Now back to the present. The McCarthy era is widely seen in retrospect as a terrible time for the state of public discourse. For those with less familiarity with the period I hope I have given you an understanding of why that is so. The best I can conclude with respect to my initial thesis is "it's bad now but it has been roughly as bad in the past". That's not good. But it's the best I have. And, on the bright side, things eventually did get better after the McCarthy era peaked and subsided. So there is hope for a brighter future now.
I have a dog in this fight. But instead of going straight at it I want to approach things from a different angle. Were things as bad then as they are now? With this ultimate objective in mind I am going to dive into the McCarthy era and see what things looked like then. To do that I need a short, concise example of thinking from that bygone time. I found it in the most unlikely of places.
I read a lot. And a some of what I read, or in this case reread, is the fiction of my youth. That and the fiction my parents read and filed in the bookshelf in the hall. One example is the "Shell Scott" books by Richard S. Prather. Even in the less exalted halls of '50s detective pulp fiction, Shell was definitely considered Junior Varsity. When I read one a few years ago I still found it to be JV level work. But I also found it quite entertaining. So I set out to read a bunch more.
Shell was your standard issue two fisted hunk who beat people up and got beat up in return. And did I mention he screwed a lot of women. And, like many of his contemporaries, he was actually a model of gentlemanly behavior when it came to the ladies. It's just that the babes kept throwing themselves at him. They were all voluptuous and, in the case of the Scott books, frequently made it known to our boy that they were not wearing a brassiere. And this was in spite of the fact that these women's proportions proudly advertised the need for substantial support.
So anyhow, here I am peacefully plowing through what is supposed to be '50s escapist pulp fiction when I come across this tirade. It was unexpected to say the least. Mostly this type of fiction restricts tirades to the subject of the bad guy kicking the crap out of the good guy or vice versa. This tirade was actually on a political subject. Generally speaking these books avoid politics. Why jeopardize a part of your customer base unnecessarily?
Our boy Shell is supposed to be chasing, or being chased by babes, beating people up and getting beaten up, drinking far beyond wretched excess while suffering little or no consequence, and generally providing me with a good time. But in this book he stops to provide a lecture in the form of a "debate". The only thing I can think of is that the author thought that the words he was put into Shell's mouth were so non-controversial that it was unlikely that they would turn off a significant portion of his readership. As such, I think it is fair for me to take them as representative of mainstream thought of the time about the subject at hand. So I will.
The book in question is "Pattern for Panic" published in 1954. It was originally rejected by the publisher so Prather changed the name of the lead character to something else and got it published by someone else. But the Shell Scott books continued to gain in popularity. So Prather re-edited his manuscript to change the name of the lead character back to Shell Scott and republished it in 1961. The version I read was the 1961 version. As far as I can tell, other than a quick edit to change the name of the lead character back and forth, the books are the same.
As I indicated, the book contains a "debate" between a minor character and our boy Shell. The books are first person narrations so we only get Shell's perspective. So let's start by looking at Shell's description of his debating opponent. He is described as:
A modern "liberal" [quotation marks in the original] and egghead; and he was always shooting off his mouth on subjects he knew nothing about.
He is also described as "something of a fathead". He is, in fact, a microbiologist. Egghead attempts to make an argument for peaceful coexistence with communists in general and the USSR (the then name for what is now roughly Russia) in particular. Scott belittles his arguments and interrupts a lot. Here's Scott's argument:
We've got a very clever gang of pro-Red salesmen in the States -- in the press, television, and radio, publishing, movies, schools, government agencies -- any place where spoken and written words can be used to shape opinion or policy.
And a little later in a response to Egghead he says:
O.K. But the next time some fat issue involving Russia or Communism comes up, watch the same pink and pro-Red cats start shoving the Russia-Communist angle into the background and concentrating instead on America's errors, or Red hysteria, or McCarthyism -- whatever the current party line happens to be. Watch them play up Russia's successes and America's failures, while at the same time playing down Russia's failures and America's successes. With the natural result that a lot of people gradually start believing the Russian molehill is a mountain, and the American mountain is a molehill.
He completes his thesis with the following statement:
Shaping public opinion, whether it's in international relations or homegrown subversion is like the public relations business. Control enough of the words reaching people and you can make them believe damn near anything. You can make them believe -- falsely -- that Chiang is a 'corrupt dictator' and Mao Tse-tung is simply an agrarian reformer; that Batista is a 'corrupt dictator' and Castro's a Cuban Robin Hood; that Rhee and Trujillo are 'corrupt dictators' but Red dictators are democratic reformers -- hell , isn't that's what happened, what's happening right now?At this point let me go through the players as many may be unfamiliar to modern audiences.
Fidel Castro is still well known as the dictator of Cuba until a few years ago. Cuba is now run by his brother Raoul. Fidel, either directly or indirectly through his brother, is still running Cuba 60 years later. It may be that he has maintained a first class propaganda machine all that time. But he was and still is liked, trusted, and respected by the bulk of Cuban people. Cuba is a poor country. But it has little abject poverty and probably the best medical system in the third world. The rest of the players are no longer with us.
Mao Zedong (as it is now spelled) led an uprising against Chiang Kai-shek, the leader (his title changed regularly) of the government of China. The communist forces eventually routed the Chiang forces on the mainland and Chiang retreated to Formosa, also known as Taiwan, in 1950. At that point the communists took complete control of mainland China. They maintain that control to this day although many argue that their economic system can no longer accurately be described as communist.
Syngman Rhee was the President of South Korea from 1948 to 1960. South Korea was created when Korea was partitioned in to North and South areas at the end of World War II. He was described as a strongman and was only ousted from power after an election dispute.
Rafael Trujillo was President of the Dominican Republic from 1930 until 1961. His nickname was El Jefe, a phrase popularized by movies and TV shows. El Jefe is inevitably a corrupt, sneering, Hispanic bad guy. Trujillo ruled until he was assassinated. And that leaves us with McCarthy.
Joseph McCarthy was a Republican Senator from Minnesota. He took office on January 3, 1947 and served until his death on May 2, 1957. He ended up lending his name to an era, "The McCarthy Era" and to a movement, "McCarthyism". So what's the story?
The Democrats became immensely popular under President Franklin Delano Roosevelt. Most people gave him credit for fighting the Great Depression. Then he was Commander in Chief during World War II, an eventually very popular war. At the end of the war the Republicans were at a loss as to what to do. Then the whole Iron Curtain thing happened. A (very) little background:
During World War I the Germans smuggled some prominent communists, notably Trotsky and Lenin, into Russia with the idea that they would be a disruptive force and weaken one of the countries fighting against the Germans. The plan worked far better than it was supposed to. The communists not only caused disruption. They took over the government in a coup and took Russia out of the war. So far so good.
The problem was that they stuck around and kept running the country in spite of substantial allied efforts to prop up the "White Russians" in the '20s. The German government was taken over by the Nazis in the early '30s and Russia, now known officially as the USSR, came in against the Germans in World War II. That was not so good for the Germans. That was all well and good in the US while the War was still going on, however. But it quickly became a problem starting in the immediate postwar period. And by this time Stalin had been running things in the USSR for some time.
Stalin decided he needed a buffer to prevent the Germans from invading a third time. (The Germans attacked Russia in both World War I and World War II.) So he forcefully took control of Eastern Europe. He used, shall we say, non-democratic means to do this. The Republicans seized on this as an opportunity. They started loudly asking "who lost Eastern Europe?" And, of course, the obvious answer, as far as they were concerned, was "the Democrats". And this led to the obvious follow on question: How was it done? The Republican answer was "subversion from the inside done by the communists but aided and abetted by the Democrats". Why? Because "they are soft on communism". And, of course, "the evidence is all around us". The principal spokesman for this line was Senator McCarthy.
He launched numerous "investigations" in the expectation of finding communists everywhere. Why do I use scare quotes around the work "investigations"? Because McCarthy and his staff did little or no investigating. The FBI and some other agencies actually did some investigating and they actually turned up some spies and sympathizers. They decided it was to their political advantage to reveal these results through McCarthy so they did and McCarthy got the public credit. But what little actual investigation was done was done poorly.
Stalin was running a large spy operation in the US at the time. The most obvious success this spying operation chalked up was the theft of both of the Atomic Bomb designs that the U.S. developed during the War. They were the "implosion" and the "gun" designs. The gun design was subsequently abandoned but the implosion design is the basis of all subsequent A-bomb and H-bomb designs developed since.
There were several spies at Los Alamos. The most important one was Claus Fuchs. He came to the program as part of the British contingent. So he was vouched for by the Brits. But it turns out that British intelligence was riddled with Soviet spies. A group called the Cambridge 5 had been successfully recruited by Soviet Intelligence in the '30s. They became long term assets that were able to continue operating into the '60s. With their fingers everywhere in British Intelligence the Russians were able to use the British pipeline to get their people into all kinds of places both in the UK and in the US.
After the theft of the bomb designs was completed Soviet Intelligence later had a second less well known but equally important success. The CIA attempted to infiltrate agents into Eastern Europe starting in the late '40s. All of these efforts, 100%, were failures. It was obvious that there was a leak somewhere. A man by the name of James Jesus Angleton was put in charge of finding the leak. He had "all access" within the CIA. And it was important that no one be able to track his activities as this might allow the bad guys to counter them. So Angleton had access to pretty much anyone and anything in the CIA but no records of his activities were kept. So if you tried to track down who knew about a certain blown program Angleton's name would not show up even if he actually knew all about it.
And it turned out that Angleton was the source of the leak. Was he a spy? No! He just chose his friends poorly. He was friends with Kim Philby, the most highly placed of the Cambridge 5 Russian spies. Angleton trusted Philby. So when Philby came from London to Washington DC, which he did frequently, the two would get together. They would then have off the record discussions of what the CIA in general and Angleton in particular was up to. Philby was able to suss out information on all the CIA infiltration efforts and pass enough information about them back to Moscow for the Russians to thwart them all.
None of this, the Cambridge 5, the Angleton to Philby hemorrhaging of operational information, nothing was turned up by McCarthy or his operatives. Actual spies were all uncovered by investigations happening elsewhere. And little enough of that happened either. A lot of this was only uncovered after McCarthyism died down and ceased to be a distraction. McCarthy's ineffectiveness was well known to insiders. But this secret was carefully kept from the public by those who benefitted politically from his antics. He was only brought down when ABC decided to broadcast the Army-McCarthy hearings on TV. There the public got to see for themselves how he operated instead of having to rely on the highly sanitized media version of his activities.
Angleton was well liked by the CIA top brass. So in spite of his spectacular blunder and the incredible amount of damage he did he was kept on at the CIA in a senior position until 1975. This was in spite of the fact that Philby had defected in 1963. So if Angleton, who richly deserved it, wasn't dumped, who was?
During World War II it was a US priority to keep Russia in the war and there was good reason for this. Russia suffered tremendous losses and inflicted tremendous losses on the Nazis. As an example the largest tank battle in history was fought between the Nazis and the Russians. This "largest" is also true if you measure the number of soldiers involved, the number of airplanes involved, the number of casualties involved, the amount of land involved. Clashes on the "Eastern Front" were either the largest or one of the largest clashes of the War as measured by any of these criteria. So keeping Russia in the war was sound strategic thinking. And who manned the front lines of diplomatic efforts to achieve this critical strategic objective? State Department officials. After the war they became targets of opportunity for cheap shots, all as a result of them doing their jobs.
There was another group of people who became targets of opportunity. The communist takeover of Russia in roughly 1920 was the first example of communist thinking put into action. Up until then it was just a theoretical concept. As such it had its good points and its bad points. And no one knew whether it was practical or not. So in the '20s communism became fashionable in some circles. In these early days a lot of things happening in the USSR could be dismissed as "teething problems" or working out the kinks. But particularly after Stalin came to power excusing bad behavior became more and more difficult. First, industry was nationalized. This could be excused as being only fair because the rich and powerful (the owners), often abused their powers.
Then Stalin collectivized agriculture. Many small family farms were consolidated into a few large collective farms. This too might have been a good idea. But it quickly became obvious that the whole endeavor was a giant disaster that benefitted no one and definitely did not benefit the former family farmers. Then Stalin started the purges. He killed and imprisoned large numbers of people. It quickly became apparent that the only goal of this was to consolidate power into the hands of Stalin and his close associates. This shed a new light on the collectivization of agriculture. It too could now be seen as part of a plan to consolidate power into the hands of Stalin and his close associates. The purges were so destructive that the Soviet military was unable to operate effectively early in the War because so many senior officers had been purged.
So we have an early period where communism is legitimate. Then we have the time when communism loses its legitimacy under the leadership of Stalin. Then, for pragmatic reasons, the USSR (and by extension Stalin and communism) is rehabilitated so it can contribute to the War effort. A lot of people were swept up in one or the other of these honeymoon periods. In many cases well before the late '40s start of the McCarthy era, and definitely by the late '40s, most of those who had gotten involved with communism at one point or another had rightly abandoned it. But the cry became "are you or have you ever been" a communist or a communist sympathizer. If the answer is yes then you should be hounded out of your job and your place in polite society. Any subtlety was lost on McCarthy and his sympathizers. You can see this in the statements I have quoted above. They are the oft voiced sentiments of the pro-McCarthy side. Let's go back and take another look at them.
In the second quotation we are given a list of "the press, television, and radio, publishing, movies, schools, government agencies" where communist stooges are supposed to running things and brainwashing the masses. Yet many of these same institutions, and specifically "the press, television, radio, publishing, the movies" are all part of large companies run by people who are strongly anticommunist. And if you look at what was actually printed, broadcasted, etc., it is loaded with material that is similar to what I have reproduced above. It is loaded with "don't trust those dirty commies and their dirty propaganda" messages rather than "trust what those fine communist gentlemen are saying" messages. As for the schools, they were run by local groups and those groups by and large followed the anticommunist line. That left government agencies.
The State Department and many of its staff was guilty of saying nice things about the USSR. And many of those nice things were wrong. But those people engaged in misleading the American public were doing their jobs. And their job was "do what it takes to keep the USSR in the war". And unfortunately "what it takes" frequently involved looking the other way when Russian bad behavior was involved and saying nice things about the USSR that were not justified. But diplomats are employed and paid to do what they are told whether it comports with their beliefs (or even the truth) or not. It makes their jobs incredibly difficult in the best of times. So the whole second statement was B.S.
Let's look at the third statement. The first thing the third statement does is accuse the media of playing up Russian successes and playing down American failures. Yet that is exactly what the pro-McCarthy statement does. It plays up Russian successes and plays down American failures. Why? Because the McCarthyites needed a powerful enemy to justify the drastic measures they wanted to enact. If the bad guys are wimps then you don't have to work very hard to beat them. You only need maximal effort if the bad guys are formidable. So the McCarthyites accuse the other side of the very bad behavior they themselves are engaged in. But wait, it gets worse.
Look at the last quotation. We are told that Chiang Kai-shek has been falsely accused of being a 'corrupt dictator' as have Syngman Rhee and Rafael Trujillo. The noted historian Barbara Tuchman has written a wonderful biography of General Joseph Stillwell. Stilwell had extensive dealings with Chiang. He concluded that Chiang and his entire administration was deeply corrupt. The matter is no longer in dispute. You can find plenty of other sources that come to the same conclusion and provide abundant documentation to support their conclusion.
I alluded to Trujillo's nickname. I note that it took decades of effort to turn Taiwan (Chiang), South Korea (Rhee) and the Dominican Republic (Trujillo) into democracies after these three characters were removed from power, typically by death. So all three actually were corrupt dictators. And now that the McCarthy era is long over that characterization is no longer in dispute. So in case of all four individuals cited above as being unfairly characterized as 'corrupt dictators' they actually were fairly characterized as corrupt dictators. So we see McCarthyites being apologists for corrupt dictators while loudly protesting that these same individuals are not corrupt dictators.
And no one referred to Mao as merely an "agrarian reformer". Instead he was seen as either the head of the Chinese government or a powerful player in the Chinese government. His power was diminished by the failure and chaos that marked the Cultural Revolution. But that happened long after the time period we are examining. In spite of the damage the Cultural Revolution did to his reputation and power base he was still considered a powerful figure right up until his death in 1976.
So McCarthyites are wrong on who is not a corrupt dictator. They are wrong on how people characterized Mao's position in China. The only place they come close is where they claim it is inaccurate to characterize Fidel Castro as a "Cuban Robin Hood". But many Cubans see him just that way. So the most you can do is give the McCarthyites credit for being possibly right in this one instance. That's a pretty poor track record.
Finally, let me circle back to the first quotation. Here Scott assassinates the character of his opponent then opines that he is "shooting off at the mouth about subjects he knew nothing about". But, what are Scott's credentials? He is a pulp private detective. But let's put that aside and enter his fictional world. There he is an L.A. private detective who does a lot of work in Hollywood. He is not known for his intellectual prowess nor for his careful research into the issues of the day. Instead he is a guns, fists, and babes, kind of guy. It is possible but unlikely and out of character that Shell has studied up on the subjects of communism and McCarthyism. But we are told by none other than Shell himself that Egghead has also studied up on the subject.
So, at worst, it is studious amateur versus studious amateur. In other words, their credentials are about equal, and that's assuming Scott has done something out of character and studied up on the subject himself. But it is more likely that Shell has not studied up. Instead he has absorbed the McCarthyite ranting that populated much of the, dare I say it, sensational press of the day. I have demonstrated that Scott is singularly uninformed on the subject he claims expertise on. And that's exactly what you expect from the sensational press. So it is more likely that he is the one who is soaked up a bunch of propaganda and is "shooting off at the mouth about subjects he knew nothing about".
In defense of the author, he needs a good conflict to drive the action. To that extent, and only to that extent, his approach is justified. But if he frames his conflict in a way that is at odds with the understanding of a large segment of his reading audience he risks losing a lot of readers. So it behooves him to stick to sentiments he believes are broadly held. I think that's just what the author did.
This book was in the middle of the ascent of the Shell Scott phenomenon. The ascent was not impeded by this book so I judge that Prather's calculation was correct. He in fact put beliefs in the mouth of his main character that were widely held at the time by a large segment of the general public. That's why I think my selection of this source, odd though it may seem, as an accurate and reliable representation of the arguments advanced at the time by McCarthyites to support their position is justified.
Now back to the present. The McCarthy era is widely seen in retrospect as a terrible time for the state of public discourse. For those with less familiarity with the period I hope I have given you an understanding of why that is so. The best I can conclude with respect to my initial thesis is "it's bad now but it has been roughly as bad in the past". That's not good. But it's the best I have. And, on the bright side, things eventually did get better after the McCarthy era peaked and subsided. So there is hope for a brighter future now.
Wednesday, October 21, 2015
Steve Jobs' Mac Sucked
The new movie "Steve Jobs" opened in limited release a few days ago. As I write this it is set to open wide in a couple of days. I haven't seen it yet but I plan to. As I understand it the movie is a fictional account of the events preceding three product launches. Liberties have been taken. But, if you believe the makers of the movie, these liberties have been taken in order to better understand Mr. Jobs and his relationship with the people around him. I leave the question of how successful this aspiration is to others. And, as all successful movies do, the movie focuses on the relationships between people. I am not going to go there. I am a "tech" guy and tech is what I will focus on. Specifically I am going to focus on the original Macintosh, the "Mac 1", if you will. Why? Because I believe it illuminates some of the same issues the movie is addressing, just from a different perspective.
And in the interests of perspective, I want to start by illuminating the environment that surrounded the Mac's original introduction in 1984. Computers have been evolving rapidly since the very beginning. Initially people would figure out better ways to do things and incorporate them into subsequent designs. But this evolution came to be dominated by the evolution of the computer chip. It started out as the Transistor, a device with a single "gate". Then the integrated circuit was developed. It initially allowed several gates to be built into a single chip. The gate count increased rapidly to dozens, hundreds, thousands, etc. It is now many billions.
With this increased gate count came increased capability and diminishing costs. Why? Consider the picture postcard. (I know that now everybody takes selfies and posts them to their Facebook page. But stick with me anyhow.) A picture postcard from New York might feature say the iconic "I 'heart' N Y" logo. Or it might instead feature a view of the skyline of the city. What's my point? The "I 'heart' N Y" image includes only a few simple shapes while the skyline includes many complex shapes. Yet the production cost for each is the same. You print the picture on one side of the card and you put the message and mailing stuff on the other. It literally doesn't matter what the picture is of. The cost is the same.
It turns out that computer chips are manufactured with essentially photographic techniques. A simple chip with a few gates costs about the same amount to manufacture as a complex chip with many gates. There's a little more to computer chip manufacture (manufacturers spend fantastic amounts of money developing new techniques for manufacturing more complex chips more cheaply) but that is the key insight. As manufacturers have gotten better and as the demand for computer chips has increased the cost of ever more powerful chips has stayed the same or gone down. This continuous innovation process has worked out over decades. But at any specific point in time, say 1984, the best chip available at a reasonable cost has only a limited amount of capability. It is way more capable than a chip from 1980 but way less capable than a chip from 1990. So that's the hardware side.
On the software side things advance over time too. People have lots of good ideas for what would be a cool thing to do. And during this time there was a company called XEROX. XEROX pioneered a technology for making copies that was head and shoulders better than anything else. They leveraged this into a ton of money. But copy machines are a one trick pony. XEROX management were very aware of this so they put together XEROX PARC (Palo Alto Research Center) to brainstorm new ideas. Money was not an object because it was pouring in from the copy machine business. What was important was to come up with a great new idea. And PARC did.
They invented all of the pieces of modern computing. They started with the mouse. They added to it high resolution (for the time) screens that allowed you to display proportional font text, images, all kinds of things. They tied it in with a new networking technology called Ethernet and added a laser printer based on their copier machine technology. It was really cool and it all worked great. There was just one problem. It was way too expensive.
XEROX put the whole package together and marketed it. No one bought. The problem was that the terminals cost a lot of money. XEROX's idea was that you would put them on the desks of executives. They were not too expensive for this. But what did they do? In the opinion of the executives they did secretarial work. They did all the things that secretaries at the time did like keeping track of a person's calendar, handling phone calls, typing things, etc. (Spread sheets and other high powered business tools were all in the future at this time.) To an executive of the time that was all secretary's work and the whole XEROX system was far to expensive to put on a secretary's desk. So no one bought it.
At some point Steve Jobs was informed of what was going on at PARC and got a tour. He could see the potential. Anyone could. The problem was cost. Jobs figured he could solve that problem so he took all the things he saw at PARC back to Apple and said "do it". He was CEO at the time so they did it. The result was something called the Lisa. You have never heard of the Lisa because it flopped. The problem was still that it was too expensive. Jobs decided that if he was going to make the whole thing work he needed to get the cost down to $1,000. So he set out to do that. The result was the Mac.
In the early days Jobs' partner was Steve Wozniak. The Woz, as he was frequently called, was a hardware genius. What he was particularly good at was combining parts in very creative ways so that something could be made with fewer parts than the "standard" design required. This made his designs cheaper to manufacture so they could be sold at a lower price. This is one of the key reasons that the Apple II, the first mass market Apple product, was so successful. It did everything the others did, and sometimes, more, but it cost less. This Woz concept of "make it but with fewer parts" was seen as key to meeting the cost target for the Mac. Woz was instrumental in the early work on the Mac but a number of "Woz clones" ended up doing most of the work in the end. And Jobs was ultimately unable to hit is target. The original Mac sold for $1,100. Initially that looked like it was close enough.
In the first few months the Mac sold very well. But then sales tailed off rapidly and never recovered. Jobs had essentially bet the company on the success of the Mac. When it tanked the Apple board of directors sensibly forced him out. So why did the initial Mac flop? It was cheap enough and it had all the cool stuff the PARC system had pioneered. (Actually it didn't have any networking capability but at that time people were not networking that much so that wasn't the problem.) The problem was a simple and fundamental one. You couldn't actually do anything with it. What? People bought those early Macs. Then they tried to run the programs they wanted to run on them and they couldn't. To understand why we need to take a look under the hood.
First of all, it came in exactly one model. You got what you got and that was it. You got a really sweet processor chip called a Motorola 68000. But that was pretty much it for the good news. The box came with 128KB (K - thousand, not M - million or G - billion) of RAM. You got a nice GUI (Graphical User Interface) screen. But it was built in so it was not upgradable. And it was small and it was black and white only. Then you got a 600KB (K - thousand again) floppy drive.
You couldn't add a second internal floppy drive and you couldn't add an internal hard drive. So you had to boot off the floppy and it had to hold the operating system, your application, and your data. Parts of the operating system could be saved in RAM after you booted up but there was only a little space and the more of the operating system you put there the less room was left over for your application and data. That meant you were swapping floppies all the time. But wait, there's more.
Remember the "leave as many parts out as you can" thing. To get the price down they left a lot of parts out. They used software running on the processor chip instead. And many of these functions were dependent on tight timing. So at predetermined intervals you had to stop running the application so that some time critical function could take place on schedule. This made writing applications very tricky. If they didn't defer to the operating system properly the computer would start misbehaving. This made writing applications much more difficult.
The result of this was that there weren't many applications available on the Mac. They were very hard (expensive) to write what with the fact that they had to be small and properly behaved (defer to the operating system at appropriate intervals). So it turned out that the original Mac was a cute toy.
Apple and Steve Jobs couldn't have been so stupid as to allow for no expandability could they? No! They weren't that stupid. They incorporated something called the SCSI (Small Computer Systems Interface) bus. This was a jack on the back that you could hook say a hard drive into. But that meant you had a lot of cables and a lot of boxes littering up your desk. Most people found that they didn't want to do this. So the practical effect was that a lot of people gave up on the Mac. They also warned their friends off them so sales quickly dried up.
Somebody fixed the Mac. It wasn't Jobs as he was fired. Who was it? It was John Sculley. He is a much reviled figure among Apple aficionados but the fact is that he saved the Mac and, by extension, Apple. He is the guy who took over after Jobs was fired. The first thing he did was increase the RAM size to 512KB. It is not much by modern standards but it did a lot to relieve the pressure. You could now run a bigger program on the Mac. It was an easy change. It increased the price slightly. But by this time people could see the Mac's potential so they were willing to pay a little more.
The next fix came not from Apple but from an outsider. Someone figured out how to fit a hard drive inside the case. Sculley was smart enough to adopt the idea even though it was "not invented here". That fixed the two most pressing problems with the Mac. You could load a larger program onto the hard disk and then run it from the hard disk and within the expanded 512KB RAM now available. But there was a lot more that needed fixing.
Subsequent Macs added parts back in so operating system functions became far less time sensitive. That made it much easier to write applications so people did. Apple also changed to an external monitor so you could get a much bigger screen than the original one. Finally, a way was found to support color monitors and things like networking. Macs got substantially more expensive than the original Mac that was the brainchild of Steve Jobs. But by this time customers liked what they saw and Mac sales increased to the point where Apple was on sound financial footing.
So why is Sculley the goat and not the hero? The problem was that once he had fixed the Mac he did not know what to do. So Apple drifted and soon started drifting downward. This is when Jobs swooped in and "saved the company". He then went on to success after success. But he had learned a valuable lesson.
The reason Microsoft did so well over this period is that Microsoft did a much better job of working within the capabilities of cost effective hardware. The PCs of the 1984 era were much more clumsy than the Mac but they did useful things. The original iterations of Windows were crude and clumsy compared to the Mac. But they lived within the abilities of the hardware available at the time. As the hardware got more capable Windows got more capable. Microsoft did not do a nice GUI until Windows 95. But by that time the kind of hardware consumers were able to afford could handle the requirements.
I am not going to go into why Microsoft stalled out a few years after Windows 95 (hint: Antitrust lawsuit). Instead I am going to return to Steve Jobs. It was no secret that he wanted to do the iPhone years before he actually did it. But he had learned his lesson. He couldn't do the kind of device he wanted to do at the price point he thought he needed to do it at. So he waited and did things like the iPod.
But when he finally judged the moment right he brought out the iPhone. And part of what made the moment right was a decision to outsource manufacturing to China. All the Apple II computers and early Mac computers were built in the US. Jobs decided he couldn't meet his price target for the iPhone if he built it in the US so he didn't.
The evolution of the iPhone has followed the Microsoft model. Apple has regularly introduced new models with additional capabilities. They have closely followed the increase in capabilities of the available hardware. A classic example of this is camera resolution. The camera on the latest model takes much better pictures than the camera on the older models.
It is unclear whether Jobs had an idea for the next new thing. We'll never know because he is no longer with us. It is too soon to tell if Tim Cook, Jobs' successor, can keep the momentum going. Time will tell.
I am not a fan of Steve Jobs as a person. I am also not a fan of him as a manager. The justification for this bad behavior is the results he achieved. And that's enough for a lot of people but not for me. In spite of that I do want to end with some praise for him in two areas where I think he was truly superior.
He was a spectacularly good salesman. This evidenced in his ability to sell Apple products to consumers. But it also evidenced itself within the company. He is now known for his "reality distortion field". He was able to find a way to motivate people to achieve what they thought was impossible. Often what he wanted really was impossible. But those instances tend to get lost to the mists of time. And just often enough he was right. He convinced somebody that they could find a solution to an "impossible" problem and improbably they did. And those occasions turned out to be extremely important so we now remember and revere them.
But beyond that Jobs was the best person at industrial design in at least the last 50 years and possibly longer. We celebrate architects for designing beautiful buildings. There have been several notable and extremely influential car designers. But Jobs eclipsed them all both in the quality of his work and in his influence on us all. He sweated the tiniest details. There are stories about him spending days deciding exactly where a screw should be placed.
The result was series of devices that are characterized by many uses as magical. They are both extremely functional (they work and they work the way people want them to work) and elegant. They just look cool and feel cool. There are designers that are good at functional (Bill Gates) and designers that are good at a cool look (any number of architects). There are very few that could do either nearly as well as Jobs did both.
And in the interests of perspective, I want to start by illuminating the environment that surrounded the Mac's original introduction in 1984. Computers have been evolving rapidly since the very beginning. Initially people would figure out better ways to do things and incorporate them into subsequent designs. But this evolution came to be dominated by the evolution of the computer chip. It started out as the Transistor, a device with a single "gate". Then the integrated circuit was developed. It initially allowed several gates to be built into a single chip. The gate count increased rapidly to dozens, hundreds, thousands, etc. It is now many billions.
With this increased gate count came increased capability and diminishing costs. Why? Consider the picture postcard. (I know that now everybody takes selfies and posts them to their Facebook page. But stick with me anyhow.) A picture postcard from New York might feature say the iconic "I 'heart' N Y" logo. Or it might instead feature a view of the skyline of the city. What's my point? The "I 'heart' N Y" image includes only a few simple shapes while the skyline includes many complex shapes. Yet the production cost for each is the same. You print the picture on one side of the card and you put the message and mailing stuff on the other. It literally doesn't matter what the picture is of. The cost is the same.
It turns out that computer chips are manufactured with essentially photographic techniques. A simple chip with a few gates costs about the same amount to manufacture as a complex chip with many gates. There's a little more to computer chip manufacture (manufacturers spend fantastic amounts of money developing new techniques for manufacturing more complex chips more cheaply) but that is the key insight. As manufacturers have gotten better and as the demand for computer chips has increased the cost of ever more powerful chips has stayed the same or gone down. This continuous innovation process has worked out over decades. But at any specific point in time, say 1984, the best chip available at a reasonable cost has only a limited amount of capability. It is way more capable than a chip from 1980 but way less capable than a chip from 1990. So that's the hardware side.
On the software side things advance over time too. People have lots of good ideas for what would be a cool thing to do. And during this time there was a company called XEROX. XEROX pioneered a technology for making copies that was head and shoulders better than anything else. They leveraged this into a ton of money. But copy machines are a one trick pony. XEROX management were very aware of this so they put together XEROX PARC (Palo Alto Research Center) to brainstorm new ideas. Money was not an object because it was pouring in from the copy machine business. What was important was to come up with a great new idea. And PARC did.
They invented all of the pieces of modern computing. They started with the mouse. They added to it high resolution (for the time) screens that allowed you to display proportional font text, images, all kinds of things. They tied it in with a new networking technology called Ethernet and added a laser printer based on their copier machine technology. It was really cool and it all worked great. There was just one problem. It was way too expensive.
XEROX put the whole package together and marketed it. No one bought. The problem was that the terminals cost a lot of money. XEROX's idea was that you would put them on the desks of executives. They were not too expensive for this. But what did they do? In the opinion of the executives they did secretarial work. They did all the things that secretaries at the time did like keeping track of a person's calendar, handling phone calls, typing things, etc. (Spread sheets and other high powered business tools were all in the future at this time.) To an executive of the time that was all secretary's work and the whole XEROX system was far to expensive to put on a secretary's desk. So no one bought it.
At some point Steve Jobs was informed of what was going on at PARC and got a tour. He could see the potential. Anyone could. The problem was cost. Jobs figured he could solve that problem so he took all the things he saw at PARC back to Apple and said "do it". He was CEO at the time so they did it. The result was something called the Lisa. You have never heard of the Lisa because it flopped. The problem was still that it was too expensive. Jobs decided that if he was going to make the whole thing work he needed to get the cost down to $1,000. So he set out to do that. The result was the Mac.
In the early days Jobs' partner was Steve Wozniak. The Woz, as he was frequently called, was a hardware genius. What he was particularly good at was combining parts in very creative ways so that something could be made with fewer parts than the "standard" design required. This made his designs cheaper to manufacture so they could be sold at a lower price. This is one of the key reasons that the Apple II, the first mass market Apple product, was so successful. It did everything the others did, and sometimes, more, but it cost less. This Woz concept of "make it but with fewer parts" was seen as key to meeting the cost target for the Mac. Woz was instrumental in the early work on the Mac but a number of "Woz clones" ended up doing most of the work in the end. And Jobs was ultimately unable to hit is target. The original Mac sold for $1,100. Initially that looked like it was close enough.
In the first few months the Mac sold very well. But then sales tailed off rapidly and never recovered. Jobs had essentially bet the company on the success of the Mac. When it tanked the Apple board of directors sensibly forced him out. So why did the initial Mac flop? It was cheap enough and it had all the cool stuff the PARC system had pioneered. (Actually it didn't have any networking capability but at that time people were not networking that much so that wasn't the problem.) The problem was a simple and fundamental one. You couldn't actually do anything with it. What? People bought those early Macs. Then they tried to run the programs they wanted to run on them and they couldn't. To understand why we need to take a look under the hood.
First of all, it came in exactly one model. You got what you got and that was it. You got a really sweet processor chip called a Motorola 68000. But that was pretty much it for the good news. The box came with 128KB (K - thousand, not M - million or G - billion) of RAM. You got a nice GUI (Graphical User Interface) screen. But it was built in so it was not upgradable. And it was small and it was black and white only. Then you got a 600KB (K - thousand again) floppy drive.
You couldn't add a second internal floppy drive and you couldn't add an internal hard drive. So you had to boot off the floppy and it had to hold the operating system, your application, and your data. Parts of the operating system could be saved in RAM after you booted up but there was only a little space and the more of the operating system you put there the less room was left over for your application and data. That meant you were swapping floppies all the time. But wait, there's more.
Remember the "leave as many parts out as you can" thing. To get the price down they left a lot of parts out. They used software running on the processor chip instead. And many of these functions were dependent on tight timing. So at predetermined intervals you had to stop running the application so that some time critical function could take place on schedule. This made writing applications very tricky. If they didn't defer to the operating system properly the computer would start misbehaving. This made writing applications much more difficult.
The result of this was that there weren't many applications available on the Mac. They were very hard (expensive) to write what with the fact that they had to be small and properly behaved (defer to the operating system at appropriate intervals). So it turned out that the original Mac was a cute toy.
Apple and Steve Jobs couldn't have been so stupid as to allow for no expandability could they? No! They weren't that stupid. They incorporated something called the SCSI (Small Computer Systems Interface) bus. This was a jack on the back that you could hook say a hard drive into. But that meant you had a lot of cables and a lot of boxes littering up your desk. Most people found that they didn't want to do this. So the practical effect was that a lot of people gave up on the Mac. They also warned their friends off them so sales quickly dried up.
Somebody fixed the Mac. It wasn't Jobs as he was fired. Who was it? It was John Sculley. He is a much reviled figure among Apple aficionados but the fact is that he saved the Mac and, by extension, Apple. He is the guy who took over after Jobs was fired. The first thing he did was increase the RAM size to 512KB. It is not much by modern standards but it did a lot to relieve the pressure. You could now run a bigger program on the Mac. It was an easy change. It increased the price slightly. But by this time people could see the Mac's potential so they were willing to pay a little more.
The next fix came not from Apple but from an outsider. Someone figured out how to fit a hard drive inside the case. Sculley was smart enough to adopt the idea even though it was "not invented here". That fixed the two most pressing problems with the Mac. You could load a larger program onto the hard disk and then run it from the hard disk and within the expanded 512KB RAM now available. But there was a lot more that needed fixing.
Subsequent Macs added parts back in so operating system functions became far less time sensitive. That made it much easier to write applications so people did. Apple also changed to an external monitor so you could get a much bigger screen than the original one. Finally, a way was found to support color monitors and things like networking. Macs got substantially more expensive than the original Mac that was the brainchild of Steve Jobs. But by this time customers liked what they saw and Mac sales increased to the point where Apple was on sound financial footing.
So why is Sculley the goat and not the hero? The problem was that once he had fixed the Mac he did not know what to do. So Apple drifted and soon started drifting downward. This is when Jobs swooped in and "saved the company". He then went on to success after success. But he had learned a valuable lesson.
The reason Microsoft did so well over this period is that Microsoft did a much better job of working within the capabilities of cost effective hardware. The PCs of the 1984 era were much more clumsy than the Mac but they did useful things. The original iterations of Windows were crude and clumsy compared to the Mac. But they lived within the abilities of the hardware available at the time. As the hardware got more capable Windows got more capable. Microsoft did not do a nice GUI until Windows 95. But by that time the kind of hardware consumers were able to afford could handle the requirements.
I am not going to go into why Microsoft stalled out a few years after Windows 95 (hint: Antitrust lawsuit). Instead I am going to return to Steve Jobs. It was no secret that he wanted to do the iPhone years before he actually did it. But he had learned his lesson. He couldn't do the kind of device he wanted to do at the price point he thought he needed to do it at. So he waited and did things like the iPod.
But when he finally judged the moment right he brought out the iPhone. And part of what made the moment right was a decision to outsource manufacturing to China. All the Apple II computers and early Mac computers were built in the US. Jobs decided he couldn't meet his price target for the iPhone if he built it in the US so he didn't.
The evolution of the iPhone has followed the Microsoft model. Apple has regularly introduced new models with additional capabilities. They have closely followed the increase in capabilities of the available hardware. A classic example of this is camera resolution. The camera on the latest model takes much better pictures than the camera on the older models.
It is unclear whether Jobs had an idea for the next new thing. We'll never know because he is no longer with us. It is too soon to tell if Tim Cook, Jobs' successor, can keep the momentum going. Time will tell.
I am not a fan of Steve Jobs as a person. I am also not a fan of him as a manager. The justification for this bad behavior is the results he achieved. And that's enough for a lot of people but not for me. In spite of that I do want to end with some praise for him in two areas where I think he was truly superior.
He was a spectacularly good salesman. This evidenced in his ability to sell Apple products to consumers. But it also evidenced itself within the company. He is now known for his "reality distortion field". He was able to find a way to motivate people to achieve what they thought was impossible. Often what he wanted really was impossible. But those instances tend to get lost to the mists of time. And just often enough he was right. He convinced somebody that they could find a solution to an "impossible" problem and improbably they did. And those occasions turned out to be extremely important so we now remember and revere them.
But beyond that Jobs was the best person at industrial design in at least the last 50 years and possibly longer. We celebrate architects for designing beautiful buildings. There have been several notable and extremely influential car designers. But Jobs eclipsed them all both in the quality of his work and in his influence on us all. He sweated the tiniest details. There are stories about him spending days deciding exactly where a screw should be placed.
The result was series of devices that are characterized by many uses as magical. They are both extremely functional (they work and they work the way people want them to work) and elegant. They just look cool and feel cool. There are designers that are good at functional (Bill Gates) and designers that are good at a cool look (any number of architects). There are very few that could do either nearly as well as Jobs did both.
Wednesday, September 30, 2015
DHCP and a wrap up
This is really, really, really, going to be the last post in this series, for now. At this point I have no plans to add anything. But who knows what the future will bring. All of these posts can be found in the September, 2015 section the of my blog. (See Blog Archive in the navigation section at the right.) But, for consistency sake, here is a link to the first post in the series: http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html. And here is a link to the immediately previous post: http://sigma5.blogspot.com/2015/09/internet-dns.html. So . . .
DHCP
DHCP stands for Dynamic Host Configuration Protocol. This mouthful is almost completely meaningless so what's up? Back in the olden days IP configurations were set up manually. When my company first rolled out TCP/IP that's what we did. Several of us maintained extensive lists of which computer would use which IP address. Besides being tedious and error prone it had a problem. You had to go to each computer and plug the "static IP configuration" parameters in individually. That was bad enough but my company had facilities in several states. So if we wanted to change things up on a computer in another state we had to actually send someone there at a great cost in time and money.
And, hey, this is a computer thing. Why not automate things? Early attempts went by the names RARP and BOOTP. In these cases you plugged everything into one computer. That computer served the information out to the other computers. But it was still a lot of trouble. My company did not use either to any extent. Then along came DHCP. This helped a lot. DHCP added the "dynamic" part. Instead of having a list of computers and IPv4 addresses DHCP let you use a pool. With DHCP you allocated a group of IPv4 addresses to the DHCP server and said "go for it".
DHCP depends on Ethernet or some other protocol that supports a the ability to broadcast to all other local computers. Your computer sends out an "are you a DHCP server?" broadcast message. Hopefully, exactly one machine answers back. Then your computer sends a message directly to that computer that says "my name is X - do you have IP configuration information for me?". The DHCP server looks in its mini-database. If it already has an entry for that computer it sends out the information. If not, it picks an unused IPv4 address from the pool, adds in whatever other information it is supposed to provide, plugs that into the mini-database and sends it out. Your computer catches the information, plugs it in, and boom! it is on the air.
Here's a little more detail. Each entry in the mini-database has a time to live. It can vary from a few months to a few hours, depending on your estimate of the rate of turnover. In a coffee shop where turnover is quick a few hours is probably a good choice. In a corporate environment where turnover is slow, a few months is a good choice. If an entry goes unused too long it is purged and the IPv4 address goes back into the "available" pool. The other thing is that the DHCP server can send out more than just an IPv4 address for the computer to use. Typically it sends out the subnet mask, the gateway, and the addresses of the primary and backup DNS servers. That's the standard set of "IP configuration" information machines need.
By shifting to DHCP we just had to set up a DHCP server at each location and make sure it had a big enough pool of IPv4 addresses to work with. At this time we also converted from using our 5 class "C" public IPv4 nets to using private IPv4 nets so having lots of IPv4 addresses available at each location was easy to arrange. Now we just had to make sure each machine had a unique name. That was much easier to pull off. Then when the machine booted up it would find the local DHCP server and get the information it needed to operate at that location. This is usually referred to as a "dynamic IP configuration" for obvious reasons. We might very occasionally need to change the DHCP setup at a location but that was it.
When you set up a DHCP server you have to plug in the address pool. Usually you say something like "all IPv4 addresses in the range going from 192.168.0.100 to 192.168.0.200". It is possible to specify multiple ranges but this is rarely necessary. Most DHCP servers are set up pretty much the same way. So it is easy to plug in the subnet mask, the gateway address, and one or both DNS server addresses. If you have a home network that shares one public IP address among several machines you have a home DHCP server lurking somewhere. How does that come about?
wrap up
I have left a couple of questions dangling. I am going to answer them in this section. But first I am going to do a quick review of my home setup. My setup is more complicated than most home setups. I am going to explain why I do things the way I do and then move on to some recommendations for how most people should operate.
Comcast is my ISP. They have provided me with an Arris cable modem. This box behaves somewhat like a combination DHCP server and gateway. Comcast provides a pair of public IPv4 addresses to the box. One is attached to the Arris box and one is available to be served up to whatever machine I hook up to it. The hookup is an Ethernet connection. If I only had one computer I could hook it up directly to the Arris box, set it to do DHCP (the default) and everything would work fine. My computer would make a DHCP request to the Arris box and the Arris box would serve up a public IP address. It would also serve up a subnet mask, gateway address (the address of the Arris box), and the addresses of two DNS servers operated by Comcast. That's how that would work.
But I want to hook several boxes up to the Internet. How does that work? I have another box. It is a Netgear FVS318G ProSafe 8 port Gigabit VPN Firewall. This is a box specifically designed for the home networking market. It runs about $120 at Amazon. (BTW, avoid the FVS318. It has slower ports.) It has a special port, the WAN port, that you connect to your cable modem. That's the "external" interface of the box. Then it has 8 more Ethernet ports, marked as LAN ports. Together they constitute the "internal" interface. I hook all my home network equipment up to them. The 8 "internal interface" ports behave just like an Ethernet switch. And between the "internal" interface and the "external" interface this box does NAT. So now all my inside boxes can share my one public IPv4 address.
I also have a separate Linksys WAP54G Wi-Fi base station. It is just plugged into one of the 8 inside ports on my Netgear box. I also have a computer running Windows 2008 Server Edition. This turns my home network environment into a small corporate network, also sometimes referred to as a "Microsoft NT Domain". This is something most people don't need to mess with. But I did system administration work on corporate NT domains for a couple of decades. I like them and I know how to administer them so I did. It is total overkill for most home users.
Nestled into this mess is a standard PC running Windows 7. It has a private IPv4 address. As does the Windows Server box, the Netgear box (on its internal interface), the Linksys box, my TiVo, my DVD player, and my printer. I also have a couple of other boxes I occasionally use but I think you get the idea. All the boxes I listed (and the ones I didn't) are able to get out to the Internet whenever they need to. And they can talk to each other. I can send files between my computers. I can print from either one. My home network does everything I want it to. But it is overkill for most people.
But like more typical home network users there are some capabilities that all of us need. So let me run down the list. I have already indicated that the NAT capability resides in my Netgear box. My DHCP capability resides in my Windows server box. My DNS capability also resides in my Windows server box. That means I had to configure both. DHCP is pretty easy. With Windows server DNS is harder but not a lot harder. (And it probably will not surprise you to know that Microsoft uses its own code rather than BIND.)
First of all, remember that list of "root" servers. Microsoft plugs that list in for me automatically. And it changes very occasionally. But when it does Microsoft sends out the updated list through Windows Update. As long as I keep my Microsoft maintenance current, which I do, my "root" server list stays current. I did have to configure a "forward zone". It holds the "A" records for all my local devices. And I had to configure two "reverse zones" to hold all my PTR records. I won't go into the details on how this was done. Suffice it to say it was not hard.
So I have this expertise built up from decades of doing this sort of thing for a living. What's a regular person supposed to do? Many people who don't know much about any of this have home networks that work just fine. How is that possible? The solution to both of these questions lies in looking at a more common setup.
Most home users do not have an NT domain operating. So what do they do about DHCP and DNS? The answer to both questions lies in the Netgear box. That box can do DHCP. I just turned it off. You don't want competing DHCP servers running around and I knew that I wanted to use my Windows server to do DHCP. (Why? 'Cuz!) So I went in and turned it off. Most people leave DHCP turned on and this is the right thing to do for them. And the defaults for the box set up a home network using the 192.168.0.0/24 private net and configure DHCP accordingly. I don't remember the range of IPv4 addresses the Netgear box selects as its default but it works for all but a few home setups.
That leaves DNS. Here too the Netgear box has a cute trick. It has a setting called "Get Automatically from ISP". Remember that the box uses DHCP through its outside interface to get the information in needs to communicate with Comcast. Well, included in that information are the IP addresses of two Comcast DNS servers. When turned on (I turned it off manually) this option copies the two DNS entries it gets from Comcast into the DHCP information it serves out to the internal network. The down side of this is you can't find the IPv4 addresses of the boxes on your internal network using DNS. I like to be able to do that so I went through all those "NT domain" hoops. But most people can and do live without this capability just fine.
So if I had not installed a Windows server and I had left all the settings on my Netgear box at their defaults I would have a perfectly functional network. One final thing on this, the Netgear box assumes it is king of the 192.168.0.0/24 net. As such it automatically assigns the IPv4 address of 192.168.0.1 to its inside interface. Then it plugs this address in as the default gateway in the DHCP information it serves out. So, if they used DHCP, and if they used the default DHCP setup the Netgear box comes with, my home computers would get all the settings they needed to fully participate in IPv4 networks.
The Netgear box would serve out to each machine a unique IPv4 address from the pool of 192.168.0.0/24 addresses it maintains. It would automatically serve out a subnet mask of 255.255.255.0. It would automatically serve out a gateway address of 192.168.0.1 (itself). And it would automatically serve out the two DNS server addresses it got from Comcast. That's how it works.
There are other boxes that have essentially the same capabilities as my Netgear box. Some of them have 4 ports or none. Some of them include a built in Wi-Fi base station. Pick the box you like. In some cases you can even get the box from your ISP. And pretty much all the ISPs operate along the same lines as Comcast. They might provide you with a different brand or model of cable modem. And it might not technically be a cable modem. But in all likelihood it will behave pretty much like the Arris box I got from Comcast.
And you actually don't need to know all this stuff. Unless something goes wrong. Then this information may be very helpful. Do you have no (or bad) DNS server addresses? Check your NAT box and make sure it is doing the "Get automatically from ISP" thing. Are IPv4 addresses changing around in weird ways? Check to make sure you have exactly one DHCP server running and check that the DHCP server is serving out sensible settings. Are things physically connected up correctly? In my case the Arris box needs to be connected to the WAN jack and everything else needs to be connected to LAN jacks. If you have a separate Wi-Fi box is it trying to do the same things (i.e. DHCP) as your NAT box is doing. For some services you can pick either box to host the service but there needs to be only one box hosting each service.
And then it may be those that just want to know how things work. I hope these posts give you enough foundation so that you can explore anything or everything in more depth, should you choose to. Wikipedia is a great source of more information on these subjects. There are many hard core geeks among its contributors. And they delight in drilling down much further than I have. And in a lot of cases, they will provide you with references that will allow you to dive even deeper. It's a lot of fun, if you get off on that sort of thing. And I do. For the rest of you, I hope you have found this group of posts entertaining and enlightening.
DHCP
DHCP stands for Dynamic Host Configuration Protocol. This mouthful is almost completely meaningless so what's up? Back in the olden days IP configurations were set up manually. When my company first rolled out TCP/IP that's what we did. Several of us maintained extensive lists of which computer would use which IP address. Besides being tedious and error prone it had a problem. You had to go to each computer and plug the "static IP configuration" parameters in individually. That was bad enough but my company had facilities in several states. So if we wanted to change things up on a computer in another state we had to actually send someone there at a great cost in time and money.
And, hey, this is a computer thing. Why not automate things? Early attempts went by the names RARP and BOOTP. In these cases you plugged everything into one computer. That computer served the information out to the other computers. But it was still a lot of trouble. My company did not use either to any extent. Then along came DHCP. This helped a lot. DHCP added the "dynamic" part. Instead of having a list of computers and IPv4 addresses DHCP let you use a pool. With DHCP you allocated a group of IPv4 addresses to the DHCP server and said "go for it".
DHCP depends on Ethernet or some other protocol that supports a the ability to broadcast to all other local computers. Your computer sends out an "are you a DHCP server?" broadcast message. Hopefully, exactly one machine answers back. Then your computer sends a message directly to that computer that says "my name is X - do you have IP configuration information for me?". The DHCP server looks in its mini-database. If it already has an entry for that computer it sends out the information. If not, it picks an unused IPv4 address from the pool, adds in whatever other information it is supposed to provide, plugs that into the mini-database and sends it out. Your computer catches the information, plugs it in, and boom! it is on the air.
Here's a little more detail. Each entry in the mini-database has a time to live. It can vary from a few months to a few hours, depending on your estimate of the rate of turnover. In a coffee shop where turnover is quick a few hours is probably a good choice. In a corporate environment where turnover is slow, a few months is a good choice. If an entry goes unused too long it is purged and the IPv4 address goes back into the "available" pool. The other thing is that the DHCP server can send out more than just an IPv4 address for the computer to use. Typically it sends out the subnet mask, the gateway, and the addresses of the primary and backup DNS servers. That's the standard set of "IP configuration" information machines need.
By shifting to DHCP we just had to set up a DHCP server at each location and make sure it had a big enough pool of IPv4 addresses to work with. At this time we also converted from using our 5 class "C" public IPv4 nets to using private IPv4 nets so having lots of IPv4 addresses available at each location was easy to arrange. Now we just had to make sure each machine had a unique name. That was much easier to pull off. Then when the machine booted up it would find the local DHCP server and get the information it needed to operate at that location. This is usually referred to as a "dynamic IP configuration" for obvious reasons. We might very occasionally need to change the DHCP setup at a location but that was it.
When you set up a DHCP server you have to plug in the address pool. Usually you say something like "all IPv4 addresses in the range going from 192.168.0.100 to 192.168.0.200". It is possible to specify multiple ranges but this is rarely necessary. Most DHCP servers are set up pretty much the same way. So it is easy to plug in the subnet mask, the gateway address, and one or both DNS server addresses. If you have a home network that shares one public IP address among several machines you have a home DHCP server lurking somewhere. How does that come about?
wrap up
I have left a couple of questions dangling. I am going to answer them in this section. But first I am going to do a quick review of my home setup. My setup is more complicated than most home setups. I am going to explain why I do things the way I do and then move on to some recommendations for how most people should operate.
Comcast is my ISP. They have provided me with an Arris cable modem. This box behaves somewhat like a combination DHCP server and gateway. Comcast provides a pair of public IPv4 addresses to the box. One is attached to the Arris box and one is available to be served up to whatever machine I hook up to it. The hookup is an Ethernet connection. If I only had one computer I could hook it up directly to the Arris box, set it to do DHCP (the default) and everything would work fine. My computer would make a DHCP request to the Arris box and the Arris box would serve up a public IP address. It would also serve up a subnet mask, gateway address (the address of the Arris box), and the addresses of two DNS servers operated by Comcast. That's how that would work.
But I want to hook several boxes up to the Internet. How does that work? I have another box. It is a Netgear FVS318G ProSafe 8 port Gigabit VPN Firewall. This is a box specifically designed for the home networking market. It runs about $120 at Amazon. (BTW, avoid the FVS318. It has slower ports.) It has a special port, the WAN port, that you connect to your cable modem. That's the "external" interface of the box. Then it has 8 more Ethernet ports, marked as LAN ports. Together they constitute the "internal" interface. I hook all my home network equipment up to them. The 8 "internal interface" ports behave just like an Ethernet switch. And between the "internal" interface and the "external" interface this box does NAT. So now all my inside boxes can share my one public IPv4 address.
I also have a separate Linksys WAP54G Wi-Fi base station. It is just plugged into one of the 8 inside ports on my Netgear box. I also have a computer running Windows 2008 Server Edition. This turns my home network environment into a small corporate network, also sometimes referred to as a "Microsoft NT Domain". This is something most people don't need to mess with. But I did system administration work on corporate NT domains for a couple of decades. I like them and I know how to administer them so I did. It is total overkill for most home users.
Nestled into this mess is a standard PC running Windows 7. It has a private IPv4 address. As does the Windows Server box, the Netgear box (on its internal interface), the Linksys box, my TiVo, my DVD player, and my printer. I also have a couple of other boxes I occasionally use but I think you get the idea. All the boxes I listed (and the ones I didn't) are able to get out to the Internet whenever they need to. And they can talk to each other. I can send files between my computers. I can print from either one. My home network does everything I want it to. But it is overkill for most people.
But like more typical home network users there are some capabilities that all of us need. So let me run down the list. I have already indicated that the NAT capability resides in my Netgear box. My DHCP capability resides in my Windows server box. My DNS capability also resides in my Windows server box. That means I had to configure both. DHCP is pretty easy. With Windows server DNS is harder but not a lot harder. (And it probably will not surprise you to know that Microsoft uses its own code rather than BIND.)
First of all, remember that list of "root" servers. Microsoft plugs that list in for me automatically. And it changes very occasionally. But when it does Microsoft sends out the updated list through Windows Update. As long as I keep my Microsoft maintenance current, which I do, my "root" server list stays current. I did have to configure a "forward zone". It holds the "A" records for all my local devices. And I had to configure two "reverse zones" to hold all my PTR records. I won't go into the details on how this was done. Suffice it to say it was not hard.
So I have this expertise built up from decades of doing this sort of thing for a living. What's a regular person supposed to do? Many people who don't know much about any of this have home networks that work just fine. How is that possible? The solution to both of these questions lies in looking at a more common setup.
Most home users do not have an NT domain operating. So what do they do about DHCP and DNS? The answer to both questions lies in the Netgear box. That box can do DHCP. I just turned it off. You don't want competing DHCP servers running around and I knew that I wanted to use my Windows server to do DHCP. (Why? 'Cuz!) So I went in and turned it off. Most people leave DHCP turned on and this is the right thing to do for them. And the defaults for the box set up a home network using the 192.168.0.0/24 private net and configure DHCP accordingly. I don't remember the range of IPv4 addresses the Netgear box selects as its default but it works for all but a few home setups.
That leaves DNS. Here too the Netgear box has a cute trick. It has a setting called "Get Automatically from ISP". Remember that the box uses DHCP through its outside interface to get the information in needs to communicate with Comcast. Well, included in that information are the IP addresses of two Comcast DNS servers. When turned on (I turned it off manually) this option copies the two DNS entries it gets from Comcast into the DHCP information it serves out to the internal network. The down side of this is you can't find the IPv4 addresses of the boxes on your internal network using DNS. I like to be able to do that so I went through all those "NT domain" hoops. But most people can and do live without this capability just fine.
So if I had not installed a Windows server and I had left all the settings on my Netgear box at their defaults I would have a perfectly functional network. One final thing on this, the Netgear box assumes it is king of the 192.168.0.0/24 net. As such it automatically assigns the IPv4 address of 192.168.0.1 to its inside interface. Then it plugs this address in as the default gateway in the DHCP information it serves out. So, if they used DHCP, and if they used the default DHCP setup the Netgear box comes with, my home computers would get all the settings they needed to fully participate in IPv4 networks.
The Netgear box would serve out to each machine a unique IPv4 address from the pool of 192.168.0.0/24 addresses it maintains. It would automatically serve out a subnet mask of 255.255.255.0. It would automatically serve out a gateway address of 192.168.0.1 (itself). And it would automatically serve out the two DNS server addresses it got from Comcast. That's how it works.
There are other boxes that have essentially the same capabilities as my Netgear box. Some of them have 4 ports or none. Some of them include a built in Wi-Fi base station. Pick the box you like. In some cases you can even get the box from your ISP. And pretty much all the ISPs operate along the same lines as Comcast. They might provide you with a different brand or model of cable modem. And it might not technically be a cable modem. But in all likelihood it will behave pretty much like the Arris box I got from Comcast.
And you actually don't need to know all this stuff. Unless something goes wrong. Then this information may be very helpful. Do you have no (or bad) DNS server addresses? Check your NAT box and make sure it is doing the "Get automatically from ISP" thing. Are IPv4 addresses changing around in weird ways? Check to make sure you have exactly one DHCP server running and check that the DHCP server is serving out sensible settings. Are things physically connected up correctly? In my case the Arris box needs to be connected to the WAN jack and everything else needs to be connected to LAN jacks. If you have a separate Wi-Fi box is it trying to do the same things (i.e. DHCP) as your NAT box is doing. For some services you can pick either box to host the service but there needs to be only one box hosting each service.
And then it may be those that just want to know how things work. I hope these posts give you enough foundation so that you can explore anything or everything in more depth, should you choose to. Wikipedia is a great source of more information on these subjects. There are many hard core geeks among its contributors. And they delight in drilling down much further than I have. And in a lot of cases, they will provide you with references that will allow you to dive even deeper. It's a lot of fun, if you get off on that sort of thing. And I do. For the rest of you, I hope you have found this group of posts entertaining and enlightening.
Tuesday, September 29, 2015
Internet - DNS
This is, the sixth post in this series. I recommend reading them in sequence. The first post is at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html. The immediately previous one is at http://sigma5.blogspot.com/2015/09/internet-routing-and-nat.html. I had hopes that I could wrap things up with this post but to do so would make it run too long. So this post is short and only covers one subject, DNS.
What is DNS and why do we care? Computers love numbers. People love text. So people like dealing with something like www.google.com rather than 172.25.11.23 (not Google's actual IP address). DNS, short for Domain Name Server, is what gets us from here to there. The thing that does this is called a DNS server. In the earliest days of ARPANET there were only a few computers to keep track of. A simple list worked fine. But now there are "millions and billions" of web sites, mobile applications, etc., on the 'net. We have long since moved past the time when a list could get the job done.
This was recognized fairly early in the transition from that first ARPANET to the modern Internet. The first practical and comprehensive solution was called BIND - the Berkley Internet Name Daemon. The University of California at Berkley Computer Science department was an early and active participant in Unix, the "C" computer language, and the Internet. They developed a lot of tools and enhancements and one of them was BIND. BIND is still around although responsibility has been turned over to the Internet Systems Consortium. You can find out more at https://www.isc.org/downloads/bind/. But most of us only need to use DNS servers. We don't need to install or operate our own DNS server. But let's find out a little about how they work anyway.
DNS is not a single server. It is a swarm of cooperating servers. And their job is to turn something like www.google.com into something like 172.25.11.23. So how do they do this? A DNS server is mostly just a simple database. You ask questions. It provides answers. Actually it serves up the data from records that are appropriate responses to our query. The most popular record type is the "A" for Address record. If a DNS server has an A record that has a key of www.google.com it serves up the data value (172.25.11.23, in our example). But in many cases it does not have an A record that is an exact match. What then?
Would it surprise you to know that, as with a lot of Internet things, DSN is into this whole delegation thing. Various servers provide part of the answer then pass things on to a server that knows more. So all blind queries (queries that are not in the wheelhouse of whichever DNS server we are using) are passed on to a "root" server. Where are the "root" servers? That' one of those Internet "well known" things. There is an official list of root servers and you can find out what they are if you know where to look. Anyone who messes with DNS servers knows where this list is found and it is plugged into each DNS server. So what our server does is pass the question on to a root server.
And this whole delegation thing proceeds from back to front. The only thing root servers know about is where that last part is. You know, the ".COM" or ".EDU" or ".GOV" or whatever thing. The root servers don't care about the rest of it. They know about that last part and where to send you to find out more information. They have a list of DNS servers that specialize in ".COM" DNS entries, for instance. So they just pass any ".COM" queries along to a ".COM" DNS server. (We don't need to know where the ".COM" DNS servers are because the root servers know.) This server knows about a whole bunch of "anything.COM" things. It has no clue about "anything.EDU" or "anything.GOV" or "anything.whatever" if "whatever" is anything but ".COM". But different servers do know about ".EDU", for instance. The root server passes queries about "anything.EDU" to one of them. The same is true for ".GOV", ".NET", and so on.
There is an official list of all the legal top levels. (They are the last part of the name but they are called "top" levels because they represent the top of the tree of interconnected DNS servers.) That's because every root server needs entries for all of them. The servers that handle ".COM", ".NET", etc. are called "top level servers". Originally there were only a few top levels. But the criteria has been loosened up several times now. Why? Well, there is a little work necessary to get a new top level to work. But the main reason has been a concern over confusion. But people are now pretty comfortable with things like ".CO" or ".TV" or ".US" so there now seems little reason to keep the list short. Also, the Internet has slowly become more international and part of that has been support for various languages. There are now top levels in Chinese and Arabic, for instance, languages that don't even use our standard alphabet. Returning to our example, hopefully the ".COM" DNS servers have an entry for ".GOOGLE.COM".
But that entry is just the name of another DNS server. This server knows about "anything.GOOGLE.COM". It knows nothing about "anything.MICROSOFT.COM" or "anything.FACEBOOK.COM". And it especially knows nothing about "WH.GOV" (different top level). So this ".GOOGLE.COM" DNS server has (again, if everything is working ok) an entry for WWW.GOOGLE.COM. It serves it up to your computer and your computer finally knows what IPv4 address to send its traffic to. Lots of messages are exchanged between lots of DNS servers. But that's traffic that happens in the background where you don't see it. And it is perfectly legal to go more levels. The process just starts at the top level and works its way through the layers until it gets to the end at whatever level that might be.
That's the standard way things work. I am now going to talk about a couple of variations. Let's say we ask our local DNS server about WWW.FACEBOOK.COM. The first time through it goes through the elaborate process I described above. But we often go to the same place many times. So DNS servers have a cache. This is a place where the answers to recent questions are saved. For any query that is outside the direct responsibility of that DNS server the query is first checked against the cache before the elaborate process described above is undertaken.
If it finds an entry for WWW.FACEBOOK.COM in the cache (because somebody asked about it recently) it just sends a "non-authorative" answer back. Why non-authoritative? Because only one or a few DNS servers are designated as the official respositories for WWW.FACEBOOK.COM DNS information. They are the "authoritative" DNS servers. All DNS entries in the cache of non-authoritative servers are also marked with a "time to live", usually a few days. If Google wants to move their servers around they will need to update the DNS entries for these servers. Confusion will reign if the old information does not eventually flush out of the Internet.
Another non-standard thing is an ALIAS record. Let's say your company buys another company and you want to merge web sites. You could just shut one down but that would mean that customers who had links to the shut down site would be stuck. The ALIAS record solves that. You just put an ALIAS record in the appropriate DNS servers. The ALIAS record says "if someone asks about WWW.X.COM answer the question as if they had actually asked about WWW.Y.COM instead". This way everything that was linked to the old location gets automatically connected up with the new location. ALIAS records actually have a lot more uses than this but that's enough to give you the idea.
And it turns out that there are many different kinds of DNS records. I am only going to talk about two more. One is the MX record. You can ask a DNS server "what's the IP address of the mail server for WWW.GOOGLE.COM?" The DNS server will go looking for MX records instead of A records. This kind of thing is handy in a number of ways. Have you noticed that sometimes companies leave the "WWW" off. DNS tricks like these allow computers to find the same web site either way. Try just "GOOGLE.COM" sometime. It gets you to the same place as you would get to by putting the "WWW." on the front.
The other record I want to talk about is the PTR record. "A" records get you from the name to the IPv4 address. PTR records get you back the other way. Not everybody sets up PTR records all the time but in a lot of cases people do. If so you can find out what the text name that goes with an IP address is. This turns out to be tricky. To make this work you need to know the net/subnet boundaries. But it is literally impossible, in general, to know what they are. So what to do? The PTR system assumes that IPv4 addresses are broken on octet boundaries. And, to allow for this delegation thing to work, we have to go at things backwards. 172.25.11.23 is temporarily turned around. We pretend for a minute it really is 23.11.25.172. That's because we want to look the "172" part up first, and so on. Cutting to the chase, instead of eventually looking for "WWW" in the ".GOOGLE.COM" DNS server we look for "23" in the "11.23.172" DNS server. If you are not a little confused at this point I have failed.
But let me see if I can confuse you even more. We have these authoritative DNS servers. Why? So everyone everywhere who asks a specific DNS question gets the exact same answer. But what if we want different people to get different answers? Google has server farms all over the place. They want to send you to one of their local farms rather than requiring you to gallivant across the country or the world to get to a Google server. A number of other large companies like Microsoft, Facebook, and Amazon want to do the same thing. There's the trick.
What if we have a super-DNS server that not only looks at the question but looks at the IP address of the computer asking the question? If we know the general location of the computer making the DNS query (you can usually get at least an OK idea by doing a lot of research) then you can serve up an IPv4 address answer that is for a server that is in a server farm that is close (in terms of the Internet) to the computer generating the query. There is a company called Akami that does just that.
Their services are not cheap but they are "cheap at the price" for large companies. The ".COM" DNS server just points to a special Akami DNS server whenever someone asks about ".GOOGLE.COM" (or any of the others). Akami fakes things up on the fly so that people asking for the IPv4 address of WWW.GOOGLE.COM get a different answer if they are asking from a computer located in Europe than they get if they are asking from a computer located in San Francisco, which is close to Google's US headquarters
Let me finish up with several quick items.
DNS entries are not case sensitive. So www.google.com is the same as WWW.GOOGLE.COM is the same as WwW.gOoGle.CoM, or any other pattern of capitalization that floats your boat.
IPv6 - DNS servers can handle a mixture of IPv4 and IPv6. A new record type has been added for IPv6. The "A" record is now an "AAAA" record. The data part of an AAAA record contains an IPv6 address. As far as I can tell the PTR record has been extended to handle IPv6 without requiring a new record type. But I haven't really looked into this.
You can directly query DNS servers by using a command called NSLOOKUP. You can find information on it on the web, if you are interested.
If you run a Windows PC you can execute the IPCONFIG command in a "CMD" box. "ipconfig /all" will tell you more than you probably wanted to know about various network settings.
If you poke around you will probably find that a "primary" and a "backup" DNS server are configured on your machine? Why? Because DNS is so critical. The only time the backup entry is used is when there is a problem using the primary entry.
DNS server entries are always specified as an IP address (usually IPv4 but possibly IPv6). Why? Because if DNS is not working how do you turn a name into an IP address in order to know where to go to get DNS information?
Finally, where do the addresses for a primary and backup DNS server come from? In most cases "magic". I will get into this a little more in a future post.
What is DNS and why do we care? Computers love numbers. People love text. So people like dealing with something like www.google.com rather than 172.25.11.23 (not Google's actual IP address). DNS, short for Domain Name Server, is what gets us from here to there. The thing that does this is called a DNS server. In the earliest days of ARPANET there were only a few computers to keep track of. A simple list worked fine. But now there are "millions and billions" of web sites, mobile applications, etc., on the 'net. We have long since moved past the time when a list could get the job done.
This was recognized fairly early in the transition from that first ARPANET to the modern Internet. The first practical and comprehensive solution was called BIND - the Berkley Internet Name Daemon. The University of California at Berkley Computer Science department was an early and active participant in Unix, the "C" computer language, and the Internet. They developed a lot of tools and enhancements and one of them was BIND. BIND is still around although responsibility has been turned over to the Internet Systems Consortium. You can find out more at https://www.isc.org/downloads/bind/. But most of us only need to use DNS servers. We don't need to install or operate our own DNS server. But let's find out a little about how they work anyway.
DNS is not a single server. It is a swarm of cooperating servers. And their job is to turn something like www.google.com into something like 172.25.11.23. So how do they do this? A DNS server is mostly just a simple database. You ask questions. It provides answers. Actually it serves up the data from records that are appropriate responses to our query. The most popular record type is the "A" for Address record. If a DNS server has an A record that has a key of www.google.com it serves up the data value (172.25.11.23, in our example). But in many cases it does not have an A record that is an exact match. What then?
Would it surprise you to know that, as with a lot of Internet things, DSN is into this whole delegation thing. Various servers provide part of the answer then pass things on to a server that knows more. So all blind queries (queries that are not in the wheelhouse of whichever DNS server we are using) are passed on to a "root" server. Where are the "root" servers? That' one of those Internet "well known" things. There is an official list of root servers and you can find out what they are if you know where to look. Anyone who messes with DNS servers knows where this list is found and it is plugged into each DNS server. So what our server does is pass the question on to a root server.
And this whole delegation thing proceeds from back to front. The only thing root servers know about is where that last part is. You know, the ".COM" or ".EDU" or ".GOV" or whatever thing. The root servers don't care about the rest of it. They know about that last part and where to send you to find out more information. They have a list of DNS servers that specialize in ".COM" DNS entries, for instance. So they just pass any ".COM" queries along to a ".COM" DNS server. (We don't need to know where the ".COM" DNS servers are because the root servers know.) This server knows about a whole bunch of "anything.COM" things. It has no clue about "anything.EDU" or "anything.GOV" or "anything.whatever" if "whatever" is anything but ".COM". But different servers do know about ".EDU", for instance. The root server passes queries about "anything.EDU" to one of them. The same is true for ".GOV", ".NET", and so on.
There is an official list of all the legal top levels. (They are the last part of the name but they are called "top" levels because they represent the top of the tree of interconnected DNS servers.) That's because every root server needs entries for all of them. The servers that handle ".COM", ".NET", etc. are called "top level servers". Originally there were only a few top levels. But the criteria has been loosened up several times now. Why? Well, there is a little work necessary to get a new top level to work. But the main reason has been a concern over confusion. But people are now pretty comfortable with things like ".CO" or ".TV" or ".US" so there now seems little reason to keep the list short. Also, the Internet has slowly become more international and part of that has been support for various languages. There are now top levels in Chinese and Arabic, for instance, languages that don't even use our standard alphabet. Returning to our example, hopefully the ".COM" DNS servers have an entry for ".GOOGLE.COM".
But that entry is just the name of another DNS server. This server knows about "anything.GOOGLE.COM". It knows nothing about "anything.MICROSOFT.COM" or "anything.FACEBOOK.COM". And it especially knows nothing about "WH.GOV" (different top level). So this ".GOOGLE.COM" DNS server has (again, if everything is working ok) an entry for WWW.GOOGLE.COM. It serves it up to your computer and your computer finally knows what IPv4 address to send its traffic to. Lots of messages are exchanged between lots of DNS servers. But that's traffic that happens in the background where you don't see it. And it is perfectly legal to go more levels. The process just starts at the top level and works its way through the layers until it gets to the end at whatever level that might be.
That's the standard way things work. I am now going to talk about a couple of variations. Let's say we ask our local DNS server about WWW.FACEBOOK.COM. The first time through it goes through the elaborate process I described above. But we often go to the same place many times. So DNS servers have a cache. This is a place where the answers to recent questions are saved. For any query that is outside the direct responsibility of that DNS server the query is first checked against the cache before the elaborate process described above is undertaken.
If it finds an entry for WWW.FACEBOOK.COM in the cache (because somebody asked about it recently) it just sends a "non-authorative" answer back. Why non-authoritative? Because only one or a few DNS servers are designated as the official respositories for WWW.FACEBOOK.COM DNS information. They are the "authoritative" DNS servers. All DNS entries in the cache of non-authoritative servers are also marked with a "time to live", usually a few days. If Google wants to move their servers around they will need to update the DNS entries for these servers. Confusion will reign if the old information does not eventually flush out of the Internet.
Another non-standard thing is an ALIAS record. Let's say your company buys another company and you want to merge web sites. You could just shut one down but that would mean that customers who had links to the shut down site would be stuck. The ALIAS record solves that. You just put an ALIAS record in the appropriate DNS servers. The ALIAS record says "if someone asks about WWW.X.COM answer the question as if they had actually asked about WWW.Y.COM instead". This way everything that was linked to the old location gets automatically connected up with the new location. ALIAS records actually have a lot more uses than this but that's enough to give you the idea.
And it turns out that there are many different kinds of DNS records. I am only going to talk about two more. One is the MX record. You can ask a DNS server "what's the IP address of the mail server for WWW.GOOGLE.COM?" The DNS server will go looking for MX records instead of A records. This kind of thing is handy in a number of ways. Have you noticed that sometimes companies leave the "WWW" off. DNS tricks like these allow computers to find the same web site either way. Try just "GOOGLE.COM" sometime. It gets you to the same place as you would get to by putting the "WWW." on the front.
The other record I want to talk about is the PTR record. "A" records get you from the name to the IPv4 address. PTR records get you back the other way. Not everybody sets up PTR records all the time but in a lot of cases people do. If so you can find out what the text name that goes with an IP address is. This turns out to be tricky. To make this work you need to know the net/subnet boundaries. But it is literally impossible, in general, to know what they are. So what to do? The PTR system assumes that IPv4 addresses are broken on octet boundaries. And, to allow for this delegation thing to work, we have to go at things backwards. 172.25.11.23 is temporarily turned around. We pretend for a minute it really is 23.11.25.172. That's because we want to look the "172" part up first, and so on. Cutting to the chase, instead of eventually looking for "WWW" in the ".GOOGLE.COM" DNS server we look for "23" in the "11.23.172" DNS server. If you are not a little confused at this point I have failed.
But let me see if I can confuse you even more. We have these authoritative DNS servers. Why? So everyone everywhere who asks a specific DNS question gets the exact same answer. But what if we want different people to get different answers? Google has server farms all over the place. They want to send you to one of their local farms rather than requiring you to gallivant across the country or the world to get to a Google server. A number of other large companies like Microsoft, Facebook, and Amazon want to do the same thing. There's the trick.
What if we have a super-DNS server that not only looks at the question but looks at the IP address of the computer asking the question? If we know the general location of the computer making the DNS query (you can usually get at least an OK idea by doing a lot of research) then you can serve up an IPv4 address answer that is for a server that is in a server farm that is close (in terms of the Internet) to the computer generating the query. There is a company called Akami that does just that.
Their services are not cheap but they are "cheap at the price" for large companies. The ".COM" DNS server just points to a special Akami DNS server whenever someone asks about ".GOOGLE.COM" (or any of the others). Akami fakes things up on the fly so that people asking for the IPv4 address of WWW.GOOGLE.COM get a different answer if they are asking from a computer located in Europe than they get if they are asking from a computer located in San Francisco, which is close to Google's US headquarters
Let me finish up with several quick items.
DNS entries are not case sensitive. So www.google.com is the same as WWW.GOOGLE.COM is the same as WwW.gOoGle.CoM, or any other pattern of capitalization that floats your boat.
IPv6 - DNS servers can handle a mixture of IPv4 and IPv6. A new record type has been added for IPv6. The "A" record is now an "AAAA" record. The data part of an AAAA record contains an IPv6 address. As far as I can tell the PTR record has been extended to handle IPv6 without requiring a new record type. But I haven't really looked into this.
You can directly query DNS servers by using a command called NSLOOKUP. You can find information on it on the web, if you are interested.
If you run a Windows PC you can execute the IPCONFIG command in a "CMD" box. "ipconfig /all" will tell you more than you probably wanted to know about various network settings.
If you poke around you will probably find that a "primary" and a "backup" DNS server are configured on your machine? Why? Because DNS is so critical. The only time the backup entry is used is when there is a problem using the primary entry.
DNS server entries are always specified as an IP address (usually IPv4 but possibly IPv6). Why? Because if DNS is not working how do you turn a name into an IP address in order to know where to go to get DNS information?
Finally, where do the addresses for a primary and backup DNS server come from? In most cases "magic". I will get into this a little more in a future post.
Sunday, September 27, 2015
Internet - Routing and NAT
This is the fifth post in this series. I recommend reading them in sequence. The first one is at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html. The immediately previous post is at http://sigma5.blogspot.com/2015/09/internet-classescidripv6.html. In this post I am going to attack two subjects. The first is how routing works on the public internet. And the second is NAT - Network Address Translation. The two subjects turn out to be interconnected. Onward.
I have touched briefly on the subject of routing in an earlier post. Two computers are local, in the IPv4 sense, if they are both in the same "net". You apply the subnet mask to the computer's IPv4 address and extract the net part. You do the same thing to the IPv4 address of the "to" computer, the one the message is being sent to. If the net part of both IPv4 addresses is the same then the computers are local and the message is sent directly to the "to" computer. Any IPv4 address that is not local is, by definition, remote. What happens in this case, as far as our computer is concerned, is simple. The message is just sent to the gateway. But what does the gateway do?
Theoretically there this specific thing called a "gateway" and there is this other different specific thing called a "router". As I have pointed out in several places previously, roles now frequently get mixed together in the modern Internet. That's true of gateways and routers. A router's job is to route messages along toward their destination. It turns out that, to some extent, a gateway does the same thing.
All gateways have at least two interfaces. One is connected to the local net so that it can catch traffic that is intended specifically for it. But that same interface also catches gateway traffic. Once the gateway has caught the message, which is marked as a gateway message, it does its gateway thing. First it looks at its other interfaces. There may be several but, in most cases there is just one. Each of these other interfaces has a net associated with it. Let's say the message destination is within that net. Then the gateway gets rid of the extra gateway stuff in the message and sends it directly to its destination using the appropriate interface. But what if there is no match? Then, if the box is actually just a simple gateway, the box looks up its own gateway address and forwards the message along.
Now consider a pure router. It typically will have several interfaces. Again each will have a net associated with it. If the "to" of the message is a match to any of these interfaces it behaves just like the gateway. It strips off the gateway stuff and sends it directly on to its destination. So far that's just the same as a gateway. But what if we have a miss? What if the "to" IPv4 address doesn't match anything. Then what? Here is where router behavior differs from gateway behavior. A router has a routing table. The routing table has a bunch of net-like entries. The entry says something like "if the 'to' IPv4 address fits the net specification for this entry send the message out interface 4". A properly constructed routing table has at least one entry that matches any IP address.
There are some special ones that say "If you match this entry just throw the message away". These entries are used to handle 0.0.0.0/8, 127.0.0.0/8, 255.0.0.0/8 and other similar cases. There is usually a "default gateway" type entry that says "if you otherwise don't know where to send it, send it here". But basically what a router does is get a message, look it's "to" IPv4 address up in the routing table, and send it out using whatever interface the routing table specified. That's it. But how does the routing table get set up and maintained? That's where the magic is.
There are a number of "discovery" protocols used to create and maintain routing tables. Each router knows what nets it is directly connected to. It plugs that information into its routing table. But it can also use one of these protocols to send the information to all the routers it is in contact with. And yes, there is a protocol for routers finding each other. More than that, a router builds up information on what its nearest neighbor routers can access. It sends that along too. And so on. So a router can eventually find out what routers 3, 5, 10, etc. hops away have access to. The details of the several systems for doing this vary but they all perform the same basic function. Special procedures have to be used when one router uses one system and another router uses another. But that too has been worked out.
So theoretically we wait a while and the router knows where everything on the Internet is. But the Internet is redundant so there are usually two or more paths from point A to point B. Each of the routing table maintenance systems uses a different "weighting" system. A simple "metric" is hop count. If the destination is three hops away using one path and two hops away using another path, go with the shorter path. The weighting function can get quite complex but that gives you the idea.
And that's how the Internet actually worked before it became the Internet. Back then the federal government owned it all and paid for it all. There was no particular reason to prefer one route over another so the systems just fought it out and as long as the message got delivered everyone was happy. But now all the pieces of the Internet are in private hands. Companies want to minimize costs and maximize revenue. All of a sudden path B might look a lot better to the bottom line than path A. So a lot of the components of the old system are still in place but a lot of biases have been added in to make sure the answer comes out the way the company wants it to.
Routing tables are no longer just built automatically by using some kind of discovery process. A lot of manual effort now goes into their construction. And the details of their construction are closely guarded secrets. So I don't think anyone now knows all the details. But here's a guess that I think is pretty close.
Let's construct a simple model of the Internet. You have your local network, say at home or at work. For the moment we will assume that all your computers have "public" Internet addresses. (I'll get into what that means below.) So you have some computers on a local net using Ethernet. One of these computers is the gateway. The gateway has two interfaces. One is connected to your local network. The other is connected to an ISP, an Internet Service Provider. If the message is going to another computer in the ISP's coverage area the ISP shoots it over to that other gateway and it disappears into the local network where it is delivered.
But let's say the "to" computer is in the control area of a different ISP. For simplicity sake, I am going to assume that other ISP is not directly connected to our ISP. In this case the ISP passes the message along to what I am going to call a long haul service provider. There are a number of these LSPs and the actual situation is much more complicated but stay with me. There are routers all over the place. Some belong to one LSP. Some belong to another. The LSP's job is to see that everything gets delivered. But the more traffic he can dump on to some other LSP's equipment (routers, long haul Internet circuits, etc.), the lower the first LSP's costs are. But then, to make things complicated, there are companies like Netflix that are willing to pay for premium service. So we have competing agendas.
What happens is that the ISPs and the LSPs have people called traffic managers. Their job is to firstly make it all work. But beyond that they need to arrange things in the manner most advantageous to their employer. So they spend a lot of time figuring out what path is the "best" (for their employer's objectives) path. They then ship out routing table data to make it so.
You can see this kind of thing in action for yourself. I did some experimenting with a command called TRACERT. It tells you the names of all the routers in the path between "from" and "to. If I trace the route of a message to Boston College a company called Level 3 ferries the message across country. But a message to the University of Massachusetts, a school located in the same city, uses Northern Telecom. Why?
My theory is that the ISP that Boston College contracts with subcontracts its long haul needs to Level 3. Whereas the ISP that U Mass contracts with subcontracts its needs to Northern Telecom. Am I certain of this? No. But What I do know is that two messages going to almost exactly the same place take wildly different routes. And, by the way, all traffic to the same destination seems to always take exactly the same route.
I also know that routing tables have gotten extremely complicated. There was an outage a year or so ago that was caused by some routing tables exceeding an approximately 30,000 entry limit. I also have a neighbor that does this kind of thing for one of the telephone companies. He has told me a couple of war stories about tweaking certain traffic to go this way instead of that way. Why? Because somebody was writing a big check to his employer.
Anyhow the magical thing is that somehow it works. The routers with their routing tables do get out messages wherever we want. And "wherever" can be anyplace in the world. It's pretty magical. And fortunately we don't have to have anything to do with routers or routing tables. That's definitely something to be grateful for. And that brings me to NAT.
Your little home network and mine don't actually use public IPv4 addresses. They use private ones. Let's start out with the basics. Fortunately for us someone came up with a brilliant idea a while ago. "Let's carve out some special IPv4 nets", this person said. These nets will be specifically designed to NOT work on the public Internet. Instead they will be "internal use only" nets. The official name for these babies is private IPv4 nets. Since these nets don't work on the public Internet they can be used over and over. You don't need to worry if someone else is already using the net you want to use. Their use does not interfere in any way with you using the exact same private net.
And at least one net was set up for each "class" size. So we can pick which private net to use in a specific situation based on how many IPv4 addresses we need. This was done back in the old "class" days so these reserved private nets follow the old rules. The private nets are:
Class A - 10.0.0.0/8 (1)
Class B - 172.16.0.0/16 through 172.31.0.0/16 (16)
Class C - 192.168.0.0/24 through 192.168.255.0/16 (256)
Many home networks, including mine, use 192.168.0.0/24. The company I used to work for uses a number of class C private nets but it also uses several class B private nets. (It pretty much doesn't use its public class C nets at all but it turns out to be hard to give Cs back.) These IPv4 nets are carefully selected because they don't work on the public Internet. But many of the devices using these private addresses need and get access to the Internet. How is all that possible?
The answer is NAT. And to understand what is going on you need to know a little more technical detail. When you send a message to a computer you don't just simply send it to that computer. You send it to a specific port that belongs to a specific protocol type. That's the foundation of the trick.
There are several protocol types. But the only ones we care about are UDP and TCP. (Now we finally get around to the discussion of TCP I promised you in the second post in the series.) The protocol rules for the two are different. TCP is like making a telephone call. You set up the call (dial - answer). Then you talk back and forth. When you are done you both hang up. TCP works the same way. You set up a session. Then you exchange messages. Then you tear down the session. TCP also makes sure the messages are delivered and in the correct order.
UDP is often called a "datagram" service. It is like exchanging telegrams (or, in the modern era, Tweets). Each message is a stand alone entity. It has to have a full set of from/to information. And there is no guarantee a message will be delivered or, if two messages are delivered, they will be delivered in the correct sequence. There are advantages and disadvantages to each. The point is that each computer has a set of 64,000 TCP ports and a different set of 64,000 UDP ports. A message not only goes to a specific IPv4 address but it goes to a specific port belonging to a specific protocol type. If any one of these three things is different (IPv4 address, protocol type, port number) it is a path to a different place.
NAT takes advantage of this. Your NAT box has two interfaces, an "outside" interface and an "inside" interface. Let's say you have one public IPv4 internet address. It gets attached to the outside interface. Then an IPv4 address from your private network is attached to the inside port. And, by the way, your NAT box is also your gateway box. So now your computer, which has a different IPv4 address, one from your private network, is hooked to your internal network. And lets say all the boxes on your private network are connected together using Ethernet technology. This is all probably sounding pretty familiar by now.
So you want to send a message, say a Google query from the browser on your computer. Well, that message needs to get to the Google server somehow. What happens? Well, Google's server is not on your local net so the message gets sent to the your gateway, which also happens to be your NAT box. The message comes in on the inside interface and is marked as a gateway situation. It is also marked with the eventual destination of Google. And it's a web message. That means it uses the TCP protocol type and it is supposed to go to TCP port 80. (I know this because the Internet has all these "well known" things and I know where to look them up.) But the point is that the NAT box knows that your computer generated a TCP message from port 80 on your computer to port 80 on the Google computer.
What it then does is pick a different TCP port on its outside interface and fix the message up so that that it looks like exactly the same TCP message came for this other port number and from the outside interface's IPv4 address. It still sends it to the Google server and to TCP port 80. It also remembers that it made this specific translation for this specific session. When the response comes back from the Google server it is addressed to the IPv4 address of the external interface of the NAT box. It is also addressed to a specific funny TCP port on that interface. The NAT box looks all this up and says "Oh - I need to translate things back so that the message goes to the IPv4 address of your computer and to TCP port 80". So a fixed up version of the message from Google shoots out the NAT box's internal interface destined for your computer.
The NAT process is "stateful". It keeps a lot of specific information about the state of each connection. And it maintains separate state information for each connection. This way it can handle multiple web sesstions at the same time or multiple sessions of different kinds from the same computer. The entire process is very complicated. Fixing everything up so nobody is the wiser can be very complicated in some situations. Newer protocols are now carefully designed so that they are easy to NAT, the complexity of the translation process is kept to a minimum. But that's the basic idea.
The NAT box is able to fix everything up so that all the traffic in the public part of the Internet appears to come from the same single public IPv4 address. Computers on your local network think they are talking directly to the boxes out on the Internet with nothing funny happening along the way. What makes all this possible is that there are tens of thousands of port numbers available and that only a few of them are in use at any one time.
And the best news of all is that all this complexity is taken care of for us by the NAT box. You just turn it on and it magically works. And what NAT has done is to drastically reduce the need for public IPv4 addresses. I have a number of different kinds of boxes at home. Most of them need Internet access at least once in a while. This is all easily accommodated by my home NAT box (which is also an Ethernet switch) so everything works fine even though I only have one IPv4 address available to handle all of it.
My old company actively uses several public IPv4 addresses. But they had more than 1,500 devices when I retired, and that was several years ago, and a goodly percentage of them needed Internet access, at least occasionally. And the same is true up and down the line. NAT boxes and private nets are now the norm. It has gotten to the point where they are often more convenient than using public IPv4 addresses would be. By switching to NAT and private networks Microsoft was able to abandon at least one class B so that it could be added back to the CIDR pool.
A final note on these private nets. Above I indicated that routers have routing table entries that throw traffic from certain nets like 0.0.0.0/8 away. Well, they also have "throw it away" entries for the private net addresses. There is an entry for 10.0..00/8 and for 172.16/0.0/12 (check it out - it catches all of them in one entry) and for 192.168.0.0/16 (same here).
And a quick note on IPv6, things work in pretty much the same way as they do with IPv4. But the Internet handles IPv4 and IPv6 traffic separately. It may go down the same wire and it may be processed by the same router. But both the wire and the router behave pretty much like they are two wires and two routers. There is separate logic in the router for handling IPv4 and IPv6 traffic. The general approach is the same and both kinds of traffic may come in or go out on the same interface but the two routing tables are completely separate. Inside the router the traffic is completely segregated.
There's not much in this post that average people will use. But I hope it clears up how things work and provides some perspective that is helpful when wrestling with the parts of networking that you are forced to deal with.
I have touched briefly on the subject of routing in an earlier post. Two computers are local, in the IPv4 sense, if they are both in the same "net". You apply the subnet mask to the computer's IPv4 address and extract the net part. You do the same thing to the IPv4 address of the "to" computer, the one the message is being sent to. If the net part of both IPv4 addresses is the same then the computers are local and the message is sent directly to the "to" computer. Any IPv4 address that is not local is, by definition, remote. What happens in this case, as far as our computer is concerned, is simple. The message is just sent to the gateway. But what does the gateway do?
Theoretically there this specific thing called a "gateway" and there is this other different specific thing called a "router". As I have pointed out in several places previously, roles now frequently get mixed together in the modern Internet. That's true of gateways and routers. A router's job is to route messages along toward their destination. It turns out that, to some extent, a gateway does the same thing.
All gateways have at least two interfaces. One is connected to the local net so that it can catch traffic that is intended specifically for it. But that same interface also catches gateway traffic. Once the gateway has caught the message, which is marked as a gateway message, it does its gateway thing. First it looks at its other interfaces. There may be several but, in most cases there is just one. Each of these other interfaces has a net associated with it. Let's say the message destination is within that net. Then the gateway gets rid of the extra gateway stuff in the message and sends it directly to its destination using the appropriate interface. But what if there is no match? Then, if the box is actually just a simple gateway, the box looks up its own gateway address and forwards the message along.
Now consider a pure router. It typically will have several interfaces. Again each will have a net associated with it. If the "to" of the message is a match to any of these interfaces it behaves just like the gateway. It strips off the gateway stuff and sends it directly on to its destination. So far that's just the same as a gateway. But what if we have a miss? What if the "to" IPv4 address doesn't match anything. Then what? Here is where router behavior differs from gateway behavior. A router has a routing table. The routing table has a bunch of net-like entries. The entry says something like "if the 'to' IPv4 address fits the net specification for this entry send the message out interface 4". A properly constructed routing table has at least one entry that matches any IP address.
There are some special ones that say "If you match this entry just throw the message away". These entries are used to handle 0.0.0.0/8, 127.0.0.0/8, 255.0.0.0/8 and other similar cases. There is usually a "default gateway" type entry that says "if you otherwise don't know where to send it, send it here". But basically what a router does is get a message, look it's "to" IPv4 address up in the routing table, and send it out using whatever interface the routing table specified. That's it. But how does the routing table get set up and maintained? That's where the magic is.
There are a number of "discovery" protocols used to create and maintain routing tables. Each router knows what nets it is directly connected to. It plugs that information into its routing table. But it can also use one of these protocols to send the information to all the routers it is in contact with. And yes, there is a protocol for routers finding each other. More than that, a router builds up information on what its nearest neighbor routers can access. It sends that along too. And so on. So a router can eventually find out what routers 3, 5, 10, etc. hops away have access to. The details of the several systems for doing this vary but they all perform the same basic function. Special procedures have to be used when one router uses one system and another router uses another. But that too has been worked out.
So theoretically we wait a while and the router knows where everything on the Internet is. But the Internet is redundant so there are usually two or more paths from point A to point B. Each of the routing table maintenance systems uses a different "weighting" system. A simple "metric" is hop count. If the destination is three hops away using one path and two hops away using another path, go with the shorter path. The weighting function can get quite complex but that gives you the idea.
And that's how the Internet actually worked before it became the Internet. Back then the federal government owned it all and paid for it all. There was no particular reason to prefer one route over another so the systems just fought it out and as long as the message got delivered everyone was happy. But now all the pieces of the Internet are in private hands. Companies want to minimize costs and maximize revenue. All of a sudden path B might look a lot better to the bottom line than path A. So a lot of the components of the old system are still in place but a lot of biases have been added in to make sure the answer comes out the way the company wants it to.
Routing tables are no longer just built automatically by using some kind of discovery process. A lot of manual effort now goes into their construction. And the details of their construction are closely guarded secrets. So I don't think anyone now knows all the details. But here's a guess that I think is pretty close.
Let's construct a simple model of the Internet. You have your local network, say at home or at work. For the moment we will assume that all your computers have "public" Internet addresses. (I'll get into what that means below.) So you have some computers on a local net using Ethernet. One of these computers is the gateway. The gateway has two interfaces. One is connected to your local network. The other is connected to an ISP, an Internet Service Provider. If the message is going to another computer in the ISP's coverage area the ISP shoots it over to that other gateway and it disappears into the local network where it is delivered.
But let's say the "to" computer is in the control area of a different ISP. For simplicity sake, I am going to assume that other ISP is not directly connected to our ISP. In this case the ISP passes the message along to what I am going to call a long haul service provider. There are a number of these LSPs and the actual situation is much more complicated but stay with me. There are routers all over the place. Some belong to one LSP. Some belong to another. The LSP's job is to see that everything gets delivered. But the more traffic he can dump on to some other LSP's equipment (routers, long haul Internet circuits, etc.), the lower the first LSP's costs are. But then, to make things complicated, there are companies like Netflix that are willing to pay for premium service. So we have competing agendas.
What happens is that the ISPs and the LSPs have people called traffic managers. Their job is to firstly make it all work. But beyond that they need to arrange things in the manner most advantageous to their employer. So they spend a lot of time figuring out what path is the "best" (for their employer's objectives) path. They then ship out routing table data to make it so.
You can see this kind of thing in action for yourself. I did some experimenting with a command called TRACERT. It tells you the names of all the routers in the path between "from" and "to. If I trace the route of a message to Boston College a company called Level 3 ferries the message across country. But a message to the University of Massachusetts, a school located in the same city, uses Northern Telecom. Why?
My theory is that the ISP that Boston College contracts with subcontracts its long haul needs to Level 3. Whereas the ISP that U Mass contracts with subcontracts its needs to Northern Telecom. Am I certain of this? No. But What I do know is that two messages going to almost exactly the same place take wildly different routes. And, by the way, all traffic to the same destination seems to always take exactly the same route.
I also know that routing tables have gotten extremely complicated. There was an outage a year or so ago that was caused by some routing tables exceeding an approximately 30,000 entry limit. I also have a neighbor that does this kind of thing for one of the telephone companies. He has told me a couple of war stories about tweaking certain traffic to go this way instead of that way. Why? Because somebody was writing a big check to his employer.
Anyhow the magical thing is that somehow it works. The routers with their routing tables do get out messages wherever we want. And "wherever" can be anyplace in the world. It's pretty magical. And fortunately we don't have to have anything to do with routers or routing tables. That's definitely something to be grateful for. And that brings me to NAT.
Your little home network and mine don't actually use public IPv4 addresses. They use private ones. Let's start out with the basics. Fortunately for us someone came up with a brilliant idea a while ago. "Let's carve out some special IPv4 nets", this person said. These nets will be specifically designed to NOT work on the public Internet. Instead they will be "internal use only" nets. The official name for these babies is private IPv4 nets. Since these nets don't work on the public Internet they can be used over and over. You don't need to worry if someone else is already using the net you want to use. Their use does not interfere in any way with you using the exact same private net.
And at least one net was set up for each "class" size. So we can pick which private net to use in a specific situation based on how many IPv4 addresses we need. This was done back in the old "class" days so these reserved private nets follow the old rules. The private nets are:
Class A - 10.0.0.0/8 (1)
Class B - 172.16.0.0/16 through 172.31.0.0/16 (16)
Class C - 192.168.0.0/24 through 192.168.255.0/16 (256)
Many home networks, including mine, use 192.168.0.0/24. The company I used to work for uses a number of class C private nets but it also uses several class B private nets. (It pretty much doesn't use its public class C nets at all but it turns out to be hard to give Cs back.) These IPv4 nets are carefully selected because they don't work on the public Internet. But many of the devices using these private addresses need and get access to the Internet. How is all that possible?
The answer is NAT. And to understand what is going on you need to know a little more technical detail. When you send a message to a computer you don't just simply send it to that computer. You send it to a specific port that belongs to a specific protocol type. That's the foundation of the trick.
There are several protocol types. But the only ones we care about are UDP and TCP. (Now we finally get around to the discussion of TCP I promised you in the second post in the series.) The protocol rules for the two are different. TCP is like making a telephone call. You set up the call (dial - answer). Then you talk back and forth. When you are done you both hang up. TCP works the same way. You set up a session. Then you exchange messages. Then you tear down the session. TCP also makes sure the messages are delivered and in the correct order.
UDP is often called a "datagram" service. It is like exchanging telegrams (or, in the modern era, Tweets). Each message is a stand alone entity. It has to have a full set of from/to information. And there is no guarantee a message will be delivered or, if two messages are delivered, they will be delivered in the correct sequence. There are advantages and disadvantages to each. The point is that each computer has a set of 64,000 TCP ports and a different set of 64,000 UDP ports. A message not only goes to a specific IPv4 address but it goes to a specific port belonging to a specific protocol type. If any one of these three things is different (IPv4 address, protocol type, port number) it is a path to a different place.
NAT takes advantage of this. Your NAT box has two interfaces, an "outside" interface and an "inside" interface. Let's say you have one public IPv4 internet address. It gets attached to the outside interface. Then an IPv4 address from your private network is attached to the inside port. And, by the way, your NAT box is also your gateway box. So now your computer, which has a different IPv4 address, one from your private network, is hooked to your internal network. And lets say all the boxes on your private network are connected together using Ethernet technology. This is all probably sounding pretty familiar by now.
So you want to send a message, say a Google query from the browser on your computer. Well, that message needs to get to the Google server somehow. What happens? Well, Google's server is not on your local net so the message gets sent to the your gateway, which also happens to be your NAT box. The message comes in on the inside interface and is marked as a gateway situation. It is also marked with the eventual destination of Google. And it's a web message. That means it uses the TCP protocol type and it is supposed to go to TCP port 80. (I know this because the Internet has all these "well known" things and I know where to look them up.) But the point is that the NAT box knows that your computer generated a TCP message from port 80 on your computer to port 80 on the Google computer.
What it then does is pick a different TCP port on its outside interface and fix the message up so that that it looks like exactly the same TCP message came for this other port number and from the outside interface's IPv4 address. It still sends it to the Google server and to TCP port 80. It also remembers that it made this specific translation for this specific session. When the response comes back from the Google server it is addressed to the IPv4 address of the external interface of the NAT box. It is also addressed to a specific funny TCP port on that interface. The NAT box looks all this up and says "Oh - I need to translate things back so that the message goes to the IPv4 address of your computer and to TCP port 80". So a fixed up version of the message from Google shoots out the NAT box's internal interface destined for your computer.
The NAT process is "stateful". It keeps a lot of specific information about the state of each connection. And it maintains separate state information for each connection. This way it can handle multiple web sesstions at the same time or multiple sessions of different kinds from the same computer. The entire process is very complicated. Fixing everything up so nobody is the wiser can be very complicated in some situations. Newer protocols are now carefully designed so that they are easy to NAT, the complexity of the translation process is kept to a minimum. But that's the basic idea.
The NAT box is able to fix everything up so that all the traffic in the public part of the Internet appears to come from the same single public IPv4 address. Computers on your local network think they are talking directly to the boxes out on the Internet with nothing funny happening along the way. What makes all this possible is that there are tens of thousands of port numbers available and that only a few of them are in use at any one time.
And the best news of all is that all this complexity is taken care of for us by the NAT box. You just turn it on and it magically works. And what NAT has done is to drastically reduce the need for public IPv4 addresses. I have a number of different kinds of boxes at home. Most of them need Internet access at least once in a while. This is all easily accommodated by my home NAT box (which is also an Ethernet switch) so everything works fine even though I only have one IPv4 address available to handle all of it.
My old company actively uses several public IPv4 addresses. But they had more than 1,500 devices when I retired, and that was several years ago, and a goodly percentage of them needed Internet access, at least occasionally. And the same is true up and down the line. NAT boxes and private nets are now the norm. It has gotten to the point where they are often more convenient than using public IPv4 addresses would be. By switching to NAT and private networks Microsoft was able to abandon at least one class B so that it could be added back to the CIDR pool.
A final note on these private nets. Above I indicated that routers have routing table entries that throw traffic from certain nets like 0.0.0.0/8 away. Well, they also have "throw it away" entries for the private net addresses. There is an entry for 10.0..00/8 and for 172.16/0.0/12 (check it out - it catches all of them in one entry) and for 192.168.0.0/16 (same here).
And a quick note on IPv6, things work in pretty much the same way as they do with IPv4. But the Internet handles IPv4 and IPv6 traffic separately. It may go down the same wire and it may be processed by the same router. But both the wire and the router behave pretty much like they are two wires and two routers. There is separate logic in the router for handling IPv4 and IPv6 traffic. The general approach is the same and both kinds of traffic may come in or go out on the same interface but the two routing tables are completely separate. Inside the router the traffic is completely segregated.
There's not much in this post that average people will use. But I hope it clears up how things work and provides some perspective that is helpful when wrestling with the parts of networking that you are forced to deal with.
Subscribe to:
Posts (Atom)