DHCP and a wrap up

This is really, really, really, going to be the last post in this series, for now.  At this point I have no plans to add anything.  But who knows what the future will bring.  All of these posts can be found in the September, 2015 section the of my blog.  (See Blog Archive in the navigation section at the right.)  But, for consistency sake, here is a link to the first post in the series:  http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.  And here is a link to the immediately previous post:  http://sigma5.blogspot.com/2015/09/internet-dns.html.  So . . .

DHCP

DHCP stands for Dynamic Host Configuration Protocol.  This mouthful is almost completely meaningless so what's up?  Back in the olden days IP configurations were set up manually.  When my company first rolled out TCP/IP that's what we did.  Several of us maintained extensive lists of which computer would use which IP address.  Besides being tedious and error prone it had a problem.  You had to go to each computer and plug the "static IP configuration" parameters in individually.  That was bad enough but my company had facilities in several states.  So if we wanted to change things up on a computer in another state we had to actually send someone there at a great cost in time and money.

And, hey, this is a computer thing.  Why not automate things?  Early attempts went by the names RARP and BOOTP.  In these cases you plugged everything into one computer.  That computer served the information out to the other computers.  But it was still a lot of trouble.  My company did not use either to any extent.  Then along came DHCP.  This helped a lot.  DHCP added the "dynamic" part.  Instead of having a list of computers and IPv4 addresses DHCP let you use a pool.  With DHCP you allocated a group of IPv4 addresses to the DHCP server and said "go for it".

DHCP depends on Ethernet or some other protocol that supports a the ability to broadcast to all other local computers.  Your computer sends out an "are you a DHCP server?" broadcast message.  Hopefully, exactly one machine answers back.  Then your computer sends a message directly to that computer that says "my name is X - do you have IP configuration information for me?".  The DHCP server looks in its mini-database.  If it already has an entry for that computer it sends out the information.  If not, it picks an unused IPv4 address from the pool, adds in whatever other information it is supposed to provide,  plugs that into the mini-database and sends it out.  Your computer catches the information, plugs it in, and boom! it is on the air.

Here's a little more detail.  Each entry in the mini-database has a time to live.  It can vary from a few months to a few hours, depending on your estimate of the rate of turnover.  In a coffee shop where turnover is quick a few hours is probably a good choice.  In a corporate environment where turnover is slow, a few months is a good choice.  If an entry goes unused too long it is purged and the IPv4 address goes back into the "available" pool.  The other thing is that the DHCP server can send out more than just an IPv4 address for the computer to use.  Typically it sends out the subnet mask, the gateway, and the addresses of the primary and backup DNS servers.  That's the standard set of "IP configuration" information machines need.

By shifting to DHCP we just had to set up a DHCP server at each location and make sure it had a big enough pool of IPv4 addresses to work with.  At this time we also converted from using our 5 class "C" public IPv4 nets to using private IPv4 nets so having lots of IPv4 addresses available at each location was easy to arrange.  Now we just had to make sure each machine had a unique name.  That was much easier to pull off.  Then when the machine booted up it would find the local DHCP server and get the information it needed to operate at that location.  This is usually referred to as a "dynamic IP configuration" for obvious reasons.  We might very occasionally need to change the DHCP setup at a location but that was it.

When you set up a DHCP server you have to plug in the address pool.  Usually you say something like "all IPv4 addresses in the range going from 192.168.0.100 to 192.168.0.200".  It is possible to specify multiple ranges but this is rarely necessary.  Most DHCP servers are set up pretty much the same way.  So it is easy to plug in the subnet mask, the gateway address, and one or both DNS server addresses.  If you have a home network that shares one public IP address among several machines you have a home DHCP server lurking somewhere.  How does that come about?

wrap up

I have left a couple of questions dangling.  I am going to answer them in this section.  But first I am going to do a quick review of my home setup.  My setup is more complicated than most home setups.  I am going to explain why I do things the way I do and then move on to some recommendations for how most people should operate.

Comcast is my ISP.  They have provided me with an Arris cable modem.  This box behaves somewhat like a combination DHCP server and gateway.  Comcast provides a pair of public IPv4 addresses to the box.  One is attached to the Arris box and one is available to be served up to whatever machine I hook up to it.  The hookup is an Ethernet connection.  If I only had one computer I could hook it up directly to the Arris box, set it to do DHCP (the default) and everything would work fine.  My computer would make a DHCP request to the Arris box and the Arris box would serve up a public IP address.  It would also serve up a subnet mask, gateway address (the address of the Arris box), and the addresses of two DNS servers operated by Comcast. That's how that would work.

But I want to hook several boxes up to the Internet.  How does that work?  I have another box.  It is a Netgear FVS318G ProSafe 8 port Gigabit VPN Firewall.  This is a box specifically designed for the home networking market.  It runs about $120 at Amazon.  (BTW, avoid the FVS318.  It has slower ports.)  It has a special port, the WAN port, that you connect to your cable modem.  That's the "external" interface of the box.  Then it has 8 more Ethernet ports, marked as LAN ports.  Together they constitute the "internal" interface.  I hook all my home network equipment up to them.  The 8 "internal interface" ports behave just like an Ethernet switch.  And between the "internal" interface and the "external" interface this box does NAT.  So now all my inside boxes can share my one public IPv4 address.

I also have a separate Linksys WAP54G Wi-Fi base station.  It is just plugged into one of the 8 inside ports on my Netgear box.  I also have a computer running Windows 2008 Server Edition.  This turns my home network environment into a small corporate network, also sometimes referred to as a "Microsoft NT Domain".  This is something most people don't need to mess with.  But I did system administration work on corporate NT domains for a couple of decades.  I like them and I know how to administer them so I did.  It is total overkill for most home users.

Nestled into this mess is a standard PC running Windows 7.  It has a private IPv4 address.  As does the Windows Server box, the Netgear box (on its internal interface), the Linksys box, my TiVo, my DVD player, and my printer.  I also have a couple of other boxes I occasionally use but I think you get the idea.  All the boxes I listed (and the ones I didn't) are able to get out to the Internet whenever they need to.  And they can talk to each other.  I can send files between my computers.  I can print from either one.  My home network does everything I want it to.  But it is overkill for most people.

But like more typical home network users there are some capabilities that all of us need.  So let me run down the list.  I have already indicated that the NAT capability resides in my Netgear box.  My DHCP capability resides in my Windows server box.  My DNS capability also resides in my Windows server box.  That means I had to configure both.  DHCP is pretty easy.  With Windows server DNS is harder but not a lot harder.  (And it probably will not surprise you to know that Microsoft uses its own code rather than BIND.)

First of all, remember that list of "root" servers.  Microsoft plugs that list in for me automatically.  And it changes very occasionally.  But when it does Microsoft sends out the updated list through Windows Update.  As long as I keep my Microsoft maintenance current, which I do, my "root" server list stays current.  I did have to configure a "forward zone".  It holds the "A" records for all my local devices.  And I had to configure two "reverse zones" to hold all my PTR records.  I won't go into the details on how this was done.  Suffice it to say it was not hard.

So I have this expertise built up from decades of doing this sort of thing for a living.  What's a regular person supposed to do?  Many people who don't know much about any of this have home networks that work just fine.  How is that possible?  The solution to both of these questions lies in looking at a more common setup.

Most home users do not have an NT domain operating.  So what do they do about DHCP and DNS?  The answer to both questions lies in the Netgear box.  That box can do DHCP.  I just turned it off.  You don't want competing DHCP servers running around and I knew that I wanted to use my Windows server to do DHCP.  (Why?  'Cuz!)  So I went in and turned it off.  Most people leave DHCP turned on and this is the right thing to do for them.  And the defaults for the box set up a home network using the 192.168.0.0/24 private net and configure DHCP accordingly.  I don't remember the range of IPv4 addresses the Netgear box selects as its default but it works for all but a few home setups.

That leaves DNS.  Here too the Netgear box has a cute trick.  It has a setting called "Get Automatically from ISP".  Remember that the box uses DHCP through its outside interface to get the information in needs to communicate with Comcast.  Well, included in that information are the IP addresses of two Comcast DNS servers.  When turned on (I turned it off manually) this option copies the two DNS entries it gets from Comcast into the DHCP information it serves out to the internal network.  The down side of this is you can't find the IPv4 addresses of the boxes on your internal network using DNS.  I like to be able to do that so I went through all those "NT domain" hoops.  But most people can and do live without this capability just fine.

So if I had not installed a Windows server and I had left all the settings on my Netgear box at their defaults I would have a perfectly functional network.  One final thing on this, the Netgear box assumes it is king of the 192.168.0.0/24 net.  As such it automatically assigns the IPv4 address of 192.168.0.1 to its inside interface.  Then it plugs this address in as the default gateway in the DHCP information it serves out.  So, if they used DHCP, and if they used the default DHCP setup the Netgear box comes with, my home computers would get all the settings they needed to fully participate in IPv4 networks.

The Netgear box would serve out to each machine a unique IPv4 address from the pool of 192.168.0.0/24 addresses it maintains.  It would automatically serve out a subnet mask of 255.255.255.0.  It would automatically serve out a gateway address of 192.168.0.1 (itself).  And it would automatically serve out the two DNS server addresses it got from Comcast.  That's how it works.

There are other boxes that have essentially the same capabilities as my Netgear box.  Some of them have 4 ports or none.  Some of them include a built in Wi-Fi base station.  Pick the box you like.  In some cases you can even get the box from your ISP.  And pretty much all the ISPs operate along the same lines as Comcast.  They might provide you with a different brand or model of cable modem.  And it might not technically be a cable modem.  But in all likelihood it will behave pretty much like the Arris box I got from Comcast.

And you actually don't need to know all this stuff.  Unless something goes wrong.  Then this information may be very helpful.  Do you have no (or bad) DNS server addresses?  Check your NAT box and make sure it is doing the "Get automatically from ISP" thing.  Are IPv4 addresses changing around in weird ways?  Check to make sure you have exactly one DHCP server running and check that the DHCP server is serving out sensible settings.  Are things physically connected up correctly?  In my case the Arris box needs to be connected to the WAN jack and everything else needs to be connected to LAN jacks.  If you have a separate Wi-Fi box is it trying to do the same things (i.e. DHCP) as your NAT box is doing.  For some services you can pick either box to host the service but there needs to be only one box hosting each service.

And then it may be those that just want to know how things work.  I hope these posts give you enough foundation so that you can explore anything or everything in more depth, should you choose to.  Wikipedia is a great source of more information on these subjects.  There are many hard core geeks among its contributors.  And they delight in drilling down much further than I have.  And in a lot of cases, they will provide you with references that will allow you to dive even deeper. It's a lot of fun, if you get off on that sort of thing.  And I do.  For the rest of you, I hope you have found this group of posts entertaining and enlightening.


  

Internet - DNS

This is, the sixth post in this series.  I recommend reading them in sequence.  The first post is at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.  The immediately previous one is at http://sigma5.blogspot.com/2015/09/internet-routing-and-nat.html.  I had hopes that I could wrap things up with this post but to do so would make it run too long.  So this post is short and only covers one subject, DNS.

What is DNS and why do we care?  Computers love numbers.  People love text.  So people like dealing with something like www.google.com rather than 172.25.11.23 (not Google's actual IP address).  DNS, short for Domain Name Server, is what gets us from here to there.  The thing that does this is called a DNS server.  In the earliest days of ARPANET there were only a few computers to keep track of.  A simple list worked fine.  But now there are "millions and billions" of web sites, mobile applications, etc., on the 'net.  We have long since moved past the time when a list could get the job done.

This was recognized fairly early in the transition from that first ARPANET to the modern Internet.  The first practical and comprehensive solution was called BIND - the Berkley Internet Name Daemon.  The University of California at Berkley Computer Science department was an early and active participant in Unix, the "C" computer language, and the Internet.  They developed a lot of tools and enhancements and one of them was BIND.  BIND is still around although responsibility has been turned over to the Internet Systems Consortium.  You can find out more at https://www.isc.org/downloads/bind/.  But most of us only need to use DNS servers.  We don't need to install or operate our own DNS server.  But let's find out a little about how they work anyway.

DNS is not a single server.  It is a swarm of cooperating servers.  And their job is to turn something like www.google.com into something like 172.25.11.23.  So how do they do this?  A DNS server is mostly just a simple database.  You ask questions.  It provides answers.  Actually it serves up the data from records that are appropriate responses to our query.  The most popular record type is the "A" for Address record.  If a DNS server has an A record that has a key of www.google.com it serves up the data value (172.25.11.23, in our example).  But in many cases it does not have an A record that is an exact match.  What then?

Would it surprise you to know that, as with a lot of Internet things, DSN is into this whole delegation thing.  Various servers provide part of the answer then pass things on to a server that knows more.  So all blind queries (queries that are not in the wheelhouse of whichever DNS server we are using) are passed on to a "root" server.  Where are the "root" servers?  That' one of those Internet "well known" things.  There is an official list of root servers and you can find out what they are if you know where to look.  Anyone who messes with DNS servers knows where this list is found and it is plugged into each DNS server.  So what our server does is pass the question on to a root server.

And this whole delegation thing proceeds from back to front.  The only thing root servers know about is where that last part is.  You know, the ".COM" or ".EDU" or ".GOV" or whatever thing.  The root servers don't care about the rest of it.  They know about that last part and where to send you to find out more information.  They have a list of DNS servers that specialize in ".COM" DNS entries, for instance.  So they just pass any ".COM" queries along to a ".COM" DNS server.  (We don't need to know where the ".COM" DNS servers are because the root servers know.)  This server knows about a whole bunch of "anything.COM" things.  It has no clue about "anything.EDU" or "anything.GOV" or "anything.whatever" if "whatever" is anything but ".COM".  But different servers do know about ".EDU", for instance.  The root server passes queries about "anything.EDU" to one of them.  The same is true for ".GOV", ".NET", and so on.

There is an official list of all the legal top levels.  (They are the last part of the name but they are called "top" levels because they represent the top of the tree of interconnected DNS servers.)  That's because every root server needs entries for all of them.  The servers that handle ".COM", ".NET", etc. are called "top level servers".  Originally there were only a few top levels.  But the criteria has been loosened up several times now.  Why?  Well, there is a little work necessary to get a new top level to work.  But the main reason has been a concern over confusion.  But people are now pretty comfortable with things like ".CO" or ".TV" or ".US" so there now seems little reason to keep the list short.  Also, the Internet has slowly become more international and part of that has been support for various languages.  There are now top levels in Chinese and Arabic, for instance, languages that don't even use our standard alphabet.  Returning to our example, hopefully the ".COM" DNS servers have an entry for ".GOOGLE.COM".

But that entry is just the name of another DNS server.  This server knows about "anything.GOOGLE.COM".  It knows nothing about "anything.MICROSOFT.COM" or "anything.FACEBOOK.COM".  And it especially knows nothing about "WH.GOV" (different top level).  So this ".GOOGLE.COM" DNS server has (again, if everything is working ok) an entry for WWW.GOOGLE.COM.  It serves it up to your computer and your computer finally knows what IPv4 address to send its traffic to.  Lots of messages are exchanged between lots of DNS servers.  But that's traffic that happens in the background where you don't see it.  And it is perfectly legal to go more levels.  The process just starts at the top level and works its way through the layers until it gets to the end at whatever level that might be.

That's the standard way things work.  I am now going to talk about a couple of variations.  Let's say we ask our local DNS server about WWW.FACEBOOK.COM.  The first time through it goes through the elaborate process I described above.  But we often go to the same place many times.  So DNS servers have a cache.  This is a place where the answers to recent questions are saved.  For any query that is outside the direct responsibility of that DNS server the query is first checked against the cache before the elaborate process described above is undertaken.

If it finds an entry for WWW.FACEBOOK.COM in the cache (because somebody asked about it recently) it just sends a "non-authorative" answer back.  Why non-authoritative?  Because only one or a few DNS servers are designated as the official respositories for WWW.FACEBOOK.COM DNS information.  They are the "authoritative" DNS servers.  All DNS entries in the cache of non-authoritative servers are also marked with a "time to live", usually a few days.  If Google wants to move their servers around they will need to update the DNS entries for these servers.  Confusion will reign if the old information does not eventually flush out of the Internet.

Another non-standard thing is an ALIAS record.  Let's say your company buys another company and you want to merge web sites.  You could just shut one down but that would mean that customers who had links to the shut down site would be stuck.  The ALIAS record solves that.  You just put an ALIAS record in the appropriate DNS servers.  The ALIAS record says "if someone asks about WWW.X.COM answer the question as if they had actually asked about WWW.Y.COM instead".  This way everything that was linked to the old location gets automatically connected up with the new location.  ALIAS records actually have a lot more uses than this but that's enough to give you the idea.

And it turns out that there are many different kinds of DNS records.  I am only going to talk about two more.  One is the MX record.  You can ask a DNS server "what's the IP address of the mail server for WWW.GOOGLE.COM?"  The DNS server will go looking for MX records instead of A records.  This kind of thing is handy in a number of ways.  Have you noticed that sometimes companies leave the "WWW" off.  DNS tricks like these allow computers to find the same web site either way.  Try just "GOOGLE.COM" sometime.  It gets you to the same place as you would get to by putting the "WWW." on the front.

The other record I want to talk about is the PTR record.  "A" records get you from the name to the IPv4 address.  PTR records get you back the other way.  Not everybody sets up PTR records all the time but in a lot of cases people do. If so you can find out what the text name that goes with an IP address is.  This turns out to be tricky.  To make this work you need to know the net/subnet boundaries.  But it is literally impossible, in general, to know what they are.  So what to do?  The PTR system assumes that IPv4 addresses are broken on octet boundaries.  And, to allow for this delegation thing to work, we have to go at things backwards.  172.25.11.23 is temporarily turned around.  We pretend for a minute it really is 23.11.25.172.  That's because we want to look the "172" part up first, and so on.  Cutting to the chase, instead of eventually looking for "WWW" in the ".GOOGLE.COM" DNS server we look for "23" in the "11.23.172" DNS server.  If you are not a little confused at this point I have failed.

But let me see if I can confuse you even more.  We have these authoritative DNS servers.  Why?  So everyone everywhere who asks a specific DNS question gets the exact same answer.  But what if we want different people to get different answers?  Google has server farms all over the place.  They want to send you to one of their local farms rather than requiring you to gallivant across the country or the world to get to a Google server.  A number of other large companies like Microsoft, Facebook, and Amazon want to do the same thing.  There's the trick.

What if we have a super-DNS server that not only looks at the question but looks at the IP address of the computer asking the question?  If we know the general location of the computer making the DNS query (you can usually get at least an OK idea by doing a lot of research) then you can serve up an IPv4 address answer that is for a server that is in a server farm that is close (in terms of the Internet) to the computer generating the query.  There is a company called Akami that does just that.

Their services are not cheap but they are "cheap at the price" for large companies.  The ".COM" DNS server just points to a special Akami DNS server whenever someone asks about ".GOOGLE.COM" (or any of the others).  Akami fakes things up on the fly so that people asking for the IPv4 address of WWW.GOOGLE.COM get a different answer if they are asking from a computer located in Europe than they get if they are asking from a computer located in San Francisco, which is close to Google's US headquarters

Let me finish up with several quick items.

DNS entries are not case sensitive.  So www.google.com is the same as WWW.GOOGLE.COM is the same as WwW.gOoGle.CoM, or any other pattern of capitalization that floats your boat. 

IPv6 - DNS servers can handle a mixture of IPv4 and IPv6.  A new record type has been added for IPv6.  The "A" record is now an "AAAA" record.  The data part of an AAAA record contains an IPv6 address.  As far as I can tell the PTR record has been extended to handle IPv6 without requiring a new record type.  But I haven't really looked into this.

You can directly query DNS servers by using a command called NSLOOKUP.  You can find information on it on the web, if you are interested.

If you run a Windows PC you can execute the IPCONFIG command in a "CMD" box.  "ipconfig /all" will tell you more than you probably wanted to know about various network settings.

If you poke around you will probably find that a "primary" and a "backup" DNS server are configured on your machine?  Why?  Because DNS is so critical.  The only time the backup entry is used is when there is a problem using the primary entry.

DNS server entries are always specified as an IP address (usually IPv4 but possibly IPv6).  Why?  Because if DNS is not working how do you turn a name into an IP address in order to know where to go to get DNS information?

Finally, where do the addresses for a primary and backup DNS server come from?  In most cases "magic".  I will get into this a little more in a future post.

Internet - Routing and NAT

This is the fifth post in this series.  I recommend reading them in sequence.  The first one is at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.  The immediately previous post is at http://sigma5.blogspot.com/2015/09/internet-classescidripv6.html.  In this post I am going to attack two subjects.  The first is how routing works on the public internet.  And the second is NAT - Network Address Translation.  The two subjects turn out to be interconnected.  Onward.

I have touched briefly on the subject of routing in an earlier post.  Two computers are local, in the IPv4 sense, if they are both in the same "net".  You apply the subnet mask to the computer's IPv4 address and extract the net part.  You do the same thing to the IPv4 address of the "to" computer, the one the message is being sent to.  If the net part of both IPv4 addresses is the same then the computers are local and the message is sent directly to the "to" computer.  Any IPv4 address that is not local is, by definition, remote.  What happens in this case, as far as our computer is concerned, is simple.  The message is just sent to the gateway.  But what does the gateway do?

Theoretically there this specific thing called a "gateway" and there is this other different specific thing called a "router".  As I have pointed out in several places previously, roles now frequently get mixed together in the modern Internet.  That's true of gateways and routers.  A router's job is to route messages along toward their destination.  It turns out that, to some extent, a gateway does the same thing.

All gateways have at least two interfaces.  One is connected to the local net so that it can catch traffic that is intended specifically for it.  But that same interface also catches gateway traffic.  Once the gateway has caught the message, which is marked as a gateway message, it does its gateway thing.  First it looks at its other interfaces.  There may be several but, in most cases there is just one.  Each of these other interfaces has a net associated with it.  Let's say the message destination is within that net.  Then the gateway gets rid of the extra gateway stuff in the message and sends it directly to its destination using the appropriate interface.  But what if there is no match?  Then, if the box is actually just a simple gateway, the box looks up its own gateway address and forwards the message along.

Now consider a pure router.  It typically will have several interfaces.  Again each will have a net associated with it.  If the "to" of the message is a match to any of these interfaces it behaves just like the gateway.  It strips off the gateway stuff and sends it directly on to its destination.  So far that's just the same as a gateway.  But what if we have a miss?  What if the "to" IPv4 address doesn't match anything.  Then what?  Here is where router behavior differs from gateway behavior.  A router has a routing table.  The routing table has a bunch of net-like entries.  The entry says something like "if the 'to' IPv4 address fits the net specification for this entry send the message out interface 4".  A properly constructed routing table has at least one entry that matches any IP address.

There are some special ones that say "If you match this entry just throw the message away".  These entries are used to handle 0.0.0.0/8, 127.0.0.0/8, 255.0.0.0/8 and other similar cases.  There is usually a "default gateway" type entry that says "if you otherwise don't know where to send it, send it here".  But basically what a router does is get a message, look it's "to" IPv4 address up in the routing table, and send it out using whatever interface the routing table specified.  That's it.  But how does the routing table get set up and maintained?  That's where the magic is.

There are a number of "discovery" protocols used to create and maintain routing tables.  Each router knows what nets it is directly connected to.  It plugs that information into its routing table.  But it can also use one of these protocols to send the information to all the routers it is in contact with.  And yes, there is a protocol for routers finding each other.  More than that, a router builds up information on what its nearest neighbor routers can access.  It sends that along too.  And so on.  So a router can eventually find out what routers 3, 5, 10, etc. hops away have access to.  The details of the several systems for doing this vary but they all perform the same basic function.  Special procedures have to be used when one router uses one system and another router uses another.  But that too has been worked out.

So theoretically we wait a while and the router knows where everything on the Internet is.  But the Internet is redundant so there are usually two or more paths from point A to point B.  Each of the routing table maintenance systems uses a different "weighting" system.  A simple "metric" is hop count.  If the destination is three hops away using one path and two hops away using another path, go with the shorter path.  The weighting function can get quite complex but that gives you the idea.

And that's how the Internet actually worked before it became the Internet.  Back then the federal government owned it all and paid for it all.  There was no particular reason to prefer one route over another so the systems just fought it out and as long as the message got delivered everyone was happy.  But now all the pieces of the Internet are in private hands.  Companies want to minimize costs and maximize revenue.  All of a sudden path B might look a lot better to the bottom line than path A.  So a lot of the components of the old system are still in place but a lot of biases have been added in to make sure the answer comes out the way the company wants it to.

Routing tables are no longer just built automatically by using some kind of discovery process.  A lot of  manual effort now goes into their construction.  And the details of their construction are closely guarded secrets.  So I don't think anyone now knows all the details.  But here's a guess that I think is pretty close.

Let's construct a simple model of the Internet.  You have your local network, say at home or at work.  For the moment we will assume that all your computers have "public" Internet addresses.  (I'll get into what that means below.)  So you have some computers on a local net using Ethernet.  One of these computers is the gateway.  The gateway has two interfaces.  One is connected to your local network.  The other is connected to an ISP, an Internet Service Provider.  If the message is going to another computer in the ISP's coverage area the ISP shoots it over to that other gateway and it disappears into the local network where it is delivered.

But let's say the "to" computer is in the control area of a different ISP.  For simplicity sake, I am going to assume that other ISP is not directly connected to our ISP.  In this case the ISP passes the message along to what I am going to call a long haul service provider.  There are a number of these LSPs and the actual situation is much more complicated but stay with me.  There are routers all over the place.  Some belong to one LSP.  Some belong to another.  The LSP's job is to see that everything gets delivered.  But the more traffic he can dump on to some other LSP's equipment (routers, long haul Internet circuits, etc.), the lower the first LSP's costs are.  But then, to make things complicated, there are companies like Netflix that are willing to pay for premium service.  So we have competing agendas.

What happens is that the ISPs and the LSPs have people called traffic managers.  Their job is to firstly make it all work.  But beyond that they need to arrange things in the manner most advantageous to their employer.  So they spend a lot of time figuring out what path is the "best" (for their employer's objectives) path.  They then ship out routing table data to make it so.

You can see this kind of thing in action for yourself.  I did some experimenting with a command called TRACERT.  It tells you the names of all the routers in the path between "from" and "to.  If I trace the route of a message to Boston College a company called Level 3 ferries the message across country.  But a message to the University of Massachusetts, a school located in the same city, uses Northern Telecom.  Why?

My theory is that the ISP that Boston College contracts with subcontracts its long haul needs to Level 3.  Whereas the ISP that U Mass contracts with subcontracts its needs to Northern Telecom.  Am I certain of this?  No.  But What I do know is that two messages going to almost exactly the same place take wildly different routes.  And, by the way, all traffic to the same destination seems to always take exactly the same route.

I also know that routing tables have gotten extremely complicated.  There was an outage a year or so ago that was caused by some routing tables exceeding an approximately 30,000 entry limit.  I also have a neighbor that does this kind of thing for one of the telephone companies.  He has told me a couple of war stories about tweaking certain traffic to go this way instead of that way.  Why?  Because somebody was writing a big check to his employer.

Anyhow the magical thing is that somehow it works.  The routers with their routing tables do get out messages wherever we want.  And "wherever" can be anyplace in the world.  It's pretty magical.  And fortunately we don't have to have anything to do with routers or routing tables.  That's definitely something to be grateful for.  And that brings me to NAT.

Your little home network and mine don't actually use public IPv4 addresses.  They use private ones.  Let's start out with the basics.  Fortunately for us someone came up with a brilliant idea a while ago.  "Let's carve out some special IPv4 nets", this person said.  These nets will be specifically designed to NOT work on the public Internet.  Instead they will be "internal use only" nets.  The official name for these babies is private IPv4 nets.  Since these nets don't work on the public Internet they can be used over and over.  You don't need to worry if someone else is already using the net you want to use.  Their use does not interfere in any way with you using the exact same private net.

And at least one net was set up for each "class" size.  So we can  pick which private net to use in a specific situation based on how many IPv4 addresses we need.  This was done back in the old "class" days so these reserved private nets follow the old rules.  The private nets are:

Class A - 10.0.0.0/8 (1)

Class B - 172.16.0.0/16 through 172.31.0.0/16 (16)

Class C - 192.168.0.0/24 through 192.168.255.0/16 (256)

Many home networks, including mine, use 192.168.0.0/24.  The company I used to work for uses a number of class C private nets but it also uses several class B private nets.  (It pretty much doesn't use its public class C nets at all but it turns out to be hard to give Cs back.)  These IPv4 nets are carefully selected because they don't work on the public Internet.  But many of the devices using these private addresses need and get access to the Internet.  How is all that possible?

The answer is NAT.  And to understand what is going on you need to know a little more technical detail.  When you send a message to a computer you don't just simply send it to that computer.  You send it to a specific port that belongs to a specific protocol type.  That's the foundation of the trick.

There are several protocol types.  But the only ones we care about are UDP and TCP.  (Now we finally get around to the discussion of TCP I promised you in the second post in the series.)  The protocol rules for the two are different.  TCP is like making a telephone call.  You set up the call (dial - answer).  Then you talk back and forth.  When you are done you both hang up.  TCP works the same way.  You set up a session.  Then you exchange messages.  Then you tear down the session.  TCP also makes sure the messages are delivered and in the correct order.

UDP is often called a "datagram" service.  It is like exchanging telegrams (or, in the modern era, Tweets).  Each message is a stand alone entity.  It has to have a full set of from/to information.  And there is no guarantee a message will be delivered or, if two messages are delivered, they will be delivered in the correct sequence.  There are advantages and disadvantages to each.  The point is that each computer has a set of  64,000 TCP ports and a different set of 64,000 UDP ports.  A message not only goes to a specific IPv4 address but it goes to a specific port belonging to a specific protocol type.  If any one of these three things is different (IPv4 address, protocol type, port number) it is a path to a different place.

NAT takes advantage of this.  Your NAT box has two interfaces, an "outside" interface and an "inside" interface.  Let's say you have one public IPv4 internet address.  It gets attached to the outside interface.  Then an IPv4 address from your private network is attached to the inside port.  And, by the way, your NAT box is also your gateway box.  So now your computer, which has a different IPv4 address, one from your private network, is hooked to your internal network.  And lets say all the boxes on your private network are connected together using Ethernet technology.  This is all probably sounding pretty familiar by now.

So you want to send a message, say a Google query from the browser on your computer.  Well, that message needs to get to the Google server somehow.  What happens?  Well, Google's server is not on your local net so the message gets sent to the your gateway, which also happens to be your NAT box.  The message comes in on the inside interface and is marked as a gateway situation.  It is also marked with the eventual destination of Google.  And it's a web message.  That means it uses the TCP protocol type and it is supposed to go to TCP port 80.  (I know this because the Internet has all these "well known" things and I know where to look them up.)  But the point is that the NAT box knows that your computer generated a  TCP message from port 80 on your computer to port 80 on the Google computer.

What it then does is pick a different TCP port on its outside interface and fix the message up so that that it looks like exactly the same TCP message came for this other port number and from the outside interface's IPv4 address.  It still sends it to the Google server and to TCP port 80.  It also remembers that it made this specific translation for this specific session.  When the response comes back from the Google server it is addressed to the IPv4 address of the external interface of the NAT box.  It is also addressed to a specific funny TCP port on that interface.  The NAT box looks all this up and says "Oh - I need to translate things back so that the message goes to the IPv4 address of your computer and to TCP port 80".  So a fixed up version of the message from Google shoots out the NAT box's internal interface destined for your computer.

The NAT process is "stateful".  It keeps a lot of specific information about the state of each connection.  And it maintains separate state information for each connection.  This way it can handle multiple web sesstions at the same time or multiple sessions of different kinds from the same computer.  The entire process is very complicated.  Fixing everything up so nobody is the wiser can be very complicated in some situations.  Newer protocols are now carefully designed so that they are easy to NAT, the complexity of the translation process is kept to a minimum.  But that's the basic idea.

The NAT box is able to fix everything up so that all the traffic in the public part of the Internet appears to come from the same single public IPv4 address.  Computers on your local network think they are talking directly to the boxes out on the Internet with nothing funny happening along the way.  What makes all this possible is that there are tens of thousands of port numbers available and that only a few of them are in use at any one time.

And the best news of all is that all this complexity is taken care of for us by the NAT box.  You just turn it on and it magically works.  And what NAT has done is to drastically reduce the need for public IPv4 addresses.  I have a number of different kinds of boxes at home.  Most of them need Internet access at least once in a while.  This is all easily accommodated by my home NAT box (which is also an Ethernet switch) so everything works fine even though I only have one IPv4 address available to handle all of it.

My old company actively uses several public IPv4 addresses.  But they had more than 1,500 devices when I retired, and that was several years ago, and a goodly percentage of them needed Internet access, at least occasionally.  And the same is true up and down the line.  NAT boxes and private nets are now the norm.  It has gotten to the point where they are often more convenient than using public IPv4 addresses would be.  By switching to NAT and private networks Microsoft was able to abandon at least one class B so that it could be added back to the CIDR pool.

A final note on these private nets.  Above I indicated that routers have routing table entries that throw traffic from certain nets like 0.0.0.0/8 away.  Well, they also have "throw it away" entries for the private net addresses.  There is an entry for 10.0..00/8 and for 172.16/0.0/12 (check it out - it catches all of them in one entry) and for 192.168.0.0/16 (same here). 

And a quick note on IPv6, things work in pretty much the same way as they do with IPv4.  But the Internet handles IPv4 and IPv6 traffic separately.  It may go down the same wire and it may be processed by the same router.  But both the wire and the router behave pretty much like they are two wires and two routers.  There is separate logic in the router for handling IPv4 and IPv6 traffic.  The general approach is the same and both kinds of traffic may come in or go out on the same interface but the two routing tables are completely separate.  Inside the router the traffic is completely segregated.

There's not much in this post that average people will use.  But I hope it clears up how things work and provides some perspective that is helpful when wrestling with the parts of networking that you are forced to deal with.

Internet - Classes/CIDR/IPv6

This is the fourth post in a series.  This posts builds on the previous posts so it is probably best to read them all in sequence.  The first post in the series is http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.  The post immediately preceding this one is http://sigma5.blogspot.com/2015/09/internet-ethernet.html.  I have posted them one after the other so it should be pretty east to walk your way through all of them.  So where are we?

In the second post I devoted considerable time to IPv4 in general and to "local" computers in particular.  The third post described Ethernet, the most common way local computers are connected together.  This post delves into how remote computers are handled.  A key concept to this local/remote determination is the subnet mask.  The subnet mask divides the address into a "net" .part and a "subnet" part.  It should be already obvious that the subnet mask is important.  It may also have occurred to you that we are making things too verbose.  Here are some typical subnet masks:  255.0.0.0, 255.255.0.0, 255.255.255.252.  That's a lot of verbiage to convey something that is actually pretty simple.  So what is being conveyed?

Actually it's a small number.  The subnet mask consists of a number of 1s followed by a number of 0s.  The number of 1s plus the number of 0s always adds up to 32.  So if we know the number of 1s we can instantly calculate the number of 0s.  We don't need to keep track of the two numbers separately.  The number of 0s is just 32 - "the number of 1s".  So we can boil the whole subnet mask business down to one number - the number of 1s.  This idea is combined with the network name to compactly convey everything we need to know.  Instead of saying something like "the 10.0.0.0 (subnet mask 255.0.0.0) network" we just say "the 10.0.0.0/8 network".  The "/8" part indicates a subnet mask of 8 1s followed by 24 0s (255.0.0.0).  So we can now say "the 172.16.0.0/16 net" or the "192.168.0.0/24 net" or even, to go back to my complicated example, "the 10.20.0.0/20 net".  This makes talking about this sub-netting business much easier.  And, once you know the tricks, turning "/20" into 255.255.240.0 is pretty easy.  So from here on out I am going to use the "/20" style.

Now let's dive more deeply into the whole process of how IPv4 addresses are handed out.  There are 4 billion of them so picking one person or even a small group of people to keep track of all of them is not practical.  As I noted before this quickly led to the idea of delegation.  A top level group (ICANN) would hand a big chunk to this group and a different big chunk to that group and so on.  Let's start by looking at the version of this that was in use for a long time.

It was obvious that some groups needed a lot of addresses and other groups needed not so many.  Back in the day when 4 billion seemed like a gigantic number a simple system looked like it could get the job done.  So a simple system it was.  Well, it was pretty simple system.  It was called the "class" system.  5 classes were defined:  A, B, C, D, and E.  We could now talk about a "class A net", for instance.  Sounds pretty simple so far.  But of course we have to throw some complexity in.  Fortunately it's not that much complexity.

Three exceptions were made to the class system.  The 0.0.0.0/8, 127.0.0.0/8, and 255.0.0.0/8 nets were exempted.  Two of them make perfect sense.  Remember we said that the 0 subnet address is never assigned to a computer.  It is reserved for the name of the network.  Well, it might have been possible for computers to handle 0.0.0.0 but people got lazy.  It was just easy to say "no 0.0.0.0".  That simplified a lot of computer coding.  And, since we have lots of IPv4 addresses (remember this is the early days), let's just exclude any IPv4 address whose first octet is 0.  Similar logic applies to 255.255.255.255".  It made lots of computer code simpler if we just said "not going there".  And again why not just exclude anything with 255 as the first octet.

That leaves 127.  Another trick that turned out to be handy was "how about we have an IPv4 address that is just a way of saying 'this computer'?"  For reasons lost to the mists of time the IPv4 address selected was 127.0.0.1.  The name generally associated with this IPv4 address is "localhost".  And again by the rule of "we have more IPv4 addresses than we know what to do with", any IPv4 address with a first octet to 127 came to be excluded.  That's it for complexity.  In the original "class" scheme" all other IPv4 addresses conform to the class rules.  So what are the class rules?

Class "A" is the giant class.  Each class A net consists of about 16 million (the exact number is 16,777,216) addresses.  Any IPv4 address with a first octet of 1-126 is a class A net.  And the subnet mask for a class A net is /8.  So there are 126 class A nets.  Together they represent a little over 2 billion IPv4 addresses.

Class "B" is the big class.  Each class B net represents about 64,000 (the exact number is 65,536) addresses.  Any IPv4 address whose first octet is 128-191 is a class B net.  So there are 63x256=16,128 class B nets.  And the subnet mask for a class B is /16.  Together they represent a little over 1 billion IPv4 addresses.

Class "C" is the small class.  Each class C net represents 256 addresses.  Any IPv4 address whose first octet is 192-223 is a class C net.  So there are 31x256x256=2,031,616 class C nets.  And the subnet mask for class C is /24.  Together they represent about a half a billion IPv4 addresses.

Class D is the "multicast" class.  It appears to be used to some extent but I know pretty much nothing about it.  Any IPv4 address whose first octet is  224-239 is a class D net.  I don't know enough about class D to talk about subnet masks so you are on your own here.  Together class D represents about a quarter of a billion IPv4 addresses.

Class E is the "experimental" class.  I know less about this class than I do about class D.  Any IPv4 address whose first octet is  240-254 is a class E net.  I don't know enough about class E to talk about subnet masks so you are on your own here.  Together class E represents about a quarter of a billion IPv4 addresses.

I have long thought that classes D and E should be retired and their IP address ranges repurposed.  Together they represent a half a billion IPv4 addresses.  That's a lot.  But for whatever reason this has not happened.  So these addresses are effectively lost.  Oh well.

Anyhow, back in the day you would apply for a class A, B, or C net.  It was very difficult to get an A, fairly difficult to get a B, and pretty easy to get a C.  The company that I used to work for controls 5 class C nets to this day.  Big companies like IBM and GE were able to get class As.  So was the US military.  But it quickly became obvious that handing out class As like jellybeans was a bad idea.  So a lot of companies, Microsoft included, opted for class Bs.  You had to make your case to get a B but it was pretty easy for any medium sized company to do so.  Then the small fry like my old company went for one or a couple of class C nets.

And this system worked pretty well until ARPANET completed the transition to the Internet in the early '90s.  (It went through some intermediate transitions like NSFNet that I am going to skip over.)  All of a sudden everyone wanted IP addresses.  The introduction of Windows 95 by Microsoft accelerated this trend to warp speed.  The most obvious response was to replace the "class" system with something more complex.

The old rule was that if you needed more than a few class X nets you were bumped up to the next larger sized class.  So my company was ok with its 5 Cs.  But what if they needed 30?  At about that point they were bumped up to a B.  But 30 Cs=7,680 IPv4 addresses.  That's only a little over 1% of a Pv4 addresses in a B.  That means that more than 98% of the addresses will be wasted.  When there were lots of IPv4 addresses to go around that was not a problem.  But when things got tight all of a sudden things looked very wasteful.  But at the time there was no intermediate step between a C and a B.  The obvious solution was to introduce some intermediate steps.

Class A is a /8 net.  Class B is a /16 net.  Class C is a /24 net.  Why not introduce some intermediate sizes like /12 or /20.  That was done.  Now you could get a net corresponding to any legal subnet mask size.  The other change was to make the whole system less US-centric.  The Internet was invented in the US.  But it spanned the world.  Why should it be run exclusively by the US?  The Internet quickly got internationalized.  ICANN was still top dog.  But now the second level was composed of RIRs, Regional Internet Registries.

Well, actually this whole "give out numbers" responsibility was spun off into something called the IANA, the Internet Assigned Number Authority.  It took over the whole IPv4 address business.  But right under it were 5 regional registries, one for Africa (AFRINIC), North and Central America (ARIN), Asia-Pacific (APNIC), Latin America/Caribbean (LACNIC), and Europe (RIPE NCC).  IANA will hand out a big chunk of addresses to an RIR.  They in turn will hand out smaller chunks to whoever applies.  The combination of all these changes is called CIDR - Classless Internet Domain Registration.  CIDR quickly took over for the class system (all unallocated addresses iommediately went into the CIDR pool) but referring to a "class A" or a "class B" or a "class C" net stayed in common usage as short hand for a /8, /16, or /24 net.

And this whole thing has held together pretty well.  It has definitely lasted long past the time anyone thought it would.  For more see http://sigma5.blogspot.com/2014/09/the-great-internet-disaster.html.

All this has to do with something called "public" IPv4 addresses.  You need a public IPv4 address to access the Internet as a whole.  A combination of three things have saved us from the internet disaster that never happened.  Two of those things are "private" internet addresses and NAT.  I will cover them in a future post.  The other thing is IPv6.  As I have mentioned elsewhere, cell phones (or "mobile" phones as they are called in the rest of the world) use IPv6.  A "mobile application", of which there are hundreds of thousands now, is really just the equivalent of a web site on the Internet.  But the important thing is that the mobile world is pretty much a universe unto itself.  It is possible to access web sites but nobody does it.  Why?  Because "there's an app for that".  So we now have the IPv4 computer/web site world and we have a mostly separate mobile phone/app world.  The fact that there is little overlap makes the whole thing work.

But let me spend just a little time on IPv6.  An IPv4 address is a 32 bit number.  If that isn't enough why not just use more bits?  That's the IPv6 solution.  Instead of 32 bits an IPv6 address is 128 bits.  How big is that?  It is 4 billion x 4 billion x 4 billion x 4 billion.  Its a really big number.  Every time we go through one of these transitions it is more difficult than it was the last time.  The idea of going with so many bits is "we don't want to do this again - ever".  I think they have succeeded.  And in any case I don't expect them to run out of numbers while I'm still alive.  And that's a good practical definition of infinity.

So what else changes with IPv6?  Not much.  The a.b.c.d format got abandoned.  It has been replaced by an A:B:C:D:E:F:G:H format.  Each number goes from 0-65,536.  But instead of decimal numbers they now use hexadecimal numbers.  So they go from 0000-ffff.  And there is a ".." trick that allows you to drop out a part in the middle if it is all zeroes.  But those are just practical changes to make it possible to write and read the giant numbers involved.  Other than that, not much has changed.  It pretty much works like IPv4.  There are some other differences but only the techies will care.  They don't make a practical difference to normal people.  I will briefly note some of these changes here and there in future posts.  But that's pretty much it except . . .

All modern computers are designed to handle both IPv4 and IPv6.  Ethernet and other common technologies see messages passing back and forth, just like before.  There are a few tiny differences but mostly a message is a message is a message.  No changes had to be made to how Ethernet worked to accommodate IPv6.  But if you choose to you can have the NIC on your computer have an IPv4 address and a separate IPv6 address at the same time.  There are bits in the messages that say "this is an IPv4 message" or "this is an "IPv6" message.  The computer sends and receives messages in pretty much the same way in either case.  Most application software doesn't even care.  If it does there are ways of flagging something to "do this the IPv4 way" or "do this the IPv6 way" but it is pretty much transparent.

Given that traditional computers live in an IPv4 world I recommend turning the IPv6 stuff off.  The only exception to this applies to corporate networking people when it comes to some of the computers they deal with.  As those people are not the target audience for this post (sorry) I include the following for informational purposes only.  This is how it works.

Many companies need to maintain web sites to deal with computer traffic and app servers to deal with mobile traffic.  A lot of the business logic will be common to both.  You will just "display" things a little differently depending on the device.  So it makes sense to host the web site and the app server on the same computer.  This is easy to do.  And the web site and the app server can then share a lot of code and other application components.  This means many features can be implemented once but will apply to both the web site and the app server.

These corporate networking people need to make sure these servers have both a public IPv4 address and a public IPv6 address.  It is easy to "bind" the web site to the IPv4 address and the app server to the IPv6 address so they coexist very nicely.  If the installation is big enough so that it makes sense to use separate servers then it is still just a matter of making sure app servers have valid IPv6 address.  But if it's not part of your app server infrastructure you can do what the rest of us do and just go with IPv4.

Thus endeth, for the most part, the IPv6 tutorial.

Internet - Ethernet

This is the third post in this series.  The first post is at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.  The second post is at http://sigma5.blogspot.com/2015/09/internet-tcpip.html.  In the second post I talked about computers having either a local or a remote (in the IPv4 sense) connection.  Leaving aside remote connections for the purposes of this post, if two computers are local to each other it means that either one can send a message directly to the other one.  How does this work?  Let's start by looking at some very early networks.

As discussed in the second post in this series, early networks were hub and spoke networks.  They consisted of a bunch of terminals (spokes) connected to a computer (the hub).  In the earliest of these networks wires ran directly from the terminal to the computer.  Then modems were introduced.  This turns a terminal-type signal into one that telephone (or Teletype or TELEX) equipment can handle.  This allowed the terminal to be positioned perhaps thousands of miles away from the computer.  Nice but still pretty inflexible.  The next variation was a multi-drop connection.   Here you still have the single computer (with perhaps many connections) at one end of the connection.  But by means of clever techniques several terminals could share that one connection.

This helped some more but you still had two problems.  In most cases you needed a separate terminal for each computer and the computers could not talk directly to each other.  Oh, the "modem" scheme meant you could use one terminal to dial into several computers, if they all supported dial up and the same kind of terminal.  But in many cases the computer maker made you use a special type of terminal that only worked on his computers so the whole thing was clunky.  And the computer makers didn't see any reason to make things easier.  All this incompatibility was good for business.

ARPANET tackled a large part of this problem.  It got the computers talking to each other.  But in its early incarnation it left the terminal part to the computer companies.  With ARPANET, assuming your computer was connected to it, you used a manufacturer supported terminal to connect to your local computer.  Then you could with some difficulty ARPANET over to that other computer.  That was an improvement but it still left a lot of hassle.

And by this time computers were getting cheaper and more powerful and terminals were getting cheaper and more powerful.  But the result was a lot of terminals connected to each large computer.  Assume for the moment that your computer is not connected to ARPANET and all you are dealing with is hooking a lot of terminals to your one big "mainframe" computer.  It was still hard.  You ended up running wires all over your building.  And every time you wanted to remodel or move people around it was a big pain in the butt.  That's where Ethernet came in.

Imagine you are using radio circuits instead of wires to hook everything together.  Then you can move things around within reason and there is no problem.  The radio waves still connect everything together.  And imagine you use a radio system where the radio receiver on every device, terminal and computer alike, can hear the transmitter of every other device.  If this could be pulled off it would be magic.  There are some technical details to work out (see below) and the whole thing would be very expensive.  But gee wouldn't it be great?   That was the fundamental insight of the inventers of Ethernet.

Instead of radio they would use a single common wire, a wire that everyone would hook up to.  Since everyone was hooked up to the same wire everyone could listen to (receive) and talk to (transmit) everyone else.  It would work just like radio.  A bunch of technical details needed to be worked out and a catchy name was needed.  The name derives from a long abandoned (because it turned out to be wrong) physics theory about how radio waves worked.  The idea was that they were vibrations in something called the "luminiferous aether".  The "luminiferous" part was dropped and the modern spelling of ether (the "a" on the front was dropped) was combined with the ubiquitous "net" suffix to get Ethernet.  With the name issue solved let's move on to the technical details.

The developers came up with an acronym to describe some of the key technical details.  It is CSMA/CD.  This mouthful stands for Carrier Sense Multiple Access with Collision Detect.  A radio-like signal would be put out on the wire.  A radio signal is a modulated carrier.  The "carrier" is a standard frequency and the "modulated" part just means you jigger it a little to add in the data.  The first decision they made was that every device would use the same carrier frequency (and the same modulation system).  That meant everyone could hear all the messages.  But it also meant that you could end up with two devices trying to transmit at the same time.  The specification was carefully designed to make it easy to sense when that happened.

Multiple Access just meant that many devices could be hooked to the same wire.  And Collision Detect just meant that when two devices tried to send a message at the same time everybody (all the devices) could easily and reliably detect it.  This situation was called a Collision.  Since all the devices used the same carrier frequency and modulation technique these components could be easily and cheaply manufactured.  The system did require that the Ethernet network card had to be complex.  Specifically, it had to have some computer smarts built in.  This increased the expense.  But by the time Ethernet came along the required amount of computer power could be manufactured cheaply and would fit on a standard computer card.  (Now, everything fits into a single chip.)

Let's dig into what goes on a little more deeply.  The computer sends the Ethernet card a message.  The card monitors the wire.  If it sounds like no one else is using it the card starts sending out the message.  While it is sending the message it monitors things for a collision.  If everything goes out ok then we are done.  But if a collision is detected a "jam" signal is sent out to make sure all the other cards know that a bad thing just happened.  Then the card waits for a "random back off delay time".  The back off time doesn't have to be very random.  The idea is simply to avoid the situation where two cards keep trying to resend their messages after waiting exactly the same amount of time.  If the card detects another card starting to send a message during the back off time it politely waits till the message finishes.  Then it waits another random back off interval and tries again.  If the wire is not too busy ("busy" is being active more than 30% of the time) then all messages should get through without much delay.

The "wire" part of Ethernet has gone through several generations of evolution.  The original Ethernet wire was a 1/2" diameter coaxial cable.  That was strung around the building.  A very clever device called a "tap" was used to hook network cards up to the cable.  You could fasten the tap to the cable at any point along the way.  A clever design that I am not going to get into, allowed taps to be hooked and unhooked well after the cable was installed.  So you had to string the heavy cable around your building once.  That was expensive and a hassle.  But after that you could tap any number of devices into the cable on an "as needed" basis.  And, if you were careful, you could even un-tap them.  This was a big improvement over the necessity to string wire all over the place every time you wanted to add or move a terminal.

But the cable was heavy, expensive, and hard to work with.  It didn't bend well so it was hard to snake around.  This led to a second generation "thin coax" specification.  A much lighter and smaller cable was specified.  A new generation of taps, etc. was also developed.  This was an improvement over the first generation but it was still a pain.  Neither the thick coax or the thin coax versions of Ethernet garnered wide acceptance.

Computers have almost since the beginning been about smoke and mirrors.  If you can come up with a new system that acts from the outside just like the old system then there is really nothing stopping you from swapping the new system in to replace the old one.  And that's what happened with Ethernet.  What are the key attributes of Ethernet?  (1) This business of all equipment seeing all the messages.  (2) This business of collision detection and retry.  (3) A third attribute turns out to be fast.  It needs to be able to handle a lot of messages quickly.  If we can swap out our old Ethernet card (and all the downstream equipment) and replace it with a new "Ethernet" card (and downstream equipment) that behaves just like the old card as far as the computer can tell, who cares if it actually works differently?  Nobody!  And that's what happened.

The simple "string a single wire all over the place and go from there" idea was replaced by a more complex system that emulated the old system.  But this additional complexity ultimately turned out to be more flexible and cheaper.  The three components of the new system were:  (1) A new NIC (Network Interface Card);  (2) A different kind of wire (CAT-n) and lots of wires instead of a single one;  and (3) A new component, an Ethernet switch.  (Again, don't get hung up on the nomenclature.  I will define what I mean by an "Ethernet switch" shortly.)

The new NIC behaved exactly the same way as the old one from the computer's perspective so the fact that it was quite different under the covers didn't make any difference.  In this new scheme a separate CAT-n cable connected each NIC to a nearby Ethernet switch.  The same type of CAT-n cable was also used to connect the switches together.  The secret sauce was in the switch.  The switch contained several ports, say 8.  Each port had an RJ-45 Jack that fitted an RJ-45 plug on the end of the CAT-n cable.  So, for instance, NIC 1 could be connected by a CAT-n cable to port 1 of switch 1.  So far so good.  But the only device that looks like it would be visible to the NIC is the switch on the other end of the cable.  What gives?

The switch cross connected all the ports together.  Whatever message came in on one port it was echoed out all the other ports.  More than that, the switch monitored what was going on.  If it saw an incoming signal on more than one port it sent a Jam signal out all of the ports.  If a Jam signal came in on one port it was immediately copied to all the other ports.  This behavior was the secret to making this collection of many cables and potentially many switches behave just like the old coax based Ethernet system.

And let me address one piece of complexity right now.  A CAT-n cable actually has 8 separate wires in it.  This version of Ethernet actually uses separate transmitters and receivers.  The transmitter uses certain wires and the receiver uses other different wires.  It is important to hook the transmitter on one end to the receiver on the other end.  How is this done?  A standard CAT-n cable is "straight through".  Pin 1 on one end is hooked to pin 1 on the other end.  But if the jacks on the NIC and the switch are wired the same then using a straight through cable will connect the transmitter on one end to the transmitter on the other end.  That won't work.

The original solution was to have the jacks on switches wired backwards.  The receiver would listen on the transmitter wires and vice versa.  And that worked fine if you always had switches connected to computers.  But then you couldn't hook a switch to another switch.  The solution here was a custom "cross over" cable.  The cable would be wired so that the "transmit" pins on one end were crossed over to connect to the "receive" pins on the other end.  If the network installer people used the right CAT-n cable in the right place then everything would be fine.  But sometimes they didn't and everybody would get very confused.

The second generation solution was to have a special port on the switch.  It was wired backward so that it could be used to connect with the other switch.  That meant that sometimes you had a port you couldn't use and sometimes you didn't have enough "switch" ports.  The third generation solution was to put a switch on the special port.  The switch would cause the port to use the standard pin configuration or the cross over configuration.  That was a little more flexible but it still meant that the network people could screw up.  The fourth (and final) generation solution was autosensing ports.  The port would sense whether it should be in normal mode or cross over mode.  Network people could throw all their cross over cables away and always use straight through cables.  They could still screw up by hooking two switches together with two cables (or some more complicated version of the "loop" problem) but it was now much easier to get things right and keep them that way.

But how are devices kept track of?  Would it surprise you to learn that the answer is again a number.  In this case the number is called a MAC address.  A MAC address is 48 bits long so there are lots more of them than IPv4 addresses.  In fact, there are 64,000 times as many.  So there are 256 trillion of them.  That sounds like plenty.  But it looks like it is not enough.  So this is now called a MAC-48 address.  There is a new MAC-64 specification where the number is now 64 bits long.  I'll let you figure out how big this number is all by yourself, if you care.  For me, it seems big enough.

And in most cases the MAC address is pre-assigned.  The way this works is that the 48 bit MAC address is broken into two 24 bit pieces.  NIC card vendors go to a central authority and request a MAC block.  They get a 24 bit number that goes into the first 24 bits of the MAC address.  They then make sure that each NIC they make has a unique 24 bit sub-address number that goes into the last 24 bits.  When they run out of a 24 bit group (that's 16 million addresses but it happens to the high volume guys) they request a new block.

Think about every NIC card that has ever been made.  And while you are thinking try to remember all the devices you own or have owned that had a sticker on it listing a MAC address.  Then think about all the manufacturers that have gone out of business, etc.  You can see why 16 million "manufacturer numbers" might not last forever.  There are tricks where MAC addresses can be changed or spoofed or dynamically assigned.  But this is rarely done so I am going to ignore it.

As with IPv4 messages Ethernet messages are sent from a device with one MAC address to a device with another MAC address.  There is a difference.  There is no such thing as a gateway.  All messages are sent directly to wherever they are going in one hop.  There is also something called a "broadcast" message.  As with IPv4, there is a "from" MAC address and a "to" MAC address in each message.  But there is a way to flag a message as a "broadcast".  That means that everybody gets it.

One thing NIC cards do is listen to all the traffic going by.  They look at the "to" MAC address of each message.  If the MAC address does not match the MAC address of the NIC the message is thrown away and the computer never knows it was there.  But the exception to this is a broadcast message.  It is received and sent along to the computer in a manner similar to a message that was addressed to the computer.

These broadcast messages are used for several purposes.  But one of them is for something called ARP, Address Resolution Protocol.  This is how computes find each other on an Ethernet network.  Let's say you need to send a message to "computer 5".  You send out a broadcast message that asks "are you computer 5?".  If everything is working ok then exactly one computer sends a "yes - I'm computer 5" message back.  Your NIC can now remember the "from" MAC address of this response.  The next time your NIC wants to send a message to "computer 5" it sends a direct message to the NIC card with the right MAC address.  All the other NIC cards see this message but all of them except computer 5 silently discard it.

Let's step back now and ask how one computer sends an IPv4 message to another local (in the IPv4 sense) computer over an Ethernet network.  It first sends out the appropriate ARP broadcast message asking what NIC card handles the IP address in question.  If everything is working ok then one computer sends back an "It's me" message.  After that anything going to that IPv4 address is sent as an Ethernet message directly to the appropriate MAC address.

Now is probably an appropriate time to talk about "encapsulation".  This too is a popular computer trick.  Let's say we want to send a query to a web site.  Our browser sends a request to the IPv4 component of Windows (say) to "send this message to IPv4 address 10.20.61.12".  The message is "encapsulated".  The message itself is left unchanged but IP stuff is added to the front and back of it.  This information includes the "from" and "to" IPv4 addresses involved.  This bigger message is now sent down another level to the Ethernet component.  There Ethernet stuff is added onto the front and back of the bigger message.  It is then sent to the NIC card to be transmitted.

In this case what goes out over the Ethernet network is an Ethernet header followed by an IPv4 header followed by the original message followed by an IPv4 trailer and finally an Ethernet trailer.  We have encapsulated the original message twice.  It is possible to encapsulate three or more times, if circumstances warrant.  The benefit is that each "layer" only needs to know about its part of the overall process.  And usually it only has to process its own header and trailer information.  The part in the middle, the "payload", just needs to be passed along intact.

And now let me revisit Ethernet switches.  Computer power keeps getting cheaper and cheaper.  So what I have described above is a "classic" Ethernet switch.  (Warning:  The terminology is about to get warped yet again.)  They don't work that way any more.  Modern switches have store and forward capability.  Each port is monitored separately.  As each message comes in it is sent to a common buffer inside the switch.  This buffer has enough storage capacity to handle several messages.  That means if two messages come on two ports at the same time no Jam is generated.  Instead each goes separately into the buffer.

And most messages are no longer sent out all ports.  If it is a broadcast message it is sent out all ports.  But as messages come in the switch monitors their "from" MAC addresses.  That allows the switch to build up a list that tells the switch that a certain specific MAC address is hooked to a specific port.  If the "to" MAC address is in the MAC table in the switch messages for that MAC are only sent out the correct port.  That cuts way down on traffic.  If the switch can't find the MAC address in its MAC table it reverts to sending the message out all ports as before.

This buffering capability allows modern switches to perform another very nice trick.  Why have I been saying CAT-n so far?  Because there are several "CAT" specifications.  The ones that are important to us are the ones that have to do with speed.  CAT-3 is designed to handle a 10 megabit speed.  CAT-5 is designed to handle a 100 megabit speed.  It turns out that both CAT-5E and CAT-6 are designed to handle 1000 megabit speed.  You can safely run a lower speed signal down a wire that is designed to handle a higher speed.  But it is asking for trouble to send a higher speed signal down a wire that is designed for a lower speed.  This problem can be avoided by always using CAT-5E or CAT-6 wire.  That's what I do.  But beyond that, modern Ethernet switches can now be used to do speed matching.

I used to have a old printer that ran at 10 megabits.  But most of my network devices wanted to run at 100 megabits or 1000 megabits.  But besides sensing whether a port should run as a straight through port or a cross over port, the ports on modern switches can speed sense.  They can figure out whether the NIC on the other end is running at 10, 100, or 1000 megabits.  Since all messages are received into the switch buffer and sent from the switch buffer it is no longer a problem if one port is running at one speed while another port is running at a different speed.

My computer could send a message to my printer at 1000 megabits.  It would actually go first to the buffer in the switch so this link could and did run at 1000 megabits.  The switch would then forward it along to the printer at 10 megabits.  The same thing would be true going the other way.  The printer might send a status message at 10 megabits.  It would go into the buffer in the switch at 10 megabits.  It would then go out of the buffer and into my compute at 1000 megabits.  Devices can now run at whatever speed is appropriate for that device.

So that's a quick look at Ethernet.  But remember the "smoke and mirrors" discussion above.  There are now lots of other technologies that parrot Ethernet behavior.  The most obvious example of this in the home environment is Wi-Fi.  In practical terms Wi-Fi works just like Ethernet.  The Wi-Fi card in your laptop behaves just like an Ethernet NIC.  It probably even has a MAC address.  And a Wi-Fi base station behaves pretty much like an Ethernet switch.  And radio signals behave like the CAT-n wires and there is a CSMA/CD-like process for detecting and dealing with collisions.  That's why Wi-Fi can be spliced seamlessly into an Ethernet network.

And it turns out it's not just Wi-Fi.  You may or may not have heard of SONET or FDDI or ATM or Token Ring or a number of other technologies.  They all more or less parrot Ethernet behavior.  If you have to administer them then you need to know the details of exactly how each works.  For the rest of us we just need to know that they provide an Ethernet-like capability and we need to have some idea of how fast they can move data under real world conditions. The rest we can leave to the experts.

Internet - TCP/IP

This post is the second in a series.  To make sense of it you need to either be very familiar with binary and other forms of computer arithmetic or have read the first post in the series.  (Even if you are very familiar with binary and other forms of computer arithmetic I recommend you at least skim the first post so you know what's in it.)  That post is located at http://sigma5.blogspot.com/2015/09/internet-bits-bytes-and-numbers.html.

The acronym TCP/IP is commonly used.  But TCP and IP are actually two separate things.  I am going to start with the more foundational and important of the two, IP.  IP stands for Internet Protocol.  In the computer business "protocol" is a set of rules and formats that let one computer component interact with another one.  IP is how computers talk to each other.

The very first computers didn't talk to each other or to much of anything, for that matter.  If you wanted to interact with ENIAC, the very first computer, you had to be in the same room with it.  And there was only one computer at that time so computer to computer networking wasn't in the cards.  But even then there had long existed a technology called Teletype (for a company that made this type of equipment), or sometimes TELEX.  The idea was simple.  Use the phone system to hook two electric typewriters together.  What was typed on one would be printed on the other.  The very first computer (and all computers for a long time) was very expensive.  So it was obviously a good idea to make it more accessible because that made it easier to  keep it busy at all times.

And early computer builders quickly latched on to the idea of hooking teletypes or similar devices up to computers.  Thus was born the first computer network.  But these early computer networks were "hub and spoke" operations.  You had a very expensive computer as the hub and you had teletypes or other kinds of terminal equipment as the spokes.  And in the early days there was just one hub, the very expensive computer, at the center.  That was fine in the very early days because there were only a few computers around.

Things quickly evolved into quite elaborate networks as computers got more powerful and more of them were built.  And even then teletypes could be used to send messages thousands of miles.  So computer networks quickly came to span thousands of miles, at least potentially.  And it soon came to be that a very few very important people had hookups to two or more computers in their offices.  But each computer had its own terminal.  For these people it was a hassle to have one terminal for this computer and a different one for that one.  So they were interested in a common terminal setup.  They also quickly became interested in hooking one computer to another so that data located on one computer could be sent to another.

This was technically feasible even at this early point.  You just have one computer look like a teletype to the other one.  But by this point computers had ceased to be custom one off devices created by a group of  engineers and scientists and paid for by the government.  Computers instead had come to be designed and built by private companies.  The first successful computer company was Univac.  And if there had only been one company that might have been ok.  But soon there was also IBM, GE, and many others.  And the problem was that for business reasons they all designed hub and spoke networks that could only connect their own brand of equipment.

Then along came the Advanced Research Projects Agency (ARPA) within the Pentagon.  By this time the US military had figured out that computers were important for war fighting and they had become frustrated by the proprietary networking technology offered by the companies they bought computers from.  So they funded a project called ARPANET.  The idea was to figure out how to hook computers from different manufacturers together.  The project succeeded, eventually spectacularly.  The ARPANET project spawned IP.  Well actually, several versions of IP.

There are two versions of IP in use today:  IPv4 and IPv6.  I am not going to talk about IPv6, although I may do a future post about it.  So what do people need to know about IPv4?  Some but not that much.  And that's what the rest of this post is about.

Computers at bottom are about numbers.  So it will come as no surprise (especially since I telegraphed this in the first post in this series) to learn that each computer is assigned a number called an IP address in the IPv4 scheme.  This number is 32 bits long, which means that it represents a number between zero and four billion.  When IPv4 was designed four billion seemed like a good approximation to infinity.  But by the late '90s it no longer looked so big.  People started worrying about running out of numbers.  I have addressed this subject in a previous post (see http://sigma5.blogspot.com/2014/09/the-great-internet-disaster.html).

In the mean time, what good does our computer having a specific number assigned to it do us?  IPv4 is about the business of letting any computer that speaks IPv4 send a message to any other computer  that speaks IPv4.  And, of course, the other computer can send a message back the other way.  The idea is to provide a means for every computer to talk to every other computer.  IPv4 does this by including a "from" IP address and a "to" IP address in every message.  To respond to a message the other computer just needs to flip the "from" and "to" IP addresses and the new message goes back the other way.  It's just that simple.

But how does one computer figure out where the other one is?  One idea would be to encode some kind of geographical information.  But, as I indicated in my previous post, that wasn't done.  An IP address is just a number.  There is no geographical information in the IP address.  Well, almost no geographical information.  I'll get return to this subject later in this post.  And I'll go into how a message gets from one computer to another one in later posts.  In the mean time let's look at another simple question.  How does a computer get an IP address?

The original ARPANET was set up to hook as many as 64 computers together.  With only a few computers involved you could just have someone keep a list and assign the next free number off the list.  With four billion numbers to keep track of that's impractical.  The solution actually adopted was delegation.  One super-group of people would assign a big chunk of addresses to a subgroup.  This subgroup could slice and dice things further into sub-subgroups each with their own administrators.  And this process could go on until someone was responsible for a pool of only a few numbers and a few computers.  Then they could assign the numbers in their pool however they wanted.  This idea was initially implemented using classes and subnets.  Let me start with subnets.

Some of this was telegraphed by the "subnet" discussion in the previous post.  But I will now go into more detail.  The subnet mask is used to divide an IP address into a "net" part and a "subnet" part.  The first part of the subnet mask, the part that is all 1s, is the net part.  The rest, the part that is all 0s, is the subnet part.  There is a super-group agency out in internet-land called ICANN, the Internet Committee of Assigned Names and Numbers.  They are responsible for the bigest chunks.  They dole a big chunk to this group, a different big chunk to that group, and so on.  And they follow the subnet mask rules.  So the big chunk is identified by the first few bits.  In the early days the net part was frequently only 8 bits long.  So a big chunk might be 10.x.y.z (indicated by a subnet mask of 255.0.0.0).  This chunk contains over sixteen million addresses.  But the "10 net" owner could then start subdividing.  They could give a still pretty big chunk, say 10.20.x.y (indicated by a subnet mask of 255.255.0.0), to a large division within the organization.  They, in turn, could give a 256 address chunk, say 10.20.30.x (subnet mask of 255.255.255.0), to a relatively small department.  That's a highly artificial example of how delegation worked from a technical point of view.

Above I promised that there was a little bit of geography in IP addresses.  The subnet mask rules are taken into consideration when IP addresses are delegated.  But the subnet mask primarily fulfills a much more important role.  It is used to determine what is "local" versus what is "remote".  If the "from" and the "to" IP address are both in the same net then both computers are on the "local" network.  In this case the "from" computer just sends the message directly to the "to" computer.

As a specific example, I talked about a "net" with an IP address of 10.20.30.x (our "department" net).  We only know what's going on if we can see the subnet mask.  In this case it is 255.255.255.0.  That means that the "10.20.30" part is the "net" part so the "x" must be the subnet part.  How is this sort of thing actually handled?  The "0" subnet address is used for the name of the net as a whole.  So this would be called the 10.20.30.0 net.  Remember that by itself "10.20.30.0" doesn't mean anything.  It can only be interpreted after we have applied the appropriate subnet mask.

Here's some supplemental information.  To avoid confusion no computer is given the IP address corresponding to a 0 in the subnet part.  There is no computer with an IP address of exactly 10.20.30.0.  Also, for reasons I am not going to get into, the highest subnet address, 255 in our example, is also left open.  So 10.20.30.255 is not assigned to any computer either.  This means that the number of IP addresses that can actually be used in a subnet is 2 less than the theoretical maximum.  In the example we are working, only 254 addresses would actually be available for use.  This need to leave 2 addresses out means that the smallest practical subnet we can have has a mask of 255.255.255.252.  This subnet has 4 IP addresses in it. But after we subtract the two we don't want to use there are only 2 left.  And that's as small as it gets.

Let's take a slightly more complex example.  I talked about a subnet mask of 255.255.240.0 in the earlier post.  Let's use it in the context of our 10.20.0.0 network.  Why not the 10.20.30.0 network?  Look at the third octet.  The subnet mask byte is 11110000 but the net byte is 00011110.  The binary version of "30" is partly in the net part and partly in the subnet part.  This is asking for trouble.  What would make sense in this situation?  Well, let's say that our organization controlled the 10.20.0.0 net (mask 255.255.0.0).  That's a lot of IP addresses but let's assume its a big organization.  Now let's say that this organization has divisions and that each division is pretty big, big enough to require  in the neighborhood of 4096 IP addresses.  Then we could have a 10.20.0.0 (subnet 255.255.240.0 subnet) sub-subnet assigned to "division 0".   Why division 0?

Look at the chunk between where the net part for the company ends and where the net part for the division ends.  Both net parts share all of the first two octets and none of the fourth octet.  But the net part of the company subnet contains none of the bits in the third octet whereas the net part of the division subnet contains the top 4 bits.  This middle chunk (the top 4 bits of octet 3) becomes critical.  So let's look at them in isolation.  (We can ignore the bottom 4 bits of octet 3 for the moment because they are subnet bits in both cases.)

In this critical chunk the bits in 10.20.0.0 are 0000.  In decimal that's zero.  Now let's flip the bottom bit in our chunk to a 1.  Our chunk now looks like 0001 and octet 3 looks like 00010000.  In decimal that's 16.  So the "division 1" net would be 10.20.16.0 (same subnet mask of 255.255.240.0).  Adding 1 again (division 2) we get a chunk of 0010 in binary and an octet of 00100000.  Division 3 would have a chunk of 0011 and an octet of 00110000.  This translates to a net of 10.20.48.0 (same again on the subnet).  We then continue in a similar manner.  We can have 16 divisions (0000 through 1111) in total and stick with this scheme.

Finally, notice that 10.20.30.0 falls somewhere in the middle of the address range that belongs to division 1.  (Octet 3 would be 00011110).  That's why it is a bad choice for a sub-subnet in this scheme.  This is a good example of a situation where it's complicated if you just look at the decimal numbers and don't translate to the underlying bits.  Welcome to the world of corporate networking.  Fortunately there are tricks for avoiding this kind of complexity. 

Our local/remote test works the same way in our apparently more complicated situation.  But the rule is the same.  Two computers are local to each other if they have the same net number.  So if our subnet mask is 255.255.240.0 a computer with an IP address of 10.20.48.251 is local to a computer with an IP address of 10.20.61.12.  What?  It's confusing if you look at the numbers and not if you look at the bits.  The third octet tells the tale.  In one case the bits are 00110000.  In the other case they are 00111101.  But we only care about the top 4 bits.  In both cases they are 0011 so the net is the same.

So what if the net part is different?  That means the two computers are remote from each other.  What happens then?  Well that's where the gateway comes in.  Every network needs a gateway.  Usually the gateway has a subnet number of 1 but that's just common usage.  The gateway can use any IP address in the subnet.  The rule is that gateway address must be a local address.  Any computer in the net must be able to send messages directly to the gateway.  Why?  Because the gateway is the only computer that knows how to access other nets.  How?  Magic?  The slightly more complex (and useful) response is this question will be covered in a later post.

So the trick is a simple one.  If a computer wants to send a message to another computer it uses the subnet mask to figure out if the "to" computer is local or remote.  If it is a local computer just sends it directly.  This means that it must be possible to send a message directly from every local computer to every remote computer.  That's one of the things network administrators must understand and get right.

For everything else (the remote computers) the computer just sends the message to the gateway.  So network administrators must also make sure that gateways know how to process non-local messages.    The format of the message is slightly different.  It is marked in a special way that tells the gateway "here's the IP address I really want this message to go to".  And, by the way, all gateways have at least two network connections.  One is attached to the local network.  The others are attached to other networks.  If the message is destined for one of those networks then the gateway reformats the message and sends it directly to where it goes.  It just gets rid of the special gateway information, leaves the "from" address alone, and changes the "to" address to the correct value.  Then it sends it along and it gets to where it is supposed to go.

But what if none of the networks that are directly attached to the gateway are the right one?  Then we just go through the gateway thing again.  The gateway computer has a gateway entry of its own.  It changes the "to" address to this new gateway IP address and sends the message along.  If everything is working ok then the message gets a little closer to its ultimate destination.  This "forward to the next gateway" process can happen many times.  To send a message from the West Coast of the US, where I live, to a computer on the East coast typically takes about 20 "hops".

The Internet is called a "store and forward" network because many of its components get a message in one interface and store it for a very short amount of time.  They then decide where it should go next and quickly forward it along to its next stop along the way.  Machines that specialize in this "store and forward" functionality are frequently called routers.  But gateways function something like a router and routers function something like a gateway.  It's a good idea to not get too hung up on the vocabulary.

I think I have probably exhausted you by this point so I am going to stop here.  But there will be more installments in the series.