Scientific Method

The "Scientific Method" is (or at least was) presented in school in an early grade.  The standard version is something like this:

It's a 4 step cyclical process consisting of:
1.  Gather the facts (observations, results of experiments, etc.)
2.  Look for patterns and regularities in the facts.
3.  Formulate or revise theories, hypotheses, etc.
4.  Devise and execute experiments or plans for additional observations.
Go back to step 1 and repeat.

Science is done by people and is a much more chaotic process than the one listed above.  If you want to see a good and entertaining demonstration of the Scientific Method in action I strongly recommend a TV show called "Mythbusters".  The two main hosts (Jamie Hynaman and Adam Savage) have exactly NO standard scientific credentials.  They come out of the movie special effects industry.  This is handy because they have a good sense of how to put together an entertaining show.  But they also do good science.  Which leads to my first observation:  Science is a process or method and anyone who does it is a "scientist" doing "scientific work".  Science is not about who you are or what you know.  It's about how you go about doing things.  And the standard characterization of science and the scientific method obscure as much as they help.  So let's go straight to the basics:

• Science is the pursuit of truth.
• Whatever works is right.

These two observations are the core of what science is really about.  In fact, from a practical point of view, the reverse of the second observation is of more practical consequence to the pursuit of science.  Namely:  "Whatever doesn't work is wrong".  Modern science is not performed as it once was.  Why?  Because those old procedures sometimes led to things that turned out to be wrong.  So scientists went back and revised the procedure to avoid whatever problem was perceived to have led to the error.  Modern scientific technique as practised by professional scientists is far more complex and includes many more "checks and balances" steps than it used to in an effort to more reliably attain truth.

Does that mean that the simple procedures employed by the Mythbusters people will sometimes lead them astray.  Yes!  And they know it.  So, like good professional scientists, they occasionally do "revisited" shows where they look at earlier work after it has been criticized by fans of the show.  They will redo things a different way.  Sometimes the result comes out the same.  Sometimes it does not.  I think they have done the "chicken gun" test 4 times now.  Bitter experience has taught scientists the lesson that "always be prepared for your idea to be shown to be wrong" is critically important to science.

Scientists even go one farther.  A modern scientific concept is that of "falsifiability".  Is there some way, some set of data or some experimental result, that would definitely prove that your idea is wrong?  Now it may be that the experiment is very hard to do, even impossible at this time, but the concept of falsifiability is very important.  Scientists have learned to be very leery of situations where the answer is "no".  The easier it is to come up with and do an experiment where some result would falsify the theory, the happier scientists are.  So they like theories that make predictions that can be tested.  When Einstein came up with General Relativity he predicted that the Sun would bend the light from a star when light from that star came close to grazing the Sun.  Scientists came up with a situation where this prediction could be tested and went out and did the measurements, measurements that had never been done before.  The result came back the way the theory predicted and a different way than it would have come out if General Relativity was bunk.  This and other tests of General Relativity were why the theory, which is truly weird, was accepted fairly quickly in the scientific community.

And that brings me to another point.  Scientific "truth" is determined by a popularity contest.  Scientists argue about everything all the time.  For a scientific theory to become accepted wisdom that vast majority of scientists have to accept it.  There is no outside objective way to determine scientific truth.  What differentiates the scientific endeavor form others in not the "popularity contest" aspect but how the contest is scored.  So your mission, should you choose to accept it, is to convince the vat majority of scientists that something is true. And what convinces scientists is that thing I noted above, namely "whatever works is right".  The emphasis here is on the word "works".  So why would you believe that something works?

The obvious and traditional answer is that "some important and respected person says it".  The obvious problem is that important and respected people often believe things that turn out to be wrong.  Say an I&RP says "the sky is green".  But one day you go outside and look up and it looks awfully blue to you.  (Actually it is possible for small portions of the sky to be green under certain circumstances, but I'm talking about the whole sky.)  Now what if you poll 50 friends, enemies, and strangers and they all agree that the sky is blue?  At that point most people would agree that I&RP got it wrong.  And that's where science began.  About 400 years ago some people got together and they decided that I&RP frequently get it wrong.  They said "we can do better".

And they did do better a lot of the time.  But they also made a lot of mistakes.  So they tried to learn from their mistakes and figure out how to avoid them in the future.  And that's where modern scientific techniques come from.  Over the years since scientists have tried a lot of things to get to "truth that works".  Most of them ended up not working very well.  The few that consistently delivered the goods evolved into modern scientific technique.  And one common component of modern scientific technique is repeatability.  Is someone does an experiment and gets a certain result is that the end of the story?  No!  Time after time in the history of science people have tried to replicate the result of an experiment and failed.  And in many cases it has turned out that the original experimenter goofed in some way.  So cornerstones of modern science are:

• Publish not only your results but the procedures necessary to reproduce your results.
• Someone else needs to use your published procedures to reproduce your results.

If your results can't be replicated then they should not be relied upon.  And this has resulted in the creation of the modern science lab that movie and TV show makers like to have fun with.  An easy way for things to go wrong is for there to be something going on that no one noticed.  Imagine you are doing a soil acid study in an open field and a dog comes along and pees on your soil when you are not looking.  It is easy to see that your results might get skewed.  But if you move your study "into the lab" you can easily make sure that no dog can pee on your experiment.  This is a silly example of what can go wrong.  But the idea is simple.  You need to control your environment to make sure nothing is going on that you don't know about.

Another issue that led to the "lab" idea is complexity.  It turns out to be really hard to figure out what is going on if you change too many things at once.  If you change 100 things at once you don't know which changes are important and which aren't.  So scientists are most happy when they can change one and only one thing at a time.  I have only scratched the surface of all the ways scientists have found to get the wrong answer over the years.  But these modern procedures are all driven by the same simple idea:  whatever works is right.

Think about it.  If you want to convince me that something is true what is more likely to work:  "It's true because I say so" or "I did all these things and I was very careful and here's what happened".  Better yet "do this then this then this yourself and see what happens.  Now do you believe me?"  The thing that makes professional scientists different from you and me is training.  It's not the training in a bunch of "scientific facts" or the training on how to operate some piece of complex lab equipment.  It is the training on procedures, what procedures work, what procedures don't work, and why various procedures work or don't.

Science can be done by anyone.  It can be done by a couple of special effects guys with no formal scientific training.  It can be done by a nine year old girl with no special training of any kind.  (The "New England Journal of Medicine", one of the most respected scientific journals in the world, published a paper whose lead author was a nine year old girl).  The important thing is not who they are, or even for the most part what training they have had.  What is important is what they do.  Do they do things in a proper scientific manner?  Where professional scientists can make a contribution is to review their work to see if they did it right.  There has been a big change in Astronomy in the last 10-20 years.  Much of the work professional Astronomers now do depends on contributions by amateurs.  Professional astronomers have learned that many amateurs do work in a proper scientific manner.  And the use of amateurs results in a lot more eyeballs looking up.  And some amateurs even have equipment that is professional or near professional in its quality.

Now the scientific ideal is to do the easily replicable experiment in the lab then have others replicate it.  But often this is not possible.  A classic example is space shots.  The one and only space shot to Pluto is on its way as I write this.  Scientists can't bring Pluto into the lab and put it under a microscope.  They won't even get a chance to send a second space shot to Pluto any time soon.  So this is a case of "you go with what you can get" even though it is not "in the lab" and its not repeatable.  Scientists have a choice between getting nothing (no data about Pluto) and a less than optimal situation.  They go with the less than optimal situation, in this case, way less than optimal.  Scientists treat these situations very carefully.  The farther away they get from the optimal situation of a repeatable situation in the lab, the more careful they try to be.  So they looked a everything on the Pluto mission eight ways from Sunday while they were designing and constructing the mission.  That's one reason space shots are so expensive.  But the alternative is to spend less money but to get junk (e.g. not reliable) data.

Even so, it doesn't always work.  The first mission to Mars looking for life found it.  At least that's what the experiments said.  So did scientists say "we found life on Mars"?  No!  They said "we got interesting results from our Mars experiments".  Why?  Because this was not the first time a scientist had gotten a strange result in a situation where scientists knew little about what to expect.  Lots of things had gone wrong in this situation in the past so scientists had learned to be very careful.  And it turned out they had not found life on Mars.  They found out that the chemistry of Martian soil was not what they expected and the differences messed up their results.

Finally, science is done by people.  People make mistakes all the time.  They believe that they are doing an experiment in the proper scientific manner when they aren't.  They think they know how an experiment will come out or they hope that an experiment will come out in a certain way.  So they see what isn't and don't see what is.  This "wishing can make it appear to be so" is why scientists like to do "double blind" experiments.  These are experiments where the scientists don't know everything (like whether the patient they are examining was given the drug or not) while they are collecting the data.  If the scientist doesn't know (until later when the data is "unblinded") then the scientist can't "accidentally" bias the result.

Scientists are trained to do things in the proper scientific manner.  So they are more likely than an untrained amateur to get it right.  But doing things in the proper scientific manner only ups your chances of getting it right.  It may be that there is some yet undiscovered flaw in the current version of proper scientific manner.  But it may also be that doing things using a flawed procedure (e.g. not proper scientific manner) gets the correct result because the flaws do not turn out to be important in this case.  Looking up in the sky one day by yourself on a clear day and noting that the sky is blue gets you the correct answer even though "your sample size is too small to be significant".  In this case the effect is so great (always blue, never green) that you don't need to go through all the scientific rigmarole to get the right answer.  So doing things the wrong way may get you the right answer and doing things the right way might give you the wrong answer.  Is this really better than the alternative.

Yes!  And for two reasons.  First, doing things in a proper scientific manner is more likely to get you the right answer.  But more importantly, science is really good at determining that a particular answer is wrong.  Doing things in the proper scientific manner has demonstrated that what was generally believed to be true is actually false.  In the General Relativity example above most people believed that Newton got it right.  Einstein came along and said "Newton got it wrong in the following ways".  Scientists then followed up by looking very carefully in the places Einstein said to look.  And they decided that Einstein was right.  And before Einstein, Newton had come along first and said "these traditional ideas of how the universe works are wrong".  He then presented his theories and the experiments that showed that the new theories were right and the old ideas were wrong.

And in both cases Newton and later Einstein won popularity contests.  There was no august body that awarded either of them a "truth" prize.  In both cases they made convincing arguments that their ideas were better than those ideas that came before them.  Essentially, that the old ideas were wrong.  Both of their new ideas sounded radical and weird to people of their time.  People were comfortable with the old ideas.  They were very popular both with every day people and with elites, in this case scientists of the time.  But both of them won their respective popularity contests, especially among scientists.  Why?  Because they provided compelling arguments.

Science has become very complex.  It frequently involves mathematics far beyond what normal people can handle.  And many modern scientific concepts are truly weird.  But the way we got here is very basic and sensible.  Scientists went about the process of looking hard for the truth.  They tried to get the best data they could lay their hands on by, for instance, continually improving their instruments and tools.  Then they tested the ideas of the day to see if they fit the data.  Frequently they did not.  So they were discarded as false.  Sometimes all of the ideas of the day were found to be defective.  This forced scientists to come up with truly weird ideas.  Why?  Because none of the non-weird ideas worked.  And "[o]nce you eliminate the impossible, whatever remains, no matter how improbable, must be the truth".  Am I now quoting some august scientist or philosopher?  No!  I am quoting Sherlock Holmes from "The Adventure of the Beryl Coronet".  But Holmes, in spite of his fictional nature, is just speaking common sense.  If an idea is false, it's false.  If you are looking for truth it matters not if the idea is popular or "common sense" or any other thing not having to do with whether it is true or not.  Science would be a lot easier to do if only sensible ideas were true and all weird ideas were false.