The so-called "scientific method" is, I think, rather poorly understood. For example, let us consider one of the best-known laws of nature, often simply referred to as the Law of Gravity:
Newton's Law of Universal Gravitation: Every object in the universe attracts every other object toward it with a force proportional to the product of their masses, divided by the square of the distance between their centers of mass.
Now here is a series of questions for you, which I often ask audiences when I give lectures on the philosophy of science:
- Do you believe Newton's Law of Universal Gravitation is true?
- If so, how sure are you that it is true?
- Why do you believe it, with that degree of certainty?
The most common answers to these questions are "yes", "very sure", and "because it has been extensively experimentally verified." Those answers sound reasonable to any child of the Enlightenment -- but I submit, on the contrary, that this set of answers has no objective basis whatsoever. To begin with, let us ask, how many confirming experiments do you think would have been done, to qualify as "extensive experimental verification." I would ask that you, the reader, actually pick a number as a rough, round guess.
Whatever number N you picked, I now challenge you state the rule of inference that allows you to conclude, from N uniform observations, that a given effect is always about from a given alleged cause. If you dust off your stats book and thumb through it, you will find no such rule of inference rule there. What you will find are principles that allow you to conclude from a certain number N of observations that with confidence c, the proportion of positive cases is z, where c < 1 and z < 1. But there is no finite number of observations that would justify, with any nonzero confidence, that any law held universally, without exception (that is, z can never be 1 for any finite number of observations, no matter how small the desired confidence c is, unless c = 0). . And isn't that exactly what laws of nature are supposed to do? For Pete's sake it is called the law of universal gravitation, and it begins with the universal quantifier every (both of which may have seemed pretty innocuous up until now).
Let me repeat myself for clarity: I am not saying that there is no statistical law that would allow you to conclude the law with absolute certainty; absolute certainty is not even on the table. I am saying that there is no statistical law that would justify belief in the law of universal gravitation with even one tenth of one percent of one percent confidence, based on any finite number of observations. My point is that the laws of the physical sciences -- laws like the Ideal gas laws, the laws of gravity, Ohm's law, etc. -- are not based on statistical reasoning and could never be based on statistical reasoning, if they are supposed, with any confidence whatsoever, to hold universally.
So, if the scientific method is not based on the laws of statistics, what is it based on? In fact it is based on the
Principle of Abductive Inference: Given general principle as a hypothesis, if we have tried to experimentally disprove the hypothesis, with no disconfirming experiments, then we may infer that it is likely to be true -- with confidence justified by the ingenuity and diligence that has been exercised in attempting to disprove it.
In layman's terms, if we have tried to find and/or manufacture counterexamples to a hypothesis, extensively and cleverly, and found none, then we should be surprised if we then find a counterexample by accident. That is the essence of the scientific method that underpins most of the corpus of the physical sciences. Note that it is not statistical in nature. The methods of statistics are very different, in that they rest on theorems that justify confidence in those methods, under assumptions corresponding to the premises of the theorems. There is no such theorem for the Principle of Abductive Inference -- nor will there ever be, because, in fact, for reasons I will explain below, it is a miracle that the scientific method works (if it works).
Why would it take a miracle for the scientific method to work? Remember that the confidence with which we are entitled to infer a natural law is a function of the capability and diligence we have exercised in trying to disprove it. Thus, to conclude a general law with some moderate degree of confidence (say, 75%), we must have done due diligence in trying to disprove it, to the degree necessary to justify that level confidence, given the complexity of the system under study. But what in the world entitles us to think that the source code of the universe is so neat and simple, and its human denizens so smart, that we are capable of the diligence that is due?
For an illuminating analogy, consider that software testing is a process of experimentation that is closely analogous to scientific experimentation. In the case of software testing, the hypothesis being tested -- the general law that we are attempting to disconfirm -- is that a given program satisfies its specification for all inputs. Now do you suppose that we could effectively debug Microsoft Office, or gain justified confidence in its correctness with respect to on item of its specification, by letting a weasel crawl around on the keyboard while the software is running, and observing the results? Of course not: the program is far too complex, its behavior too nuanced, and the weasel too dimwitted (no offense to weasels) for that. Now, do you expect the source code of the Universe itself to be simpler and friendlier to the human brain than the source code of MS Office is to the brain of a weasel? That would be a miraculous thing to expect, for the following reason: a priori, if the complexity of that source code could be arbitrarily large. It could be a googleplex lines of spaghetti code -- and that would be a infinitesimally small level of complexity, given the realm of possible complexities -- namely the right-hand side of the number line.
In this light, if the human brain is better equipped to discover the laws of nature than a weasel is to confidently establish the correctness an item in the spec of MS Office, it would be a stunning coincidence. That is looking at it from the side of the a priori expected complexity of the problem, compared to any finite being's ability to solve it. But there is another side to look from, which is the side of the distribution of intelligence levels of the potential problem-solvers themselves. Obviously, a paramecium, for example, is not equipped to discover the laws of physics. Nor is an octopus, nor a turtle, nor a panther, nor an orangutan. In the spectrum of natural intelligences we know of, it just so happens that there is exactly one kind of creature that just barely has the capacity to uncover the laws of nature. It is as if some cosmic Dungeon Master was optimizing the problem from both sides, by making the source code of the universe just simple enough that the smartest beings within it (that we know of) were just barely capable of solving the puzzle. That is just the goldilocks situation that good DM's try to achieve with their puzzles: not so hard they can't be solved, not so easy that the players can't take pride in solving them
There is a salient counterargument I must respond to. It might be argued that, while it is a priori unlikely that any finite being would be capable of profitably employing the scientific method in a randomly constructed universe, it might be claimed that in hindsight of the scientific method having worked for us in this particular universe, we are now entitled, a posteriori, to embrace the Principle of Abductive Inference as a reliable method. My response is that we have no objective reason whatsoever to believe the scientific method has worked in hindsight -- at least not for the purpose of discovering universal laws of nature! I will grant that we have had pretty good luck with science-based engineering in the tiny little spec of the universe observable to us. I will even grant that this justifies the continued use of engineering for practical purposes with relative confidence -- under the laws of statistics, so long as, say, one anomaly per hundred thousand hours of use is an acceptable risk. But this gives no objective reason whatsoever (again under the laws of statistics) to believe that any of the alleged "laws of nature" we talk about is actually a universal law. That is to say, if you believe, with even one percent confidence, that we ever have, or ever will, uncover a single line of the source code of the universe -- a single law of Nature that holds without exception -- then you, my friend, believe in miracles. There is no reason to expect the scientific method to work, and good reason to expect it not to work -- unless human mind was designed to be able to uncover and understand the laws of nature, by Someone who knew exactly how complex they are.
Jump in the discussion.
No email address required.
Notes -
It is different from more aggressive forms of skepticism in that I take for granted that the universe is governed by unchanging laws and that inductive reasoning is valid in theory. The principle of abductive inference says, in effect, if I cannot produce a counterexample, there probably are no counterexamples. This requires a certain level of facially hubristic confidence in the power of your mind, relative to the complexity of the system under study -- even if that form of reasoning would work on that same system for a sufficiently intelligent agent.
I must admit, though, that the law of conservation of apple pies strikes me as pretty non-miraculous. I will think that over and get back to you.
I'm interested but not sure I understand your argument.
If inductive reasoning is valid why can't we go from "all observed masses follow Newton's law" to "therefore all masses follow Newton's law."?
Simply because there could be an object that doesn't?
I mean yes, there could be (in fact, we know there are), but assuming I don't know that Newton's Law fails, that I've only ever seen otherwise, why am I not justified in believing it?
This is a good question.
I think this puts the burden of proof in a strange place. The question is always why should we be able to make the inference, and according to what articulable rule of inference. But I will pick up the burden of proof and try to explain why we can't make that inference from all observed P are Q* to all P are Q, using the Raven Paradox.
Imagine that I see a few crows and note that they are all black, and I form the hypothesis that all crows are black. I begin to seriously pursue the matter by looking for crows, counting them, and noting their color. How many crows would I need to see, all of which are black, before I can conclude that all crows are black, or, more conservatively, that probably (more than 50% likely) all crows are black? Pick a number you think is reasonable. I'll say a hundred thousand; that sounds conservative.
Now the following is a theorem of first order logic: (for all x, P(x) => Q(x)) <=> (for all x, -Q(x) => -P(x)). Or to instantiate the symbols, all crows are black is equivalent to everything that is not black is not a crow. One way to see that that is a theorem is to see that whichever form you consider, a counterexample would consist of a crow that is not black.
But now the alternative formulation gives me an idea. It's not that easy to find crows, but it's really easy to find things that aren't black. Now there are about 150 million blades of grass in an acre of land, so I can go into my 1/8 acre back yard and find about 19 million non-black things (namely, blades of grass) that are not crows. That's waaaaay over what seemed like a reasonable threshold to establish that probably, everything that is not black is not a crow, which is logically equivalent to all crows are black. Hypothesis confirmed!
But seriously, can I prove that probably most crows are black -- let alone that definitely all crows are black -- by looking at blades of grass in my back yard? of course not. So that shows that this reasoning is not valid, even if some forms of inductive reasoning are:
I won't spoil the fun by resolving the paradox for you. Unless want me to.
Looking at blades of grass won't help you because you have prior knowledge that blades of grass aren't crows, and actually looking at them provides you with no additional evidence that is not subsumed by your existing knowledge.
If you started picking random things in the universe without prior knowledge of whether they are crows, and then it turned out that they were all non-black non-crows, that would be evidence. It would be very weak evidence since the universe is filled with lots and lots of things, but if you kept doing it you'd be gathering more and more evidence and if you somehow managed to look at every object in the universe and they were all non-black non-crows (or black crows), you would indeed have proven the idea.
Thanks for the statistically literate post. So please tell me,
by what rule of inference? If you say Bayes, it would be nice if you sketch your priors and your sampling method, to lend some plausibility to the answer.
That already describes my priors and sampling method.
A prior is a function from subsets of a given sample space to [0,1]. A sampling method is a description of a procedure that gives enough detail that I should be able to repeat the procedure and expect to get results not statistically significantly different from yours. I don't see either of those things here. Did I miss something?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Sure, I get the crows,and have an opinion on it too, but I thought you were making a point about justification for physical laws uniquely?
Is there something that singles out the laws of physics as uniquely unjustifiable, or are you simply saying that you can't prove a physical law the same way you can't prove all crows are black?
This applies to all universal generalizations over any set with large numbers of members we cannot directly test. The first critical part of my top level post is this:
So, statistical arguments cannot establish universal generalizations; nothing unique to physics about that. The second critical part is what I said in my first reply to your first comment:
There is an old joke that is relevant to the application of the abductive inference principle [credit to Kan Kannan, my doctoral advisor]: I tried whiskey and coke, rum and coke, gin and coke, tequila and coke, and vodka and coke, and got drunk every time. Must be the coke! Maybe nobody would be that dim in real life, but the principle is real. When we are doing experiments to gather evidence for a universal principle (coke and anything gets you drunk), we might be dim witted to actually look where the counterexamples are.
Here is a real-world example. I once assigned a homework problem to write a function in Python that would compute the greatest common divisor of any two integers a and b, and test it on 5 inputs to see if it worked. One student evidently copied the pseudocode found on Wikipeda (which is fine; real life is open book and open google), and submitted this program:
and these 5 test cases:
gcd(5,10) = 5
gcd(8,7) = 1
gcd(9,21) = 3
gcd(8,8) = 58
gcd(1000,2000) = 100
He tested big numbers and little ones, first argument smaller than the second, second argument smaller than the first, both arguments the same, one a multiple of the other, and them being relatively prime (having no common factors other than 1), and got correct answers in every case. So, in some ways it is a highly varied test suite -- but he probably could have written ten thousand test cases and still never found that the function is incorrect, because he systematically failed to think about negative numbers in the test suite, just like he did in his code (it gives the wrong answer for gcd(-10,-5). In one way of looking at things, negative number are atypical (in that we don't bump into them as often in ordinary life), and many people wouldn't think to test them; but from an objective way of looking at things, he systematically ignored half of the number line, despite straining to come up with a highly varied test suite. Must be the coke!
The point of the joke, and the example, is to illustrate how, when analyzing complex system with nuanced twists and turns, we might not have enough ingenuity to look where the counterexamples to our hypothesis really are. But what counts as a "complex system with nuanced twists and turns" depends on the complexity of the system under investigation, relative to the mental acuity of the investigator. So, what right do we have to expect that our little brains are up to the task of finding the "bugs" in our hypotheses about the laws of nature, when they are just barely (sometimes) capable of finding the bugs in a six-line program that is wrong for fully half of its possible inputs? If the source code of the universe is that simple, relative to the power of the little meat computers between our ears, it would be a miracle.
I get that when the sample set is unbounded/the known is unbound we can not define a hard number to some confidence of a hypothesis about that set, but I don't see how the principle of abductive inference isn't a statistical argument. Isn't it just some kind of logic similar to a Bayesian update? You have some hypothesis, every time time you fail to find a counter example that is evidence for the hypothesis. Isn't it just as flawed when dealing with a potentially infinite number of crows?
Also, isn't a fundamental difference between crows, and physics, is that we expect there to be universal laws? That is, we have no reason to believe there is a universal law governing the color of crows, but we do believe there are for how objects behave, right?
Next, I would suggest that the mathematical foundations of physics give reason to believe in universal laws. That is the laws of physics are deeply related and not as arbitrary as I think you imagine.
Its kind of like if you said "All Euclidean triangles have an internal angle of 180 degrees". And I said, "Well I can imagine a triangle with 181 degrees." I might think that I can imagine a Euclidean triangle with 181 degrees, but really I can't.
If we were pulling Euclidean polygons out of a box, I think you would be justified in saying "all triangles in the box have an internal angle of 180 degrees". This would have started out as an empirical observation, we would have pulled out polygon after polygon out of the box, counted the sides, and measured the angles, and noticed that the ones with 3 sides always had 180 degrees. However eventually someone would have discovered the mathematics that justifies saying all triangles have 180 degrees.
However, at the same time, it could be true that some of what we thought were triangles, weren't. Maybe they had a microscopic fourth side.
If we are pulling triangles out of the box, then they all have 180 degrees, and we will never see one that doesn't. But maybe they aren't triangles. And it turns out that the laws of physics are very, very much like this.
Good question. To answer it, we have to have a concrete picture of what statistical arguments really are, and not just a vague intuition that says "make observations and allow them to change your beliefs" -- see also this post :https://www.themotte.org/post/907/the-scientific-method-rests-on-faith/195677?context=8#context).
Statistical arguments are based, first and foremost, on random samples, and this is a premise of the theorems that justify statistical methods. Abductive inference is not based on random samples. On the contrary, statistical in based on decidedly nonrandom samples chosen in a deliberate search for counterexamples. In a random sample, you must pick with your eyes closed or the test is no good, and sample size is crucial; in abduction, you must cherry pick as the devil's advocate, trying to disprove the hypothesis, or the test is no good. This means you must be an effective enough advocate to have a good chance of finding counterexamples if they actually exist -- which is why abductive inference is not objective evidence, but rests on an article of faith in the capabilities of reasoner as an effective advocate to disprove the hypothesis in case it is false.
This seems like mostly a discussion of definitions. To me, what justifies the claim that failing to find a counter example makes the hypothesis more likely is statistics.
Regardless, I don't see how abductive inference solves the problem, if you claim that the laws of physics are crows. Why does abductive inference let me say "all crows are black" when I try hard to find non-black crows, what is the logic?
Anyways, the laws of physics are decidedly not crows. We can generalize about them because they can be mapped to mathematical constructs, crows can not. That is, we can make a universal law about electrons because what an electron is, is fixed by the mathematics of the theory. Now, we could be wrong about the theory, but if we are right then we can say things like "every electron has a charge of E", not because we looked and saw only electrons with charge E, but because the math says it is.
If you point me to a statistical method that can give objective evidence for nonzero confidence in a universal generalization (such as Newton's Law of Gravity), you will have taught me the most interesting thing I have learned in a month.
More options
Context Copy link
Well, that is the trillion-dollar question. The fact is that is an inference rule often used in the physical sciences -- and it is the only inference rule that can give us any nonzero confidence in a universal generalization (such as a universal natural law, such the laws of thermodynamics or electromagnetism). Statistical methods cannot give objective evidence for such laws.
So either (1) abductive inference is only good for generating useful fictions (with zero reason to ever believe they are anything but fictions), or (2) it can sometimes be used to yield nonzero confidence in certain universal natural laws. If you choose door number 1, so be it. If you choose door number 2, and you ask me where the logic is, I will tell you that there isn't any unless we are blessed with minds so powerful, and a universe so simple, that if counterexamples to the law existed, we would be tolerably likely to find them. So, unless the corpus of physics is a useful fiction, with no reason to believe anything in it is anything but a fiction, you tell me: where's the logic?
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I think this issue turns out to be pretty deep. Note, first, that apple pie is not a natural kind in physics, and is not of a character that it ever could or would become a natural kind in the domain of physics. That is, you will not find any mention of "apple pie" in a physics text that is not interchangeable with, say, "blueberry pie". For example, there could be a problem that says "Suppose an apple pie weighs 2 kilograms, and falls from a height of twelve meters in a vacuum..." -- but in this case, the apple pie is interchangeable with any other common sense object that might way 2 kilograms, and is just there to make the problem more fun than if it were a falling rock, or a falling stick. On the other hand, if we changed kilograms to pounds, or "in a vacuum" to "in a pressure of one atmosphere", that would change the problem physically. So, to restate, apple pie is not a concept that is mentioned in any law of physics, nor a concept of the sort that would ever be mentioned in a law of physics.
In that light, an apple pie of all things popping into existence is categorically more unlikely, a priori, than the sorts of things that are explicitly ruled out by the laws physics. Even a 2Kg object (in particular, of all weights) is not a natural kind in physics. The laws that actually prohibit apple pies from materializing and disintegrating -- viz. the law of conservation of matter and energy -- could, in theory, be violated in myriad ways that do not involve apple pies in particular, or flying teapots in particular, or objects that weigh 2KG in particular. And I do stand by my argument in the case of the law of conservation of matter and energy.
I still wouldn't claim to have gotten to the bottom of it (of what makes something a candidate to be a natural kind in physics, that is), but I do think that my argument is only supposed to apply to propositions that are actually candidates to be laws of the physical sciences, and the Law of Conservation of Apple Pies, for whatever reason, does not have that property.
More options
Context Copy link
More options
Context Copy link