A MarketPlace of Ideas                        

Likely you have heard this old adage about security and liberty.  The security they talk about is economic.  Things are likely to get tougher in the near future.   We are shipping too much money out of the country for oil.   We are too dainty to allow it to be drilled on our coasts and in ANWR.   Thus things will deteriorate more until the pain threshold is found. 

Many people will want to vote for the politician that says it will be ok if we only regulate this or tax that.  These are the unDemocrats who will say anything to get elected.  Say anything to get the power.  I would like to think you as an electorate will have the sense not to listen.  But I’m doubtful about this.  You listened in the past and you likely will in the future.   The problem is this sort of thing never helps economically and in fact damages something much more precious than your temporary economic circumstances and that would be your liberty.  I know this is true.  My favorite home away from home country is Brazil.   As currently constituted Brazil has a fully implemented form of government that the unDemocrats want to have here.  The people know their government and changing it is hopeless.  Nothing will ever change in that area and the Brazilians know it.   With liberty you can improve your economic circumstances.   With economic wellbeing you can not necessarily do diddly squat about your liberty.  I know its tough to be a good soldier.  You have to be brave.  And clearly about 50% of the population lacks any impulse toward liberty. Quite the opposite.  They lick their chops at the thought of another round of incremental carving up of liberty for the sake of financial gain. 

Remember losing a job is temporary.  Living in a land of curtailed liberty is pretty much for your entire lifetime.    So don’t be dour Gus Halls who secretly long for a beer, a recliner and a black and white television.   For an inspired life you have to dream of liberty.

In many physical systems the question arises what is the probability distribution that describes a system with a given expected energy E  over the interval from -infinity to + infinity?     Again you will use the maximum entropy principle to determine this.

The constraints are as follows:

•         …. sum over all probabities must = 1
•       …. given an average value AKA "mean"
•       ….. given an "energy" or standard deviation AKA "variance"

The langrangian is formed as follows:

     ….setting equal to zero to find the extrema point

Now the problem is to solve for the lambda coefficients.   I use a trick.  I assume the curve centered at the Y axis.   The curve must have the same amount of entropy on the left side of the Y axis as on the right or entropy will not be maximized.  Because the polynomial is even degree the probability curve must be "even"….that is symmetric about the Y axis. 

Solve for the coefficients using identity of the integral of normal -infin to + infin

Using the identity and setting it to 1:    yields:   

         The resultant distribution is:          which is the normal distribution.

 Now for the other coefficients the job is made easier by observing the distribution can only retain this even about the mean form if  the polynomial is in the form :  (x-mu)^2  : this form can propagate along the X axis without distribution shape change.

Observations

• base state of quantum harmonic oscillator is gaussian : It is maximum entropy.  
• gaussian wave packet can not mishapen because it is already max entropy.  If it changes shape it decreases entropy which requires force.
• gaussian is the base state of the wave packet?  Is it possible to have forms of the higher energy states?

In a previous article entropy was defined as the expected number of bits in a binary number required to enumerate all the outcomes.  This was expressed as follows:

entropy= H(x)=  

In physics ( nature ) it is found that the probability distribution that represents a physical process is the one that has the maximum entropy given the constraints on the physical system.   What are constraints?  An example of a probabalistic system is a die with 6 sides.  For now pretend you do not know that it is equally likely to show any 1 of the 6 faces when you roll it.  Assume only that it is balanced.

In the case of a die the above summation is equivalent to the following sort of computation:

• Initial assumption set of 6 probabilities that  sum up = 1  … this is a given as it has to be at least one of the 6 faces unless it stands on edge Twilight Zone style.  Lets assume P(x_i) = 0.05, 0.05, 0.05, 0.05,0.05, 0.75  …. you know instinctively this is not correct but demonstrates the maximum entropy principle

The total entropy given these probabilities = (.05) * (4.322) * 5 + 0.75 * (.415)= 1.0805 + .311= 1.39 bits

Let us use our common sense now.  We know there are 6 equally probable states that can roll up.  So its easy to calculate the number of bits required.  

• Bits required = log₂6 = 2.585 bits 

Thus we can see our initial assumption of probabilities yields an entropy number less than we would expect from common sense.   How do we find the maximum entropy possible? 

• Use the Langrangian maximization method. 
• Maximize the entropy phrase with the constraint that  

                 …. sum over all probabities must = 1

The langrangian is formed as follows:

Now differentiating the langrangian and setting the derivative = 0 we can find the maximal entropic probability

         solving for the P_i  yields

        All the P_i= the same constant with the probabilities summing to 1….Thus P_i=1/6 since N=6

While this is alot of work to derive the obvious it there is a purpose. In the case of more complicated situations where the probability distribution is not obvious this method works.  For example in the case of the Black Body emission curve of Planck.  Given just the quantization of energy levels you can derive the black body curve!!  This principle is woven all through nature.  Learn it because it will serve you well. 

Some interesting Notes to myself — myself? I meant me.

• Maximum Entropy Modeling - including some open source software

There are many mathematical definitions of entropy.  The mental picture I find most useful is to imagine the following:  

• you are put in a room and your job is to label everything in the room with a sharpy indelible marker and masking tape.  
• You are asked to label everything in the room using the binary numbering system.  This binary number will be that particular objects I.D.

As you go about this you may want to number the objects you most commonly refer to with the lower digits that have less length.  That way since you mention "FORK" much more often than "NUMBER 6 SCREW" you will end up having to say less digits.

The measure of entropy in this room is the number of binary digits required to number all the objects.  This is entropy.  The formula for this sentence that I just said is:

             Entropy ~= log₂N    where N is the number of different types of objects in the room

Now in a probabalistic situation with outcomes  x₁ , x₂ ….  x_n    with P(x_i) = probability of x_i

            entropy = H(x)= SUM [ -P(x_i) * log₂( P(x_i) ) ]      This formula calculates the expectation of the number of required digits to enumerate the outcomes.

Now let us compare this to a realistic situation in the form of the good old fashioned coin flip with

• P(heads) = 1/2 
• P(tails) = 1/2

Entropy = -(1/2) * ( -1)  -(1/2)*(-1) = 1 bit

Thus in order to enumerate all the states of a coin you require 1 bit.  So you just call heads bit = 1 and tails bit = 0 ….

… and thus you only need 1 bit. 

If you have a strange coin that always comes up heads:  That is to say P(Heads)=1 then:

Entropy = -(1) * (0)= 0

Thus for your weirdo coin that only flips out heads you need no bits to enumerate its states.  There is no entropy.   You always get heads sucker!  Or heads I win tails you lose!

The base state of an electron in an infinite potential well has the most "space" for the electron state.  Thus it has the maximum entropy. Take that same state and imagine pinching the electrons existence to nil in the middle of the trough.  Now you have state-2.  The electron now exists in a smaller entropic state and guess what?  It contains exploitable energy now. This is like a spring compressed.  The electron can decompress and exert force / expend energy.  For example in an interaction with another atom possibly a recoil could occur.   In a crystal lattice an electron can transfer its energy to the atom next door and in effect yield conduction.  All these are preliminary suppositions subject to more scrutiny. As mentioned before since the electron exists in this potential well in the form of free fall it can not have any acceleration.  Thus its distribution must thoroughly avoid the edges of the well were it would indeed experience accelerations by bouncing and recoiling off of the walls.