Monthly Archives: April 2013

Investing Computer

One of the most interesting applications of computer technology is in the field of investing.  It is interesting that with all the sophisticated systems and all the monetary rewards possible, that there has not been a successful program that can guide a broker to make foolproof investment predictions…..until now.  It is a fact that out of all the investors and resources on Wall Street, that none of them much better than just slightly above random selection in picking the optimum investment portfolio.  Numerous studies have been done on this subject that show that the very best investment advisors have perhaps a 10% or 15% improvement over random selection and that even the best analysts cannot sustain their success for very long.

There are lots of people that are able to see very near term trends (on the order of a few days or a week or two, at most) and invest accordingly but no one has figured out how to consistently predict stock rises and falls over the long term (more than 3 or 4 weeks out).  That was the task I attempted to solve – not because I want to be rich but because it seemed like an interesting challenge.   It combines the math of finance and the psychology of sociology with computer logic.

I did a lot of research and determined that there is, in fact, no one that knows how to do it but there is a lot of math research that says that it should be able to be predictable using complex math functions, like chaos theory.  That means that I would have to create the math and I am not that good at math.  However, I do know how to design analytical software programs so I decided to take a different approach and  create a tool that will create the math for me.  That I could do.

Let me explain the difference.  In college, I took programming and one assignment was to write a program that would solve a six by six numeric matrix multiplication problem but we had to do it in 2,000 bytes of computer core memory.  This uses machine code and teaches optimum and efficient coding.  It is actually very difficult to get all the operations needed in just 2k of memory and most of my classmates either did not complete the assignment or work hundreds of hours on it.  I took a different approach.  I determined that the answer was going to be whole positive numbers so I wrote a program that asked if “1” was the answer and checked to see if that solved the problem.  When it didn’t, I added “1” to the answer and checked again.  I repeated this until I got to the answer.  My code was the most accurate and by far the fastest that the instructor had ever seen.

I got the answer correct and fast but I didn’t really “solve” the problem.  That is how I decided to approach this investment problem.  I created a program that would take an educated guess at an algorithm that would predict future stock values.  If it was wrong, then I altered the algorithm slightly and tried again.  The initial guessed algorithm needed to be workable and the method of making the incremental changes had to be well thought out.

The answer is using something called forward chaining neural nets with an internal learning or evolving capability.  I could get real technical but the gist of it is this – I first created a placeholder program (N0. 1) that allows for hundreds of possible variables but has many of them set to 1 or zero.  It then selects inputs from available data and assigns that data to the variable placeholders.  It then defines a possible formula that might predict the movements of the stock market.  This program has the option to add additional input parameters, constants, variables, input data and computations to the placeholder formula.  It seeks out data to insert into the formula.  In a sense, it allows the formula to evolve into totally new algorithms that might include content that has never been considered before.

Then I created a program (No. 2) that executes that formula created by program No. 1, using all the available input data and the selected parameters or constants and generates specific stock predictions.  This program uses a Monte Carlo kind of interruption in which all the parameters are varied over a range in various combinations and then the calculations are repeated.  It also can place any given set of available data into various or multiple positions in the formula.  This can take hundreds of thousands (up to millions) of repetitions of executing the formulas to examine all the possible combinations of all of the possible variations of all the possible variables in all the possible locations in the formula.

Then I created a program (No. 3) that evaluates the results against known historical data.  If the calculations of program No. 2 is not accurate, then this third program notifies the first program and it changes its inputs and/or its formula and then the process repeats.  This third program can keep track of trends that might indicate that the calculations are getting more accurate and makes appropriate edits in the previous programs.  This allows the process to begin to focus toward any algorithm that begins to show promise of leading to an accurate prediction capability.

I then created sort of a super command override program that first replicates this entire three-program process and then manages the results of the outputs of dozens of copies of the number 2 and 3 program and treats them as if they were one big processor.  This master executive program can override the other three by injecting changes that have been learned in other sets of the three programs.  This allowed me to setup multiple parallel versions of the three-program analysis and speed the overall analysis many times over.

As you might image this is a very computer intensive program.  The initial three programs were relatively small but as the system developed, they expanded into first hundreds and then thousands of parallel copies.  All of these copies reading from data sets placed in a bank of DBMS’s that represented hundreds of gigabytes of historical data.  As the size of the calculations and data grew, I began to divide the data and processing among multiple computers.

I began with input financial performance data that was known during the period from 1980 through 2010.  This 30 years of data includes the full details of millions of data points about tens of thousands of stocks as well as huge databases of social-economic data about the general economy, politics, international news, and research papers and surveys of the psychology of consumers, the general population and of world leaders.  I was surprised to find that a lot of this data had been accumulated for use in dozens of other previous studies.  In fact, most of the input data I used was from previous research studies and I was able to use it in its original form.

Program No. 1 used data that was readily available from various sources from these historical research records.  Program No.3 uses slightly more recent historical stock performance data.  In this way, I can look at possible predictive calculations and then check them against real world performance.  For instance, I input historical 1980 data and see if it predicts what actually happened in 1981.  Then I advance the input and the predictions by a year.  Since I have all this data, I can see if the 1980-based calculations accurately predicts what happened in 1981?  By repeating this for the entire 30 years of available data, I can try out millions of variations of the analysis algorithms.  Once I find something that works on this historical data. I can advance it forward to input current data to predict future stock performance.  If that works then I can try using it to guide actual investments.

This has actually been done before.  Back in 1991, a professor of logic and math at MIT created a neural net to do just what I have described above.  It was partially successful but the software, the input data and the computer hardware back then were far less than what I used.     In fact, I found that even my very powerful home computer systems were much too slow to process the massive volumes of data needed.  To get past this problem, I created a distributive-processing version of my programs that allowed me to split up the calculations among a large number of computers.  I then wrote a sort of computer virus that installed these various computational fragments on dozens of college and university computers around the country.  Such programs are not uncommon on campus computers and I was only using 2 or 3% of the total system assets but collectively, it was like using 500 high end PC’s or about 3/4th of one super computer.

Even with all that processing power, it was more than 18 months and more than 9,700 hours of processing time on 67 different computers before I began to see a steady improvement in the predictive powers of the programs that were evolving.  By then, the formula and data inputs had evolved into a very complex algorithm that I would never have imagined but it was closing in on a more and more accurate version.  By early 2011, I was getting up to 85% accurate predictions of both short term and long term fluctuations in the S&P and Fortune 500 index as well as several other mutual fund indexes.

Short term predictions were upward to 95% accurate but that was out only 24 to 96 hours.  The long term accuracy dropped off from 91% for 1 week out to just under 60% for 1 year out….but, it was slowly getting better and better.

By June of last year, I decided to put some money into the plan.  I invested $5,000 in a day-trader account and then allowed my software to instruct my trades.  I limited the trades to one every 72 hours and the commissions ate up a lot of the profits from such a small investment but over a period of 6 months, I had pushed that $5,000 to just over $29,000.  This partially validated the predictive quality of the formulas but it is just 2.5% of what it should be if my formulas were exactly accurate.  I have since done mock investments of much higher sums and a longer investment interval and had some very good success.  I have to be careful because if I show too much profit, I’ll attract a lot of attention and get investigated or hounded by news people.  Both of which I don’t want.

The entire system was steadily improving in its accuracy but I was also getting more and more of my distributive programs on the college systems being caught and erased.  These were simply duplicate parallel systems but it began to slow the overall advance of the processing.  I was at a point that I was making relatively minor refinements to a formula that had evolved from all of this analysis.  Actually, it was not a single formula.  To my surprise, what evolved was sort of a process of sequential interactive formulas that used a feedback loop of calculated data that was then used to analyze the next step in the process.

I tried once to reverse-engineer the whole algorithm but it got very complex and there were steps that were totally baffling. I was able to figure out that it looked at the stocks fundamentals, then it looked at the state of the economy which was applied to the stock performance.  All that seems quite logical but then it processed dozens of “if-then” statements that related to micro, macro and global economics in a sort of logical scoring process that was then used to modify parameters of the stock performance.  This looping and scoring repeated several times and seemed to be the area that was being refined in the final stages of my analysis.

By June of 2012, I was satisfied that I can accomplished my goal.  I had a processing capability that was proving to be accurate in the 89 to 95% range for predictions out two to six weeks but it was still learning and evolving when I took it offline.  I had used the system enough to earn enough to cover all the costs of the hardware and software I invested in this project plus a little extra for a much needed vacation.  I never did do this for the money but it is nice to know that it works and that if I ever need a source of funding for a project, I can get it.

B-17 Miracle

The B-17 Miracle and PVT Sam Sarpolus

A mid-air collision on February 1, 1943 between a B-17 and a German fighter over the Tunis dock area became the subject of one of the most famous photographs of World War II. An enemy fighter attacking a 97th Bomb Group formation went out of control, probably with a wounded or dead pilot.   An Me109 crashed into the lead aircraft of the flight, ripped a wing off the Fortress, and caused it to crash. The enemy fighter then continued its crashing descent into the rear of the fuselage of a Fortress named All American, piloted by Lt. Kendrick R. Bragg, of the 414th Bomb Squadron. When it struck, the fighter broke apart, but left some pieces in the B-17. The left horizontal stabilizer of the Fortress and left elevator were completely torn away. The vertical fin and the rudder had been damaged, the fuselage had been cut almost completely through – connected only at two small parts of the frame – most of the control cables were severed, and the radios, electrical and oxygen systems were damaged.   The two right hand engines were out and one on the left had a serious oil pump leak.  There was also a hole in the top that was over 16 feet long and 4 feet wide at it’s widest and the split in the fuselage went all the way to the top gunner’s turret.  Although the tail actually bounced and swayed in the wind and twisted when the plane turned, one single elevator cable still worked, and the aircraft still flew-miraculously!  The turn back toward England had to be very slow to keep the tail from twisting off.  They actually covered almost 70 miles to make the turn home.

The tail gunner was trapped because there was no floor connecting the tail to the rest of the plane.  The waist and tail gunners used straps and their parachute harnesses in an attempt to keep the tail from ripping off and the two sides of the fuselage from splitting apart more.  British fighters intercepted the All American over the Channel and took one of the pictures that later became famous – you can easily find it on the internet.  The figher pilots also radioed to the base describing the empennage (tail section) was “waving like fish tail” and that the plane would not make it and to send out boats to rescue the crew when they bailed out.

Two and a half hours after being hit, the aircraft made an emergency landing and when the ambulance pulled alongside, it was waved off for not a single member of the crew had been injured. No one could believe that the aircraft could still fly in such a condition. The Fortress sat placidly until the crew all safely existed through the door in the fuselage, at which time the entire rear section of the aircraft collapsed onto the ground and the landing gear folded. The rugged old bird had done its job.

This event topped off an impressive streak of good luck that the crew of the All American experienced.  In all of the 414th Bomb Squadron for the entire war, they were the only crew that survived without a single major injury for their entire 25 mission assignment.  This incident was on their 25th mission and as a result, the entire crew were given orders to other non-combat assignments following their return from this flight.

See  http://www.reddog1944.com/414th_Squadron_Planes_and_Crews.htm

http://garfieldsteamhouse.org/History/WWII/WWII-B17-Survival-Story.php

B-17 “All American” (414th Squadron, 97BG)


 
That is the story that has been told and repeated for the past 70 years but there is something that has only recently come to light.  Lt. Bragg was busy flying the plane but he was in constant contact with the two waist gunners SGT. Henry Jebbson and PVT Michael “Mike” Zuk, as they kept Bragg informed of the condition of the tail and made their attempts to strap it to the rest of the plane.  Henry and Mike also tried several times to reach the tail gunner – PVT Sam Sarpolus – but there just was too much body damage to the aircraft.  All of the crew have since died except Mike and Sam and this new aspect of the story comes from Mike.  Sam was the youngest member of the crew at only 19 years old – with red hair and freckles.  Mike was the next youngest

I met Mike at a Silver Eagles meeting in Pensacola in 2004.  He was 81 and very frail and talked slow because of a stroke but there was nothing wrong with his mind.  Few of the other party goers were willing to take the time to talk to Mike but I did.  I took him into another room where we talked for more than 4 hours.  He told me about the flight and his life after that.  He became an enlisted pilot (a Silver Eagle) during the war and ferried aircraft over to England from the US.  When I asked him if any of his crew was still alive, he said, “Only Sam, and of course he will be for a long time”.  I wondered what he meant and asked.  He smiled and said there was much more to the story than anyone has ever said.  It wasn’t Henry and himself that held the plane to together.  It wasn’t Lt. Bragg’s careful flying…..it was Sam.

Mike went on, “the whole time we were flying that day after the collision, Sam sat backwards in the tail gunners seat with his hands out like he was stopping traffic and his eyes closed.  He never moved from that position….except once.  One of the fighters flew too close to us and his prop wash shook the All American hard.  We heard metal cracking and one of the two beams of the frame that was holding it together snapped.  At that moment, Sam opened his eyes and looked straight at the broken beam and pointed to it with one hand while still holding the other out “stopping traffic”.  Henry and I turned to look at what Sam was pointing to just in time to see a blinding light come from the break.  When our eyes cleared, we could see that the beam had been fused back together again.  We both snapped back to looking at Sam and he had gone back to holding his hands up with his eyes closed but he had a smile on his face.

He sat like that until after we landed.  They had to cut open the front of this gunner’s position and pull him out thru the window.  All the time with him holding his hands out.  Everyone thought he was scared or frozen stiff.  When he was put down on the ground, he still had his eyes closed.  I finally told him that everyone was out of the plane and he opened one eye and looked at me and said, “Really?”.  I assured him everyone was safe and then he put his arms down.  When he did, the old B-17 broke right in half – the tail fell off, the #3 engine burst into flames and the landing gear collapsed.  Sam looked at Mike and me and smiled and said, “Don’t tell anybody – I’ll explain later”.

It was three weeks later before we met with Sam in a quiet pub and had a long talk with him.  Sam said he didn’t know how he does it but he can move stuff and make things happen just by thinking about it.  He said he’s been busy during most of the flights keeping bullets from hitting any of the crew members.  We were the only crew that ever flew 25 missions without having a single crewman shot up.  We just stared at him and then both Henry and I said “bullshit” at the same time.  Sam said, “No, really, let me show you”.  He pulled out his K-bar sheath knife and handed it to Henry and told him to stab his hand.  Henry said, “No” so Sam said, “OK, then just stab this table napkin”.  Henry raised up the knife and plunged it down onto the table.  The table made a loud thud but the knife stopped about one inch above the napkin.  Henry pushed with both hands and then leaned his entire body onto the knife but it would not go that last inch into the table.  Sam said that it was harder to do bullets but he had a lot of practice.

We spent hours talking and testing Sam over the next few days before he went back to the US and we were reassigned to a USO tour to talk up our flight in the All American.  It seems that Sam has a rather well developed ability of telekinesis that allows him to control objects with his mind.  Not just move them but manipulate them even at an atomic scale.  That was how he welded the aluminum beams in the B-17 and created a sort of force field around each crewman when we were attacked.  We wanted to tell other people and told Sam that he would be famous if he would let us but he made us promise to keep it a secret.  Mike said I was the first person that he has ever told.  After telling me, Mike sat there very quietly as if he was regretted telling me.  I waited awhile and we sipped our drinks.  Mike finally spoke, “I wonder if Sam remembers me?”.  I asked if he had seen Sam since the war.  Mike said, “The next time we talked was about 1973 or so.  We met at a Silver Eagle Reunion in San Diego.  I didn’t know Sam had gotten his enlisted pilot’s license also.  That was the only reunion that Sam ever attended.  When I saw him, I recognized him immediately and then realized that the reason I recognized him so quickly was because he looked pretty much like he did 30 year earlier.  He had grown a mustache and dyed his hair but he did not look like he had aged much at all.  He and I went off into a corner of the bar and talked for hours.  It seems he liked helping people and he got a job as a paramedic on a rescue truck.  He was very well qualified and confided in me that he often used his powers to help him in an emergency.  Because he seemed to not age very fast he could only stay for a few years at each job but his skills were in high demand and he could get a job anywhere he went.  He also had had jobs as a policeman and a highway patrol officer”.  Mike would stop and stare at the floor every so often as he would get lost in memories and thoughts.

One of these moments that Mike stopped to stare turned into several minutes.  I said his name several times but he did not respond.  Finally, I touched his arm and asked if he was OK.  Mike got a grimace on his face and then grabbed his chest and rolled out of his chair onto the floor.  I recognized the signs of a heart attack and I called for help.  In an instant, a large crowd of people had gathered around him and calls for a doctor and 911 were shouted.  Someone put a large coat over Mike to keep him warm and another put a rolled up coat under his head for a pillow.

As I was sitting in my chair, holding his hand, someone with a hat on, bent down from the crowd and leaned over Mike.  He put one hand on Mike’s forehead and the other under the coat on his chest.  I thought it might be a doctor trying to check his vital signs but the person just frozen in that position.  I watched intently and then noticed a slight glow of light coming from under the coat.  No one else seemed to notice but I’m sure I did not imagine it.  After about 15 seconds, Mike opened his eyes and looked up.  He smiled and said, “Hi Sam”.  The man in the hat then got up and melted back into the crowd.  I asked Mike if he was OK and he said he felt fine that the he wanted to get up off the floor.

As I helped him up, I saw the man with the hat go out the door of the room we were in.  I sat Mike down and rushed out the door but there was no one anywhere in sight.  I rushed back to Mike who was shooing everyone away and sipping his drink.  I sat down with him and said, “Was that Sam?”.  Mike said, “Oh yea, he seems to come whenever I need him – that’s the third time he has done that”.  “Done what?” I asked.  Mike winked at me and said, “You know, you saw it”.  Then he said, ”I’m getting tired and I need to go. It has been good talking with you”.  I asked if we could talk again but Mike told me he was traveling back home early the next morning.  I asked if he knew where I could find Sam.  Mike turned to me and smiled and said, “I have no idea where he lives but every time I have needed him, he shows up”.

I spent two years searching for Sam with no luck.  I carried a picture of him from his days of flying the B-17 but had it cropped and colored so that it did not look like an old picture.  I showed it to anyone I thought might have seen him.  He did not have a social security number and there were no public records of his name anywhere in the US.  During my travels, I passed through Las Vegas and just out of habit, I showed Sam’s picture around.  The second night I was there, the desk clerk at my hotel said he recognized Sam.  He came about twice a year for only two or three days and played the roulette and Keno for a few hours in each of several hotels and then he would leave town.  He seemed to have remarkably good luck and the desk clerk said that he was always generous with the tips and always seemed to be smiling.  I smiled and agreed.

I figured I had been looking for Sam the wrong way.  Instead of trying to find someone that had seen him by showing his picture, I took another tack.  I started by looking in newspapers and online for unusual happenings that seemed to be unexplained or that were very much out of the ordinary.  I started with the first few days after he was last seen in Las Vegas and looked in a 500 mile circle around Vegas.  I was surprised at how many such events were reported on the internet and in YouTube videos but by reading each one, I narrowed it down.

One was for a small town in central Utah called Eureka – just south of Salt Lake City.  They had reported that someone had tipped a waitress at the local truck stop with $500.  It turned out that she needed about that much to be able to pay for a home medical device that her son needed for his severe asthma.  I drove to this small town and found the waitress.  Her name was Sally.  She was reluctant to talk about it because of all the news attention she had gotten but when I showed her Sam’s picture, she clearly recognized his face but she hesitated for a minute and then said that was not him.  I assured her that I was not a reporter and that I did not want to harm him.  I showed her my previous stories about the B17 and his days in the Silver Eagles.  She sat down with me in a quiet corner of the diner and we talked.  She said he was quick to pick up on her sadness about her son and he listened intently as she described the problem.  She had saved for an aspirator for her son Jimmy but times were tough and not many people were leaving tips and business at the truck stop was slow outside of tourist season.  When Sam left, he smiled and held my hand and said “thank you and say hello to Jimmy for me”.  Sally stopped for a moment and then said, “to this day, I don’t know what he was thanking me for – I only gave him coffee and he didn’t even finish that.”

I used the date Sam was here in Eureka and began the search again.  I found another story in Ketchem, Idaho where someone paid to have a house rebuilt for a single mother with four kids.  The husband had been killed in Iraq in 2009 and she had struggled to make ends meet but when a fire burned down their house, she was faced with having to send her kids to foster homes.  Someone paid a local contractor to build an entire house on their old lot and then put $10,000 into a bank account in her name.  She never saw the donor but at Perry’s restaurant on First Ave., a waitress that received a $100 tip confirmed that it saw Sam.

I repeated this searching pattern and tracked down more than a dozen places where Sam had stopped by some remote town or obscure business and helped out someone.  Most often he paid for something or gave money to someone.  About half the time, no one knew it was him but what he did seemed to follow a pattern.  He would show up just as some situation was about as serious as it can get and he seemed to know exactly what was needed and exactly who needed it.  He never seemed to stay overnight in the towns where he helped someone and he didn’t seem to do much investigating or asking around.  He often spent less than 10 minutes at the place where he did his good deed and then he was gone completely out of town.  I didn’t meet one person that knew his name.

I followed his trail up through Idaho and western Montana, then east through North Dakota and then south all the way to southern Texas.  He did his good deeds about every 300 to 400 miles about every other day.   Sam stopped along the way at casinos that were on Indian reservations and he also bought lottery tickets the day before the drawings.  He often won.  He always paid the IRS taxes immediately but I found out that he was using different social security numbers so that no one really knew who he was.

In Kansas, I found a State trooper that told me about a 25 car pileup that happened in a major storm on I-235 just outside of McPherson.  Lots of people were hurt but when the paramedics came, they found that no one had any broken bones or life-threatening injuries.  16 of the accident victims said that someone had come to their car shortly after the crash and “fixed” them.  They described a young looking man with red hair and freckles that calmed them down and then rubbed their legs or arms where it hurt and it stopped hurting.  The medics said that the blood found in some of the cars indicated that there had been some very serious injuries but when the examined the people inside, they found no cuts or bleeding from any of them.  No one saw Sam come or leave and most of them just called him an angel.

I don’t know who or what Sam is and maybe he doesn’t either.  He roams around doing good deeds, saving lives and bringing a little peace and happiness to everyone he meets.  He obviously wanted to remain unknown and I finally decided that I needed to honor that so I went home.