Chaos Theory and an Anecdote
Oct. 16th, 2010 12:32 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
deirdreI don't often show my inner geek on my LJ, but I feel it's worth a story.
I remember very clearly when I was reading Benoit Mandelbrot's work -- I was consulting for a company that was at least as global (maybe even more global) than the company I work for now. It shall remain unnamed.
I worked on a team about the size of the one I now work on; I was the only female software engineer (we had a woman tech writer and a woman tech support staffer, though).
I got into diagnosing weird problems as a hobby. My cube was opposite tech support and I kept hearing about application problems with one office. I got a network map of the company's global infrastructure (at the time, they had one quarter of all network lines into China, and that consisted of one sub-T1 line...).
I had only logs about failures, and very very very limited information. Yet, it was true. This relatively close office dropped connections more than all other offices, even the ones over satellite links thousands of miles further away. Continents away, even.
I wrote logging into the application, discovered that one particular version of a network stack was a problem. Got all those upgraded. Still the issue persisted, but was a little less bad. When a third office's traffic got routed through the problem office -- rather than the other way -- that office's traffic also failed.
My findings were escalated all the way up to VP level, which was five or six or eight rungs or something. Meetings were held. I was flown to the other office as obviously I had a better sense of what was going on than most everyone else did.
I'm walking around with the guys and I see a bunch of wire on a floor wrapped around a small post. "Is that part of the network?"
Yes.
"Is that metal?"
Yes.
I'm not sure if that bunch of wire took the network length out of spec for distance, if it created a significant magnetic field, or if it created a significant antenna. One or more of the three for sure.
I reported my findings, the wire was re-routed and shortened, and that office never again had trouble with our applications.
I doubt, had I not read Mandelbrot, that I would have had the confidence to see that through. It was a problem solved by sheer curiosity and dogged determination -- along with some sense that there was more to the problem than anyone else seemed to think.
You see, before, they'd considered re-writing the applications, because clearly we didn't know what we were doing. But it was a physics problem, not a software one.
RIP, Benoit Mandelbrot. May you find order in the hereafter.
I remember very clearly when I was reading Benoit Mandelbrot's work -- I was consulting for a company that was at least as global (maybe even more global) than the company I work for now. It shall remain unnamed.
I worked on a team about the size of the one I now work on; I was the only female software engineer (we had a woman tech writer and a woman tech support staffer, though).
I got into diagnosing weird problems as a hobby. My cube was opposite tech support and I kept hearing about application problems with one office. I got a network map of the company's global infrastructure (at the time, they had one quarter of all network lines into China, and that consisted of one sub-T1 line...).
I had only logs about failures, and very very very limited information. Yet, it was true. This relatively close office dropped connections more than all other offices, even the ones over satellite links thousands of miles further away. Continents away, even.
I wrote logging into the application, discovered that one particular version of a network stack was a problem. Got all those upgraded. Still the issue persisted, but was a little less bad. When a third office's traffic got routed through the problem office -- rather than the other way -- that office's traffic also failed.
My findings were escalated all the way up to VP level, which was five or six or eight rungs or something. Meetings were held. I was flown to the other office as obviously I had a better sense of what was going on than most everyone else did.
I'm walking around with the guys and I see a bunch of wire on a floor wrapped around a small post. "Is that part of the network?"
Yes.
"Is that metal?"
Yes.
I'm not sure if that bunch of wire took the network length out of spec for distance, if it created a significant magnetic field, or if it created a significant antenna. One or more of the three for sure.
I reported my findings, the wire was re-routed and shortened, and that office never again had trouble with our applications.
I doubt, had I not read Mandelbrot, that I would have had the confidence to see that through. It was a problem solved by sheer curiosity and dogged determination -- along with some sense that there was more to the problem than anyone else seemed to think.
You see, before, they'd considered re-writing the applications, because clearly we didn't know what we were doing. But it was a physics problem, not a software one.
RIP, Benoit Mandelbrot. May you find order in the hereafter.