Business travel.
Some people like it, some people hate it, some people just like it to get out of the office once in a while, even though they rarely often get to pick the circumstances as to why and where. For security folks, unless we’re masochists, it’s usually the latter, and here’s my tale of one of those first trips I ever had that wasn’t conference related.
This all began when I had my first actual title with “security” in it, a year or two after I moved back from Los Angeles at the end of the dot-com bust. I had two roles prior in smaller organizations, both on the edge of teetering into the abyss of irrelevancy and solvency. My role was at a Fortune 125 energy company in Baltimore, Maryland as a security engineer. I had enough tech experience, and we also didn’t have the stratification we do nowadays to be called “junior”, but it was my first one with that officially in my title.
I shared this time with a small but growing team, David , Ian , and our leader Brandon. Due to the timing I’m not sure Mike, Jeff, Spencer, Joel, Ben, Frank and a few others were on permanent staff at the time, but I know this allowed some of them to be hired and showed our worth as a team to the company. For that I am lucky, but it also scared the living shit out of me as to the reasons why.
For most organizations in the early 2000s, the level of security tooling that was not only available, affordable and could scale at truly enterprise levels, and still be useful for teams tasked with managing them was still pretty limited. The diversity of the market still had your McAfees and Foundstones as separate entities, Symantec was beginning their consumption of smaller security products, and places like Mandiant were still a twinkle in Kevin’s eye. Lots of stuff was still coddled together open-source solutions combined with whatever you could convince management to sign a check for.
For us, we had an IDS (intrusion detection system) architected and built by Ian, with his own even viewing console and rules manager. In fact, he ended up open sourcing that, so that was probably one of the cooler tools I had become acquainted with, the other, written by Ben did firewall audits. Both of which I think would be useful to this day for smaller teams, but I believe both have been lost to history and corporate forgetfulness. Basically, we had a Aldi/Family Dollar equivalent of a SIEM (security incident event manager), but it worked.
We proved it worked through a few detections of malicious activity we then responded to. Most were AOL Messenger spread message “viruses”, but we also had SQL Slammer and others of that era that we were able to detect on and respond to. Most of my days were spent writing, maintaining, and updating rules and I got kind good at it. We had a lot of noise due to lose rules detecting traffic, and while we had them set up on our various network segments, they weren’t down to the level of being able to watch certain business units closely and adapt to their ebbs and flows of network traffic.
At a point in time, about eight (8) months before this event that this whole blog entry is about occurred, we added a new network segment in the enterprise (well, actually two, but that one will come up in another story at a later time) for some of our M&A (mergers and acquisitions) activity we were undertaking. See, at this point, even after the Enron “glitch” the idea of commercial energy markets was unregulated and you had the ability to do power trading at the same holding company that also was a utility, but you just had to keep operations and some information separate from the two sides of the business.
This came into practice quite literally during our response to Hurricane Isobel in 2003, which again, will be another story in this series, but the company was quite adamant about this control, even in times of continuity of operations and disaster recovery activity.
But yes, in 2003, energy companies were flying high after deregulation hit, and commercial trading powered (pun sort of intended) huge swaths of consolidation and expansion of new capabilities. We even investigated providing internet access over our existing power infrastructure. I was happy that sort of went nowhere because we didn’t have the right team to also operate an ISP and ensure that it remained relatively secure.
This resulted in some M&A activity that brought us new small and medium sized companies that became part of the main enterprise but were still treated somewhat less trustworthy than the older parts of the organization. The benefit was, for example, in the information security team, the new entities were on their own segment, and we got to learn what was normal communications traffic, behavior of the hosts, and easier ways to scan and manage potential vulnerabilities. I’m sure that wasn’t super intentional to help our team, but it did, and this story wouldn’t be possible without, as painter Bob Ross had noted, “happy little accidents.”
In my case, for this story, we picked up some persistent, unique, and dare I say it, odd traffic for a corporate network one day on one segment in this zone. For the most part of “internet traffic history” in that day and age, we had to provide things like AOL instant messenger, ICQ and, *shudder*, MSN Messenger for our trading floor because instant messaging at the time also pretty much only had these three as your only standard bearers that had market penetration. What we didn’t typically see was IRC, or internet relay chat. I mean, we’d see some associated with people logging on to channels to update some chats for Everquest (as we called it “Evercrack”) and very early World of Warcraft groups, but we generally knew who it was, and it never rose to the interest or volume to warrant investigation.
The volume of IRC we saw coming from one of these recently acquired companies, however, took a very different term.
Again, at that time, we were running Snort, the open-source version, and we didn’t have the benefit of fancy port replicators that are now designed and sold for enterprises, such as Gigamon and others, so most of our detection was sort of hit and miss based on what we had volume-wise and reasonable rules for as to not flood our homegrown console. We weren’t doing deep packet inspection, we didn’t have a network flight recorder, and well, we didn’t have a huge budget for lots of toys. In short, what we had is what we had, and we had to make it work.
Every so often if you had a few moments and some interesting alerts, you’d dig in to see if it was an anomaly, a possible campaign (again, early days of this, so the concepts of APTs were still sort of new for the public). IRC was interesting because it was usually in plain text, was on a known set of ports, and was easy to classify. One of these days, when I saw an uptick on this new segment, I decided to do a packet capture off the interface.
Now, packet captures, in raw form, still need to be decoded. So, this required finding some tools that would work on the packet capture to carve out the types of traffic I wanted. This was before Wireshark was called Wireshark and was known as Ethereal, and the UI wasn’t as good as it was now (nor the feature set as complete) to do visualization. To make it easier to put into a report as well, I used a Perl program to colorize and reconstruct the conversations I was seeing. I really wished I knew what that tool was that I had gussied up and made work on this capture, but it made my life sooooooo much easier for this case.
What I was able to do with the reconstructed chats over a multi-week period was to determine, one of our employees was selling pirated satellite dish decoder cards. This also wasn’t just, like Austin Milbarge said in “Spies Like Us” and getting your family free HBO, this was doing reasonably large-scale distribution of cards, both domestically and internationally.
Now, at that time, I was familiar with satellite TV because my dad lived in an area of Maryland that was too far away from a DSLAM for DSL, old enough to have a single run of copper to his house, so even ISDN was a problem, and lived in a valley so most cellular signals were impossible to have any level of strength. I used that latter detail to effectively manage my work life balance when I had my first RIM device that definitely didn’t work once I got home, and therefore, I was electronically off the clock.
So, high speed internet access for stream TV was impossible, and in the early 2000s, was a bit off from our many options today. For him, it was approximately an 8’ terrestrial dish, like not Arecibo in size, but you still had to find and aim the dish across an arc with a motor and sextant to get TV. This was a lot easier when companies such as direct broadcast satellite companies such as Dish Network/DishTV, DIRECTV and HughesNet which were more prevalent in Europe and elsewhere before booming in the US. These had smaller dishes and the set-top boxes looked like normal cable boxes. In many cases, like cable boxes, they worked on the use of a decoder card, some line TiVO receiver and recording boxes, used a PCMCIA card format, but others had that hardware built in and, like SIM cards for phones, used a large format smart card for the decoding crypto keys for each account.
If you see where this is going – the cards were listed as smart, but they were in use because they were cheap to produce, and in turn, relatively easy to duplicate. There were plenty of “home business” and forums where these cards were churned out to allow for the “stealing” of these broadcasts on both the 8’ and smaller DBS dishes which were about a foot and a half in round diameter, or if you were in North America and were looking for equatorial satellites which were primarily Spanish language, the more oblong shaped. For a short period in our history before broadband was more ubiquitous, traditional cable TV wasn’t also carrying broadband, and streaming was not a thing, this was where you got your 200+ channels. Heck, I even had a Microsoft Ultimate TV receiver with DIRECTV, and when I lived in LA and complied with our apartment rules, I had it mounted on a 5-gallon pail full of sand I could pop outside a window and pull back in when I was done to not piss off the landlord.
In short, this was TV in the early to mid 2000s for many. It was popular, and with any popularity, there were folks looking to make a buck off it. Our person I found on our network was, as is now termed, doing their side hustle while on the clock in their office.
As I found out later, when finally getting to identify the person doing these transactions for sales, they were in a small office in rural Indiana. It was a nearly 30-year-old company that sold retail power to companies and municipalities in the state. The individual, as we found out later, spent their entire working career doing this. Many of the individuals in their office were also long-term employees as well, so blind eyes for this were pretty much everywhere, and I’m sure a nod and wink were given during slow times to allow for them to pursue these activities.
When I mentioned they were a relatively new acquisition and addition to our trusted network segment, I mean this was probably all transpiring within six months of them being connected. I’m sure those in Indiana never worked for a Fortune 125 company and figured the place was big enough for them to get lost in. Surprise!
What was constructed from those captures was a clear indication that, well, if they weren’t raising attention at home with the purchase of copying equipment and some extra cash lying around, there was enough volume that if the FBI was on those forums, they probably also knew what they were doing, if not who they actually were. Luckily, maybe for them, I found them first before they did, so this was going to be a softer landing hopefully.
I brought this case to my leadership, along with the trail of evidence, where it was, how long it had been going on (detected), and the supposed importance of this as a risk for the company. If I were to say to any front-line incident responder or SOC types – this is the bare minimum you probably need to have your leadership take action. In this case they wanted to move on it. What I didn’t know was – they wanted me to move on it, including going out and confronting the perpetrator.
Now, at the time, I was frumpy, out of shape, and really never had the air of authority or malice. I mean, I’m not too much better today, but I’m not in the Peter Falk category of investigator. So I asked how this was going to actually go down. I would be surprised by the answer and the events.
I was to be paired with one of our physical security team members, one who had worked in New York City as Diplomatic Security for the United Nations prior to his hire here. I liked Sean, but we definitely disagreed on our choice of sports teams, me a Red Sox fan, he a Yankees fan, and ne’r shall the twain meet. Bald, muscular, and visually intimidating, perfect for law enforcement and suited for corporate security. He was to accompany me on a trip to Indiana by way of Chicago. In fact, my first time ever coming and going from Midway, so I was excited to add a new airport to my list of places been.
What I wasn’t excited about was where our end destination was – Chesterton, Indiana.
If you ever wanted to know what people tend to think of as a typical small Midwestern town, in a relatively conservative leaning area, this would be your place. It’s a few miles off an exit on Interstate 94 in the North part of the state. The bonus we found once we got settled there after the flight in was that the steakhouse down the road from our hotel wasn’t half bad.
I am getting ahead of myself a little bit on the tale, but to say that was the high point of the excursion would not be the understatement one would assume.
This was also my first trip where, as a security titled person, particularly on the technical side, had to develop my own jump bag. For those in this space, things like a digital camera (I borrowed my dad’s mini Sony point and shoot), evidence bags, a toolkit, and other bits and bobs made it into my shoulder bag – a very overloaded Timbuktu messenger edition. I felt like I had most of what I’d need since most of the damming evidence was digital in nature and network based, but I needed to be prepared for what I may find in the office.
It’s been years, but I believe we arrived separately in Midway, as I lived closer to Dulles and Sean lived closer to BWI, so once landed, some coordination to make sure we synced up and grabbed our rental car. I love road trips, but I think Sean does not, so immediately for the hour plus drive to Chesterton, I did the commanding of the radio.
This was also the trip where I achieved the nickname of “Rain Man” as we scraped the radio dial for something we could both agree on, I was naming artist and song within a few notes, so much so that I believe the first half hour of the drive was Sean testing me and turning it into a game rather than an annoyance. I do forget what we settled on, but it was a good distraction for the nervousness I had.
I mean, I had no idea what I was going to be doing the next day, short of showing evidence I brought with me in the form of printouts and some stuff on my computer screen. I had no idea what the office looked like, what environment we’d be walking into, and what was to happen after we confronted our suspect.
What we did do was drive to the office we would be at the next day, a facility that was upstairs from another business up by the train tracks in town and near an agricultural complex for storage and other activity. For some reason, the pale-yellow cladding on the side of the building stuck with me, or maybe it was just my brain trying to rationalize this whole thing and put little markers in my memberberries.
After we did that swing by of the office, Sean said he wanted to get a t-shirt from the town, which was a thing he did on travel as a gift for his significant other. After asking around a few places like a gas station and I believe a drug store, folks noted souvenirs were to be had at the gun shop. Oh joy.
We arrived at the gun shop, it was stark white inside, much like the majority of the patrons, with guns on the wall on hooks and in cases, with a high shelf of weapons of larger caliber.
This was the point where Sean got to display his expertise, after we collected a shirt and were in line to checkout and scanned those high shelves. Working right to left, he pointed out the caliber of each gun, and the legality of them, with the majority of them being illegal to fire, so suspicions were that they were for display, since using a .50 cal machine gun would be overkill somewhat for close quarters home invasion protection.
The auditory bonus added to this entire experience, and possibly extended our time at the checkout, was the sounds of shots being fired above us and brass hitting the floor, as the range seemed to be upstairs. We got our answer there as to where the best place was to eat in that town, which was the steakhouse noted earlier.
We left and headed to check into our hotel, where I dumped my overnight bag and jump bag, got a soda from the lobby machine, and got ready to go to dinner and go over what the next day would entail.
From there and dinner, we basically outlined the timeline of arriving, going up to the main floor, asking to use the conference room, with our backs to the windows and facing the exit door, and bringing in our suspect to lay out what we found and what the next steps would be in his employment at the company.
The rest of the evening wasn’t so much of a blur, because near zero jet lag of hopping over to Indiana, a state that never completely decides on Central or Eastern Time Zones for daylight savings time, but more of excitement that would be the coming day.
We both went to bed relatively early, full and happy after the meal, and I believe at least one alcoholic beverage to take the end of the day off. The next day brought us lunch at the hotel lobby, as much as you can get from those things, and then with a stolen granola bar from that breakfast area, headed to the local office.
We pulled up, went over things once again from the night prior.
I was asked one very special question that gave me pause with the response I got from Sean.
“Do you notice anything on my leg?”
To which I answered, “No.”
He replied, “Good,” and then proceeded to roll up the cuff on his right pant leg to display an ankle holster with a small pistol within it.
My eyes grew big, and I took a beat.
In that moment Sean noted, “If you hear me say ‘down’, you get out of your chair and under the table and stay there until I give you ‘clear’, got it?”
I replied, “Yes,” and then a beat later uttered, “Um, why the gun?”
He explained that since we were the interlopers into a tight knit office, many of the employees inside had worked together for a while, some for multiple decades. So, the unknown number of people, and given the firearm laws in the state, it was out of a modicum of caution to be packing.
In my mind, this would have been a better thing to know before I agreed to this trip and my own participation in the next half hour or so we were on site. However, he dropped his cuff, I grabbed my shoulder bag, and we exited the rental car to go to the front door of the office.
We walked upstairs. Sean asked the office manager where our individual was, and to have them join us in the conference room.
As I could see what I could see of the office area, it had about a dozen or so places where people could work, but maybe less in that number actually on-site that day. Just as I entered the conference room, I think I spotted our perpetrator and subject out of the corner of my left eye, as they muttered a few words to their coworkers before joining us.
Now, I deflect a lot. I tend to joke around when I’m nervous, and typically as an only child, got blamed for everything and became very defensive as a person growing up.
When we did intros of ourselves to our subject and explained why we were here, the Caucasian individual across the table from me became bleached sheet, proper ghoul transparent, white. They finally realized that they were about to lose their job because of what we found, but the lizard brain kicked in for self-preservation, and they pleaded forgetful or stupid for much of what we laid out just talking about it.
However, next up was me, where I pushed a folder of our printouts with the chat logs, colorized and cross-referenced to his computer and his login account, as well as what I would also do, is provide open source intelligence to get what their role was on the trading/sales website where these satellite cards where trafficked. I rotated my laptop screen around like the do on TV and in the movies, to what we’ve become accustomed in popular media.
Now, not to sound all dramatic after I set up this movie-like, pitch perfect scene that usually is the gotcha climax to events like this, but I couldn’t for the life of me remember what was on that screen. Maybe I was showing him the tools I used, the raw data, or some more of our research to prove the jig was up, but it’s been nearly 20 years, so my brain has decided to evict that detail, so, my brain said “sorry, I can’t do that.”
At this point, I think we did see the start of tears. I mean, this person was near retirement, and not entirely sure of what the motivations were for this extracurricular activity in the middle of Indiana by a person who really didn’t appear super computer or tech savvy. I did applaud them for not exactly fitting a profile I had suspected somebody like this would be. But, I don’t think I’d want to be in the room when they got home and had to tell their spouse, who they’ve probably been with the same length of time or longer, that they’d be out of a job, and while not knowing what the separation agreement was to be like, may have not only lost benefits, but possibly had their retirement seized as well.
Regardless, this was going to be ruinous for them.
Sean accompanied them to their office to collect their personal items and anything that would allow them access into the office area after they were escorted out. We showed some compassion with the escort out and allowed them to say some select goodbyes. I don’t know who was instructed to let them know that corporate would be in touch with them about their exit details, but as we neared the door, we reiterated that same amount of detail on that procedure as we also reached the door.
As my body was still pumped on a little adrenaline, the next couple of minutes were a little bit of a blur. I’m not sure if other teams called into the office, indicating what they needed to do and not due in relation to our subject, but more or less there was a definite chillier feeling in the space as we wrapped up.
Sean and I went back to the hotel to get our bags, and we collected ourselves to the rental car and drove back to Chicago and Midway.
I reviewed some of the photos I had taken using the back screen of the digital camera to pass some of the time. Sean left the radio to the last station we ended on, and that accompanied us and traffic back to the airport. Upon arriving I grabbed my overnight bag and shoulder bag. Sean grabbed his suitcase, and, as I now know, a gun case.
Now, I noted we arrived separately, hence why I didn’t catch the gun and its case earlier. I’ve always had a question of how folks travel with this, since this was also the first time, I traveled with anybody who had a reason to be carrying. At the counter I saw him open the case, and mind you this was only a few years after 9/11, display the weapon and show it didn’t have a chambered round. Then he checked it, the same way you’d do it for your golf bag or skis, or you family’s large steamer trunk. All my computer and tech gear were in my Timbuktu, so that was my carry-on, and Sean just had himself in a nice suit and shoes, wallet, and his travel docs to contend with.
This is when I realized the weakness in my own travel plan, this being my first jump bag, and hadn’t learned the right way to do this until years later at Mandiant. This was also in the days where TSA sort of standardized on containers for the boarding security.
I had my entire tech bag spread out among like five or six trays, everything lying flat and unencumbered. Of course, with a train of gadgets like what I was carrying, it got rechecked a few times. Upon clearing, I thought I was late for the gate and plane and tried to recombobulate that pile of gear like a squirrel on tainted amphetamines. I’d only find out later that the camera my dad lent me and had photos of the facility and other travel pics remained at that security checkpoint in Chicago. Oh well, and fudge.
When we did get back into the office for a debrief of the trip, we went over our memories of how it went down. This was the first time, as I was told, that the physical security team worked along with our information and computer security teams and given the level of sophistication that this subject had done as a crime, both groups were placed under our corporate Chief Risk Officer.
It wasn’t until many years later that companies really considered that every organization needs a risk executive, and even less so had their security teams stationed under that leader rather than a CIO or similar tech executive. This provided us more independence away from IT since we had “IT Security” as our team’s title up an until this point. We were later retitled as the “Information Protection Unit” and resided under the CRO and worked very closely with that physical security team, who’s other purviews included securing buildings, our plants, and other facilities.
The move, as we were told at the time, was to avoid the “fox guarding the hen house” when it came to us having to report up through the CIO function. It felt like a wise move, and honestly still stand by the reasoning in a lot of cases, depending on how your business is structured.
As for the person who was the subject of all this effort. I was told the case was turned over to the FBI, as it constituted very illegal interstate fraud. One would suspect that when the individual returned home and went through the time taken to reason with their family and think about what’s next, and that “oh goodness, thank god law enforcement didn’t get involved,” only to have a knock on the door moments after that and then get arrested.
I mean, I know it sucks, and many of you from the security community who may read this had possibly done stuff much worse or feel that the idea of the entire subject matter, that of cracking encryption is a fun challenge, and that this isn’t a big deal. At the time, the second order effects on the company, if not shared with law enforcement could have resulted in reputational harm that would adversely affect the company, especially one that rides a fine line in many areas of interest.
You will be getting another story shortly on why this matters as well, so stay tuned. It’s also the tale that got me out of forensics as my main occupation at the time and allowed Spencer and Mike to build up their cyber investigations and forensics team. So, it’s not that bad.
In the end, this was the first criminal I ever caught in my career. I didn’t get to go to the trial or anything and I don’t know what happened. Maybe I’ll try to google bits and pieces of the story I know to see, but I still keep this in my mind whenever I think there may be a victimless crime. The penalties for a little thing can be magnified if you pick the wrong thing to do, and I know I’m not smart enough to really make those calls, so I do keep my nose clean, you probably should to. I think Dunning-Kreuger was primarily designed for criminals and wannabe criminals, because you’d be surprised who’s watching when you’re not dancing.
Till next time kids.
