I swear to god I plan on actually watching all of “Schitt’s Creek” someday, but I get the reference. I grew up watching SCTV, so Eugene and Catherine are very familiar to me, and I’m more likely to allude to Bob & Doug Mackenzie than I would, say “Friends”.
But, as I noted in the last episode of stories, I wanted to move on to this tale as it carries on the reasoning as to why our little “IT Security” team became the “Information Protection Unit” in short order after a series of events.
The energy company I worked for seemed less and less like a utility when I was there and more like an extension of a Wall Street firm. When one of our traders was off-line for any reason, that was possibly six million dollars an hour that could be lost just with that one person. Then, business was good, and when you were diversified across the country, you weren’t going to fail like Enron did.
I lived in Los Angeles when Enron managed to screw over 99% of California during the Summer of 2000. I didn’t get the full experience of this because my apartment was in the still, regulated, city of Los Angeles, versus the surrounds who didn’t incorporate into the city and surrounding counties, who were de-regulated areas, and had power delivered via market prices on a day-to-day basis. Pretty interesting way to extract a profit if you can, and one way to keep a captive audience at best.
Since by the time I got to my energy company, Enron had, for the most part imploded, and some of the castoffs in that process found their way to the company I was at. They at least learned the hard way about energy trading, and with the “starter marriage” with their time at Enron, our floor and operations were a fair bit cleaner and legal. But it doesn’t mean we could do what we wanted, as the Enron debacle had close eyes by the Federal government, regulators and others to ensure we were operating by the book.
What that does to leadership is to keep them in a relatively risk adverse operating mode, and any threat to toppling the apple cart, so to say, and ending up in the spotlight is highly avoided.
This too, was probably a main reason for our move under the Chief Risk Officer (CRO) versus remaining under the Chief Information Officer (CIO) – as a way to manage other aspects of risk by treating what we do in our technical capacity as protecting “information.” The tale about to be told, pretty much sealed the deal.
So, in our last story, I mentioned I had a bodgered up Perl script to mark and color code IRC conversations, and only did that because reconstructing them via Snort alerts and truncated captures wasn’t exceedingly useful. In other cases, and thank goodness, back in the day, and by that, I mean early 2000s, end-to-end encryption wasn’t demanded nor the norm for chat programs, and “HTTPS Everywhere” wasn’t quite a universal mantra for websites.
All of this made detecting events on the wire, and when I mean wire, I mean our corporate network, a lot easier and required less hardware. Nowadays, SSL/TLS intercept requires your network to be architected in such a way, and the use of proxies or other tools to do interception. I left this job just as they started becoming more prevalent, and while at the World Bank, was the first time I was working with teams to spec out the architecture for a web proxy that did that work that was somewhat affordable and manageable.
Given that we were in the earlier days of instant messaging and end-to-end encryption, eavesdropping on messages outside the normal sections of the network we expected to have higher volumes of that traffic like our trading floor, the more interesting the content when it showed up from a place it didn’t normally come from. This reasoning also helped us refine our intrusion detection system (IDS) rules better, as we had a map of how the network was overlaid on our business units and could determine if things seemed abnormal. This was also before behavioral network analysis tools were widely available, and this would have possibly been easier if we had that as one of our tools.
I know one of the questions on the US Department of Homeland Security (DHS) “Cybersecurity Service” exams asks about IPS, “intrusion prevention system” design and use, and sadly, I have yet to hear about one of these things actually being installed correctly or running efficiently. You need business rules to inform how and where you set things up, so, like before while Snort had IPS capabilities, we didn’t use them because we felt, even our well mapped and documented network normality, it wouldn’t work without causing business disruption.
This feature is important, because much like firewalls, IPS devices are more policy devices than any security device. They particularly trigger on what you know and what you are expected to know or be aware of. If you know the business flows and what exactly your network devices are supposed to be using in the way of protocols and application services at the network level, you can just say “no” and it doesn’t happen. Even DHS’ continuous mitigation and diagnostics (CDM) program along with the EINSTEIN program were just watching the wire (egress filtering), with no real capability or, as it seems, desire, to prevent badness from happening.
How these play into this story, is that if we had these harder rules in place, the length of time of these events could occur would have been zero, but we’d not have been able to also, in the words of a popular NBC news show, catch a predator.
Now, in most corporate environments, when you log on to your computer, you’re presented with an acceptable use policy, or AUP in tech person parlance. Some are long and very much authored in the best legalese money can buy, and others are simpler, and economize on words in only a way I wished I could. They do, however, govern what people should and shouldn’t be doing on their work provided technology equipment. I mean, I have another story off of just that, and a reason some technologies are not quite architected for enterprise consumption, but again, for another time.
My co-worker at the time, and peer, Ian, gave me the best rule of thumb about how to behave within a corporate environment. While I can’t do this in his light Scottish brogue, it was “don’t do anything on your computer at work that, if discovered, wouldn’t mind having your mother read about in the newspaper.” Generally, to be honest, a good rule of thumb for everybody to be following to ensure they keep their jobs in some situations, so take this as a well-worn nugget of advice.
Our events of this story were a combination of detected web traffic, IRC and primarily instant messenger traffic. The person who violated our AUP was located at our emergency operations center (EOC) on the West side of Baltimore outside the Interstate 695 beltway. The proximity here to a few locations, the role the facility plays in keeping power on for millions of people and thousands of businesses in the state, and the nature of the potential offense is important.
For the power company, and not just the trading arm, has yards and other management and offices around a good portion of Maryland, which is primarily its service area. This is important as it’s part of the network of grids that make up regional power control and distribution. The same way Texas’ ERCOT manages and distributes power from various generation and distribution partners. We were part of the Pennsylvania, New Jersey and Maryland Interconnect, PJM for short, and played a role in the Northeast Blackout of 2003 from being more of a national security event, but again, another story.
Our EOC has a literal underground bunker, for when shit really hits the fan, there’s a bunch of “lucky” folks that spend all day down there ensuring the safety and operations of the grid they have control over. I was on our trading floor during the blackout and it weas crazy, I could only imagine how many cigarettes were smoked chain-wise down there during that. They are trusted entities, folks who work there and have access because of their potential impact to people’s lives and, because we’re next to DC and service the national security apparatus like Fort Meade, which houses the National Security Agency (NSA), it’s good to keep the lights on more than just proverbially.
Well, one of our trusted entities went rogue, and by that, they appeared to be soliciting sex with a minor using our company equipment and network.
This was before the days of smartphones and the like, unless you were really good at T9 texting, and somebody else was willing to wait, these types of online conversations occurred over a desktop computer. If you didn’t have one at home, and in some cases, it was still not as widely an appliance in most homes in the early 2000s, so you use some equipment at work to do things. Play on-line games, gamble, listen to streaming music, and maybe do some personal errands and billpaying. Some use was definitely in the bounds of personal use, but soliciting text was definitely not in those bounds.
With any investigation, whether it be digital, or something in the more physical parts of the world, you need to collect evidence for the case, even if it’s handled internally and isn’t for law enforcement. Legal would need it for any separation where actions required proof, to learning from for other events and knowledge sharing among the security teams, or in some cases, where the offense is so egregious that it is not only handled by law enforcement, but by special divisions of law enforcement.
For things turned over to law enforcement, at the time, what was established as admissible into a legal suit and what actually was a chain of custody was still being regularly tested. Some of my friends are lawyers who were involved in some of these earliest cases, and whenever there is a new tort or case precedent which is set, cheer loudly because it often makes their jobs easier.
I’ve become a lot better note taker recently, especially as I’ve been in places where it’s vitally important to get detail and context from a meeting or other event. Then, back at the time of this story, I was still learning the ropes, so, I’m not saying I was sloppy, just that now being more verbose in what’s collected and kept is now an involuntary way I approach things and wasn’t before. Like these stories, I also heavily relied upon my memory, I wished more people had that backstop – but wouldn’t recommend to often utilize it when in a domestic discussion.
For this instance, I had to provide a reasonable preponderance of guilt to get leadership action, and once again we got paired up with our physical security folks to help close this out. The other part here which was unique, was this perpetrator was regular, almost like clockwork, but dependable enough to be able to pinpoint who it was after a while.
Since those evidentiary processes for live network activity were a little ill-formed, our ability to catch this person, was, to catch them in the act.
Remember when I noted our EOC had parts of it like a bunker, and this was also in the days before smartphones and high-speed mobile data. Imagine trying to coordinate a live takedown of a perp in a bunker. It’s not cool like in the movies. There’s no quick switching between CCTV (I should know, I helped with our SOC there too) following both parties. There’s no cool Hans Zimmer-esque soundtrack. And, while the anticipation is similar to the movies and TV, you really don’t look like James Bond doing it.
So, on the day where we had planned to bring in our pervert from the cold, we had our physical security person on the phone. Remember those really ugly wired earpieces for cellphones (and not the cool ones for say, security details), our security staffer had one. I think the reason we didn’t use radio or similar was the distance form my office and the EOC, and they wouldn’t let me use one of those radios, since I was just one of those network techies.
I got my revenge later by extracting a promise for them to teach us how to shoot their automatic rifles at the training range in exchange for allowing them to store the safe with those guns in my lab and our disaster recovery facility. Sadly, I left there before I got range time, but I felt we’d be less “geeky” if we knew how to shoot straight. (Insert some wise analogy or metaphor sadly taken from Sun-Tzu.)
Anyhow, back to the guy creeping up on the creep in our offices.
I did forget to mention what was notable about this person and why we got a little more attention than some random person doing it a lot less frequently. And by notable, I re-emphasize the frequency comment. I did say they were predictable. Enough so for us to stage this sort-of-raid. Frequency also was related to how many simultaneous chat sessions they had open at one time.
If you probably accounted for chat delay because of people being away from their keyboard (“AFK” in there for you gamer types), I think they topped out at seven or eight simultaneous chat sessions. When we pulled them in that morning, I had three separate chat sessions I was monitoring and logging.
Mind you, this person has two arms, and before you head goes into the gutter, this was also on one computer and since they weren’t a trader and had a plethora, at the time, very expensive flat screen monitors. So this was an early 2000s era 15” LCD screen. Having three going was pretty amazing with those constraints, so let’s give credit where credit is ADHD due.
We had a sense at this time in the day, the subject’s two office mates were not in the office at this time (which played into how regular we could expect this traffic to pick up) and having a capture to look for network traffic on their two computers was also running. It was a trick later used in the largest single forensics acquisition I ever performed, and it also luckily included pizza, but was a useful tripwire.
Oddly, now I think about it, most of my really good stories are from this period and I truly miss that era, but there’s always a time and place.
So, we guide or team member down the office. We give him the okay to apprehend them and the office appeared clear. While I know, given their role on our facilities, they could legally carry guns, but I had hoped the “all clear” to them was to keep one from being drawn. Again, this wasn’t Hollywood, you can’t follow everything on CCTV for the viewer to see that the exposition leads to the good guy always being safe or escaping, so they had to trust us.
When our staff found him, he did truly have three sessions open. I believe two IM sessions open, one IRC channel running, and was on a forum. That forum part there, and since we were running a targeted capture for the messaging protocols to review, and not web browsing (oh, how naive were we then), is the important decision-making step in what was to be done next.
Getting the URL of that open forum, and then also digging in our captures, we determined the subject’s username and began to also get a picture of what was going on there. This was a forum where, like many of these specialized “interest” areas, exchanged things such as hookups and other adult passions (as it were).
Okay, they traded sex on the forum. Sheesh.
For some reason, we decided to investigate the forum. I think as part of standard practice by our physical security team, was to cross check with state, local and possibly Federal law enforcement if this person was wanted or a target of any of their actions.
What would you know, our subject was the target of the FBI for the exact thing we picked him up for.
That forum. Yep, you guessed it. An FBI honeypot for folks soliciting underage minors.
Oh yeah, this wasn’t adults hooking up kind of forum. This was sex trafficking.
Pretty awful, right?
So, our subject was turned over to the police. A few days later the sting for that content was brought down and a lot of folks around the US were picked up in it. It was revealed details about the website and sting operation that it was involved in. It kind of felt good to possibly get a person like that off the streets, even though they were talking to law enforcement for a good part of the time.
We were also told, in no uncertain terms, that if we hadn’t got him when we did, they’d be asking to pick him up at our facility or at his home. At work was easier since that was more predictable. Also, I think, because the FBI field office was about two miles from our EOC, so it was a shorter “commute” to get their perp.
We avoided having the company in the news for the exact worse reasons you’d want to be in the news. But never fear, I have one of those tales to share as well. At least, in that case, we saw signs of what was to become, and when the shoe dropped, the outcome was not surprising at all. It too has a link to this tale, so stay tuned for that one. It has my recalling one of the funnier phone calls I’ve had in my professional career.
