By all accounts the show did go on, and we have some very interesting results to share with you all.

Take the Top Challenges question. Once again, “Supporting New Technologies/Enabling Innovation” was most popular. But that’s a no-brainer and as one memorable respondent told me, “the definition of what I do”. What was more important was seeing the big jump that “Reducing Management Costs” made on the list, from #5 last year to #2 this year and only 1 percentage point behind #1. Tightening the belt is top of mind for everyone. (As I write, the Dow closed down today over 700 points)

Overall, IT professionals told us they were tackling the practical projects that should and could get done – from deploying Security Information Management solutions to getting Asset Management and Inventory Tools in place. For the first time, we saw a close correlation between what people said was important and what actually got done. Of low importance and even lower actual deployments – ITIL and CMDB, IPv6, Green IT and Cloud Computing.

And perhaps people “fessed” up about virtualization. Instead of the usual “high importance, not so many deployments now, but more deployments planned” theme we’ve been seeing around virtualization adoption, this year the very hot trend seemed to lose a bit of steam. Across the board, the numbers were down for virtualization management, with close to 50% of respondents telling us that their businesses were less than 10% virtualized (4% of that with no virtualization at all).

2008 Detailed Results – showing trends year over year

Comparison of Results from Interop NY 2008 vs FOSE 2008 (government IT)

Well here’s a story where the tradeoff for speed and efficiency caused a massive stock dump erroneously.

Apparently, many traders use automation software that trolls the Web for news stories and then, depending on what it finds, executes stock trades automatically. It was United Airline’s bad luck that an old article about its 2002 bankruptcy-court filing showed up on Google’s news service and somehow made it to the list of most popular stories. In one of a series of mistakes here, the story had no date on it – which means Google’s algorithm for assessing popularity didn’t have a way to exclude it as an “old” story – OR (because there are conflicting accounts) the South Florida Sun-Sentinel actually put “today’s” date on the page that the story appeared on. This got picked up by the Income Security Advisors newsletter and sent over to Bloomberg News as a one-line brief. Plus there’s the inevitable conspiracy theory that people manipulated the web traffic for this story to adversely affect UAL. Regardless, on Monday afternoon, the stock plunged 76% in less than a day.

But the real problem here is the growing use of automated programs to trigger stock trades without any human interaction – instead based on news headlines and earnings data. According to the Wall Street Journal, these automated programs were responsible for a very surprising 25% of NYSE trades in the last week of August.

I’m sure we’ll hear more as the lawyers are now involved trying to figure out who should get the blame.

Photo Credit: www.heelys.com

Since the Ninja has such a hypnotizing voice, we decided to leave this one as a podcast: ScienceLogic Q&A with the Splunk Ninja

ScienceLogic: How long have you been involved in InteropNet?

Katsev: I started at Coyote Point 3 years ago and InteropNet 2006 was my first “big” assignment. This was the first time Coyote Point had put in a proposal to participate, so we were very excited when we were selected.

ScienceLogic: How long has Coyote Point been involved in Interop overall?

Katsev: We’ve been exhibiting at Interop for a number of years, and after seeing the InteropNet in action, we decided to submit a proposal in ’06. We were actually one of the first companies in the load balancing/traffic management space (we’ve been doing this for almost 10 years), so we have a lot of experience to share with InteropNet.

ScienceLogic: What is your role at Coyote Point?

My official title is “Engineering Project Manager”. Basically, that means that I’m in charge of product releases and maintenance. It sounds like a weird title for someone participating in InteropNet, but I’ve actually found it extremely useful since my position means that I don’t get to see our systems out in the field a lot. We’ve added several features and have ideas for others just from my experiences at InteropNet.

ScienceLogic: What do the Coyote Point products do?

Katsev: Coyote Point makes a Traffic Management appliance called Equalizer. What this means is that any traffic destined for a datacenter’s servers goes through our appliances and we make sure that the server which is best equipped to handle it, does. Our systems sit between the clients and the servers and monitor the client traffic and the state of the servers. If the clients start sending more traffic, we’ll balance it out so that no server is overloaded. If one of the servers stops responding or starts responding very slowly, we’ll steer traffic away from that server.

ScienceLogic: In what way are your products being used as part of InteropNet?

Katsev: In the InteropNet, we’re utilizing a lot of our expertise: We’re making sure that traffic is balanced and servers are redundant for show services such as DNS and SMTP. We’re also using our geographic load balancing technology to ensure that the ScienceLogic EM7 appliances and some other internal NOC services are available from anywhere, with the lowest latency, with our SSL acceleration and GZIP compression technology. Finally, we’re helping logistics in the NOC by allowing a physical separation between systems located in the NOC and those in an emergency rack outside of the NOC. If either of these two locations were to fail, the network will continue operating without a glitch.

ScienceLogic: Are there any special considerations for Interop that cause you to deploy your systems there differently that any other place?

Katsev: Interop is definitely different than most of our customer installations. One difference from a standard environment is that the network (at least this year) is one large flat network, with pieces carved out where extra security is needed. Because of this, we can actually run our failover pairs of Equalizer systems in a non-standard configuration where the two peers are in different racks, or even on different floors. That’s one of the things that I really like about InteropNet — it definitely brings new ideas to mind, which end up becoming ‘special configuration’ white papers after the show.

ScienceLogic: Has InteropNet taught you anything that caused you to actually change your product?

Katsev: In addition to the failover configuration differences I mentioned above, participating in InteropNet has actually caused us to add several new features and allowed configurations. One example is the “no-spoof” option for Layer 4 clusters. Prior to the 2006 shows, we always ‘spoofed’ the client’s IP address when talking to a server so that the server would see the client’s IP address instead of our own. At Interop, we ran into a special configuration which would’ve been very difficult to set up in this manner, so our engineers added this feature, and it’s been very a very popular configuration with our customers ever since.

We have also had a couple of business relationships that extended outside of the show. In 2006, we had a good experience using Spirent Communications gear to benchmark the network, so we ended up purchasing a couple of these systems to test our products. More recently, we have found a way to bundle our Equalizer e350si load balancers with the ScienceLogic EM7 collector appliances to help ScienceLogic get the best performance in load balancing large quantities of syslog messages to be processed. If it wasn’t for our participation in InteropNet, neither of these relationships would’ve happened.

ScienceLogic: What’s the best part of being involved with InteropNet? What do you most look forward to?

Katsev: InteropNet is an amazing networking opportunity (no pun intended). The group of engineers that put the network together every year is, well, amazing. There is so much combined experience that any question instantly has several possible answers, and the best answer is chosen very quickly. One of the ‘sayings’ at Interop is “if you run into a problem, ask someone… we’ve probably seen that problem before… five times.” One would think that being part of InteropNet is the same thing, year after year. However, in the two years that I’ve been part of this (for four shows), there have been huge differences in the way that the network is designed and put together. These are both because the vendors selected every year are different, and because the engineers who design the network change from year to year. Somehow, though, when all is said and done, we have a network that works.

ScienceLogic: You don’t have to answer this one if you’re not comfortable… What would you like to see changed with the way things are done at InteropNet?

Katsev: This isn’t a cop-out… I really can’t think of anything I would do differently. Sure, there are small problems that pop up sometimes, but every project has those, and the people at InteropNet are more than capable of figuring them all out. In fact, I know that Interop started out as a show to test the interoperability of devices… but I’m still amazed that all of these devices actually talk to each other and “play nice” together.

ScienceLogic: What’s your real job when you’re not here?

Cummings: I’m a consultant. I have a networking services company through which I offer services all the way from Layer 1 to desktop support.

ScienceLogic: How long have you been involved with Interop?

Cummings: I attended my first show in 1996. I volunteered for my first shown in 1999 and haven’t missed a year since.

ScienceLogic: What makes you want to come back each year for the additional punishment?

Cummings: Working with the team, which are long-term established friendships at this point. That and the excitement of working with the new technologies as they or even before they come out.

ScienceLogic: In Las Vegas you were Team Lead for Help Desk. What are you going be doing in NY?

Cummings: Same thing. That position incorporates some management over the show floor and off-show floor area. That’s kinda where they put me and I’ve been doing it solidly for about 5 years.

ScienceLogic: What are the biggest changes you’ve seen in the show over the years, what sticks out?

Cummings: The amount of monitoring that we have and what we do with it has really been changing. We went from more, to almost none and now back to more. We’ve been through numerous vendors and apps over the years and until recently weren’t overly happy.

ScienceLogic: Did the integration between Service Desk and Monitoring that ScienceLogic created help streamline things in a meaningful manner?

Cummings: Absolutely. In the short time that we have to get things setup there’s no way to integrate multiple products in this area. Having things pre-integrated allowed us to quickly link network events and the related tickets together in the management system [EM7].

ScienceLogic: Moving forward on the Service Desk, do you think you can move away from your current paper driven process to a completely paperless process?

Cummings: I could potentially see it changing as we get the process down and fine tune it. We might be able to get an electronic interface for people. It’s tough. There’s always going to be an aspect of the shows we have to hand off on paper and get to legacy people such as electricians and movers.

ScienceLogic: If there was one thing you could improve that you think would make the overall show or help desk operate better, what would it be?

Cummings: We need to keep refining processes down to get information into EM7. Get better at using the integration and automation that already exists in EM7.

The big difference between the two cities is the amount of time that the InteropNet team gets to produce a live, fully operational and redundant network. In Las Vegas, this was nearly a full week of time – a tight timeframe across 17 different vendors, but now we’re looking back at that timeframe as a luxury. In NY, we’ll be getting started Saturday morning, and the network needs to be delivered on Sunday morning for the registration desk and exhibitor move-in to begin. If you’re keeping score, that’s about 24 hours to deliver a working network. Sounds hard, but it’s even harder when you consider that this means four DS-3s from two different locations, 17 full and 7 half racks of network gear, all the fiber and copper that the network is delivered over, etc all have to get done. Good thing that with 2 and 3/4 kids, I’m not planning on much sleep, and I don’t think the rest of the team is either.

In order to try and get the network delivered in that short timeframe, we worked hard at Hot Stage to assure that everything is ready to go. With some luck, the work that we’ve done here will allow us simply to roll the network gear into place, run the cables, fire up and go.

Now, things never really work out that way but that’s what EM7 is going to be there for. We’ll watch in real time as the network elements come live and be able to let the other InteropNet vendors know if their gear isn’t behaving as expected or is not visible for all the areas of the network that it should be. We’ll keep track of all of this in the EM7 ticketing system so that after the show we’ll be able to analyze the behavior of the network and systems as we did after Vegas.

I’m looking forward to the show and once again working with some of the top engineers in the country on a complex and rapidly deployed network.  Speaking of which, we’re still looking for volunteers to help in the NOC.  Volunteers get to work with some really smart people, get an education that would be hard to get anywhere else, and get a trip to NY where your expenses (for things like hotel accommodations and food provided by the show) are taken care of.  Sound interesting?  Be sure and check out the application.

(Photo credit: The Tech Stop)

ScienceLogic: How long have you been involved with Interop?

Geoff Horne: Since about 1996.

ScienceLogic: How has it been changing? Does the show get more complex with new technologies or because of the constantly changing size of the show?

Geoff Horne: The technologies have changed. Every year there’s a different market environment. Since we build on customer needs, things change every year. Things like ScienceLogic for Network Monitoring actually simplify things, it’s a new thing that Network Management tools are completely web based. In general, it doesn’t really get any better or worse because every year we’re building it again. You don’t get the stability of a standard environment. The upside is that we’re always doing a full upgrade, a full technology refresh and not using old code.

ScienceLogic: Do those kinds of changes influence the types of vendors you look for for InteropNet?

Geoff Horne: The base categories don’t change. You always need to forward packets. You always need switches, you always need routers. We’ve tried to open it up to everyone that has products involved with networks to see if we have the time or space for it.

ScienceLogic: The kind of cooperation that you get between the vendors is what seems to be an unachievable nirvana for Enterprises. What’s the secret to getting 17 vendors to work together in such a short time? Enterprises would kill for that.

Geoff Horne: The honest answer is don’t trust the vendors. If they try and build something the way they want to, its not going to interoperate. You have to pull them out of their safety zone, make them do things that you think the product can/should do to ensure interoperability.

ScienceLogic: In a blog post prior to Interop Vegas 2008 you stated three major goals for InteropNet. They were Education, Monitoring and Statistics. How did you do against these goals?

Geoff Horne: I think we did pretty well. They’re 3 things we really didn’t have before. They’re things that just weren’t focused on the right way. For the first round of changing the focus, changing the way people look at the network (statistics rather than packets), it worked quite well, it gave people a much better idea as to what’s going on.

ScienceLogic: If we look at NY as take two for Interop 2008, are there things you are going to do differently based on lessons learned in Vegas?

Geoff Horne: We’re building more physical redundancy in the core network, geographic distribution of the infrastructure within the show. This will allow us to bring up chunks of the network independently. It isn’t something that we really thought of before. This helps us take the single point of failure (the NOC) out of the equation.

ScienceLogic: Are there any lessons learned from Interop that you think would help enterprises?

Geoff Horne: Visibility is key. Your network is significantly more functional when more people can see what’s going on. If the only guy that can see what’s going on is the guy with his fingers on the terminal, no one can make good decisions. You have to make people loosen up their control so that everyone can see and therefore make educated decisions.

A total of 155 trouble tickets were opened in the four days that the help desk was operational. Of these tickets:

• 91 were opened by exhibitors, these were opened by 75 different booths (of about 500).
  • 28 were to report slow or no connections. Of these only 6 were related to the network (all before the show opened), usually they were things like patch cables not pushed all the way in. The remaining 22 were user error. Overall, this is a very small number of tickets compared to previous years. We attribute this to our decision to provide DHCP on the show network this year. This eliminated a majority of the user tickets which generally were due to users not understanding the need to put a static IP on their laptop or show device.
  • Another interesting stat is that four of the user tickets came from the same networking vendor, and in each case it was their own gear that was misconfigured. I guess as users we shouldn’t feel bad when we have trouble getting configs right.
  • The other 63 were change requests with the most common being a request to move an internet drop from one location in a booth to another.
• Two tickets were proactively opened by the InteropNet NOC team to notify/warn a vendor that a machine in their booth was infected and performing malicious scans of the network, in order to try and spread the infection. Without naming names here, it’s interesting that one of the companies was a security company and the other a very large software company.
• The remaining tickets were largely opened to track that activities of the NOC and InteropNet deployment teams as they deployed, tuned and maintained the network over the course of the show.

So what does this mean? It means that less about 15% of the exhibitors ran in to something that required them to open a ticket with the help desk and that in reality only 21% of those tickets were for valid issue, meaning only about 3% of the exhibitors actually had any issues. Further analysis shows that for the tickets where there actually was an issue, the issue was resolved in an average of 50 minutes, with the quickest in 11m and the longest at 2hr 39m. Finally, not a single valid exhibitor ticket was open during show floor hours. All issues occurred before the show began during the set-up phase.

Overall I think that these stats point to an efficient help desk and trouble shooting process that was facilitated by the link between the EM7 Trouble Ticketing system and Network Monitoring components that allowed quick validation of tickets so that the right teams could be dispatched.

1) We ended up monitoring 209 nodes in the official show network. They broke down as follows:

• 73 switches (Enterasys and Netgear),
• 4 routers (Enterasys),
• 28 power distribution units (APC),
• 5 IDSes (Enterasys Dragon),
• 20 environmental monitors (APC),
• 6 load balancers (Coyote Point),
• 2 VMware servers,
• 5 DNS and DHCP Servers (BlueCat Networks),
• 27 IP KVMs (Avocent),
• 27 IP Power Strips (Server Technologies),
• 1 Master Wireless Controller (Aruba Networks),
• 2 IP-PBX Boxes (Digium Asterisk),
• 4 Optical Taps (NetOptics),
• 1 Splunk server and
• 4 external WAN links (Qwest).

EM7 pulled data from all of these devices and delivered a single view of the data to the NOC.

2) Uptime for the network was 100%. That isn’t to say that there weren’t some device failures, but each of them was handled properly by the redundancy in the network and the show exhibitors and attendees saw no impact from these failures. This is a real testament to the design and build of the network. It’s hard enough to build a complicated network in two weeks, but then to keep it up and running 100% of the time in the wild west environment that is Interop, is really phenomenal.

3) The average monitored device in the show network didn’t even hit 10% CPU utilization. This is interesting because many items were virtualized using vmWare this year and yet, there was still a lot of hardware overhead available. (Maybe we should run Folding@Home on the show network?)

4) The show network was busy. By our calculation over 864 gigabytes of data was pulled in and 1.01 terabytes of data were pushed out of the WAN links in the 3 days that the show floor was open. That’s a sustained 56Mbps average, including off hours. At peak the show network hit about 102Mbps of WAN utilization.

5) In the three days the show floor was open the network and its supporting NOC gear used 600 kwh (kilowatt hours) per day. As a comparison, the town of Rockport, Missouri (1,300 residents) uses about 35,600kwh per day. On a side note, they are completely powered by wind power and in fact sell 3,000,000kwh per year back to the local power utility. I’m thinking next year Interop should bring some wind turbines as part of the InteropNet kit?

Next I’ll be doing some analysis on the trouble tickets opened. I think it’ll be interesting to see the kinds of issues that vendors experienced and how quickly the InteropNet staff handled them. Look for that in the next couple of days.