Twitter? Time Warner Cable gets it

I have a love/hate relationship with Twitter.  Some days, I think it’s greatest time waste since they invented the meeting.  Other days, it’s an endless source of amusement.  And on some days, it’s actually useful.  Yesterday was one of the useful days.

At about 9m last night, my 10 year daughter came running into my home office.  Kathryn was convinced that she had broken the TV.  Since the destruction of electronic devices is her sister’s department, I went in to the bedroom to take a look.  Kathryn was watching a show on TLC HD and it was not coming in well.  The audio was distorted and the image had tiling artifacts.  I assured my daughter that it wasn’t her fault and sent her off to read a book.

I checked a few channels and sure enough, it wasn’t just TLC HD that was having problems.  Most of the channels had the same problem.  And yet, a small number of channels were just fine.  My decidedly non-scientific search had determined that the SDV channels were having the problem and the non-SDV channels were just fine. 

SDV or Switched Digital Video is a technical hack that allows cable companies to provide more channels than they actually have the physical bandwidth to carry them to every home.  What they do is to only provide a channel when a view switches to that channel.  A cable node will supply somewhere between 1000 to 2000 homes in a neighborhood.  If the cable company provides 200 channels, only a fraction of that number is actually being watched at any time. 

The “Switched” part of SDV means that the frequency that the channels goes out on is not fixed, it can changed based on usage.  When you select a SDV channel from your set top box, the channel request goes back to the cable office.  They check to see if any other box on your node is watching that channel.  If it’s in use, the cable office sends back the frequency that the channel is on, and you get to watch “Matlock”. 

It’s a clever hack, but it’s still a hack.  If you get enough people on your node to watch a different channel, it will fall apart like a cheap suit.  While I doubt that would actually happen, it’s technically feasible.  They can spend the money running Fiber optic to each node, and making the last run to house over copper.  Running fiber to each house is very expensive.  Just ask Verizon how much it costs to wire a neighborhood for FiOS.

If no one else is currently watching that channel, the cable company assigns the channel to an open frequency for your node and sends that frequency back down to your set top box.  If you have a TiVo, the process works the same way.  Your cable company provides a device called a Tuning Adapter, that manages the frequency negotiation between the TiVo and the cable office.

I have both a set top box and a TiVo HD with a Tuning Adapter.  The set top box was having the audio distortion and video artifacts.  The TiVo wasn’t even displaying the SDV channels.  Since the non SDV channels were crystal clear, I guessed that it wasn’t a signal problem and the problem was more likely than not an issue at the cable office.  Since I could reproduce the problem with just the cable company’s equipment, I decided to call their tech support line.

My cable provider is Time Warner Cable of Albany, NY.  I went to their web site to look up their support number.  I called it and heard a recorded message explaining that there would be a long wait time for technical support.  TWC Albany is pretty well staffed, that only happens when something bad has happened. 

While I was in the hold queue, I saw an option for chat support on their web site.  I tried that and the best thing that I can say about that experience was that it occupied 45 minutes of my time without any resolution.   The sum of the support rep’s skill set was to have me reboot the set top box and when that did not resolve the issue, he then scheduled a service call for a time slot that was 9 hours in the past.  Since my cable package did not include any time travel options, I wrapped up that chat session and tried calling the Albany support number again.  This was about 10pm.

While I was in a never ending on hold queue, I decided to see who from Time Warner Cable was on Twitter at that hour. As it turns out, the corporate office of Time Warner Cable has a bunch of people on Twitter.  Their main support account is @TWCableHelp and I could see on Tweetdeck that they were active at this late hour.  I tweeted the following message:

Can @TWCableHelp help with local issues? My SDV channels are unwatchable and I made the mistake of trying TWC online chat

The message was a little snarky, but 45 minutes of chatting with a CSR in India will do that to you.  Within two minutes, I got the following response:

@anotherlab Can certainly take a look into your issue. Had eChat already rebooted the box? ^BP

We then traded a few tweets and Bryan (the ^BP in the message) ended up calling me.  He was very professional and had a pretty good sense of humor.  We immediately ruled out time travel as an option and he was able to remotely check the signal strength coming into my set top box.  Everything looked normal.  While we were discussing what the root cause could be, the problem cleared up.  What ever had been going on at the local office, the problem appeared to have been resolved. 

We checked the channels on the set top box and on the TiVo.  From his end, he could see that about 50 channels where being sent over SDV on my node.  So we picked a few channels were not being sent at that moment. Everything was normal. 

We chatted a bit about my TiVo and I mentioned that a few people had been complaining about the latest software update that went out to the DVRs that TWC supplies.  He wanted to know more about those complaints.  I told him that I didn’t know too much about those problems, but I had read a few complaints on some local sites.  I gave him a few links (Kristi Gustafson at the Albany Times Union and Albany HDTV) and he checked them out while we were talking.

Bryan then asked if I had any other questions.  So I asked him about the CCI Flag.  Almost all of the digital channels (except for local stations) carried by Time Warner Albany have the CCI Flag set to 0x02, which is usually referred to as “Copy Once”.  “Copy Once” means that the show can be recorded to a DVR, but it can not be copied from the DVR.  The DVR must respect the CCI Flag, otherwise it would not be allowed to use a CableCARD, which is needed to access the digital content.

This is not an issue with the DVR that the cable company supplies, but TiVo can do quite a bit more than a cable company DVR.  If you have multiple TiVo DVRs in your house, you can copy shows from one to another.  If you have young children like I do, that is a very handy feature.  We can also copy the shows to a PC for storage and transfer them back at a later date.  The video files are copy protected, it’s not like you can just upload them to the great unwashed masses on the Internet.

I have a couple of TiVo boxes and it’s been handy to watch a show recorded on one TiVo from another.  When the CCI flag is set to “Copy Once”, the only copy allowed is the copy on the first TiVo.  It just wont copy it to another TiVo in the house.  Many cable providers set most of the channels (excluding Pay Per View and premium channels like HBO) to 0x00, which is usually called “Copy Freely”.

Bryan told me that it was a local cable office decision. I told him that the local cable office had made it clear to me that the decision was mandated by Time Warner Corporate.  He said that he would look into it and then would get back to me on that.  i think that he will find that it’s a dead end, but it was worth a shot.

All and all, I was very satisfied with the support provided by Time Warner Cable through their Twitter account.  But please change the CCI flag.

What to do when an antivirus vendor has flagged your software as a virus

I usually don’t talk about where I work, this is my personal blog and where I work really doesn’t matter to the content of blog posts.  This post is a little different, it’s work related and knowing where I work adds some needed context.  I’m a senior software engineer for Tyler Technologies, in their VersaTrans Solution. The VersaTrans product line provides school transportation software and services for the K-12 schools.

About a month ago, I came into work to find people waiting at my cubical.  That’s usually not a good way to start off a morning, and it wasn’t.  Apparently McAfee pushed out a virus definition file out the night before and it was flagging our software as a trojan.  Two of our executable files were being flagged.  Our customers were calling in to our technical support department, and they were understandably concerned.  The McAfee antivirus software wanted to quarantine our applications.  That would prevent our customers from using our applications, which is not a good thing.  They use our bus routing and planning software on a daily basis.  Not being able to use our software can cause a lot of problems for a school’s transportation department.  We needed to immediately resolve this.  I usually don’t get involved with technical support issues, but I had the bandwidth to immediately jump on this problem.

The immediate order of business was to replicate the actions of the A/V software.  We wanted to make sure that our software was being actually being flagged before we reported it to McAfee.  We don’t use any of the McAfee products in house, so one of our QA specialists (David) started downloading a trial copy of McAfee’s Enterprise solution.  This is the application that our clients were reporting that they were using.  David would install that copy into a virtual machine and test it in an isolated environment.

Thursday, 10:00am.  I decided to save some time and upload a copy of our application to my home PC.  I have a copy of the home version of the McAfee A/V product.  It comes free as part of my FiOs package.  I’ve been planning on removing it and replacing it with one of the free A/V products, but fortunately it was still installed.  Sure enough, the app was flagged as “BackDoor-AWQ.svr.gen.e”, which is some type of generic server type of trojan that was added to version 5460 of the McAfee A/V database.  Version 5460 was the one that had been just pushed out to all of McAfee’s clients.  David would eventually duplicate the A/V hit, but it would take a while.  Armed with immediate proof from my home PC, I could proceed to the next step.  By the way, it can be very handy to be able securely access your home PC from your office PC.

10:15am.  I made my initial call to McAfee.  I went was passed to several CSRs, most of them refused to provide any identification other than their first name.  With each CSR, I was asked for a “grant number”, which was some form of customer identifier.  With each CSR I needed to explain that I was not calling as a McAfee customer, but as a representative of Tyler Technologies and they were false identifying our software as malware.  Each CSR tried to redirect me to their home product support.  With each CSR, I would request to speak with their supervisor when it became obvious that I would not get the support.

10:30am. I spoke with “Mike”, who identified himself as Corporate Service Program Supervisor.  He refused to escalate the issue or provide a last name and phone number.  Once again, I asked to speak to his supervisor.  He did not want to do this and we went back and forth a few times and I was persistent.  He finally transferred me to their Consumer Customer Department and provided that phone number (866-622-3911) with great reluctance.

10:30-10:42am.  Waited on hold.

While I was on hold, our sales department sent out a mass email to all of our clients.  The email explained that McAfee was falsely identifying our application as a virus and they should not be alarmed and that we were working with McAfee to remedy the situation.  As soon as you can reliably identify that some version of your software is being falsely flagged as a virus, contact your customers.  Trust me, they will appreciate it.

10:42am. Spoke to another CSR who couldn’t help resolve the problem, but could transfer me to someone who would be able to help.

10:45-11:00am.  Waited on hold.

11:01-11:30am. I reached a CSR named Juan at the Dallas office of McAfee.  Juan’s title is Tier III Customer Service and he provided me with his direct phone number and email address.  He asked me to submit a copy of the file being flagged to a web site that they provide for online scanning of files.  This is McAfee Avert® Labs WebImmune and is located at https://www.webimmune.net/default.asp.   I registered an account on their site and tried to upload the Routing and Planning (RP) executable known to trigger the McAfee scanner.  I was unable to upload the file because the RP executable is 7.7MB in size and WebImmue would only accept files up to 3MB in size.  Our executable is pre-compressed and there isn’t any other method of shrinking the size down to below 3MB.  This 3MB limitation basically defeats the purpose of having that site.

I then tried emailing the file to Juan, but our own email server blocked the email because of the file attachment.  That’s bad on our part, but there is more than one way to email a file.

11:45am.  I emailed the file to Juan via my personal Hotmail account.  It took a while to make it to Juan, but by 3:00PM, I had an email confirmation from Juan that he had received the file.

11:58am.  I received an email from Juan with the following text:

Mr. Miller,

I just received an email in regards to the issue that you are currently facing.

An issue that was incorrectly indentifying some large applications executables files as Backdoor.awq.svr.gen.e has been resolved.

Attached is an ED to suppress this detection. We have tested the ED against the limited number of files we have received thus far. If the ED doesn't work and DAT 5461 hasn't shipped then please gather a sample (FTP site in pwd protected zip if larger than 3mb) and escalate via MSTeg.

I forwarded the message to David in QA.  David had set up a virtual machine with a trial version of the McAfee Enterprise virus scanner and could duplicate the problem using the 5460 DAT file.  The “ED” supplied by Juan did not fix the problem, but I am not 100% certain that we had installed it correctly.  Their ED file did not come with any instructions on how to install it and we I promptly notified Juan that it was still a problem for us.

3:00pm.  Juan emailed me to notify me that he had received the RP executable from my Hotmail account and he requested a copy of the other executable that was being falsely identified (Ascii Scheduler).  I immediately sent him that file from my Hotmail account.  I did not receive emails from Juan (or any other McAfee representative) after that point.  Both files share about 98% of the same code, I was pretty sure if they fixed the scanner for one of them, it would be fixed for the other.  Especially since we were not the only company affected.  Juan wouldn’t tell me who else had been false identified, but he hinted that they were big companies and they were not happy.

Friday 9:00am.  I was told by our QA Manage that DAT version 5461 was pushed out by McAfee overnight and it no longer flags our files as false positives.  We received many happy emails from clients thanking us for the prompt attention paid to the matter.

So what do you do if this happens to you?  I recommend doing the following when a customer reports that your software is being flagged as a virus:

• Ask the customer for the product name and version number of the antivirus software.  If possible, get the version number and date of the virus database used by the software.  Start taking detailed notes during this process.  You want an audit trail to give your boss to explain where your day went, plus it gives you ammunition when you get stuck with a CSR that wont help your or escalate your call.
• Ask for the name of the virus or trojan that your software is being flagged as.
• Get the version number of your executable that is being flagged as a virus.
• Get a trial version of the A/V software and attempt to duplicate the false positive match.  If you can not duplicate the results, ask the client to send a password encrypted zip archive of the executable being flagged.  You have to consider the possibility that the client’s machine has been infected with a virus and that your application was infected on the client’s machine.  By sending it in a password encrypted zip file, it should pass through the client’s email server and your server without flagging any additional virus checks.
• Submit your executable to the A/V vendors online support.  Most of them have a web page for submitting a questionable file.  You wont get a fast response back, but when you call their support, they will ask you to do this step so you might as well get it out of the way.
• Call the A/V vendor and ask for support.  Do not identify yourself as a customer of their A/V software, but as an ISV who has been falsely identified by their software.  If you identify yourself as a customer, you may get shunted down lower priority support queue.  If you have an expensive support contract, that would be different.  Basically, just use the best tools at your disposal.
• Ask for the CSR’s name and phone number, with extension.  Tell them that you need this information to provide an audit trail for your supervisor.  Be polite but persistent.  If they can not help you, ask for their supervisor and just work your way up their chain of command until you get a CSR that can help you.
• Stay polite and professional with the CSR.  It’s not their fault that their software is falsely identifying your software as a virus.  They are more likely to expedite a remedy if you are not beating up on them.  It’s likely that you are not the only one being affected by this.
• When you have a CSR that can help you, work out a plan with target times for how to resolve this.  Let them know that you will be in phone and email contact with them until the matter is resolved.  Basically, you want to get the file or files to them and have enough time for them to test the files and work out a solution.  You do not want to be waiting for hours just to get them the file.  You will want acknowledgement when they have received the file and when they have duplicated the problem.  Your goal is to get them to issue out a new virus update file on the same day.
• Once you get the problem resolved, send a “thank you” email to the CSR that helped resolve the problem.  If the problem ever occurs again, you want to be able to start with that CSR and not have to waste time going up the CSR tree.

Having to go through all of this was not how I planned on spending my day. I lost the better part of the day dealing with McAfee’s mistake when I could have been working on my own mistakes.  This kind of stuff happens and it’s good to have a plan for dealing with it.