Gibberish

$GiantFirm is angry.

You say you are responding to all tickets within 24 hours. Why did you take five days to answer ours?!?!

Well, Dieter, perhaps it's because no one understood your fucking ticket. If you'd submitted it in English, some monkey somewhere in the world could've picked it up. Instead it sat waiting for some schmuck (yours truly, natch) to translate it.

And after I'd translated it none of the other monkeys picked it up because they figured even if they responded in English you'd come back to them in your own language and turn what started out as a PITA ticket into a full-blown nightmare with melting landscapes, tall buildings and hungry clowns. We've had it happen many times before.

Every couple of weeks we get a ticket submitted in French, German, Portuguese. We've also had them in Dutch, Swedish, and other less-popular languages. About five years ago we got one in Czech. That was fun.

We make it clear to every customer that support is done in English. This often means mangled English, and I don't mean screwing up they're/there/their or lose/loose like so many supposed native speakers do. We get quite a lot of Googlefish translations (popular in France and Germany) which we'll often have to reverse-translate to understand and then get a colleague to properly translate for us.

We also occasionally get the word-for-word dictionary translations so that the entire content is incorrect English words in using some other language's grammar. This can be amusing for some but for most it's just frustrating and annoying.

This means that we have to write the answers in such a way that when the customer heads back to Googlefish to translate our answers, they remain clear, something rather hard to judge if you don't know the language yourself.

Once in a while we get a ticket from a guy who's either on so many drug he makes Keith Richards look like a sober schoolgirl or he's been up for four days straight and the full hallucinations have kicked in, something like this:

In giving to you the question of the manage to be system, where comes flowering and that it is seen as otherwise. Given to send, she waits rather and our doing is none.

Pure poetry.

I more than understand language barriers and am often on the other side, but the requirement for communications through our support system is the use of English or something at least closely resembling it. If you can't be bothered to get one of your English-speaking colleagues (for example, that colleague of yours we actually trained... in English), you can wait a week for someone who speaks that language to get around to it.

You fuckwit.
x-posted to HuSi

Ricky Don't Lose That (Ticket) Number

So I was off by a week. It took three before Enrique, the lazy admin who didn't want to install $Balance -- despite it taking less time and effort than a beginner level round of Minesweeper -- submitted another ticket which boiled down to:

Help! Our production system is have many problems since we install new server last month! Downtime! Logs attached!

Dumbass.

I scanned the logs just to be sure and saw exactly the messages I'd expected -- DLL call failures.

Hi, Enrique.

Remember how I told you three weeks ago that you really ought to install $balance because without it being installed on only the one server, a number of DLLs would differ throughout Production? Remember how I also said that we've seen cases of odd and irreproducible problems in mixed systems? This is what I was referring to.

Your only recourse is to shut down this server, uninstall $OurBigApp completely, clean the Windows Registry according to FAQ #xxx, install $Balance, re-install $OurBigApp and reconfigure it manually. If you still have problems you'll have to do the same thing for all the other $OurBigApp servers.

There's no schadenfreude here, only consigned resignation. I tried to warn him. I wrote it again and again and again. And again and again. Five times I responded telling him he really ought to install $Balance but he didn't want to.

I wonder how much the downtime for 1300 users cost his company.

Root Cause: 17-Fuckwit.

x-posted to HuSi

LOL What?

LOLLERSKATES!
Cross-posted from Husi.

I can't stop laughing!

I had a look at the queue of incoming tickets and stumbled across this one:

Can you tell us if $YourBigApp is secure from the following:

1. Buffer Overflow
   
2. SQL Injection
   
3. Script Injection
   
4. URL Injection
   

Now guess which customer it was.


Microsoft

Yes, that company actually submitted those questions. I forwarded it in a mail to the worldwide SysAdmin team with my dream response:

It will be when yours is.
I can't help wondering that if we answer "Yes" that they'll come back with "So how do you do it?"

There's no way in the world I'm taking that ticket because there's no possible way I'd be able to stop myself including my dream response, so I have to settle for a big glass of LMAOnade and leave this one for someone else.

While technically correct, my answer would get way too much attention from upper management. Not the good kind of attention but rather the kind that could quite possibly interfere with my addictions to food and four walls.

One guy who this was forwarded to wrote back saying his dream response would've been "Sure, if you install it on UNIX." Unfortunately $OurBigApp runs in MainWin, so even though $OurBigApp runs on stable machines, it does so in an inherently unstable and insecure manner.

I'll be in a good mood for a while today.

Brutal

I've been working 17 hours each day since Wednesday. I hate Windows. I hate Windows Updates. I despise Windows "Genuine Advantage" some of the worst double-plus-good newspeak ever, right up there with "Digital Rights Management". I'm plagued by Windows' Automatic Updates. I'm getting boned by Terminal Services Application vs. Server modes.

But I really hate Citrix.

Tuesday:
The Citrix set-up was failing but I got the other three virtual servers built and running, and I installed various tools like Ethereal and Sysinternals Monitors.

Wednesday:
We figured out Citrix was failing because the dum-dum-doody-head version 4 software is language-dependent and the German Windows 2000 server we had wasn't really German but rather MUI; installed in ENU (US-English) it was then "switched" to German. There are huge differences between base language and MUI installs. I didn't get out of the office until 9pm.

Thursday:
I arrived at 7:15am and spent half the day just trying to find the damned German Win2K image. MSDN no longer has Windows 2000 on-line so it was up to the lab guy in England to search for old MSDN CDs. Once they were found, loading started but configuration took ages because he doesn't speak German.

Then there were problems getting the server into the Active Directory. By 3:30 we we'd managed to complete the OS install. Wolfie, the customer's tech guy, started installing Citrix through a WebEx session running on another virtual server which I was controlling through a Terminal Services session. This was more problematic because the physical server was on a different rack and mis-named: we'd had to change the host name to the original machine's name due to Citrix binding the host name and OS language into the license key.

At 7pm I walked to Lidl to get a bottle of vodka -- it was going to be a long night. I was drinking 60-40 vodka&Cokes as I worked with a contact on other parts of the test build.

At midnight the name was sorted and the software copied over but we couldn't install it until someone in IT could bring the machine back into our domain. I went home. The vodka bottle was empty but I didn't notice any effects until I tried to do some sudoku puzzles on the train. You know things are bad when you manage to put three sevens in the same box, but on the plus side, that was me coming out of work mode and finally relaxing. I hadn't been making such mistakes an hour earlier and Wolfie's praise in some later mail confirms this.

Friday:
Back in the office before 7:00a.m. I was able to start testing. There were more conf calls. I had to again remind the guy from Citrix that the fault was on them, and not us or Microsoft. It doesn't matter that you can reproduce the error over RTP without Citrix 4; $OurBigApp worked in Citrix 3. It didn't change, the Windows environment didn't change, only Citrix changed.

At noon-thirty or thereabouts we were up and running. There were more WebEx sessions to terminal servers and Citrix was installed around 3:30.

At 4pm, Wolfie prepared for the final round of testing as I wrote out the protocol for the test based on a recorded session of all the crap I'd done on the Citrix v3 system.

And then it happened. $OurBigApp wouldn't start. It would crap out just before the log-in finished, ad it did so with error messages no one had ever seen before. Wolfie had to catch a plane at 5pm and called it a day. I contacted our California people for help, went home and worked from there with a colleague until midnight. Nada.

Saturday:
I woke up early and went right back to work on my laptop. I tried running the application locally on the server and got the same error so I knew I had a problem with $OurBigApp on the application server even though nothing had been changed in the past two days.

I checked out another machine and loaded it again with the same environment. That took hours . Since it was the weekend no lab admin was there to bring the machine into the domain so that SQL Server could start, so I figured I could run the server on this machine and connect to the database on the fuX0r3d machine. That worked once I found all the connection settings.

At 5:00pm it was time to go to the bar and work my shift. I was in no condition to work on it at 2am when I returned, but when I woke up four hours later, I was logged back in trying desperately to finish building the environment and have the tests done by Monday morning.

Sunday:
I'd logged into $OurBigApp from the wrong box and the application worked with my new set-up. That wrong machine was Win2K. I realised I needed to put the XP SP2 patch on $OurBigApp. It took a few hours to get the files over from the US and onto the box. I gave up Sunday afternoon and took a well-deserved few hours' break.

Monday:
I woke up at 5:30 and was in the office before 7:00. I finished installing the patches, confirmed it had completed correctly and things were starting to look better. There was a small problem because I'd missed a data connector but then -- BAM! -- it was connecting.

I fired up the client again and -- BAM! -- it failed. I was struck by another revelation: Windows XP updates. I spent the next hour downloading the damned "Jen-you-whine Ass-vantage" crap and the fixes afterwards, all the while fighting the installer to prevent it from including the 912812 rollup. Do what I want, not what you think I probably want!

At 8:30am I was done. The only problem was that I couldn't connect to the Citrix Server through a terminal session, but I was able to run $OurBigApp normally from the XP box. So I tried a session through Citrix.

Big surprise: it failed. Some sort of Java error. I sent a message to Wolfie hoping to sort this before the 9:00am conf call. He sent me some info and screenshots back. It can only be done by connecting to the Citrix Server... which we couldn't do because Citrix requires that the server be in Applications mode.

We managed to set up a Citrix session on the test client so that we could get back to the Citrix server and everything was available. We performed the normal test and it was normal. Then we tried to do the test over Citrix and started getting weird failure messages. One of the messages was really odd and we tried Help: About. IE 5 SP1. D'oh! I left for my bar shift as Wolfie started installing IE6.

Today:
I arrived this morning to half a dozen E-Mails from different people all with more questions just about this particular Citrix problem. They'd searched through our tickets and noticed my name associated with most Citrix issues. When I switched windows to the virtual server with the client, I saw Wolfie'd reproduced the problem. It had only taken us a week!

The 10am conference call was long and tedious. We have a good idea what's behind the problem and the guy from Citrix admitted to certain changes between versions 3 and 4 which could cause this. It's about damned time. He only did so after we confirmed the primary method we use for popping and hiding windows.

All I have to do now is perform the tests following the same protocol to compare what's being sent or not sent. And then I get to do it again with both Ethereal and some DLL explorer/watcher tools. That a answer eight other tickets which need and response by today.

Update:
I don't even need to do that! The customer and Citrix are writing little scripts to pinpoint the hidden window call method usage and why Citrix sucks screws up. Looks like this is just about done and I'll have one big write-up which I can cut and paste to eight different tickets. If this happens by the end of next week I'll be doing the w00t dance.

Tomorrow I'll be around 30,000ft over the Atlantic after a brief stop in Paris. Considering the number of hours I've put in since last Wednesday I made it clear they would not charge me a day off for the flight. I haven't even done the laundry much less started packing. I'll spend a lot of the plane ride slogging through Ethereal logs, with a couple breaks for The Queen's Jewels.

I hate Citrix thiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiis much. Times a-hundred. Plus infinity.

Bad Dog!

I feel dirty.
So very dirty.
I had to do it. I was in the office before 8:00a.m. and didn't get out until the little hand had done a full revolution.

There's some heavy testing going on with $OurBigApp and Citrix (even though it's not supported) because quite a few major customers are experiencing a problem. Because Microsoft and Citrix are directly involved and working with us, this issue trumps all.

My current ticket load (not including the 8 relating to Citrix) is around 25. Not only did three of those customers send updates yesterday, the due date for sending a response to three others had arrived. I could only quickly answer two of them so I did it.

I did the Support Shuffle.

* * * * *

It wasn't intentional, at least, not in the "Fuck 'em, I'm too busy playing Bookworm" sense, because one thing I wasn't doing yesterday was playing games.

Virtual Servers are slow. Terminal Sessions to three different virtual servers are even slower. Add WebEx into the mix so that third parties can access the virtual machines which are hidden behind a couple firewalls and NAT and you've got molasses. We got Citrix 3 installed in only two hours and setting up the test only took another two. I was finally able to run my tests.

Then we tried to get Citrix 4 installed. The server crashed. There went 15 minutes for a reboot, restart, prep, WebEx session initiation and reconnections. Then the installation crapped out. And so on. And so on. And at around 6:00p.m. WebEx decided to die and no one could connect through anymore.

Every minute past 4:30 p.m. that a German actually works demonstrates the exponential seriousness of the situation. It was already 6:00 p.m.

WebEx finally came back to life around 7:00p.m. so I initiated another session and sent out E-Mails to the rest of the people working on this. They'd gone for food but they'd be back. I answered the two tickets I could knock out relatively quickly and then I did the deed.

Every ticket requires we respond to any update within three working days. There has to be some sort of activity which involves contact with the customer. So I sent a stupid question which was pathetically tangent to the issue to buy myself some time. I have no clue why this guy's having trouble with large files and I don't have time to build a test environment to replicate his as well.

Twenty minutes later I was still at my desk and received notification that he replied. I finished the server crap and went home.

And from home I logged in only to find the server crapped out again shortly before 11p.m., so I restarted it, restarted WebEx and mailed everyone again. There was a problem when I woke up at 6:00 this morning and I got the servers back on-line and made them available before heading to the office.

We have more Citrix testing today and the problems we're having just installing it will keep me busy most of the day. This is being written on my laptop while riding an S-Bahn to the hell I call "hell", an 8'x6' cubicle with a few computers, a bunch of code page charts and a lot of customers waiting for me.

Update: At the time of posting, testing has been pushed back even further. That Win2K-DEU server wasn't DEU, it was MUI. I'm installing a new, German Win2K that's really German this time. What's the difference? Citrix won't install in MUI and they also bind the license key to the server name. This is so totally upgefuckt.

I don't have to write back to Mr LargeFiles until Monday and I know I won't, either.
Bad monkey.

Cleaning up

Ah, it's a fine day. Ripa's not around. All of my tickets have been answered and most of them are final. The only thing I expect to receive back is mail telling me the problem is resolved and that I can close the ticket.

I play a game of Age of Empires and completely wallop 5 other civilisations by building a giant army right at the beginning and whacking the others into nothingness before I even reach Tool Age. Huge map and game over inside 45 minutes.

Time to read some blogs but first, a look at the incoming tickets. Nothing special here; most of them are in areas outside my knowledge. Oh, wait! Here's one. One workstation is having problems. That's right up my alley. Take ownership but first, it's time to read a few blogs.

Right. Let's have a look at this ticket.

Hi, we have one user whose computer constantly crashes whenever he uses $YourBigApp. I've attached screenshots, dumps, Windows settings, a Hijack This log and Dr Watson. I can't reproduce at will yet but I'm working on it. There are no unsupported programs or Browser Helper Objects running.

Screaming Servers, Batman! A clueful admin! This is a rarity and he's got my complete attention. Look through the attachments and what do I see? IE is dying left and right. But something doesn't make sense.

Hmm...

I need a round of Zuma.

Halfway through my game it hits me, so badly in fact that I blow the level.

I look back at the logs and see calls to system DLLs which aren't being returned. This is Not Good^®. It normally means they're corrupted or missing.

I write back: "Please get me a directory listing for the Windows\System32 directory using the command prompt: 'dir c:\windows\system32 o:e >> sys32dir.txt' and send me the file. Here's my direct, personal E-Mail address."

I don't normally give that out but this guy was clueful and I saw a chance to resolve a problem inside two hours.

Within fifteen minutes comes the response. I look through and see that half the damned DLLs are missing. How this machine's even booting is beyond me. I break down and actually call this admin on the phone.

"Hi, this is REC from $BigCorp. Your ell-user told you he didn't do anything with the computer, right?"
"How'd you guess?" came the response, dripping with sarcasm.
"Half the Windows system DLLs are missing."
"What the...?!"
"Compare that file you sent me with your sys32 directory."
"Hold on a sec..."
...
...
"Son of a bitch!!"
"Yeah. Re-image."
"Gladly. That son of a bitch. Thanks a lot for calling."
"No problem."

Two hours later the SR was updated. Ready to close. The admin told me exactly what kind of nothing that luser did: he deleted "unimportant" files in an attempt to free up space. Space that he desperately needed for his downloads. From BitTorrent. Films.

The customer works with the MPAA and currently has a job opening.

Root Cause 17, super-sized.

Running around in circles.

With locations in the US and Europe a customer had chronic speed problems in Europe. E-Mail was slow, adding attachments resulted in time-outs, queries could take half an hour, often timing out as a result.

Technical stuff: Click here to skip past it
No Web proxy
Available bandwidth:
US and Germany have own IPs, UK connects through them
US has a dedicated T1, Germany has a 2MB DSL, UK has 3MB "Internet Access"

Germany's Trace Route:
2 * * * Zeitberschreitung der Anforderung.
3 * * * Zeitberschreitung der Anforderung.
4 * * * Zeitberschreitung der Anforderung.
etc.

Germany's Ping:
Antwort von x.x.x.132: Bytes=128 Zeit=205ms TTL=226
Antwort von x.x.x.132: Bytes=128 Zeit=201ms TTL=226
Zeitberschreitung der Anforderung.
Zeitberschreitung der Anforderung.
...
Zeitberschreitung der Anforderung.
Zeitberschreitung der Anforderung.
Antwort von x.x.x.132: Bytes=128 Zeit=202ms TTL=226
Antwort von x.x.x.132: Bytes=128 Zeit=173ms TTL=235
Zeitberschreitung der Anforderung.
...
Zeitberschreitung der Anforderung.
Zeitberschreitung der Anforderung.
Antwort von x.x.x.132: Bytes=128 Zeit=404ms TTL=234
Antwort von x.x.x.132: Bytes=128 Zeit=398ms TTL=234
Antwort von x.x.x.132: Bytes=128 Zeit=404ms TTL=234
Antwort von x.x.x.132: Bytes=128 Zeit=398ms TTL=234
...
Antwort von x.x.x.250: Bytes=56 (gesendet 128) Zeit=3570ms TTL=228
Antwort von x.x.x.250: Bytes=56 (gesendet 128) Zeit=191ms TTL=228
Antwort von x.x.x.250: Bytes=56 (gesendet 128) Zeit=192ms TTL=228

That's some serious delay going on, and it got worse as soon as the US would come on-line. Whatever the Europeans wanted to get done, they needed to be finished with it by around 2:00p.m.

We got a network diagram and information on settings and everything looked kosher. There were no 404s, just time-outs and really slow response times. So bad was this packet loss, in fact, that they'd often get "Cannot find server or DNS Error" messages. Basic look-ups (many of which you'd expect to have been cached) were failing.

After two weeks of exhausting the possibility of hardware errors, we started inspecting the application servers. More people were brought in. We set logging to the highest possible and painfully checked through megs and megs of information. Surprisingly enough, despite the high amount of logging (normally not done during production hours), there was no worsening of the system.

There was also no answer in sight.

We turned to the database. Even more people were brought in. The generated SQL and activities were examined and everything looked fine, at least for those requests which actually made it to the database. Many didn't.

At least, not until around 4:30p.m. Eastern, when the system would slowly come back from the dead as people started going home.

After more than a month with no results, we tore into the data being sent through the network. Snort didn't detect any DOSing and Ethereal showed that the highest volume hits were to music related sites. After being informed of this, the customer installed WebTrends to better monitor what was going on. Still, while it could slow down the network, even if everyone had been streaming music, it couldn't cause the problems we saw.

So we logged every single packet of data. The Ethereal logs spanned gigabytes.

Meanwhile, the network was getting a little better. Management sent out a note that WebTrends saw some things they didn't like and anyone caught downloading music would be fired. An awful lot of their employees went straight to Add/Remove Programs that afternoon. But the problems continued.

This company had been only barely able to work for the past six weeks and we were running out of options.

More traffic analysis showed that a full 5% of all traffic was going from one machine to www.deviantart.com. That was second only to their mail server (with 6.5%). It got stranger: 30% of all incoming connections were coming from adelphia.net. They'd stopped downloading and started streaming. Clever ell-users.

Our application accounted for only 3% of network traffic. Adelphia and a few other sites were firewalled and things picked up a bit. There were still constant delay problems but time-outs had mostly disappeared.

Work had been going on for a full two months, non-stop, and it didn't seem like we were ever going to figure this out. And then we got an E-Mail:

You can close this case. Thanks to $EtherealLogReaderGuy's help we discovered that they had some jokers who were on partypoker.com most of the day. They've been fired and US performance is OK.

Nine fucking weeks. Twenty-eight gigabytes of logs. More than a dozen of our people. All because two utter fucktards were playing poker on some central servers.

Root Cause: 17-Fuckwit

Secrets

While I'm a fan of Open Source, $OurBigApp is closed source. Despite six years with the company and making a pretty strong case to a buddy who has access the source code, I was denied my request (read: repeated begging) to see a little bit of it to help a customer.

Our company is involved with many Open Source projects, but $OurBigApp ain't one of 'em, in part because we have competition, and you'd have a hard time finding bigger champions of closed source than them. Also because $OurBigApp used to be a different company which was bought by a bigger company which is the one involved in Open Source. And also because, well, if we gave away how it all worked, a lot of us would have to look for real jobs.

Seriously, there is value or no one would buy our software. And yet, we may go Open Source within a few years and move from a sales to a service model. Or maybe not. Time will tell. It doesn't really matter to me; I'll have a job either way.

Meanwhile, the secrecy results in some interesting tickets, such as a case I'm still working on. It also makes such tickets frustrating as hell.

Our software has its own file system and we even explain a lot of the inner workings. Unfortunately, someone decided we needed to keep mum on a few bits of it, and one of those bits is the method used to distribute files into subdirectories. Why for the love of all that does not suck does someone actually think that's something we need to keep under wraps? There's nothing terribly innovative about it. More to the point, opening it up completely would work to our benefit, allowing customers to rework the system to best meet their needs.

It seems that whichever jackass made the decision did so thinking, "We need to have secrets and that's one of them. They'll live without the knowledge."

Except we have customers storing hundreds of millions of files. Billions in some cases. And that's where we have the problem, and it's not limited to Windows.

Boring but brief explanation:
Windows can't store more than four billion files in any directory. Once you reach about half that, the system gets really sluggish -- so much so that requests will time out while the system is still looking for the file. It's worse in UNIX for very different reasons, getting very slow as the file count reaches only around four million despite the ability of UNIX to handle many more files overall. The reason is that at 4M, you need yet another inode: searching for a file requires slogging through three multiple indirect blocks.

I got a ticket a few weeks ago asking about some problems with the file system and our utility for setting up subdirectories. I answered quickly, sending the utility, and everything was running smoothly again. Then the guy asked about how the subdirectory distribution system worked.

"Well, I'm sorry to say that it's proprietary. But maybe I can get some additional information anyway. After all, this is an older version of our application and this particular method is no longer even used in our software."

The customer was floored and sent off a gushing note to my manager about people going the extra mile. Yay me.

I actually managed to get to the right person who would be able to tell me about it. Except he couldn't. What he told me I already knew and he couldn't get authorisation to tell me more. I tried again to see the source code just to figure out what the hell the distributor was doing and how it calculated doing it. No dice. And so I have to send the guy an apology that I couldn't do more for him and that he'll be stuck running a few scripts to constantly catalogue and track the files as they're written to the file system.

However, most of the time we get requests for information on the inner workings, it's shit like this:

Hi, we want to reverse engineer $YourBigApp. How do we do it?

Fuckwits.

Fawlty Admins

Install the Application, Manuel!

"¿Que?"

Hoo-boy!

The customer's in Spain. They're using our old-as-dirt (6 years) version of $OurBigApp. The Win32-based one. The totally Win32-based version. It had a lot of problems and we were dependent on a particular piece of software I'll call $Balance to handle -- what else? -- load balancing.

The weird thing was that it shared a few DLLs with $OurBigApp. If you installed it, it had to be installed before our software. We'd pick those DLLs up and run with 'em. If you didn't install $Balance first, you'd have to completely uninstall our software, install $Balance, then re-install our software from scratch.

We told people we recommended installing $Balance even if they didn't think they'd use it, and if they installed $Balance on one machine, the should definitely install it on all of their boxes since machines using different DLLs equals anomlies.

Comes the ticket:

Hello,

we're studying to install a new application server. The question is: Could we install the new server whithout configuring $Balance on it?

Thanks in advance,
Enrique

What I wanted to write was "No! Install $Balance! Save yourself and us a lot of headaches! Install it beeyotch!" But I couldn't. Our Supported Systems Guide (SSG) says that it's not mandatory. That means I have to say it's not mandatory. I squirmed.

If your system is currently using $Balance then you should install it on the new server. If you don't, $Balance isn't required but it is recommended. If you choose to load balance the system later, you'd have to uninstall $OurBigApp on each server, then install $Balance, and then re-install the $OurBigApp from scratch.

If you're not actually running $Balance, it doesn't have to be configured and running but it must be installed before $OurBigApp.

Cheers,
REC

There came a reply.

Thank you for the answer. Could you be more explicit with your explanation?. We're not using $Balance, but it has actually installed and configured on the five Servers on the Production Environment. When we're going to install the new server, MUST we install $Balance software in this one although we aren't going to use it, as is in the five Servers mentioned? We don't use to load balance the system in a future.

Didn't I make myself clear? Apparently not. Let's try again.

Enrique,

Since $Balance is installed on all other servers we would strongly recommend installing it on the new server as well.

Cheers,
REC

That seems to me to be pretty damned straightforward.

Excuse us, we don't understand if your answer is a recommendation or is obligatory. Please, do you mean we HAVE to install $Balance on the new server because is mandatory?

He's joking, right? Has he still not comprende'd this yet? Son of a bitch!

Enrique,

I cannot tell you that you're required to install $Balance because the SSG lists it as optional. HOWEVER, we STRONGLY RECOMMEND you install it anyway for the following reasons:

* You've installed $Balance on all the other servers
* Effort should be made to keep Production machines identical
* $Balance and $OurBigApp share certain DLLs.
* Without $Balance on the new server these DLLs will differ throughout Production
* We have seen cases of odd and irreproducible problems in mixed systems

I'm not allowed to tell you that you must install $Balance; all I can tell you is that it would be a VERY GOOD IDEA to do so for the reasons stated above.

Cheers,
REC

There. I'd spelled it out and still managed not to violate our guidelines, which include not claiming something is mandatory when the SSG doesn't say so. He wrote back a couple hours later. It seems Enrique finally saw the words I'd been writing...

You say in last letter install of $Balance is not required, we won't. Thank you. Please close ticket. Thanks, Enrique

This isn't a language barrier. Ricky doesn't want to do any sort of installation, not even one that requires five minutes and seven mouse clicks.

We'll be hearing from him again in a couple weeks. I'll cut and paste the answer to his upcoming problem from this very blog: Uninstall $OurBigApp on each server, then install $Balance, and then re-install the $OurBigApp from scratch.

Root Cause: 17-Fuckwit
Pay attention, people. Just because we Monkeys aren't supposed to tell you things doesn't mean we don't try to anyway.

Pancakes

HELP!!!!1!11 Your system is broken!

Uh-huh.

None of the graphics are being transmitted! Everything works in Development but Production is down! We followed the guide exactly!

I'm believe you.

OK, send me the Web server information and log as well as the logs from $OurBigApp. Also send me a full directory listing for the Web server. Oh, and a screenshot while you're at it. You never know.

Sure enough, the Web logs showed error 404s (file not found) for most graphic files. I looked at the directory listing and the graphics were in their directories. The screenshot showed few of our graphics being displayed properly, like the splash screen and some arrows.

And then it hit me.

* * *

Despite some strict rules (or rather, thanks to the lax enforcement of them), I occasionally send mails around with a few fun links or an appropriate picture gleaned from some site or another that I waste my employer's time reading.

One time I thought I'd embedded a picture in a reply to a mail beggin for someone to please take a ticket for a problem no one knew anything about.

Except I hadn't embedded the picture. Windows was helpful so while I actually saw the picture in the mail before hitting send, no one else could access the shortcut to C:\Documents and Settings\REC\Desktop\pics\pancake_bunny.jpg.

* * *

I asked the customer to send me the Web templates. Sure enough, they were full of "C:\Documents and Settings\JoeBlow\My Pictures\$OurBigApp\Graphics\*". The twit had made new templates with Dreamweaver or something similar and had used drag-n-drop to add new images.

It would've been a lot funnier if he'd saved the graphics to a shared directory on his notebook computer. The LoadRunner testing would've smoked his laptop's drive.

Root Cause: 17-Fuckwit.

Where's the Start Button?

"We downloaded the file system clean-up program and it won't run!"

Huh? It's not available for general download; it has to be requested for specific purposes. Check the back tickets for these guys and sure enough, their last ticket was handled by a colleague in Bangalore. A quick glance at their system: Solaris. A shudder as I click over to the attached files...

There it is: cleanup.exe. For a customer running Solaris.

Which is worse: one of our idiots sending a UNIX customer a Windows file or a UNIX admin actually trying to run a Windows program?

I swear this shit really happens. Every. Damned. Day.

Sometimes that picture just ain't worth a thousand words.

"Help! The browser-based application crashes once in a while for no reason. Here's a screenshot of the error."

Oh yes, very helpful, that. Chock full of information.

I wrote back to him telling him the next time the user has this error to click on "En savoir plus sur les modules complémentaires..." because the stuff there might actually be able to help me find the problem. Don't get the wrong impression here -- French is one langage I've mostly forgotten, but Windows is Windows in any language. I can't read a single Korean character but Installing and setting up a Korean-language Win2K3 "Advanced" Server in is a piece of piss.

Anyway, a week later I got an update.

No Dr Watson info or perhaps a log, but he was so helpful that he cut and pasted all the text from the box.

Into Word.

There are 329 pages (Arial, 12pt, single-spaced) of:

00000: 0d 00 0a 00 0d 00 0a 00 ........
00008: 55 00 6e 00 65 00 20 00 U.n.e. .
00010: 65 00 78 00 63 00 65 00 e.x.c.e.
00018: 70 00 74 00 69 00 6f 00 p.t.i.o.
00020: 6e 00 20 00 64 00 27 00 n. .d.'.
00028: 61 00 70 00 70 00 6c 00 a.p.p.l.
00030: 69 00 63 00 61 00 74 00 i.c.a.t.
00038: 69 00 6f 00 6e 00 20 00 i.o.n. .
00040: 73 00 27 00 65 00 73 00 s.'.e.s.
00048: 74 00 20 00 70 00 72 00 t. .p.r.
00050: 6f 00 64 00 75 00 69 00 o.d.u.i.
00058: 74 00 65 00 a0 00 3a 00 t.e. .:.
00060: 0d 00 0a 00 20 00 20 00 .... . .
00068: 20 00 20 00 20 00 20 00 . . . .
00070: 20 00 20 00 41 00 70 00 . .A.p.
00078: 70 00 a0 00 3a 00 20 00 p. .:. .
00080: 43 00 3a 00 5c 00 50 00 C.:.\.P.
00088: 72 00 6f 00 67 00 72 00 r.o.g.r.
00090: 61 00 6d 00 20 00 46 00 a.m. .F.
00098: 69 00 6c 00 65 00 73 00 i.l.e.s.
000a0: 5c 00 49 00 6e 00 74 00 \.I.n.t.
000a8: 65 00 72 00 6e 00 65 00 e.r.n.e.
000b0: 74 00 20 00 45 00 78 00 t. .E.x.
000b8: 70 00 6c 00 6f 00 72 00 p.l.o.r.
000c0: 65 00 72 00 5c 00 49 00 e.r.\.I.
000c8: 45 00 58 00 50 00 4c 00 E.X.P.L.
000d0: 4f 00 52 00 45 00 2e 00 O.R.E...
000d8: 45 00 58 00 45 00 20 00 E.X.E. .
000e0: 28 00 70 00 69 00 64 00 (.p.i.d.
000e8: 3d 00 33 00 37 00 36 00 =.3.7.6.
000f0: 38 00 29 00 0d 00 0a 00 8.).....
000f8: 20 00 20 00 20 00 20 00 . . . .
00100: 20 00 20 00 20 00 20 00 . . . .
00108: 4c 00 6f 00 72 00 73 00 L.o.r.s.
00110: 71 00 75 00 65 00 a0 00 q.u.e. .
00118: 3a 00 20 00 32 00 30 00 :. .2.0.
00120: 30 00 36 00 2d 00 30 00 0.6.-.0.
00128: 33 00 2d 00 30 00 36 00 3.-.0.6.

Sometimes I hate French people as much as I hate fucking Windows' use of UCS-2. Not that I truly hate the French. If he'd been from the Netherlands I'd've hated Dutch people today.

After half an hour converting this mess into something remotely legible I found the problem is a damaged ActiveX control which he should have seen himself had he bothered to look in Tools: Internet Options: Settings: View Objects.

Checking the ActiveX objects is always one of the first things you do for a browser-based client which uses them and had he done so I wouldn't've had to go through the tedious process of manually cleaning up a hex dump which, in turn, took so much time time that I forgot about my layout and strategy and the damned Choson came in and whacked my Hittites into total submission so I have to start a new round of Age of Empires.

Give me my Root Cause 17 already!

Cow-orkers II: Groundhog Day

"REC!" she screams as I walk in the door more hangover than usual.
"Hhnnnngrfmm?"
"What is German word for Alpeeeen MarMOAD?"

Ripa's shrill voice rips through the fog and interrupts my only functioning thought process, that being to continue placing one foot in front of the other until I get to the coffay machine.

"Why the hell are you asking me what an Alpine marmot's called in German?"
"Because you are the A-mair-ican here. You have to know."

What the hell does being American have to do with knowing the German name of some rodent? A rodent, I might add, which is native to Europe.

"Why the fffu... Why don't you just look it up yourself on LEO?"

Treating that question as rhetorical, she carried on. "I theenk it is Murmeltier."
"A Murmeltier is a 'groundhog', just like in the film. Go look it up at LEO."
"Then tell me what is Alpeen MarMOAD!" she hollered over the cubicles, ignoring my suggestion again.

Clickity-click. Three fucking keystrokes later and I'm on the dictionary site.

Well how about that? German only has one word for marmots, groundhogs and woodchucks. "It's called an Alpin Murmeltier," I yell back, still jonesing for my morning fix: a liter of coffay.
"I told you it was Murmeltier! Why are you don't leesten to me?"

I close the LEO tab and bang my head on my desk some more. It's much less painful.