There's a NetApp for that

PRIO-1! PRIO-0! THE SKY IS FALLING! WE'RE DOOMED!! HAAAALLLP!!!

What a way to start the day. A quick look showed Srini works for $BigInterwebs who are still in development. You can't have a Prio-1 if you're in development, and if you want 'round the clock support, $MegaCorp makes you pay extra for it. So I don't feel too bad that it was almost lunch before I even see just the tip of this iceberg.

Reading through the problem, it was going to be after lunch before I answered it. The puppy needed feeds and it's marked Prio-3 (because that's what it is).

I'm not being as much of a dick as it seems. Srini is in California; his day starts after mine is well over, and thank fuck for that because it means he's unlikely to ever get up early enough to try calling me. Copypasta continues:

$YourBigApp %datastore% param has a limit of 255 char length since we are limited with 4 filesystem folders in our Production Environment so unable to include additional folders.Please let us know below are there any available options!!

1. Reference a config file to maintain this param value or any enhancement in future.
2. Any alternative to specify a way to store more folder names

A field length of 256 characters for the directory list should be able to allow even the least creative out there to specify at least 10 sub-directory location. And since one subdirectory can safely hold around 2.1 billion (~2³¹), it's going to be a special challenge to run out of room anytime soon. And I told him so, even explaining how to map and rename in a network-neutral sort of way.

And that was Srini taken care of. If only some of these other tickets were as easy.

This morning I logged in and the IM went nuts, popping up windows from four different mooks at Central who had tried to ping me about some ticket. There were more than a dozen mails all referring to the same damned ticket number. It dawned on me that I hadn't finished building a repro yesterday and this weird international sorting problem was a big deal for $EuroTelco...

And then I saw which company had the ticket number:

Escalate!! make the priority to 2!!

We have a limitation on the filer provided NetApp storage which can accomdate 91k files per folder but our environment creates around 40K files per day since we have trasncripts,email and other attachments. This is the defect..

You have NetApp storage? Which we don't support. Which we told you {clicky-clickety-click} a whole month ago we don't support. And you went and got it anyway even though it's unnecessary, a burden on our system, causes you a performance hit, and it can't hold more than two days' worth of your files in a directory?

Had you listed to us in the first place you'd have saved time and money and aggro and you'd have a system which, based on your stated fill rate of 41K docs a day, would be able to hold no fewer than 2004 years' worth of files.

Root Cause:17 - Fuckwit.
We already told you this shit but yeah, what do we know?

When You've Got Nothing to Lose

I don't do politics here but I can't pass on this one. Lt. Col. Dr. Robert M. Bowman, (USAF, ret.) has a rather "conservative" site over at thepatriots.us. His open letter to US military commanders left me wondering just one thing: what the fuck took someone so long to write this?!

We find out early on. Dr. Bowman has terminal cancer. At the end of the two-page letter he writes how he remained silent before the US went into Iraq and that he "must not make that mistake again." Fair dinkum, but why did he remain silent? It looks like for the same reason everyone in the Pentagon who might read it has: make waves and lose your pension as well as your position on the board of a contractor once you retire. The only time for action is when you have nothing left to lose. A looming deadline, for example, and terminal cancer sure as hell gives you one of those.

Satish doesn't have terminal cancer as far as I know, but he, too, was running out of options. It started simply enough.

Did install one app server on one windows machine and fileserver got installed in c:\appz\filesrvr

Installed new server on another windows m/c in the same system but then when this server tries to access the filesystem it gives following error message . To facilitate this changed the following parameters pointing to the shared filesystem path for all the components of server 2

OK, maybe that's not so easy, but being able to decode whatever the hell he's on about makes my degree relevant to my job. Satish sent me a list of errors (or "erros).

The path 'T:\uprefs' does not exist or is not a directory. If the problem persists, please contact your systems administrator.

Over and over again for every single component of the system. Pages of these, something like two dozen per user access attempt. If you work in any remotely related field (which you probably do or you wouldn't be reading this), you're probably thinking, "He didn't map the file system properly." That's what I thought.

Hey, Satish. Check the permissions and accessibility of the T: drive for the second server. The errors you're receiving indicate a missing path/directory or incorrect permissions. See KB doc 1AT9003 for all the gory details and bits to double-check.

Love,
REC

And that should've been the end of it.

It is all set correctly.. I can map the T:\ drive from m/c 2 and access all the directories, the installation on both the m/c has been done using the same id, also the sharing and security has been given as full control

No, it hasn't. I know it hasn't. If it had been you couldn't possibly see a attmod.cpp(901) error. It's simply impossible. I'd sooner believe that Lyndon Larouche isn't a crackpot thief than I would believe that you set up the system correctly. There is only one way you can get $OurBigApp to throw a mgdir.cpp (097) error: delete/remap/fail to map the fucking path.

Which is more or less what I told him. Less. Much less. And in much nicer and simpler words. Not quite as simple as this explanation of relativity, but certainly something that my cow-orkers' kids could understand and follow.

Satish came back again, his English getting even choppier.

I tell you It all is set correctly..! Our administrator is out of station and I am mapping correctly. drive T:\ is mapping from m/c 2 with all directories access , We are now ask a third once for the solution for this problem that the T:\ is not to be reached from m/c 2.

In case you're wondering, "out of station" is Indian slang for "out of town", meaning I'm dealing with a scared PFY n00b. Worse, his company has entrusted the very expensive process of setting up $our_(very expensive)_BigApp to... him.

I fired up Paint.NET, drew a quick diagram and sent him not only explicit instructions but a load of documentation page references. Bangaloreans seem to love that (more on that in a later entry this week). You give 'em written documentation of an 87-step process and they'll take that over the easy, three-step way you tell 'em every fucking time.

Since I could see that he was having a conniption fit thinking that I wasn't taking him seriously, I also asked him to send me the server logs. All of them. Within two hours I had a 25MB zip of half a year's logs and his acknowledgment that he would continue reading through the docs I'd cited and would get back to me.

I waited. And waited. And waited. Two weeks later Vera was harping on about my huge backlog of no response tickets. I told her I preferred to give customers extra time when it appeared they need it. "It's very good for customer satisfaction." She can't touch me when I say that. It's not just my Get out of Jail Free card, it's my motherfucking Get Off Death Row and Go Straight To Paradise Island card. She can try to get around it but that phrase is golden.

Still, it doesn't behoove me to piss her off so I set to work closing out tickets, first sending a personal note asking about the subject one last time. If it's sorted they'll usually ignore it but sometimes they write back. $Telco was more than a week past the go-live date. The problem had to have been sorted by now. Satish wrote back.

Resolved this issue, by setting the UNC path for the filesystem

Uh-huh. Back up against the wall he finally gave in and did what I initially told him: check the fucking path. I can't help wondering if I'd included a "please do the needful" in my original answer that he might've actually done it. This only makes me wonder more about what the hell he was doing each time I sent an update.

Root Cause: 17-Fuckwit.

Headbanger's Ball

The my-shaped-dent in front of my keyboard is getting deeper and deeper. They're all so fucking stupid I don't know how the fuck these assmonkeys remember to breathe, and frankly I wish they'd stop already. It'd sure make my life easier.

$TouchyInsuranceCo is back. They've stopped scraping the bottom of the barrel to find IT employees and have headed over to the sewage pits with a few buckets. The current "Windows administrator" doesn't know how to copy files from within Explorer.

Here's the E-Mail and ticket update exchange. I'm not making this up, as Mini-Me can confirm.

Twaticus: We would like to know what is the maximum size limit on file system folder. Currently the size of $ABC folder is 80 GB. When do we decide to move to new file system folder?
REC: No limit. Add volumes if you want. No need to move the file system unless you have more than two billion actual files.
Twaticus: If we create new file system, do we need to create all the sub folders from Old file system to the new file system?
REC: If you insist on moving it, just move the entire file system parent directory and structure over from the old system to the new. There's nothing else to do besides update the path the cfg files.
Twaticus:You mean we need to create just the structure like [complex structure redacted]
REC: No. Just more the parent folder, the one called "FileSystem".

OK, it's coming. The part you won't believe. I sure as fuck didn't. I checked the audit trail to see if Mini-Me or someone else was fucking with me and changed what this fuckwit actually wrote.

Twaticus:So we should not copy the files which were existing under these folders in the old file system. Pls advise.

[Second mail 13 minutes later:] Moreover if we only mention the new file system path, how the users will be able to open old attachments?


Gobsmacked, I tell ya.

REC: Do not create anything. Copy over the entire File System root directory WITH all the subdirectories AND all the files contained within from the old location to the new.
Twaticus:There is a difference between What you suggest and what is written in the System Admin documents
REC: What are you talking about? Please cite the book, chapter, section and page number.
Twaticus:[Citation provided]

So I read through this big long section which, in essence, says what I did but does so over three pages. This document was clearly written by some outsourced contractor being paid by volume, not content.

REC: It says exactly what I said but includes more complicated directions for moving the file system when the system isn't taken off-line and when you're over 2 billion files and Windows isn't able to store more files in that directory.
Twaticus: What tool exists for to move the files to the different drive?
REC: ... *THUNK*

Motherfuck how can anyone be so fucking stupid? He reminds me of this guy in boot camp who was kicked out of our company inside a week because he was unable to march even when the drill instructor was screaming out "LEFT! RIGHT! LEFT! RIGHT!" Incredible. It's a special kind of stupid that lets you get confused over which foot to move next. Twaticus has drunk deeply from that well.

While I want to close the ticket with a Root Cause: 17-Fuckwit, I'm still waiting for its implementation. I did find an error in the documents he mentioned though -- a holdover from about 7 years ago -- so we close this as a Defect and make my metrics prettier. But the real defect here isn't the documentation...