jjstautt

Elizabeth May – “not in a million years”

Jack Layton – “We can’t support a budget that gets the priorities so wrong.”

This budget hurts the unemployed, the environment, and the economy. Shame.

I just got home from the LISA conference this year. I took a full day course on ZFS while I was there. Aside from being very inclined to replace my Ubuntu home media server to BSD for the sole purpose of being able to use ZFS I also learned a bit on how to speed up our 7410.

As more and more time passes since we installed the 7410, I seem to get more people complaining the system is slow. Generally, these complaints seem to be correlating with when the full backups run. Well, it makes sense that things will slow down while I am doing a front-to-back read of the data.

It turns out that the reason for slowness might actually be because of a configuration error. When creating a share on the 7410 there is an option for “Update access time on read”. This option is on by default and the Sun employee who installed our system told us we should never need to change that. What does this mean? It means that every time a file is read the system updates the meta-data of that file with the timestamp it was last read. So for every file read, the system does a write. In fact, due to the copy-on-write nature of ZFS, the system ends up copying the entire file table tree up to the root of the tree. This is a very time expensive operation to do and severely impacts performance.

When is the access time of a file used? Generally, this would only be when you need to perform forensics on the filesystem if it was hacked. Perhaps there is some obscure program that needs the atime of a file. If you run into such a program or absolutely need the ability to tell when a file was last read for forensics purposes, then you should leave atime on. Otherwise, I would suggest turning this feature off. I have turned it off on our shares. Time will tell if things speed up or not.

I am looking for an ISP that directly supports IPv6. I have done the tunnel broker thing, and it’s a pain to keep updating my IPv4 endpoint and all that fun.

I have emailed my current provider (Sentex Communications) and another ISP I hear good things about (TekSavvy). Neither offer IPv6 and neither have plans to support IPv6, at least so they tell me.

It would be appreciated for anyone who knows a DSL provider in Ontario that supports IPv6 to let me know.

It’s a sad day that 53% of voters get to rip the rights away from a minority group. This is what happened in Maine yesterday when the voters ripped the right to marry from same-sex couples. I think the picture says it all…

So we recently upgraded to ak-2009.09.01.1.0 to take advantage of the improved iSCSI software. The upgrade went smoothly. The iSCSI management is a lot nicer with this release. You get a lot more fine-grained control over the luns. We haven’t seen any issues yet.

We also recently had a failed drive. We had a really bad Sun tech working on our case. The drive went through 3 autosupports, which the system was apparently able to recover from, before the drive officially went offline and they decided to replace the drive. When they did finally decide to replace the drive, it took a week of me trying to find out what was going on before everything was coordinated for the drive to ship. I have talked with others, and the concensus seems to be that Sun support has gone way downhill since Oracle bought them out. This experience made me feel I am not getting the enterprise support I need as a sysadmin.

I just came across this website that was in the news. So far 20,000 Canadians have wrote their MPPs. I urge you to do the same. Voice your opinion and keep broadband internet from becoming more expensive and slower.

http://www.competitivebroadband.com/consumer/

We get to meet and hear from the IT consultant this Thursday about all the changes he wants to make to PI. I sit here pondering what drastic measure he will try to impose on us that’s going to push me past the brink instead of enjoying a nice labour day. I imagine since I am only an underpaid peon (Systems Administrator) I will not get the full story of recommendations.

Some things he suggested in our one-on-one meeting way back:

• Give postdocs and faculty Administrative privileges (I am ok with once we stop hoteling their desks when they’re away)
• Hire someone with (Maple, Mathematica, Matlab, Python, C++, etc) coding experience for use by researchers
• outsource web development. On-staff coders should be integrators only
• outsource networking
• outsource email (Exchange)
• outsource web hosting
• outsource ……..

If there weren’t so much stuff to do around PI I would be worried about still having a job after this meeting. The reality is there’s too much to do.

Perimeter recently had Jim Cranston come in to review the IT department and how we do things. One of the things he asked is why we don’t allow administrative access on our desktops. All of the users he talked to want administrative access over their desktop and have complained that they don’t have it. I obviously hadn’t prepared for questions like this, having not been told what we’d be talking about, so I came up with the usual reasons: with users running as admin access viruses propagate quicker, users can install unlicensed software which we may have to prove to software vendors wasn’t installed by PI, it creates a non-standard environment which is harder to troubleshoot, provides an easy way for the user to cause bad things (ie. delete command.com or otherwise break the system) to happen, and security concerns such as keyloggers when sharing the machine, among many other concerns. He responded with “Every desktop is assigned to a user so there shouldn’t be any security concerns.” This made it clear he doesn’t have a good handle on PI, as we hotel our desk space when researchers visit other institutions or go on sabatical and thus the machines in people’s offices are actually shared machines.

This got me thinking about what would be involved in granting admin access to users on their desktops. You certainly could just flick the switch and have chaos, but that wouldn’t be very smart. I’m going to create two classifications, Windows and Linux, as they both have their own quirks.

• Viruses are rampant on Windows platforms. When users are browsing the web and opening email attachments as an Administrator it is much easier for viruses, trojans, etc to install on the machine.
• NFS home directories in Linux would have to go. I don’t see a big problem with local home directories aside from it won’t be backed up, which has potential for huge issues. Other authenticated protocols (AFS, NFS4, CIFS), most of which the NAS doesn’t support, could be investigated for network-based home directories.
• Non-standard linux is hard to troubleshoot given the plethora of ways of doing things in Linux.
• It is debatable whether fire fighting viruses, non-standard environments and user mishaps which breaks the machine would generate more or less help requests. More staffing may be necessary.
• Hoteling of desk space would have to stop. As administrator, the user can install keyloggers and other monitoring tools to catch passwords and other information of the next unsuspecting user. It’s not clear this can happen until the building expansion is complete.
• Machines would have to be imaged when re-deployed (see point above for reason), which involves more staff time.
• Machine maintenance – we would be entrusting the user to do system and application updates and/or not to break or remove our automation to do these tasks
• May wish to have users sign policy stating they will abide by legal constraints (no unlicensed software)
• Machines (according to policy) last 4-5 years, postdocs last 3 years. Since there is not a one-to-one mapping of new machines to new postdocs, some of the machines will have to be upgraded before the postdoc leaves. This can cause headaches for IT and the postdoc if the postdoc has made lots of changes to the system.

Most things considered, I don’t see any show stoppers for giving postdocs and faculty administrative access on their desk machine, aside from the hoteling of desk space issue. The above would have to be addressed. It’s not clear there exists enough staff for a change with one help desk and two sysadmins. Such a change takes time to implement and there isn’t a lot of spare time available. I can’t ever see this change for users that are at PI on a temporary basis and thus hotelling desk space is a necessity (eg. visitors, associates, affiliates, etc.) as that is a definite show-stopper in my opinion.

I’ve been watching some Ted Talks lately and came across a gem. With the right plants, you can grow enough oxygen to replenish what you use with very little plantlife.

On June 30 we had another crash of the storage system. At the time we were running the 2009.Q2.1.1 release. I was on holidays at the time it happened. My co-worker ended up rebooting the storage system and everything came back up. We didn’t call Sun for support on this; we were a few releases behind, many of which applied fixes for various system panicks, crashes and such. We scheduled downtime for an upgrade of the system. We are now on version 2009.Q2.3.1, and have been running smoothly since July 9th when we did the upgrade.

Performance wise, the system seems to be a good match for our environment. We haven’t had any complaints about slowness since getting past the MS Office file locking issues. I have been adding a few more Windows-based VMWare machines lately. You wouldn’t notice that there’s any more load than there was.