As more and more time passes since we installed the 7410, I seem to get more people complaining the system is slow. Generally, these complaints seem to be correlating with when the full backups run. Well, it makes sense that things will slow down while I am doing a front-to-back read of the data.

It turns out that the reason for slowness might actually be because of a configuration error. When creating a share on the 7410 there is an option for “Update access time on read”. This option is on by default and the Sun employee who installed our system told us we should never need to change that. What does this mean? It means that every time a file is read the system updates the meta-data of that file with the timestamp it was last read. So for every file read, the system does a write. In fact, due to the copy-on-write nature of ZFS, the system ends up copying the entire file table tree up to the root of the tree. This is a very time expensive operation to do and severely impacts performance.

When is the access time of a file used? Generally, this would only be when you need to perform forensics on the filesystem if it was hacked. Perhaps there is some obscure program that needs the atime of a file. If you run into such a program or absolutely need the ability to tell when a file was last read for forensics purposes, then you should leave atime on. Otherwise, I would suggest turning this feature off. I have turned it off on our shares. Time will tell if things speed up or not.

I have emailed my current provider (Sentex Communications) and another ISP I hear good things about (TekSavvy). Neither offer IPv6 and neither have plans to support IPv6, at least so they tell me.

It would be appreciated for anyone who knows a DSL provider in Ontario that supports IPv6 to let me know.

We also recently had a failed drive. We had a really bad Sun tech working on our case. The drive went through 3 autosupports, which the system was apparently able to recover from, before the drive officially went offline and they decided to replace the drive. When they did finally decide to replace the drive, it took a week of me trying to find out what was going on before everything was coordinated for the drive to ship. I have talked with others, and the concensus seems to be that Sun support has gone way downhill since Oracle bought them out. This experience made me feel I am not getting the enterprise support I need as a sysadmin.

http://www.competitivebroadband.com/consumer/

This got me thinking about what would be involved in granting admin access to users on their desktops. You certainly could just flick the switch and have chaos, but that wouldn’t be very smart. I’m going to create two classifications, Windows and Linux, as they both have their own quirks.

• Viruses are rampant on Windows platforms. When users are browsing the web and opening email attachments as an Administrator it is much easier for viruses, trojans, etc to install on the machine.
• NFS home directories in Linux would have to go. I don’t see a big problem with local home directories aside from it won’t be backed up, which has potential for huge issues. Other authenticated protocols (AFS, NFS4, CIFS), most of which the NAS doesn’t support, could be investigated for network-based home directories.
• Non-standard linux is hard to troubleshoot given the plethora of ways of doing things in Linux.
• It is debatable whether fire fighting viruses, non-standard environments and user mishaps which breaks the machine would generate more or less help requests. More staffing may be necessary.
• Hoteling of desk space would have to stop. As administrator, the user can install keyloggers and other monitoring tools to catch passwords and other information of the next unsuspecting user. It’s not clear this can happen until the building expansion is complete.
• Machines would have to be imaged when re-deployed (see point above for reason), which involves more staff time.
• Machine maintenance – we would be entrusting the user to do system and application updates and/or not to break or remove our automation to do these tasks
• May wish to have users sign policy stating they will abide by legal constraints (no unlicensed software)
• Machines (according to policy) last 4-5 years, postdocs last 3 years. Since there is not a one-to-one mapping of new machines to new postdocs, some of the machines will have to be upgraded before the postdoc leaves. This can cause headaches for IT and the postdoc if the postdoc has made lots of changes to the system.

Most things considered, I don’t see any show stoppers for giving postdocs and faculty administrative access on their desk machine, aside from the hoteling of desk space issue. The above would have to be addressed. It’s not clear there exists enough staff for a change with one help desk and two sysadmins. Such a change takes time to implement and there isn’t a lot of spare time available. I can’t ever see this change for users that are at PI on a temporary basis and thus hotelling desk space is a necessity (eg. visitors, associates, affiliates, etc.) as that is a definite show-stopper in my opinion.

Performance wise, the system seems to be a good match for our environment. We haven’t had any complaints about slowness since getting past the MS Office file locking issues. I have been adding a few more Windows-based VMWare machines lately. You wouldn’t notice that there’s any more load than there was.

So I start looking at things. Many of the machines shat themselves when coming back up because they are dependent on other servers that hadn’t started yet. The other half were rather confused… they are all set to “Last State” in the BIOS for what to do on power restore. It seems many of the machines couldn’t remember what state they were in. I should change those to just “Power on”.

So John investigated the UPS while I’m getting things up and running again. In the logs was a brief switch to battery followed shortly after by a dead battery. The next log item was “8:15am: Power off by front panel”. Wait, what?? Yeah. Someone pushed the shiny power button on the UPS and confirmed they wanted to shut it off. They then turned it back on and didn’t tell anyone what they had done (probably for fear of their job). I blurt out while still in shock someone would do that “Maybe having security silence alarms in the server room isn’t the best idea”.

A 2 hour support call for the phone system, 8 hours of my time, 4 hours of John’s time, a dead hard drive, and many upset researchers later all because at some point someone along the line decided it was a good idea to have unqualified people pushing buttons in the server room rather than just having them report the issue and let it beep. Thanks a lot! Have you learned your lesson yet?

On April 27, the 2009 Q2 update was released. I diligently scheduled an upgrade as soon as possible. I installed the upgrade on Saturday May 2 and all seemed fine. When I arrive to work on Monday, I get some complaints about Microsoft Outlook and Adobe InDesign randomly crashing. After some investigation, the issue involved open files on the CIFS share from the Sun Storage. In the error log in all cases were “delayed write failed” messages. More investigation showed that CIFS kept restarting itself. Well, if the CIFS sessions are ending with a CIFS service crash, that would explain the delayed write failed issues.

So again, off to contact Sun technical support. They walk me through submitting a support bundle and let me know they will get back to me. Ok, fine. Until May 5 rolls around and the Storage system panicks in the middle of the day. Yay, more CIFS bugs: “reboot after panic: mutex_enter: bad mutex”.

Support finally gets back to me the next day. “There are so many bugs related to CIFS in this release that it is impossible to determine which ones you are hitting. The Q2 release is an entire new release of Solaris, so it’s not possible to backport any of them to the previous release. The developers are working on a fix. Since it’s currently in development we can’t predict a release date.” After this I had to work quite hard on trying to get a recommendation on what to do out of the support technician. Finally, he recommended that we revert to the latest 2008 Q4 release.

Ok! Let’s revert… but, wait. That would have to be scheduled. So I ask management if I can revert that night after hours so that we don’t chance losing any data. I’m told I’m not allowed to and that the Microsoft Office issue was big enough that unless the system panicks again I’m not to do anything to it. Risk of losing data is less important than having to wait a few seconds for a file to open.

Fast forward to May 12. Luckily no more system panicks yet. Sun releases patch 1 to their 2009 Q2 release. I schedule the install for the next evening. We have now been running with the 2009.Q2.1 release since May 13 without any issues. Yay!

So, my point and hard-earned experience: don’t upgrade to the first revision of a quarterly release from Sun!

Our configuration:

• 7410 Storage Controller with 16GB RAM, 2 x 2.3GHz Quad-Core processors and built-in 4 x 1Gb ethernet ports
• 22 x 1TB SATA drives, Double parity RAID (14TB formatted)
• 2 x 18GB Solid State write disks, RAID1
• 100Gb Solid State read disk

This does not provide redundant controllers. The system can be made fully redundant by doubling the price and clustering multiple controllers and J4400s.

Exported, we have:

• /export/home – NFSv3 and CIFS home directories
• /export/vmware – NFSv3 for VMWare (currently 6 VMs)
• exchangedb iSCSI lun – for exchange database files
• exchangelogs iSCSI lun – for exchange transaction logs

Configuration issues:

• The Sun tech that installed the system suggested to dedicate one of the ethernet interfaces to the administrative BUI as the analytics tool uses a lot of bandwidth. This proves to only be useful if you are hitting the administrative interface from the same network as the Sun system is on. Otherwise it sends all traffic out the interface configured for your default route, which is likely tied to your data interface. In our case we would be hitting the admin interface more-or-less always from a different network, making their suggestion a waste of an interface.
• We wanted to do a combination of IPMP and LACP. This would allow us to aggregate two interfaces each to two different network switches. If one switch died the 7410 would fail over to using the other aggregate. The problem with this is there was no option in the BUI to configure failback to a preferred aggregate in the IPMP options. As we were using iSCSI to email and NFS to VMWare with the 7410 we would prefer to use a specific switch. Because of this we ended up using LACP to aggregate all four interfaces. This makes the system not resiliant to switch stack failure, but significantly reduces the chance of overloading cross-switch traffic.
• We found that whenever we reconfigured the network on the 7410 that the system needed a reboot before it was accessible again. This isn’t an issue as network settings are a set once thing for us, but can be a pain for some people.

Performance:

• We did a JetStress test of a 173GB test database on an iSCSI lun from the system. We achieved 750 IOPS. This was while the system was otherwise idle. Our NetApp FAS270 topped out at ~250 IOPS from the SATA shelf and ~400 IOPS from the local FC disk.
• We ran iozone on the system.  I am not very good at interpreting the results nor making graphs of the results. It looks like we saw on average 80MB/s transfer for the test.
• We see on average 130 IOPS use of the system, which is more than sufficient for us.
• Performance can still be increased by adding up to 5 more (albeit expensive) 100Gb Solid State read disks.
• Aside from the Microsoft Office bug (see below) we have not heard complaints about performance in two weeks.

Backups:

• We were able to successfully do a full backup the NFS/CIFS fileshares from the 7410 using NDMP. The fileshares are not browsable via NDMP, so you must tell your NDMP client the full path to what you want to export.
• Incremental or differential backups via NDMP is still a mystery. I need to open a support case for this.
• iSCSI luns are not yet able to be backed up via NDMP. We are doing host-level exchange backups for this. There is no “SnapManager for Exchange”-like tool as NetApp had. Sun claims NDMP backups of iSCSI luns are on the development board for later this year.

Other issues:

• There is a bug for file locking on shares that are exported both NFS and CIFS. When opening Microsoft Office documents from the share there is a 15-30 second wait while a file lock is aquired. This has been explained by Sun tech support as being the result of having to delve down to the individual SATA disk for the file lock. They are implementing a fix for the next release.
• We created the fileshares with Reject non-UTF8 filenames. This is a default setting and not changable once the share is created. This caused issues copying files while using linux from the NetApp to the 7410. The NetApp had some latin1 encoded files that would not copy. We were 6 hours into the data move when the issue showed itself. We found a work-around to use CIFS to copy these specific files.
• File permissions mapping between CIFS and NFS is just as bad as using NetApp in Mixed mode. This is due to Posix file permissions being inherently incompatible with ACLs. After a lot of work one can massage the permissions to work properly, but it’s mind boggling madness. I understand this is not so big an issue with NFS4 which uses ACLs by default, but we are stuck in NFS3 world.
• Authenticated NFS (eg. kerberos) does not exist yet. Apparently this is in a future release.
• User quotas don’t exist. There is a workaround to create a seperate share for each user. This is unmanageable in my opinion. It’s a good thing Perimeter’s administration thinks it’s too draconian to implement quotas.
• Snapshots are not named in a user-intelligible way. They show up with the unix timestamp (eg. .auto-1238601600). They are only accessible from the root of the directory structure (eg. \\solar\home\.zfs\snapshot). The Windows Previous Versions tab does not show up. Thus, snapshots are a little less user-friendly, however they are still there and quite usable.

All in all, I like the system. It is significantly cheaper than anything else which was a major decision in keeping the unit and gives as much expandability as the NetApp 3410 would have. The unit comes with free software upgrades, and for a very cheap price (considering the cost of the unit) was purchased with 3 year hardware warranty.