The Complete Geek

New Blog

2010-03-08T13:28:00.000-06:00

I have made the evaluation that due to the complexity and doing a cost/benefit analysis that I am not going to go back and port my blog into WP format

If you would like to continue following my technical postings I will continue it at www.youdontevenrealize.com/wp

I will be chaining around redirects and such to make things easier to get to but the old content will stay put
Thank you for all that are following me and hope to see you all on the other side

New blog format

2010-02-02T13:10:00.000-06:00

I wanted to give you a heads up that I will soon be changing formats for my blog, The new format will likely be wordpress but I have to plan the import because of my wish to maintain the links as is.

This is because blogger announced the termination of FTP publishing support today and I wish to continue hosting my own blog

Thanks for your patience and hope to see you on the other side

Vsphere 4.0 Quick Start guide

2009-12-25T10:07:00.000-06:00

A new book, Vsphere 4.0 Quick Start guide by Duncan Epping (as well as many other talented people). I have mine on order already!

If you are not familiar With Duncan check out his blog He is one of the foremost virtualization bloggers and always has useful and concise information.

For those starting down the path of virtualization or veterans trying to expand their skills check out this book!

Point in time copies, are they backups?

2009-10-24T14:48:00.000-05:00

The simple answer is no, the more complex answer is it depends.

Point in time copies (snapshots) can drastically increase your time to recovery. And can be incredibly useful for data restoration due to an application/operating system/ file system/ user problem.

That being said, really the idea of a true backup is to handle DR and the failure of systems, disks and other problems. Being that your data and copies reside in the same hardware/location it does not protect you against such failures.

Now the reason above I stated is "it depends" is because some vendors are offering the replication of existing point in time copies to a remote system. So the idea is that when it resides on the same system it is not fully useful as a backup media but once it gets transfered/archived and if that archive container the sum total of the original data as well (So a mirror or replication) it can act as a full backup.

Current Problems
There is some possibilities here, If you are simply looking for a point in time to be able to restore to for DR for the short term you can fulfill this with a large archive of remote point in time copies (replicas) but with current vendors these technologies fall flat in two area's

1) Long term storage
A lot of vendors are trying to make improvements in this area but still fall short. While it may be feasible to store weeks or even months worth of daily snapshots is doable with some of todays vendors really keeping a lot of these around for years is non feasible. This storage space is also relativity expensive per GB compared tape and other media. At best it can be used as a "Mid Teir" place to store info to reduce your "race to daylight"

2) Archival, cataloging
No solution that I am aware of (and if you know any different please share!) really handles the cataloging on a file level that is available to some other server side tools. What happens when you have a user that asks "I deleted a very important document from 2 months ago, I don't remember its name exactly but it was financial something or other and was an excel spreadsheet" or "I have not looked at my important financial document in a year but someone changed it in the last year, I want the one that was modified in 2008"

With E discovery tools available to some backup solutions this is easy to do I just search for *financial*.xls? but with block level point in time copies this is not as easy.

Some vendors are making strides (specially if they also do file level shares) and are going in the right direction but are not there yet.

Long term storage

A nice trend I am seeing in industry is the ability to ship point in time copies off to alternative media, this is a great idea. Quite a few vendors are making an effort to be able to do things like use standard NDMP to be able to export the data directly from the SAN to something like a data domain or a VTL or directly to tape. This makes long term storage more feasible and economical.

My take on it
Point in time copies are a great way to prevent against many different types of non storage layer problems, and replicas can even help with that. But in the end without cataloging the usefulness and ability to do file level recovery and E-discovery is non existent. It may be perfectly fine for your industry but in some that is a challenge.

3par multipathing in mesh archetecture

2009-10-11T11:43:00.000-05:00

A complicated idea that seems to have quite a bit of merit, their architecture sure seems to have lots of possibilities to provide a high performing very flexible architecture, has my interest piqued.

Disabling SSH Tunneling

2009-09-27T17:12:00.001-05:00

As a corollary to my last post I wanted to describe how to disable TCP forwarding.
You will want to add or uncomment the following line and make sure it is set to no in /etc/ssh/sshd_config

AllowTcpForwarding no

The default is “yes"

Note you will want to make sure that users do not have permissions to set their own forwarders, but this is the global disabling.

This can also be done on a per user basis with the enforce key-based authentication and use per-key directives in each user's authorized_keys file. This is further explained in "AUTHORIZED_KEYS FILE FORMAT" section of sshd http://man.he.net/man5/authorized_keys

I have not done this myself but maybe something you want to evaluate if you are hosting a shared resource.

SSH Tunneling and remote administration

2009-09-25T16:30:00.004-05:00

This is another one of those old school tools that can make your administration life much easier.

What is it:
It is easiest to think of an SSH tunnel as software VPN from your computer (Client) To the server serving SSH. You can then forward any traffic to or from your PC through this tunnel. While not like a full VPN in operations it allows you to forward traffic securely through the tunnel like you are on the network.

Why this instead of a true VPN you ask?

Because it is free, does not require specific network equipment, often pre configured to be ON already on most linux distro's, and setup is easy.

How I have used it:
I have used this tool in many scenarios from my home network, where I want to be secure or access resources that are either impractical to share out individually or were of a type you do not normally publish to the internet (Like Windows File sharing, or NFS)

Also have used it when I need access to resources in networks where there are either overly concerned network guys or where it was impractical because of design to publish out needed resources to the public internet.

Example:
Here is an example of the data flow when it is configured, this example encompasses tunneling a VNC session of the SSH server back through the tunnel and tunneling data (Windows file share port 139 from server 2 back to the ssh server then through the tunnel

Figure 1

Configuration:

Linux:
Using standard open SSH simply use the following line

ssh -L localport:remotemachine:remoteport Targetaddress

So for our example of VNC in Figure 1

ssh -L 5900:localhost:5900 SSHserversaddress

In plain English this tunnels local port 5900 through the tunnel and loops it back to the local host of the SSH server also on port 5900

Following Figure 1 if we wanted to then configure port 139 (windows file share)

ssh -L 139:Server2Address:139 SSHserversaddress

What this says is listen on local port 139, forward it through the tunnel to server2's address also on port 139 at the far side of that tunnel

Windows:

I am going to show the windows configuration using Putty. While the directons will be for this SSH client as it is probably the single most popular client there are other ones that I like to use for this as well, Namely Tunnelier (Bitvice) , A very useful file transfer/SSH tunneling application.

Figure 2

Like above what line 1 says is forward port 139 through the tunnel to the remote side, from there send it to server 2 on the same port. Line 2 says forward port 5900 through the tunnel to the local host of the ssh server.

What this means:

Now you all say "thats neat but what can I DO! when configured"

Using the above example you would be able to type in on the client machine (on windows) \\localhost\share and actually see the contents of the share on server 2 across the tunnel!

Or you would be able to vnc to "localhost" on port 5900 and connect to the vnc session on the ssh server across the tunnel (because it is not published through the firewall)

Advanced Usages:

There are a couple of advanced cases that you can use this for, once you have the concept down it is easy

1) Forwarding different local ports to the same port different destinations on the remote side

Linux:
ssh -L 5900:localhost:5900 -L 5901:Server2Address:5900 SSHserversaddress

What this does is forwards port 5900 to the localhost address of the ssh server also port 5900, it also forwards local port 5901 to server 2 port 5900

In this way you can have a whole bunch of the same traffic to different machines on the far side of the wire without having to change VNC configuration on the far side to listen to a non standard port.

In addition the -N flag can be useful it launches the tunnel without starting a remote session so you can use a 2nd session to communicate through the tunnel
Windows:

Figure 3

What this does is forwards port 5900 to the localhost address of the ssh server also port 5900, it also forwards local port 5901 to server2 port 5900

2) Setting the tunnel to accept connections from other computers on your network to forward through the tunnel.

This one I am going to do in putty only, But putty is available for linux as well. I am unsure if it can be done with openssh

Figure 4

Just one more checkbox here, what this allows you to do is forward on requests from other machines through the tunnel to the far side, making this even more VPN like (Kind of like VPN + NAT through the tunnel)

*Note in all above screenshots replace server2 with the IP address of Server 2 or the host name if the ssh server knows it

Designer of Vmworld interview, Nice hardware!

2009-09-24T22:54:00.000-05:00

Rather cool video I ran across on youtube

An interview with Dan Anderson, The designer behind vmworld 2009's data center.

Also get to see a bit of the hardware close up for those of us who could not attend this year.

Also an impressive amount of money for only 2 weeks (As stated in the video a rough estimate of 35 million dollars!)

ESX memory management

2009-09-24T16:15:00.001-05:00

I ran across this a week ago or so, it was more than helpful

http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

A good clearly defined explanation of memory resource management in ESX. It was clear and concise and covered all the major topics like page sharing, ballooning and memory reclamation. As well as a good case study and basic covering of best practices

Worth the read

Remote X11 an old linux feature that can help make a seamless envyronment

2009-09-23T23:24:00.001-05:00

What is remote-X you ask?

Its a tool I have been using for years that can make multi system administration, testing, access a easy task.

The basics of how it works is this, Remote X11 forwarding is as it sounds, you can redirect the visual portion of linux apps over the network via your favorite SSH client. This works from both another linux client OR from a WINDOWS machine!

Why would you use this?
If you are like me and need to use tools from both the windows and linux world to do your job it can make things easy for gaining access to thoes tools.

Using me as an example I run Windows server 2008 as my base desktop OS and run several debian virtuals at home and the office (or on the laptop itself using vmware server) and just run the graphical apps remotely

I will note that this access method is not really designed for WAN access or in high latency environments, if that is what you are looking for there are other tools to accomplish this. The advantage of this tool is really that it is integrated and that it does the equivalent of seamless windows (so it is only the application that pops up)

How to:
Configure the server:
Step one is to configure the server to send its X11 information over tssh, with debian has this on automatically some other distro's necessitate modifying /etc/ssh/sshd_config

The line you need to make sure is in there and uncommented is

X11Forwarding yes

At that point your machine should be setup to forward X11

One caviat that I have found also is ssh as the user that you want to execute the application as, X11 gets confused by su and switching to another user, it does not know how to forward the display through the other session.

Client side:
Linux
This is the easy one simply add a -X flag to your ssh session

ssh -X yourserver

Windows
You will need only 2 things to acomplish this
1) SSH client that does remote X11 forwarding, I use putty
2) An X server, I prefer Xming (Cygwin can do this)

With Xming you will only need Xming and Xming fonts

At that point simply run Xming (it will work in the background quietly no prompts or other information) and make sure your putty session is configured as follows.

The enable x11 forwarding checkbox must be checked, at that point save your session or connect as is without saving it.

At this point you are good to go, any application you launch from the SSH session will launch locally

In my example here I launched iceweasel from my debian virtual, it is a simple example but possibly a useful one (IE trying to download a file or directory directly to your server but is on a site that uses JAVA to initiate a download instead of actually getting a URL for wget or links)

I also use it for tools like when I am network simulating with NS2 or want to use XChat or any number of a dozen tools that sometimes have windows ports sometimes not.

Conclusion:
Remote X11 forwarding is a useful tool in seamless administration between multiple linux and windows/linux environments and can get you access to your tools in a seamless way that can look act and feel like a local application on operating systems that may not have access to the tool directly (or with more difficulty)

Installing ESX(i) Vmware tools on Debian Lenny

2009-09-14T21:56:00.002-05:00

I have seen quite a few partial or old processes for installing vmware tools for some of the vmware products but not for Debian and ESX. I would like to note that I am doing this step by step, there are of course shortcuts to the way I am doing this but I wish to explain it in simple steps so you can take away why I am doing these steps not just blindly following them.

1) Install the necessary tools

aptitude install autoconf automake binutils cpp gcc linux-headers-$(uname -r) make psmisc

2) From the console select the vmware tools install\upgrade

3) Make a directory to mount the cdrom device to

mkdir /mnt/cdrom

4) Mount the cdrom device to the directory you created for it

mount /dev/cdrom /mnt/cdrom

5) Navigate to the cd-rom directory

cd /mnt/cdrom

6) Copy the Vmware tools installer file to somewhere you can work with it, I choose my home directory (Note the name of the file WILL change as new versions come out, as it is the only file in the directory it will be easy for you to find the new name)

cp VMwareTools-4.0.0-164009.tar.gz ~

7) Untarball the file then change to the extracted directory

tar -xvzf VMwareTools-4.0.0-164009.tar.gz

cd vmware-tools-distrib/

8) Run the installer perl script

./vmware-install.pl

8)Follow the steps on screen, I have left all the defaults on mine
In the past you may have needed to specify things like your kernel header location or compiler or make locations, I did not need to assuming I followed step 1 to install the tools neede

I would also note I am NOT using a custom kernel I am using an almost fresh out of the box one, it may be that the steps vary if you do so (Though vmware installer is pretty good at just building the required modules all on its own)

Converting Equallogic arrays to non standard RAID configurations

2009-08-31T20:57:00.004-05:00

Some of you may not know this but equallogic supports a whole host of useful RAID configurations available only through their command line interface.

My favorite non standard is RAID6-nospares which creates a RAID6 set with all the drives in the chassis without any spares. While we do not use it for production it is good for backup or replication as it gets you the same amount of space as their RAID 5 configuration but will handle a double failure (of course with the understanding you may want to keep disks on hand for replacement)

They also have a RAID50-nospares and RAID10-nospares though I have not had a use for this.

While I am a CLI fan I would mention that I dislike these not being in the GUI, the reason for this dislike is if you want to carve off a new group with this array and this is the first one you have to initialize it on one of their standard RAID sets and convert it (Crazy but yes equallogic can convert RAID configurations WITH data on it … there are limitations, which basically are you can always convert to a RAID set that holds more data)

The steps are this

1) Log into the member you wish to convert (or the group IP) via SSH or telnet. You must log in as an administrator account but it need not be grpadmin (though that would work too)
2) Type the following line “member select {member name} configure raid-policy {policy name} “ (no quotes)

Your available RAID policies are:

raid5, raid6, raid10, raid50, raid6-nospares, raid10-nospares, raid50-nospares

Again keeping in mind that you can always convert to a RAID container that has more space but not the other way around (so for example RAID10 > RAID50 or RAID50 > RAID 6)

The amount of time it will take to convert will vary depending on the amount of data and usage. I would note that I have ALWAYS even with backup data had enough space to extract arrays from the group (so no data on it) before converting it.

Hope that helps! Please comment if there is any additional information you are curious about that I forgot to include.

Unofficial ESX HCL

2009-08-29T16:56:00.001-05:00

My intent is to post all models that I have successfully got ESX and ESXI to run on and what versions and hints. The reasoning is many of these are "Lower" grade desktop/server hardware that works well for home testlab.

If any of you out there have experiences with this class of hardware please post a comment with a description of the hardware and Ill add it to the list. I will note that some of these may be in part or whole on the HCL as it keeps expanding as well but I am not covering the ones I KNOW are on there.

All these have been tested with ESX and ESXi 3.5 and ESX and ESXI 4.0

Manufacturer	Model	Configuration	Notes
Dell	SC430	Pentium D, 4 GB RAM max	Works, cant virtualize 64 bit operating systems otherwise works good for home system
Dell	SC440	Pentium D and Xenon, 4 GB RAM max	Depending on the model ,can not virtualize 64 bit operating system
Dell	T605	2 Socket quad core opterons, up to 32 GB of ram	Great little box for entry level or mobile hosts, dual power supplies and with up to 4 port PERC 6 RAID controller.
Dell	Dimension 4300	Intel Pentium, max of 4GB of RAM	Desktop class hardware, worked good for some home brewed linux virtuals

Limited list so far but I have some on the roadmap, anyone with good experiences weather custom builds or built boxes let me know!

New Favorite Vcenter Features

2009-08-21T23:50:00.002-05:00

There are many many improvements that I love in this post it is going to be GUI features that I like that make my management life easier. The order does not really matter here to me all are very useful.

1) Search

Absolutely love the search box, with over 300 virtuals in 5 different clusters even though they are numerically ordered by name they are still a pain to find. Combine that with the fact that it searches the note fields makes it amazing (especially if you do like us and consistently put the client name and function for every virtual in the notes)

2) Storage vmotion

While there are other changes I am focusing here on simple act of integrating it into the gui, while I am a CLI fan at heart, the ease it makes it easy, especially tracking down which volumes have free space when you have many VMFS volumes in a cluster

3) Storage Views

The ease of tracking down and mapping which VMFS volumes are attached where or which virtuals are in the wrong volume make this a big tool for us. We have phased rollouts in pre-deployment volumes and clusters for virtual sprawl reasons. But that means its easy to vmotion a virtual while forgetting to move it out of the pre-production volume.

4) Performance overview

For a quick look the overview section in the performance tab is a big feature add, I do like getting a dashboard view of some of the important stats to see the health of your system

5) Alerts

While this is not expressly a GUI addition I am going to bundle it with it. The additional alarms are a life saver. One of my favorites is being able to alert on snapshot size and being able to set thresholds. We have whole teams of people building virtual, it is easy for one or two to forget one out there. We used to track this by a lot of SQL fancyness but no more!

There are many and many other alerts that are useful that I will probably create a separate post on just these after we investigate further.

Defining Vmware view in kaseya

2009-08-20T08:30:00.003-05:00

This one is more of a little trick. We have started to use Kaseya to manage our servers and it became obvious at the start that there was a strong need to have a view defined that just shows me all vmware machines.

The simple way to do this is to create a new view

Under the advanced filters view at the bottom filter by *Vmware* in the product name field.

This is what kaseya picks up as the product name

This works for reports too

VMDK vs Direct Attached storage in the iscsi world

2009-08-11T20:35:00.001-05:00

I am approaching this from the point of view of our EquaLlogic SAN some of these (namely number 4) may not be as big of an issue. Also discussing in the 3.5 world for most of this will address what 4.0 changes in the end

Pro

1) One place that VMDK’s are a good choice to use is where you have questionable System administrators with local administrative accounts. In some cases we have vendor or client administer accounts (they are the exception) so they can handle specific processes. The problem is if the virtual actually has adapters on the SAN network they have a more direct avenue for causing problems weather on purpose or accident. Anything from re-addressing adapters or some mal-ware that could get installed as they are an admin (not likely in a controlled environment but still possible)As such VMDK’s protect that network from those admins.

2) VMDK’s also behave nicer in the case of SAN failure or performance issues, if a virtual looses connectivity to its VMDK it pauses rather than dropping the volume.

3) Svmotion, while moving volumes between pools is easy moving them between groups and or different SAN vendors is much harder. There are processes but if your data is in VMDK it is a snap. Simply mount the new SAN or volume in a new group and fly, no downtime necessary for what would have been a major move. Virtual storage is the best thing for storage administrators.

4) Not having to administer separate SAN volumes and configuration. This cuts deployment time for a SAN volume to a fraction of the time separate volumes take. Not having to setup ACLS, MPIO configurations … install HIT and configure adapters. It cuts deployment time from a half an hour to 2 Minutes and moves the knowledge out to system administrators rather than storage administrators.

5) We have run across a number of situations lately that either required a host integration tool upgrade or MSISCSI version upgrade to overcome the loss of connectivity to volumes. This is a BIG deal when that means touching 200+ servers. Reducing update and hand maintenance tasks is a huge money saver from the point of view of labor costs.

6) And the last advantage Ill discuss is administration security, unfortunately the granularity of permissions for EqualLogic is lacking. You may not want all your system administrators to be group admins just so they can expand a SAN volume. We have in the past written custom interfaces to achieve this without giving them direct permission but VMDK’s also achieves this. No longer does a sys admin need to be a group administrator on your SAN just for creating new volumes or expanding drives.

Con:

1) Performance: This is the biggest concern with 3.5 not supporting MPIO on its software initiator the performance is just not as good in the equallogic world where MPIO has a huge impact on performance.

2) Not able to take advantage right now of some of the advanced features the Host integration toolkit provides you. Most people are not using these features so it is not as large of a concern

And I am not pleased to say that we are seeing awesome performance (especially with the new beta driver) out of vsphere4 as the software initiator now does have MPIO support. We are likely to evaluate it as our default configuration (presuming successful testing in production) because of the advantages.

cerberus -wag

2009-08-05T15:06:00.003-05:00

This is not a troubleshooting process but a humorous observation.

When looking at the process list someone at equallogic has has a sense of humor when naming flags for their processes

/sbin/cerberus -wag

Someone actually named a process after the 3 headed dog and then proceeded to ad a -wag flag :)

Disabling Equallogic VSS integration

2009-07-20T15:49:00.001-05:00

Here is another rather quick one

In our situation we have HIT 3.2 installed (Host integration Toolkit) for equallogic, we have a piece of software which is taking VSS snapshots and not cleaning up after itself afterwards. Equallogic’s host integration toolkit takes a SAN snapshot when a VSS snapshot is taken.

What prompted this to be an issue is that because the application did not clean up after itself we got lots of online snapshots that the server was logging into.

We ran into a bug with the host integration toolkit that resulted in the dropping of a volume that had an online snapshot when an asynchronous logout was requested (in our case moving it between pools) This is slated to be resolved in the next HIT

The resolution in our case was to disable VSS integration on the problematic servers, this is actually a rather simply process

Simply disable and shut down the eqlvss service

The long term is to un-install the feature but I am unsure of a reboot so it may be advantageous to just disable the service, this causes no impact.

Cloud computing: Why?

2009-07-15T13:03:00.002-05:00

Speaking as virtualization guy the ideal of a cloud
computing resource has some very interesting applications specially when it
comes to DR and HA. The idea that for example at month end when your business
sucks up compute cycles for accounting you can just "borrow" or
transfer load to a rented per cycle resource is intriguing. Allowing you to
plan for the average and rent the difference without having to buy

We are closer than we think:

Yes you heard me say it ... while there is a big push across
the market, heck even vmware is marketing their Vsphere4 as the "Cloud
OS" We really were introduced to most of the idea's of a cloud (at least
an internal one) with ESX3.X

I say this because most of what you think of when talking
about the vague term "Cloud" Really is done at that level with HA and
DRS

You have groups of hosts pooling resources, as load changes
and increases on a host virtuals move around to distribute load across the
cluster. In the cases of failures in hardware virtuals spin up on alternative
resources turning Multi hour outages into seconds.

The idea of a cloud OS really to me is just being able to
vmotion my virtual to someone else’s cluster based on policy or interaction
from me. The basic tools are there, heck some people already operate multiple
"Clouds" in separate data centers and use tools like SRM to be able
to spin up workloads based on policy between datacenters simply with a click of
a button.

Why not just rent the second datacenter?

The challenges:

Some of these are basic, fundamental, and hard to overcome.

1) Latency

Workload and workload interaction is key, the idea of
latency introduces complications to the concept. These are not insurmountable
in most cases ... the idea that you have several virtuals that make up an
"Application" and you simply pool those resources and always move
them together rather than as individuals reduces a lot of those problems (For
example always keeping an exchange frontend and backend together)

2) Security

A commonly brought up problem that has no easy answer

How can we ensure data integrity?

Really the answer is through policy for the most part ...
you can ensure data transmittal is secure ... you can control where when and
how that data comes into and out of your own data center but in the remote environment
it is all done by contract and holding them to the standards

While it is hard as IT guys to relinquish control there are
plenty of hosted models that already do that very same concept (heck I work for
one) It is do-able if they are held to be responsible.

Conclusion:

In the end the concept is nice but we are a long ways off to
a full cloud computing model. I think we will still see further developments in
the internal cloud, more features (E.X. FT virtuals) before we are able to
leverage those features seamlessly remotely

Importing .ovf Vsphere4

2009-07-13T14:31:00.001-05:00

this one is really short and to the point but it is a change from 3.5 and I did spend a few min looking for how to do it.

Simply you can not import .ovf directly into ESX (I know dont know why they REMOVED this functionality ... you CAN buy ovf's off the marked place and import but not from file)

Simply use vmware converter and select "virtual appliance" and follow through

short and simple

Mass mountain SAN arrays

2009-05-15T13:43:00.001-05:00

Mass mountain SAN arrays

We purchased a Mass mountain, as opposed to our normal fair for our new backup to disk product.
The idea was separate low cost hardware with a lot of space, as it was not primary data the performance was not a huge concern. We have run across some issues that put it out of the running for A) primary storage B) This project

First the good:

Mass Mountain uses openE, this allows for a very open and feature rich product. You are able to publish via FC, ISCSI and NAS (all at the same time) along with the ability to bridge other storage devices and publish them out. (For example could connect it to another FC array and publish a volume via ISCSI). Also unlike some others you can put different disk and different RAID all on the same array (For example we wanted a 2 disk SAS raid mirror ISCSI and the other 14 disks raid 50 (with 2 global spares) published NAS in several logical volumes)
As such it is a very very flexible piece of hardware with many options
The bad:

1) No built in way to expand beyond the chassis (Other then using the RAID card to expand the RAID set)
2) Interface non intuitive and often hard to find things
3) Publishing new targets and new volumes via ISCSI restarts the ISCSI service for all volumes (this is a big one!)
4) Max logical volume with replication is 4TB (Deal breaker for our project as it needs a very large single volume)
5) We have had our volumes just “Stop” being published. All servers dropped connection to their ISCSI volumes (this included 2 different RAID containers) The only resolution was to REBOOT the array! No RCA on it.
6) To do MPIO required actually having the NICS on both the SAN and the Servers be in different subnets with the MSISCSI initiator. Which is silly ISCSI is designed to be able for the SAN to be able to control logins and balance between interfaces without having to manually set subnets so they CANT communicate with the other one
All said and done the bads outweigh the goods we have no confidence in the stability and have some deal breaker limitations on size of volumes
As such we have elected to move back to our normally arrays at a significantly higher price for the stability and features

Hyper-v problems part 2

2009-04-25T19:58:00.002-05:00

This one I want to discuss management

First of all microsoft states that best practices is to deploy hyper-v on server-core. As someone with a long history of someone with CLI experience the idea of managing a windows server, specially your single point of failure with microsofts badly implemented CLI interface makes me cringe.

In test lab we already ran across one situation which having a comprehensive interface would have saved a lot of work. In the lab we deployed a stand alone server-core test platform, and some virtuals ... we then decided to build out a domain and join it to the domain. The problem being after that we did not have WMI rights to remotly open the server manager remotly , and to join it to the SCVMM instance we needed to update hyper-v to the RC. But the virtuals we deployed earlier started before the main OS instance and we could not update hyper-v with virtuals running.

We got the problem resolved (removing and re-adding it to the domain after deleting the computer profile) resolved but simply being able to stop virtuals from the CLI would have saved a lot of troubleshooting

SCVMM VS VM

My second qualm is SCVMM which seems very 1st release, problems with naming between the two and how it behaves are common.

SCVMM also falls short in controlls in a HA mode, which is controlled through microsoft clustering not within SCVMM moving virtuals, controlling them, all of which is done outside the SCVMM the tool just does not have the power, controlls or reliability

We found ourselfs going back to VM at times as well to just perform simple actions like controlling the virtual networks. Some of this may be overcome with familiarity with the tool but a lot of it just feels more disjointed.

Monitoring

Hyper-v, from a built in standpoint hyper-v has almost NO advanced monitoring capability. Tracking either virtual or host performance is an almost imposible task. Scaling out to the cluster level and tracking performance is almost impossible

They promice this ability but they achive it through Pro-tips which is only avalaible deploying SC Opps manager which a very large and complicated and expensive product. I am a fan of opps man but deploying such a huge product for what SHOULD be integrated into SCVMM seems silly to me, not to mention requiring a lot of work to get out of the box

I will discuss some of the other shortcomings later

Hyper-v and High availability in an ISCSI environment

2009-03-31T18:57:00.002-05:00

Hyper-v and High availability in an ISCSI environment
I am writing this after doing a fairly extensive multi month test lab / first gen deployment of hyper-v and wanted to talk about some of the specific shortcomings of hyper-v in a high availability configuration. While there are many many other shortcomings of this version 1 software I wanted to address them separately as to not get overwhelmed by the topic

How it does it

For those of you not familiar hyper-v deployed as a roll on top of server 2008 integrates and uses Microsoft clustering to provide high availability and failover capacity.
To the local operating system the drive that the virtual hard disk resides on is shared among the cluster nodes, it is active on one and passive on all other nodes (even if they have other shared drives) they also use a “quorum” drive (the name has changed with 2008 clustering but the effect is the same) that shares a hive to control cluster resources.
In the case of a failure or initiating of a transfer the active drive becomes passive on the original host and it transfers over to the target machine

The problem

The problem is that that Microsoft’s NTFS file system does not handle multiple server access to the same volume.
What this means is that when you want to “move” a virtual between two hosts you have to fail the ENTIRE volume that the virtual hard disk and configuration files reside on over to the new host AT THE SAME TIME.
The only practical way in this sort of setup if you want to have a fraction of the functionality available to vmware is to setup each virtual on its own separate volume.
Personally this is ridiculous and unfeasible solution to the shortcomings of NTFS. It also causes problems with many of the large players in ISCSI SANS, for example our equallogic arrays currently (in version 4.05 of their firmware) only allow a max of 512 simultaneous ISCSI connections to a pool, assuming that we could today convert all of our 200 ESX virtuals over to hyper-v on the same host density we would be talking roughly 4800 ISCSI connections with MPIO! Or 9.4 times the maximum we can have in a pool (even if we broke our arrays into separate pools we will still be at 3 times the maximum possible)

Conclusion

While too many hyper-v is very attractive due from a cost perspective (especially if using SPLA) this combined with many other flaws in the product put it years behind ESX from a feature/performance/reliability and manageability standpoint … hopefully I will touch on some of the other issues a bit later

Updating the CEMI on equallogic PS5000 series

2009-03-27T14:45:00.002-05:00

Here is a rather interesting problem that I ran across the last 2 weeks in the process of replacing a control module
Note most of the steps provided here it is recommended that you contact your storage service provider, some of these are clearly not recommended by equallogic
On to the problem…
After swapping control module the array pages out that the primary and the secondary control module CEMI version is different. In my case the CEMI of the replaced control module is 6.08 when it should on a 6.11 with the PS5000xv at this point in time
1) Connect to the control module you want to update with the serial cable
2) Log into bash shell with the command “su exec /bin/bash”
3) You can check the CEMI version with the command “cemi_update –v” (sometimes it will show up as all 0000 this is not unexpected I will cover another way to check later)
4) Check that the file cemi611.bin exists in the folder /pss/primary/
5) Run the command “cemi_update –f /pss/primary/cemi0611.bin”
The update can take awhile and it may take even longer for it to show up (30 + min) … the second way to check the version is at the bash prompt type in “ecli” it will bring you to a new prompt. Then type the command “cmv” that will show you a bunch of info including the CEMI version

Note that the ecli screen has a bunch of environmental system information and settings.
Now in some cases (like mine) cmv did not show the actual version, it was recommended to me to reseat the control module to verify the update happened. (It did)
Also note that this should not be performed on the primary controller as there is possible downtime associated

ESX 3.5 on Dell T605 Revisited

2009-02-24T18:11:00.002-06:00

I published an earlier post about ESX on a Dell T605

It was a short post but I wanted to update it as the round of patches January 2009 fixed the SATA CD problem. Vmware has not published these to rollup therefor not CD but if you install from another source and patch you should be able to use the SATA CD rom drive and not have any problems that I have seen

Not sure if this has made it to the "Supported" list but can be benificial