I have had a couple people remark to me that it is somewhat difficult to use iostat on XenServer to gather storage performance metrics because the output format is not easily parsed. Today’s blog is about how to collect and import storage performance values from XenServer using the iostat command and a PowerShell script I wrote to make this easy.

 getting the performance counters

The first step is to configure the XenServer hosting the virtual machines to output the performance metrics. The Linux iostat command provides key storage related performance counters for the XenServer host. The easiest way to get these counters is to shell out from the XenServer console and run a command similar to the following:

iostat -x 15 5760 >iostats.txt 2>/dev/null &

Here is a breakdown of the individual parameters for this command so you can modify them for your environment:

iostat -x

15

5760

>iostats.txt

2>/dev/null

&

If you want to know more about the iostat options, just Google “iostat command parameters” and read up on the man pages which coincidentally explain all the value calculations too. Here is a sample of the iostat output from the command above:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.27    0.00    0.43    0.00    0.28   99.02

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0        0.00     1.74  0.20 14.36     1.80    78.73     5.53     0.00    0.17   0.14   0.20
cciss/c0d0p1 
               0.00     1.74  0.00  0.89     0.01    21.01    23.65     0.00    0.45   0.32   0.03
cciss/c0d0p2 
               0.00     0.00  0.00  0.00     0.00     0.00     8.40     0.00    0.59   0.59   0.00
cciss/c0d0p3 
               0.00     0.00  0.20 13.47     1.79    57.72     4.35     0.00    0.16   0.12   0.17
dm-1              0.00     0.00  0.00  0.00     0.01     0.00     7.80     0.00    0.48   0.48   0.00
dm-2              0.00     0.00  0.00  0.24     0.03     4.90    20.08     0.00    0.15   0.15   0.00
dm-5              0.00     0.00  0.00  0.23     0.05     4.69    20.20     0.00    0.13   0.13   0.00
dm-3              0.00     0.00  0.00  0.14     0.03     2.81    19.36     0.00    0.12   0.11   0.00
dm-0              0.00     0.00  0.00  0.05     0.03     1.06    20.40     0.00    0.14   0.14   0.00
dm-4              0.00     0.00  0.00  0.05     0.02     1.05    20.03     0.00    0.14   0.13   0.00
dm-6              0.00     0.00  0.00  0.05     0.03     1.09    20.35     0.00    0.15   0.15   0.00

Note that the statistical information for Drive0 is found on the same line as its devide name c0d0; however, the information for the drive partitions is found on the line after the device name c0d0p1. Similarly, the actual metrics for the avg-cpu are on the line after the declaration and header line. Since both those lines have nothing but numbers and spaces on them, they cannot be filtered or distinguished between when looking only at the line itself. The trick to making this file useful, is to be able to parse it first into a useful format.

parsing the logfile with powershell

I tend to use PowerShell to manipulate text files where possible, so it was my first choice for parsing the iostat log file. Since the values we need are after the line that identifies them, the only way to parse it is to read each line and then make a decision based on the contents. (For large logfiles, this could take a long time, so plan to run it over your lunch hour or something.) For the drive partition lines, I just needed to combine the two lines. For the CPU metrics, the best approach was to just export them to another file at the same time I was writing the new one. In the end, the script generates three files, one for the CPU metrics, one for the disk performance metrics, and one that is exactly like the original, but has the drive partition information on the same line. The two files that hold the CPU and disk performance metrics are space-delimited and can be imported into Excel for analysis.

Here is the script in its entirety. All the variables are configured at the beginning in the section labeled #Environment Variable Configuration rather than as parameters like my other scripts.

################################################################################
# Author:  Paul Wilson
# Updated: 28 Feb 2011
# Purpose: This script parses an IOSTAT results file and outputs three different
#          files. The $ExportAllFile is the target file that will include the
#          parsed results that removes the carriage returns for disk partitions.
#          The $ExportCPUFile will contain just the CPU performance data. The
#          $ExportDiskFile will contain the performance data for the selected
#          disks or partitions as defined by $DiskPartition.
################################################################################

# Environment Variable Configuration
# This section specifies the location of the input and output files and the disk/partition pattern match 

# $InFile = The location of the IOSTATS file to be parsed.
# $ExportAllFile = The path and filename to the place where the parsed file will be created.
# $ExportCPUFile = The path and filename to place the CPU performance data.
# $ExportDiskFile = The path and filename to place the disk performance data.
# $DiskPartition = The disk line or pattern you want to extract. For just the Disk0 use "cciss/c0d0 "
#                  for all the partitions on Disk0 use "cciss/c0d0". For just Partition1 use "cciss/c0d0p1"

$InFile="C:\iostats.txt"
$ExportAllFile="C:\exportall.txt"
$ExportCPUFile="C:\cpuperf.txt"
$ExportDiskFile = "C:\diskperf.txt"
$DiskPartition = "cciss/c0d0p1"

# This line adds the header info for the CPU file
add-content $ExportCPUFile "avg-cpu:  %user   %nice %system %iowait  %steal   %idle"

# The IOSTAT output format makes filtering impossible for the disk partition stats and the CPU stats because they both appear on the
# next line of the file. This section of code processes the file one line at a time and fixes both issues. If the previous line
# was the CPU line, then the line is exported to the CPU file and also to the new file. If the previous line was a disk partition line
# (ie c0d0p1), then it is combined with the current line and written to the new file.

$NextLineFlag = $false
$CombineLineFlag = $false

foreach ($myline in get-content($InFile))
{
   $newline = $myline
   if ($NextLineFlag)
   {
     add-content $ExportCPUFile $myline
   }

   if ($CombineLineFlag)
   {
     $newline = $str1 + " " + $myline
   }
   switch -wildcard ($myline)
   {
      "*avg-cpu*" {$NextLineFlag = $true; add-content $ExportAllFile $newline}
      "*c?d?p*" {$CombineLineFlag = $true; $str1=$myline}
      default {$NextLineFlag = $false; $CombineLineFlag = $false; add-content $ExportAllFile $newline}
   }

}

# This section of code writes out the disk performance file by first writing the header line then making a single pass through
# the new file looking for the DiskPartition string pattern specified earlier. Since the partition data in the new file resides
# on a single line, the filter works for both disk and partition patterns. Adding more export files for other partitions or
# drives can be done here by copying these three lines and changing the match pattern.

add-content $ExportDiskFile "Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util"
$LineItem = Select-string -pattern $DiskPartition -path $ExportAllFile
add-content $ExportDiskFile $LineItem 

# Completed
write-output "Done!"

Once the script is complete, you will have two output files, which can then be easily imported into Excel. 
The CPU file (cpuperf.txt) looks something like this: 
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.27    0.00    0.43    0.00    0.28   99.02
           0.73    0.00    4.00    0.47    1.47   93.34
           0.60    0.00    3.07    0.00    1.33   95.00
           0.47    0.00    3.40    0.00    1.60   94.54
 
While the disk performance file (diskperf.txt) file looks something like this:
Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
C:\exportall.txt:8:cciss/c0d0p1                 0.00     1.74  0.00  0.89     0.01    21.01    23.65     0.00    0.45   0.32   0.03
C:\exportall.txt:24:cciss/c0d0p1                 0.27     5.67  0.67  2.87    13.87    68.27    23.25     0.01    1.70   1.51   0.53
C:\exportall.txt:40:cciss/c0d0p1                 0.00     0.67  0.00  0.53     0.00     9.60    18.00     0.00    0.00   0.00   0.00
C:\exportall.txt:56:cciss/c0d0p1                 0.00     2.67  0.00  1.00     0.00    29.33    29.33     0.00    0.00   0.00   0.00

importing the files into excel

Importing the files into Microsoft Excel is a fairly easy process. I tend to use the data text import functionality to add each set of performance metrics to the same spreadsheet, but on different tabs. To import the data from the text files, follow these steps:

1. Open Excel and select the Data tab.

2. Choose the “From Text” option under “Get External Data”

3. Select the “Import Data from a Text File” option from the context menu.

4. In the Text Import Wizard, select the “Delimited” radio button and click Next.

5. Untick the “Tab” delimiter checkbox and instead tick the “Space” delimiter checkbox and then click Finish.

At that point you can save the data as a new Excel sheet and you are ready to begin the analysis of your XenServer data. If you have comments or questions, please feel free to add them below. If you found this article useful and would like to be notified of future blogs, please follow me on Twitter @pwilson98.