Skip navigation

Path to Laguna Beach.jpg

 

During my recent Southern California road trip, I marveled at how easy it was to avoid the usual LA traffic bottlenecks and find the quickest way to the beach using Google Maps on my mobile phone.  This was a life saver especially with two restless kids in the backseat.  Unfortunately, we don’t have these type of conveniences to help steer us in the right direction when trying to find or get around bottlenecks in the data center.  However using the proper analytics tools, that Gartner now refers to as Artificial Intelligence IT Operations (AI Ops), can certainly help to direct us onto the right data center path.

 

This is key when trying to optimize IT resources or quickly troubleshoot data center problems, which at times can feel like trying to find a needle in the haystack.  Often it's the first question that is the hardest to answer, “Where do I start?”.

 

A call might come in from the application owner, "Oracle is running slow today and its impacting my users", or multiple alerts might be going off on your monitoring dashboard.  But Oracle isn’t deployed alone.  It runs on top of the multi-vendor IT resources in the data center, including servers, hypervisors, storage, etc, that makes finding the right path to resolve a problem even more complicated.

 

To help with this analytics topic, I asked my colleague, Ojay Bahra, Global Product Manager for Hitachi Vantara, for some insights.  Ojay has extensive experience with Hitachi analytics and storage products spanning 16 years.  He has a wealth of knowledge on how analytics can direct you on the right path to optimize data center performance and diagnose IT resource problems.

 

digital pic.png

 

Richard – Hello Ojay.  What are some basic analytic tips one should follow when trying to properly monitor their data center environment?

 

Ojay – A very good question, Richard.  To properly monitor data centers today, it is vital to start by reviewing and examining some key data performance indicators such at utilization, response time and throughput for key resources like servers, virtual machines or storage systems.

 

Richard – OK, how does one know whether they are having any performance issues or not?

 

Ojay Today’s modern IT monitoring tools provide some form of base measurement point we call a threshold.  These can be applied to different resources along the data path from your host server or VM to shared storage resource.  Either you would know what values to set the threshold or perhaps your monitoring tool can tell you what the average values are.  Although the threshold doesn’t point out if you have an issue per se, it can give you an indication if a resource has hit a particular high value compared to its normal operations.  In other words, this can help to establish a  performance baseline for you to set appropriate thresholds to measure against.  This in turn could be used to trigger an alert when an anomaly is detected or a high value was observed that exceeds its normal operating range.  These threshold indicators generally can be set like a traffic light system, green is good, amber is something of concern, while red indicates you have a problem. 

 

For example, Hitachi Vantara offers a complete IT analytics solution, Hitachi Infrastructure Analytics Advisor (HIAA), providing end-to-end data center monitoring, analysis and troubleshooting.  HIAA includes a centralized dashboard (see below) with a similar traffic light monitoring system as discussed so you can easily view the status of your data center environment at a glance.

 

HIAA Dashboard.png

 

Richard - If you suspect there is an issue with a red status indicator, how do you verify you have a problem and what should you do about it?

 

Ojay - When trying to manage any performance problem, you have to narrow down the troublespot because there may be expectations or possible finger pointing that the problem is not in a particular area (application, server, storage, etc).  In other words, often times you need to follow a process of elimination.  We need to start by using analytics to give us a high level overview across our data center operations.  For example when an application is reported as performing slowly, we need to determine all the associated IT resources (server, virtual machine, network ports, storage resources, etc.) tied to that application server.  With an analytic tool like HIAA, it is very easy to check the full I/O data path end-to-end from the application server to the shared storage resources.  We can see the host server the application resides on, which SAN switch ports are in its I/O path, then show which volumes are being accessed on the storage system.

 

Richard – Once I can see the full data path for the application server, what should I be looking for?

 

OjayIf we are still unclear where the issue is, we need to look in-depth at each component on the data path from host server to storage system.  For example, first check to see if the storage system performance is OK.  By using the available health and alert mechanisms on HIAA, it’s easy to examine the storage system’s health and see if any key performance values have exceeded their assigned thresholds.  If the storage array does not appear to be a concern, then we should apply a deeper dive analysis against the host server and SAN switch components, checking threshold values applied to these various datapoints.  What we are trying to do is present a more accurate view on all points of the data path to eliminate areas and shorten the analysis time.

 

E2E View.png

 

Richard – I can see using the above view, the SAN switch looks OK, but my hosts/virtual machines (VMs) are red with alerts as are the storage systems.  What does this indicate?

 

Ojay – So immediately, we have identified that we can eliminate the switch as an area of concern.  We should isolate the red indicators from the rest of the IT view and concentrate on looking at the information provided for both the Hosts/VMs and the storage systems.

 

Richard – How should you do a deep dive examination on these resources (Hosts/VMs, storage systems), excluding the SAN switch of course?

 

Ojay – Because HIAA provides different views for analysis, one can look at each particular resource area by isolating the views.  We can verify any suspected bottleneck by reviewing each dependent resource further down the data path.  We can take any resource which have been flagged red and compare its performance against similar resources.  It is this resource comparison where we can further identify anomalies and isolate the problem.

 

Verify Bottleneck.png

 

RichardCan HIAA help further to determine the root cause of the problem and provide suggestions on how it could be fixed?

 

Ojay –  Yes, HIAA offers multiple troubleshooting aids for analyzing a bottleneck resource and identifying its root cause.  As we discussed earlier, HIAA provides various analytic views and performance charts to properly analyze the suspected bottleneck and all of its dependent resources.

 

Analyze Bottleneck.png

 

Often times the problem is usually related to resource contention also known as the noisy neighbor problem.  This is where a particular resource may disrupt the balance of usage for a shared resource like a switch or storage port.  Another common cause of recent performance impacts may be a recent configuration change that you are not aware of.  With these common scenarios, HIAA can help you verify, diagnose and determine the root cause of the problem while giving you suggested changes to correct the problem.

 

In addition, HIAA is integrated with Hitachi Automation Director (HAD) for data center management automation and orchestration to streamline any required configuration changes.  For example if HIAA suggest moving this noisy neighbor host to another (less congested) shared storage port, you could have HIAA initiate the appropriate service template in HAD to automate this host configuration change to resolve the problem.  This close integration of analytics and management automation can greatly accelerate an quick fix to similar problems moving forward.

 

Richard - Thanks Ojay for your great insights as this example shows that finding and troubleshooting data center problems can be a lot easier using a tool like HIAA.  To recap, these easy troubleshooting steps to follow in the data center are:

 

  • Monitor IT dashboards with thresholds for new alerts and anomalies.
  • Isolate trouble spots by reviewing each data path resource from host to storage and all their dependent resources.
  • Examine suspected problem resource(s) while comparing their performance against similar resources.
  • Leverage built-in tool diagnostics to determine the root cause of the problem and obtain suggested fixes.  Plus utilize any integrated automation tools to quickly implement any problem fixes.

 

Even without interactive maps to help you avoid bottlenecks in the data center, using the right analytic tools can certainly help you get started and give you the necessary insights to find the right data center path.

No one wants to be marooned on an IT management island.  Management tools need to be extensible so you can integrate ongoing administrative operations into your existing IT management environment.  I wanted to introduce the Hitachi’s Configuration Manager REST API that provides a common, REST-based application programmer’s interface (API) for creating programmable services to connect and interact with Hitachi storage platforms and software.  This can help you build an interactive bridge with your existing IT environment by leveraging standard API commands that can be integrated into management programs, interfaces or scripts.

 

56676f053279a.image.jpeg

 

To help readers learn more about what they can build and extend using the Configuration Manager REST API, I’ve wanted to share a recent conversation I had with my colleague, Craig Chan, Hitachi Data Systems Director of Product Management, who manages our REST API efforts.

 

Richard:  Craig, what is the best way to manage Hitachi storage platforms if users want to use their own programmatic interfaces?

 

Craig:  Using the Configuration Manager REST API is the easiest and most complete way to programmatically configure and manage Hitachi platforms.  It is as simple as incorporating REST API commands directly in your existing management scripts.  With this API, you can provision storage, get an inventory of your resources or even automate the setup of your Hitachi storage system.  For system setup, it covers a wide range of administrative operations such as creating and managing RAID groups, storage pools, logical storage volumes or configuring system settings such as date/time, user passwords, licenses, email notifications through SMTP or SNMP traps bound to specific communities.

 

Richard:  What are the advantages of using the API compared with other Hitachi programming interfaces, such as the Command Line Interface (CLI)?

 

Craig:  The Configuration Manager REST API is easier to use compared to the CLI because it uses standard HTTP commands (GET, PUT, POST, DELETE).  Plus, the API uses a new API architecture that provides faster operations.  You can get quicker system responses so the API operations can be up to 20x faster than using the standard CLI.

 

Richard:  What is the architecture of the Configuration Manager REST API?

 

Craig:  Here is a graphical representation of the REST API architecture.  It sits across an extensive set of micro services that span common management functions to system operations.  You can utilize the API to access and integrate any of these services into your management programs or scripts.

 

Configuration Manager REST API Image.png

 

Richard:  What else about the REST API is better?

 

Craig:  Well, we have heard from customers that they see much better performance for operating our storage systems  through the in-band (via Fibre Channel connection) interface.  We took a look at that interface and built the same interface out-of-band (via TCP connection) so you get the enhance security of encryption and avoid using the API over your Host I/O path.  The operations are almost as fast as in-band and you’ll barely notice the difference.  Then we took a look at all the information customers are clamoring for and built some custom views to deliver relationships you could not see before in our storage arrays.  These correlations make it much easier to code to and developers will be thrilled with being able to issue just one command to get their information instead of 3 or 4.  Best of all, no more refreshing.

 

Richard:  Wait! You mean the dreaded refresh button is gone?

 

Craig:  Correct. We figured out a new mechanism to automatically keep the Configuration Manager REST API’s database up to date without the need for a manual refresh.  We call this a notification service and we currently use this service to check for changes on the storage array.  Then we send out notifications to the REST API that a specific object has changed and you need to update that object’s information.

 

Richard:  Sounds great!  What are other useful management operations that the REST API could help to enable?

 

Craig:  One thing that comes to mind is the use of storage resource labeling or logical device (LDEV) labeling.  We are scripting or coding machine languages, but us humans need to understand it too.  With the REST API, you can segment and label all the storage resource LDEVs for a particular application such as for Oracle database files versus Oracle log files.  With properly labeled Oracle LDEVs, it is a lot easier to know the purpose of an Oracle application volume and create the appropriate administrative scripts or integrate it into your management orchestration tool of choice.

 

Richard:  How does someone get started using the Configuration Manager REST API?

 

Craig:  The REST API comes with the Hitachi Storage Virtualization Operating System that is bundled with each Hitachi VSP F or Hitachi VSP G storage system.  The API can be downloaded directly from the REST API Developer Site on the Hitachi Data Systems Community or from your very own login to the HDS Customer Support Connect web site.  Once you have it installed on a server, it’s easy to get started by discovering your first Hitachi storage array.  If you need more detailed API information, the REST API documentation is available on the HDS Customer Support Connect portal or you can visit our REST API Developer Site on the HDS Community.

 

Use the Hitachi Configuration Manager REST API to easily cross that divide when integrating Hitachi storage platform management into your existing IT environment.

Storage provisioning has always been one of the more complicated management tasks that administrators deal with on a regular basis.  Requests for new storage capacity from application owners are often varied requiring individual analysis and planning.  Depending on the visibility into the current storage utilization of your environment, it can be difficult to determine where is the most optimal place to deliver on the next capacity request, which often leads to over provisioning and lower storage utilization rates.  Infrastructure self-service promises easy deployment and accessibility of infrastructure resources for new business application needs.  However, can you achieve this goal within your infrastructure if you are still using manual storage provisioning processes? 

 

Hitachi's new Automation Director software application delivers new capabilities that can automate repeatable, manual storage provisioning operations saving both time and costs.  It addresses two key provisioning challenges of dealing with varied storage requests for business applications and ensuring the storage capacity fulfillment process is optimized and properly aligned with application based service requirements.

 

So how does this work?  For storage requests, Automation Director provides new service templates for storage capacity requests based on existing application provisioning best practices.  For example when allocating tiered storage resources for a new Oracle database, a standardized process could be established where the fastest Tier 1 storage would be allocated for the core Oracle database while slower, less expensive Tier 2 storage would be allocated for the database logs or temporary files.  By incorporating storage provisioning templates for Oracle, Microsoft Exchange, Microsoft SQL Server, etc., Automation Director can help you to establish a predefined catalog of service  templates to streamline and standardize storage capacity requests for common business applications.

 

Of course each application deployment might have specific needs, so you could modify these templates using Automation Director's service builder feature to suit the particular needs of your environment.  For example, you could change the defined storage service level for a particular application component or include an additional function like send an email when the scheduled provisioning task is completed.  To extend use of these templates and further reduce risks, you can restrict entry for certain fields of a service template to enable their self-service use by a wider audience.

 

For storage capacity fulfillment, Automation Director provides infrastructure abstraction to quickly discover and categorize existing virtualized storage resources into service defined storage tiers, i.e. platinum, gold, silver, etc.  This helps to fully utilize tiered storage resources that can be properly aligned with application specific storage capacity and performance requirements.  When new application capacity requests are made, Automation Director automatically provides intelligent choices to ensure placement on storage volumes with the appropriate amount of space and service level available.

 

By utilizing these automated provisioning capabilities, you can eliminate potential errors from manual configuration and provisioning operations while improving administrative productivity and storage service reliability.  By enabling storage infrastructure self-service with automated provisioning, this helps to establish one of the key automated elements of Hitachi's vision for a software-defined infrastructure. 

Richard Jew

Doing More With Less

Posted by Richard Jew May 6, 2014

Doing More with Less

 

As IT budgets have become constrained or stagnant recently,  IT organizations are constantly being asked to “do more with less”.  However how do you apply this principle in practice particularly when it comes to managing storage and dealing with unabated storage growth.  As more storage resources have become virtualized within the datacenter, IT organizations are running into similar management challenges as when they expanded their virtual server farms.  In virtualized storage and server environments, they struggle with limited visibility of actual infrastructure usage, unsure whether storage is meeting their required service level agreements and a continued rise in management costs.

 

Doing more with less typically involves trying to get more usage out of existing IT infrastructure assets or developing new management automation.  However, perhaps the most significant impact to achieving “doing more with less” may be enabling your IT staff to become more efficient in virtualized storage management practices, which can significantly lower ongoing operational costs. The recent release of Hitachi Command Suite v8 (HCS v8) can help users to address these “doing more with less” challenges to better utilize and manage virtualized storage assets with existing IT staff resources.

 

HCS v8 is fully integrated with the new Hitachi Storage Virtualization Operating System and Hitachi Virtual Storage Platform G1000.  As an integrated management framework, HCS v8 enables users to fully leverage and deploy Hitachi’s global storage virtualization into their environment.  Global storage virtualization extends virtualization across physical and virtualized storage platforms enabling new levels of pooling, abstraction and mobilization of storage resources in the form of virtual storage machines.  Virtual storage machines can eliminate isolated data silos with spanned virtualized volumes that provide a complete separation between host and storage, independent of storage system, location or vendor.  Using HCS v8, you can easily create new virtual storage machines and define its physical storage resources, such as disk, cache and ports, from a common storage resource pool.  This enables you to manage large and complex virtualized storage infrastructures with greater efficiency and less risk.

 

In addition, all core configuration operations have been consolidated under the Hitachi Command Suite v8 management console to reduce complexity.  A common graphical user interface can be used for all management tasks to further simplify administration.  HCS v8 unified management capabilities has been enhanced as Hitachi Tuning Manager v8 add new Hitachi NAS Platform file storage monitoring functionality.  Tuning Manager can now simplify performance analysis and troubleshooting across an unified storage environment, covering both  structured and unstructured data, that improves performance and availability for both business applications and file servers.

 

Ensuring storage performance and capacity service levels for mission critical business applications has always been a hallmark of the Command Suite.  Hitachi Command Director v8 provides new aids to help users to quickly define storage service level objectives for business applications.  In addition to predefined storage service level profiles for common business applications, such as Oracle databases and Microsoft Exchange, application storage performance can be monitored to form a historical performance baseline.  Service level objective recommendations can be then be applied and customized for an application thus speeding up the establishment of application based storage service level agreements.

 

HCS v8 also introduces a new standardized interface to facilitate data sharing with external management reporting systems.  A REST based API leverages standard HTTP commands and it is language independent.  Users can easily query storage performance and capacity statistics from the Hitachi Command Suite to be integrated into existing management data repository and reporting systems.

 

By utilizing some of these Hitachi Command Suite v8 capabilities to develop new storage administrative and operational efficiencies, you can rein in operational costs and move a big step forward in achieving your "doing more with less" goal.