Making data extraction less like tooth extraction

Making data extraction less like tooth extraction

As a company, we have a lot of experience with various virtualization technologies and pride ourselves on our innovative thinking, proactive approach, and ability to collect and analyse our client’s data in our software-as-a-service product. Our Capacity Planner product is very powerful and intuitive. Getting data from a client’s virtual estate has in the past sometimes been a challenge for some of our clients and something we have strived to change and make less painful and more efficient.

How it was previously done

Previously, we recommended using Powershell to extract data from vCenter as it allows lots of fine detailed tweaking but required the user to go through a steep learning curve to implement a successful integration between their virtualization estate and Sumerian. A lot of users are like ourselves and don’t mind getting their hands dirty. But we also wanted to be able to allow users to install a no hassle simple tool to both try our product out and use it in the future to seamlessly do anything to provide regular data to our SaaS Tool.

New extraction method

The Sumerian Data Collector has been developed in house by our development team which I have been a member of for a number of years. It has been designed to be straight forward to use by an Infrastructure manager whose responsibilities involve managing a virtualized estate.

The Data Collector has been written using C# running on the .NET framework. Currently, we only support running the tool from Microsoft Windows but in the future we may support other platforms such as Mac and Linux with a future release.

The product can be easily installed using the MSI installer. Just a few clicks and it is installed. You have to register for a Capacity Planner account first and enter your username and password when you first run the data collector.

Data that is extracted from a virtualization management server such as vSphere is securely sent to Sumerian over https and uses a strict token based authentication scheme. An extraction can take anywhere from a few minutes to several hours, this is dependent on factors such as the size of your estate and network speeds. Extracted data is first stored locally by the data collector in compressed form then removed once sent to Sumerian where it is securely processed.

The tool has been developed with a plugin architecture that allows easy future development for other virtualisation systems. We have written various custom based extractions using this technology and in the future it may be possibly for a client to tailor their own extraction themselves if required.

We chose to not run any services on our clients systems such as a permanently running agent. The reason for this is we are all aware that our clients are interested in using our product to either grow or contract their estate based on performance and future predicted consumption. We didn’t want to impact this by having running agents. We generally recommend an extraction every 24 hours and this is run as a scheduled task which our tool initialises.

Internally, our products are rigorously tested before every release using a combination of unit tests, integration tests, regression tests and user acceptance tests. Our high level of testing allows us to increase our development velocity, focus more on feature development and have a very stable product.

Tim Stanford
Senior Developer

By | 2017-11-16T14:21:27+00:00 November 16th, 2017|