http://blog.ylastic.com/

Updating Spending Analytics

Mon, 01 Apr 2013 09:41:00 -0400

The spending analytics in Ylastic are currently built by downloading and processing the raw CSV data from the AWS Usage activity page. Even though it works quite well, it is a rather tedious process. You can use IAM user credentials to download the data, but we want to reduce this to something even simpler. We are experimenting with the programmatic billing API access that AWS added which gives each AWS account the ability to request that billing csv files be dumped in the S3 bucket of your choice. This makes it a lot easier to both download and process the data and keeping up with the changes that AWS makes to the pricing. It also makes it a breeze to add new services. We are going to roll this out to replace the current implementation within a few weeks. Here are a few sample screenshots of a page built from this data.

AWS provides detailed documentation on how to enable each AWS account for programmatic billing access here. We will have more updates on the blog with all of the charts we will have soon.

DynamoDB Management

Thu, 21 Mar 2013 08:26:00 -0400

Ylastic now supports management for DynamoDB. Manage all your tables in single or multiple regions on the same page. The read and write capacity units consumed in the last 20 minutes are displayed in spark line graphs right next to each dynamo table in the listing, giving you a quick overview of your DynamoDB environment.

Explore the items in your tables with the built-in viewer.

An audit trail is maintained to display the history of changes made to your DynamoDB environment from Ylastic, including the name of the user making the change as well as the IP address from which the change was made.

View Cloudwatch charts for each table by clicking either of the sparkline graphs. You can change time periods, dimensions, etc and refresh to view all of the cloudwatch data for the table.

We are in the process of hooking DynamoDb into our monitoring. Next release will have the ability to monitor and alert via email, voice, SMS :)

Selecting Snapshots for Backup

Mon, 18 Mar 2013 14:11:00 -0400

It can be a bit daunting when you have to backup snapshots or to use the AWS parlance - copy snapshots to other AWS regions, especially when you have a lot of them. We have been grappling with this issue and how to make it easier and simpler for our users to pick the snapshots they want to copy to other regions on a schedule for disaster recovery purposes. You can now select the snapshots to copy in the Ylastic scheduled task in four different ways:

Select all snapshots in the source region - every single one of them.

Select all snapshots whose tag value contains specified string value.

Select all snapshots whose tag name contains a specified string value.

Select all snapshots whose tag name contains a specified string, as well as the tag value contains a specified string.

This scheme lets you select snapshots for backup from a big net that captures everything to something really fine-grained that can only pick up a single select snapshot. Manage your backups in AWS the easy way!

Copying Snapshots between regions on a schedule

Sat, 16 Mar 2013 14:38:00 -0400

You can now schedule a task in ylastic to copy snapshots of your choice to one or multiple regions on a schedule of your choice.

Select a source region, and specify the strings to match for a tag name and tag value and the task will select all snapshots in the source region that meet those criteria. You can also specify exactly how many copies of a snapshot you want in each of the regions. All previous backups are removed leaving only the latest number of backups that you want in each region.

Here is a list of the backups being created with the above task.

All tags from the source snapshot are preserved and added to each of the new snapshots.

Backups the easy, easy way :)

Hello Sydney

Mon, 17 Dec 2012 12:07:00 -0500

Management and monitoring support for the new AWS region in Sydney, along with migration of AMIs, scheduling and more.

Windows AMI migration for Sydney on the way :)

Autoscaling in Amazon Virtual Private Cloud

Thu, 31 May 2012 11:55:00 -0400

Ylastic can now configure autoscaling for VPCs. You can use all of the scaling goodies such as policies, scheduling actions, recurring actions, etc for your instances running inside a VPC.

Create a launch configuration that uses the security groups of your choice in your VPC.

Create an autoscaling group with the above launch configuration and specify the VPC subnet to use.

Kick back and watch auto scaling take over :-)

View all the changes being made to your VPC autoscaling group in its audit trail.

Change the scaling group to use a different subnet in your VPC?

Switch over to the VPCs page if you like and view all the instances (both normal and auto-scaled) that are currently inside your VPC along with their CPU util and all the other cloudwatch metric charts.

Manage your AWS cloud, the easy way :-)

Route53 Latency based routing

Sat, 14 Apr 2012 11:24:00 -0400

AWS recently enhanced Route53 with the ability to do latency based routing, which serves user requests from the EC2 region for which network latency is lowest. You create a latency resource record, and when Route53 receives a query for the domain, it will select the resource record for the EC2 region that will have the lowest latency for the requesting user. It really is as simple as that. Ylastic now supports managing these latency records. In the example below, we have a load balancer in US East(Virginia) region and one in the US West (California) region.

You can create a latency record for all seven AWS regions if you like from one screen. How does AWS figure out the latencies in order to make the routing decisions? AWS apparently gathers these latency measurements between most /24 subnets on the internet and the different AWS regions in order to create the dataset that is the basis of latency-based routing. The technology underpinning this is also used by CloudFront, the AWS CDN product.

You can also use IP addresses/Elastic IPs instead of ELBs for creating these records. This is a really nice addition to Route53, and much requested by the users. Enjoy :-)

Scheduling the AWS Account Advisor

Wed, 14 Mar 2012 12:30:00 -0400

You can now schedule the AWS Account advisor in Ylastic to run on a time period of your choice. So you can set it up to run checks against your account, say once a week and alert you via email if any flags are raised by the check. How easy is it to setup?

You will get an email if there are any warnings from the advisor run.

Simplify your AWS cloud management!

Simple Backups for EBS Instances

Fri, 09 Mar 2012 15:00:00 -0500

At Ylastic, we have been looking at backup management and ways to make it easier and simpler to both create the backups, as well as manage them easily without getting lost in looking through tons of AMIs. Introducing simplified EBS instance backup management - Select an instance, view all of its backups, launch new instances from any of the backups, and even schedule backups to happen on a time period of your choice.

Backups can be created in two different ways - on-demand, by clicking a button and filling out a few fields.

You can automate backups by setting up a scheduled task to create them on a time schedule that you want. Automate backups for multiple instances with a single task by specifyng a string to match in the name tag for instances. Ylastic will also save you storage costs by ensuring that only the specified number of latest backups are kept, and prune the older backups. The task shown below will backup all EBS instances in Virginia whose name contains Lorax at 1:00 AM everyday, and keep only the latest ten backups.

Ebs instance backup management is a feature available in the Ylastic Plus version. AWS cloud management made easier :-)

Simplifying CloudFormation Stack Management

Thu, 23 Feb 2012 12:12:00 -0500

Ylastic just released several enhancements for CloudFormation stack management. Our initial implementation of the UI was done when CloudFormation was first released, and it is now a rather large and complex service that enables you to reference and use resources from a lot of different services in the AWS stable. The increase in complexity leads to a quick proliferation of the number of resources that comprise your stack, and correspondingly needs a simpler way to get your head around what is in your stack. You launch this wonderful stack that creates instances, databases, Route53 resource records, autoscaling groups and so on. How do you actually view all of these resources that are part of your stack? No, we do not want to view a rather large incomprehensible table that lists just the physical ids for resources, as it is completely worthless in terms of getting any work done. And no, we do not mean navigating to 20 different pages to view them. We mean a single place to view your resources and additional information for each resource. Here we go ..

All the resources in a stack are displayed on this page, separated by the service that they are part of. Additional and meaningful information for each resource is also presented. For example, if you are viewing the instances that comprise your stack, you can view other info for the instance such as AMI id, zones, uptime, cloudwatch data, etc.

There is also an easier way to view the JSON template definition associated with the stack. We added a nicely formatted representation of your stack template. You can even expand and collapse various sections in the template as you like.

Stacks have a cost associated with them like most resources in AWS. We like knowing the money that we are spending on AWS, and knowing the expenses that are being incurred for a stack is very, very useful. We now display the estimated costs for each stack computed for two different time periods:

Estimated month to date cost for running the stack.
Estimated cost to run the stack for the whole year.

Finally, another thing that we use a lot when using stacks - the ability to view cloudwatch charts for a resource. Each resource such as instance, elb, volumes, etc will display a little sparkline graph for the CPU util or similar metric for the last 20 minutes. Click on the sparkline to display detailed cloudwatch charts for the selected resource.

Manage your AWS cloud the easier way :-)

AWS Account Advisor

Sun, 19 Feb 2012 13:35:00 -0500

Introducing the Ylastic AWS Account Advisor, a tool for inspecting your AWS environment and identifying opportunities for optimizing your usage of AWS.

We built it to be very simple and intuitive to use. You pick the checks you want to include in each run of the advisor (this initial release has a total of ten checks), Ylastic runs the checks and gives you a nice list of things that it found. Each advisor run is saved, and at any time you can review past runs.

The checks are broadly divided into four categories:

Cost Optimization - Opportunities for reducing costs by detecting unused volumes, elastic load balancers, elastic ip addresses and Route 53 zones. These checks will also display an estimated cost saving per month and per year from removing the unused resources.
Disaster Recovery - Check your ability to recover from system wide failures by detecting volumes that are in-use but not being backed up to snapshots. The advisor will also flag volumes that have snapshots older than several days, as that may be an indication that the backups are getting stale.
Fault Tolerance - Identifies situations that can impact your ability to recover from the failure of an EC2 availability zone, by checking if your elastic load balancers have distributed allocation of instances, as well as if you have instances distributed in more than one zone.
Security Audit - Secure access to your resources by detecting security groups that provide public access to sensitive ports or port ranges, as well as S3 buckets that can be listed by anonymous users across the internet.

As you use AWS over time, cruft builds up, and you start having unused resources in your account that are just driving up your costs. One of the cool features of the advisor is to flag these unused resources, and give you an estimate of the savings that you can get if you get rid of them. The screenshot below is from one of our customers that helped us test the advisor. Those elastic IPs, old unused volumes and balancers add up pretty quick :-)

The advisor is a feature available in the Ylastic Plus version. Coming soon, the ability to run the advisor on a schedule, as well as enhancements and additional checks based on feedback from customers that have already been trying this out.

Enjoy :-)

AWS Route53 Spending Analytics

Tue, 17 Jan 2012 15:35:00 -0500

You have all your domains nicely imported into Route53 using Ylastic. You have scheduled backups of your zones to the S3 bucket of your choice for DR using Ylastic. You do have those zones backed up, right? You can view an audit trail of all the changes/additions/updates being made to your zones in Ylastic. And now you can view the spending break-down for all of those zones in Ylastic Plus. View the spending for the current month, previous month, curent year or the last year.

Easily view the cost associated with each of the zones you are hosting in Route53. The chart also displays the total number of queries made for the displayed zones in the chosen time period.

More integrated cool tools for Route 53 in the pipeline. Manage your AWS cloud, the easy way!

Migrating Amazon Linux AMI between EC2 regions

Wed, 21 Dec 2011 13:36:00 -0500

You can now migrate Amazon Linux based AMIs between regions of your choice in Ylastic. Select your AMI, the region you want to migrate to, and that’s it.

Get an email when the migration is completed.

Launch an instance at your leisure from the new AMI and off you go.

Enjoy and happy holidays!!

EC2 Auto Scaling Management

Wed, 26 Oct 2011 09:15:00 -0400

Refreshed our auto scaling support to include all of the recent features released by AWS. Here is a quick run through of the various things that you can do with auto scaling on EC2.

Create auto scaling groups with a new easy to use wizard that also lets you set up policies and their associated alarms for scaling up/down your EC2 fleet.

View and manage all of your resources associated with the scaling groups.

Suspend scaling processes to investigate any configuration issues with your app and resume scaling processes when you are done.

Create and manage scaling policies and their associated alarms to setup the thresholds for scaling your fleet of instances.

Setup scheduled scaling actions to increase/decrease the number of instances in your fleet. You can also setup the scaling action to be recurring on a schedule of your choice.

Peruse cloud watch charts of group metrics for the scaling group of your choice.

You can still manage your auto scaling groups that are using the trigger mechanism that has been deprecated by AWS.

More goodies on the way. Enjoy :-)

Amazon EBS Snapshots in the EU-West Region

Tue, 09 Aug 2011 08:32:00 -0400

AWS has discovered a bug in their software that cleans up EBS snapshots in the EU West region. They are contacting customers that have snapshots affected by this bug. Here is the email that some of our customers are receiving:

Hello,
We’ve discovered an error in the Amazon EBS software that cleans up unused snapshots. This has affected at least one of your snapshots in the EU-West Region.During a recent run of this EBS software in the EU-West Region, one or more blocks in a number of EBS snapshots were incorrectly deleted. The root cause was a software error that caused the snapshot references to a subset of blocks to be missed during the reference counting process. This process compares the blocks scheduled for deletion to the blocks referenced in customer snapshots. As a result of the software error, the EBS snapshot management system in the EU-West Region incorrectly thought some of the blocks were no longer being used and deleted them. We’ve addressed the error in the EBS snapshot system to prevent it from recurring.
We have now disabled all of your snapshots that contain these missing blocks. You can determine which of your snapshots were affected via the AWS Management Console or the DescribeSnapshots API call. The status for any affected snapshots will be shown as “error.”We have created copies of your affected snapshots where we’ve replaced the missing blocks with empty blocks. You can create a new volume from these snapshot copies and run a recovery tool on it (e.g. a file system recovery tool like fsck); in some cases this may restore normal volume operation. These snapshots can be identified via the snapshot Description field which you can see on the AWS Management Console or via the DescribeSnapshots API call. The Description field contains “Recovery Snapshot snap-xxxx” where snap-xxx is the id of the affected snapshot. Alternately, if you have any older or more recent snapshots that were unaffected, you will be able to create a volume from those snapshots without error. For additional questions, you may open a case in our Support Center: https://aws.amazon.com/support/createCase
We apologize for any potential impact this might have on your applications.Sincerely,
AWS Developer Support

Migrating EBS instances to an AMI in a different region

Mon, 11 Jul 2011 12:33:00 -0400

You have an EBS instance running in US east region. But you want to migrate this instance to an AMI in EU, for disaster recovery, testing, whatever. And you say want to do this simply and without major contortions, preferably via a GUI. We hear you :-) Ylastic now has the ability to migrate an EBS linux instance to an AMI in a region of your choice. Pick a few options, and click a button.

And receive an email when the migration is complete :-)

Enjoy :-)

Route53 backup to S3

Sun, 19 Jun 2011 23:02:00 -0400

Ylastic can now backup all your Route53 hosted zones to S3. Each of your hosted zones will be exported to BIND zone file format and saved to the S3 bucket of your choice. Sounds complicated? How about as simple as select the zones to backup, pick a bucket and click a button.

We love simple, so we extended it to also give you the ability to schedule your Route53 backups on a recurring time period of your choice :-)

Manage your AWS environment, the easy way!

WRR balancing between EC2 regions with Route53

Wed, 01 Jun 2011 15:36:00 -0400

Route53 introduced a new feature that lets you configure DNS based balancing between EC2 regions. Yes, you heard that right. Between EC2 regions. How does this magic happen? By leveraging a new R53 concept called Weighted Alias resource records, that let you define multiple mappings between your zone apex and your ELBs, as well as assign weights to the mappings.

This works great, but do keep in mind that this is a round robin scheme and may not be suitable for all kinds of apps. This is a great start, and looking forward to more neat stuff from the route53 team. Enjoy :-)

Mapping zone apex to your ELB with Route53

Thu, 26 May 2011 12:03:00 -0400

This blog post is outdated. Route53 has since released new features and Ylastic has integrated them. Please check these posts for the updates:

AWS just released a nice feature named zone apex association for mapping the apex of your hosted zone in Route 53 with an elastic load balancer. No more jumping through various hoops and redirects to make queries for a domain such as xanthe.us resolve to your wonderful ELB in EC2. The magic all happens using the concept of Alias Resource Record Sets, which are aliases that point to DNS names in any Route 53 hosted zone. And here’s how easy it is to set it up with Ylastic.

Enjoy :-)

Weighted Round Robin DNS based load balancing with Route53

Thu, 26 May 2011 12:01:00 -0400

Route53 now gives you the ability to balance request load by configuring a DNS record to return one of multiple possible answers according to some defined weighted round robin policy :-) Here is how one would setup three A records for the sub-domain www.xanthe.us with weights of 1, 1, and 3 (sum = 5). Route 53 will then select each of the first two resource record sets one-fifth of the time, and returns the third resource record set three-fifths of the time for all queries for www.xanthe.us. And how do we set this all up in Ylastic?

Yep. As simple as that.

Enjoy :-)