Valtira Blog

On Cloud Apps

by Morgan Catlin - 27 Sep 2010, 19:02:55

We discovered and began using the cloud three years ago, which George discusses in his book, Cloud Application Architecture. As he says, Valtira took advantage of the cloud early to save money for hosting & infrastructure.

You might be wondering what we've been doing since then. We've been using it to scale out our OnDemand infrastructure, bring many client websites online and to build "cloud apps". The first two are straight-forward: hosting Valtira's applications in the cloud - not terribly exciting. The latter is what I want to focus on.

What I mean by building a cloud app is designing and implementing an application in a disconnected way, leveraging cheaper on-demand infrastructure. Amazon's cloud offers many services that I can just use, rather than have to setup, administer and maintain myself. I've built applications using EC2 (for HTML/JSP/Video Streaming), SNS (for notifcations to users), S3 & CloudFront for CDN. All of these services I could run using my own infrastructure & applications, but I don't need to anymore. All of these services exist & operate without the other, but together, provide the services users need.

My next foray will to build a cloud app that will reduce the need for instances on EC2 and focus on tying SQS, SNS and RDS together while talking to Salesforce.com.

About the author: Morgan is an application developer, focusing on rich Web-based applications. These applications solve a myriad of business problems, from managing spots on cable tv to tracking users 'click-through' rates on marketing web sites. These applications rely on database and operating system interactions, primarily Java, Ubuntu Linux and MySQL.

Comments? Suggestions? Feedback?

Secure Streaming Using EC2 and Wowza

by Morgan Catlin - 27 Oct 2010, 13:56:14

One of our clients wanted to bring their library of DVD movies to the internet. They wanted to give their customers the option to stream purchased content from home, office or wherever an internet connection exists.

They were looking at using S3 + CloudFront to deliver the streaming content and had begun the process of encoding the DVDs to MP4. I took a look at the situation and found while CloudFront does offer RTMP streaming to Flash-based players, it didn't offer the RTMPE protocol. RTMPE encrypts the video content, making it much harder to rip off.

We settled on using Wowza to stream the content since it offered several key features: RTMPE, EC2 and integration with S3. I fired up a Wowza DevPay instance and began testing. Since the bucket with the video in it was secure, I had to update the security credentials in the vods3 Wowza application. I also configured the secure token for the RTMPE streaming.

We then developed the client's application using our CMS as the framework on top of EC2, MySQL and Tomcat. The application requires users to login before streaming one of the videos they own. The actual content purchase happens on the client's website and then it calls a webservice on the application to add a video the user's library. The application uses Flowplayer with bandwidth detection and secure token to stream the video.

The client wanted to keep a close eye on abuse - e.g. is the user logged in more than once from different IPs or is the same user streaming the same video at the same time from different IPs? We setup the application to send alerts to an SNS topic so the client could subscribe both email accounts and automated systems to respond in real time.

After the application had been up for a while, the client realized that they have all these valuable assets successfully streaming to their customers. What if they wanted to use these same assets as marketing content? They couldn't just release the entire video to just anyone. I implemented a custom Wowza application that streams just parts of the videos to the browser. I call this "virtual clips" - allowing the client to configure a start and end time for a clip and streaming it out to Flowplayer based web applications. Very cool!

We have been looking into streaming to mobile devices, iOS and Android. Funny thing is that since Android devices have Flash, they can just use the current application. As you well know, iOS doesn't have Flash, so we spent some time looking into use streaming over HTTP using the cupertino and silverlight protocols. This will actually work in Wowza, but we'd have to create a custom application within Wowza to hand out encryption keys on request. We are currently designing and scoping this project. We are even looking to build apps for the latest generation TVs (think Netflix on the Wii).

About the author: Morgan is an application developer, focusing on rich Web-based applications. These applications solve a myriad of business problems, from managing spots on cable tv to tracking users 'click-through' rates on marketing web sites. These applications rely on database and operating system interactions, primarily Java, Ubuntu Linux and MySQL.

Comments? Suggestions? Feedback?

Tomcat Failover

by Morgan Catlin - 25 Mar 2011, 10:30:05

Just a quick note: when you are configuring your load balancing settings with Apache2, JK and Tomcat, remember to plan for failover. The default settings WILL NOT failover like you expect!

Here are some quick settings that illustrate my point.

workers.properties file:

I highlighted socket_timeout and connect_timeout because these settings cause JK to actually check its connections to the Tomcat servers it is load balancing. It seems like a matter of course that JK should do this automatically, but it won't! So, socket_timeout is set to 2 seconds here for the case where JK is having problems getting a network connection. connect_timeout is where the magic happens, I've set that to 500 milliseoonds which tells JK to wait that long before getting back a CPONG response to its CPING before issuing the actual request to the chosen Tomcat server. When a Tomcat server doesn't respond within the timeout window, JK uses another Tomcat server to send the request to and marks the original Tomcat server as ERR. It'll check that original Tomcat server again after its maintenance window expires. All this stuff is explained in Tomcat's timeout article.

I learned this the hard way - where I have some mis-behaving EC2 instances which freeze up now and then (working on fixing that). In the short term, I modified my JK settings to enabled failover. I've also used the jkstatus/jkmanager to monitor the situation, extremely handy!

Comments? Suggestions? Feedback?

Top 5 Cloud Features

by Morgan Catlin - 01 Aug 2011, 12:44:53

As a heavy consumer of Cloud infrastructure, I get asked by vendors what I'm looking for in a cloud (just today AWS asked me to do a survey).

I'm not going to bore you with reading from 5 to 1 to see what I think is the most important feature. I'm just going to say it:

#1: Reliability
Vendors think about reliability in terms of how the applications I'm running on their infrastructure are doing. It's true, it's my main concern! But as the AWS failure shows, the ability for other parts of the vendor's infrastructure to be up and running are critical. I'm looking at reliability from an overall perspective:

• Are the resources I provision reliable?
• Are the tools that I use to manage those resources reliable (CLI or UI)?
• Are those tools easy to understand? Do they respond quickly? Am I sitting around chewing my fingernails off during an especially stressful time?
• Is the support process quick and easy to understand? Can I find out how to get support quickly?
• Documentation!?

#2: Provisioning
When I fire up new instances, I should see them quickly. When I sign up for new services, I should have immediate access. Are there things I need to be aware of?

#3: Redundancy
I like how AWS breaks up their infrastructure into regions and zones, separating the physical resources into pools. Does the vendor follow a similar strategy?

#4: Pricing
Needs to be competitive or show a value added service! Easy to calculate cost as resources scale up. Cost should increase in a linear way, instead of in big jumps.

#5: Local "Cloud" Storage
Local permission-based cloud storage, like S3, is critical. I use it for storing back-ups and basing my disaster recovery (DR) scripts on it. I don't want to have to hit my applications to pull data off of them - I like to setup batch processes to push the data off into cloud storage. Then I can pull them down for offline storage and into my DR zones.

About the author: Morgan is an application developer, focusing on rich Web-based applications. These applications solve a myriad of business problems, from managing spots on cable tv to tracking users 'click-through' rates on marketing web sites. These applications rely on database and operating system interactions, primarily Java, Ubuntu Linux and MySQL.

Comments? Suggestions? Feedback?

Cloud Applications Presentation

by Morgan Catlin - 16 Nov 2011, 21:10:33

I've submitted my MN IT Symposium presentation, "Architecting and Deploying Scalable, Resilient, and Cost-Effective Cloud Applications", which discusses how to bring about an affective Cloud-based application.

It discusses my experiences using PaaS and rolling my own architecture. I go over what worked and some common misconceptions about using PaaS and RYO.

I think the most interesting part is showing an ideal distributed architecture for the common Java Pet Store application. I talk about how to use AWS' services to achieve high reliability and scalability.

Can't wait to meet all you great attendees!

Comments? Suggestions? Feedback?

Secure Video OnDemand

by Morgan Catlin - 18 Jan 2012, 15:24:57

It's been over a year since we launched the Video On-Demand site for one of our clients (here). We've learned a lot about streaming video since then, mostly, that it's hard!

Problem 1: Secure Streams
I spent two blog posts talking about why and how to secure your streams. Keep those dirty pirates from getting your content!

Problem 2: Content Providers
Content providers impose even weirder and harder to implement restrictions than securing the streams. For example, one client's providers want to keep people from watching free content from anywhere but the US and Canada. We had to pull down Maxmind's geo database and use it to validate IP addresses. We're rolling this feature out to Roku and Nook based apps too!

Problem 3: Scaling
Man, video is bandwidth starved. Bandwidth means three areas:

• Client Internet - sure, you might have cable but your WIFI router is crap
• Server Internet - obviously
• Server Disk I/O - just how much data can you push through?

All of these things can put a crimp on a user's happiness with your video. What we've been doing when we bump into limitations on Server Disk I/O is spool up another EC2/S3-backed Wowza instance and loop it into the production cluster. We're then able to step up the amount of I/O capability through out the system.

Problem 4: Customer Support
Well, Disk I/O causes a problem for all users streaming from a specific server but what about other issues? We spent some time building diagnostic pages (like MLB's) that will issue bandwidth tests and report to us what type of environment the user has. It's easy then to create profiles of environments that have issues, could have issues or are relatively issue-free. We're adding these types of diagnostics to devices too - consolidating all the reporting into a set of RESTful web services easily invoked by developers.

Problem 4: Devices
Getting video to play on a device is really just harder than it should be. Devices all support different standards at different levels of support. For example, Android devices don't support Apple's HTTP streaming, but Roku devices do. In fact, we have to go with a Flash player on Android - talk about hideously slow and battery killing! Other problems include setting up the content to look nice on each device - some content just needs lots and lots of massaging to get it look right.

Comments? Suggestions? Feedback?