Royalty-Free Low-Latency Streaming

2 second delay is fine…

We’ve witnessed HLS and MPEG-DASH streaming solutions taking over RTMP in just a few years, yet these are inherently delayed from real time due to their segmented nature. And we are talking default end-to-end lags of over 30 seconds. While for some live streaming setups a sub-minute long delay is harmless, others are compelled to reduce the latency to the very minimum. Think live auctions, gambling, or trading platforms.

To be clear, HTTP streaming and its big delays are no mistake. It leverages the ubiquity of HTTP, gives your player time to adapt to network fluctuations, doesn’t rush the muxer to output slices or the edges to cache them. It’s friendly to ABR. And it’s hugely beneficial to the CDN industry 🙂

While there are no protocol limitations when delivering to mobile apps, and you can easily stream over RTMP or RTSP, the browser is very restrictive and not at all straightforward to live stream to or from.

For quite a while, online platform implementers held on to RTMP, but as support for it eventually went away, the community had to adapt. Many of us tried pushing the HLS and DASH to its limits (shorter segments and shorter playlists), but that soon proved to be far from ideal, as playback smoothness would suffer on all but the best connections.

You’ll run into more and more companies and individuals willing and able to put together a custom low-latency solution nowadays. Moreover, the rise of HQ Trivia seems to have stimulated many to bring their approach to the the masses and offer plug-and-play solutions. You may hear stories of ground-breaking technology but really, they all fall into into one of the 3 categories:

  1. WebRTC basedWebRTC is not particularly new to the game, yet it’s still not ubiquitous, and it may still be a while before it is; providers like Twilio or Red5Pro offer easy to integrate SaaS or hosted solutions on top of it and CDNs like LimeLight are building it into their networks. If done right, you can expect sub-second latencies out of WebRTC, even across long distances and poor network connections
  2. WebSocket based – every modern browser supports WebSockets; while not trivial, a high-level protocol can be implemented on top of it and successfully stream video from server to client; at least in theory, capabilities similar to RTMP can be achieved. As WebSockets run over TCP (in turn WebRTC can use UDP) and the lower-level protocols introduce overheads, expect a latency of 2-3 seconds out of any solution; providers like Wowza, Nanocosmos and Nimble offer solutions based on it, while CDNs like CloudFront and CloudFlare lately support WebSockets
  3. Chunked-transfer basedchunked-transfer is built into HTTP 1.1, and makes it possible for “chunks” of data to be written to, and read from, the network before the whole data is available. Provided compatible encoder, infrastructure, and decoder, this can be taken advantage of to output and playback “not-yet-complete” video segments, and significantly reduce the latency of segment-based protocols. The technique is being employed with promising results for HLS, DASH, and newly CMAF. Companies like Periscope or TheoPlayer offer proprietary solutions based on it, and a few open approaches can be found online. Chunked-transfer is supported by some CDNs. Expect a latency of under 5 seconds, it will greatly vary with the specifics of the implement.

So what’s the best of them?

There isn’t one… Client has been presented the above knowledge and options, among others. Long story short, a solution based on Nimble was chosen. They agreed to make it public but asked to remain anonymous. And here it is for you to deploy in just a few clicks.

Does it scale?

Yes, horizontally. It has held up to hundreds of thousands and I see no reason it can’t do more. In case of sudden spikes though, the Auto Scaling group is set to only fire up a new edge every 5 minutes so you’d want to manually intervene if you expect a riot.

Vertically no. As per the specs, it would only need to stream a handful of streams. Sure, there are ways to turn this into a bunch or a million.

Is it stable?

Very. It works like a charm and it’s production sound out of the box. Don’t take my word for it, try it out.

Is it worth it?

Depends on what you’re after. At the time (summer-fall 2018) Nimble’s has been the preferred option; cost, reliability, and capability to deploy on own infrastructure taken into account. Worthy competitors were Nano, Wowza and Red5Pro.

Where’s the diagram?

Here it is, sorry…

Scalable Wowza Transcoding

I need a Wowza/AWS Engineer to reduce costs

Live transcoding is never easy. Doing it yourself from scratch may be way too complex, running it off a cloud service looks like a rip off. Middle ground solutions still involve a lot of decision making.

It gets worse if you run non-24h streams and their number fluctuates. The optimal strategy depends a lot on how many streams you are running and the dynamics of these. As they’re rarely predictable, yet tend to peak at various times of day or week, provisioning enough capacity for all is definitely a waste.

Wowza has long had transcoding built-in, but as they’re promoting the cloud they document no way of scaling the server product. And if you’ve customized a bit of it, the cloud may no longer be compatible. Not to mention costly.

Customer was running a single Wowza server in AWS, capable of transcoding a handful of simultaneous streams. It was good enough at the time but as the portal grew in popularity it would require more and more resources, up to the point where even the most powerful AWS instance would no longer be good enough. Furthermore, given the nature of the business, most events would run at the same time of day or week and the ever pricier single server would sit idle for most of the time.

The proposed solution was quite simple and them knowing exactly what they wanted helped a lot. The existing Wowza would stay in place and be turned into a “master”. It would do no transcoding on its own but instead spin up ephemeral transcoder workers in independent EC2 instances. Upon a new stream getting published to the “master”, it would start a transcoder instance, push the unprocessed stream to that, and pull the transcoded streams back from it. Very little would change for the application layer as broadcasting and playback would stay almost the same.

There’s a a downside to the approach, the processed (transcoded) feed would only be available some 1-2 minutes after it had started, this being the time needed for the instance to initialize. Yet, given the scheduled nature of streams, this would not be a problem.

They courteously agreed to share the solution with the world and here it is, complete with some instructions. Do make sure it fits your needs.

Does it scale?

Not fully. Although not doing any transcoding, there’s a limit to how many streams the single “master” can handle. But rest assured that number’s pretty high, I have a client running more than 700 streams on a common Wowza configuration.

There are ways to programatically make this a multiple-‘master’ setup so that it scales all the way. If ever needed.

Is it stable?

I would say so. It’s been running since January and I’ve had no complaints.

Is it worth it?

It depends 🙂 It sure has for them and still can’t think of a better setup to fit their needs. But again, this was a very particular use case to start with.

Unmentioned, they had customized the transcoder to display some nice scoreboard overlays. Unless ready to recreate that, the solution had to stay within the Wowza confines.

The transcoder instances management could have been set up at some other level (i.e. Lambda) but we decided to just build it into the ‘master’ Wowza as it would be compact to deploy and easy to grasp by the team in place, since they already had some exposure with coding Java for Wowza.

Not least, they had already invested in wowza licensing long term. And it would pay out as they’d continue to use it on the ‘master’ (which has to run 24/7 anyway as it’s also serving VOD) while paying by-the-hour to for the transcoders’ licensing.

Even if any of these is your case, it may still be viable. Especially if you come from a Wowza background and don’t want to spend much time looking around for options.

Simple Video Sharing Platform

I would love to allow my users to upload their own videos to [presumably] AWS S3. As usual on web, we cannot assume much about uploaded videos.

The customer actually came up with the architecture. They just wanted to know if it’s feasible and if it can be done easily.
Sure thing! The diagram hopefully says is all.

Using Presigned URLs is a great way to let your users (or anybody else) securely upload content to your S3, saving you the bother of having to proxy or manipulate large files.

Every video upload triggers a Lambda function, which in turn asks MediaConvert to process that video. It gets transcoded for Adaptive Bitrate and properly packaged for segmented delivery. When ready (or failed) it fires a notification that lets your backend know the video can now be played (or not).

Content is packaged as CMAF, which brings significant savings in transcoding, storage and bandwidth over traditional HLS and DASH, while still compatible with both.

Original videos (the videos that users upload), get archived to low-cost storage. As processed, play-ready copies of these are already in place, you may never need the originals again, yet you don’t really want to throw them away. Just in case…

Complete solution, less the URL signing, is available for grabs here. You should be able to set it up and transcode a few short videos in less than 20min.

Does is scale?

Oh yes! There’s virtually no bottleneck in the whole AWS-driven part of the architecture. As long as you can keep up with the requests for signed URLs and the SNS notifications, sky’s truly the limit.

Is it expensive?

It depends. 🙂
Transcoding alone will set you back some $7 for every hour of content. Yet the default preset being used is overkill (10 quality variants), you can easily duplicate and customize it to proportionally cut costs. You can save even more with reserved pricing for MediaConvert once your portal reaches a steady flow of uploads.

You’ll also be paying for storage, CDN, S3 and internal traffic. Finally, pennies or nothing for traffic, Lambda and notifications.

Is it worth it over other solutions?

There’s no doing better if you need it scalable and easily deployed. It’s also maintenance free, which many tend to disregard when factoring in alternatives.

Hot topic here is transcoding, and there are many different approaches to it, with pros and cons to each. Factor in volume, size, and fluctuations of video uploads, required readiness (how soon after upload you need a video to be available, at minimum), SaaS vs cloud vs on premises, development and testing capabilities etc.