Wowza CPU Performance comparative for Live Streaming sites

5

The idea of this post is share some technical details about different Wowza servers settings, running into Live Streaming sites.

This 4 years working into multiple projects allow to me recollect stats about performance, currently support more than 100 Wowza Server daily and this list of CPU’s is the common used.

One normal question from my customers are, what server need to buy for my streams?

For this reason into this post I’ll share some metrics and performance graph from multiple servers.

All the server run, with Wowza Media Server 2.2.3 with JDK 1.6.0_XX over Centos 5/6 64 bits.

The list of server for this comparative are: (benchmark from http://www.cpubenchmark.net):

Methodology of the metric recollections:

  • At the moment to take the metrics all the server need to be using 1Gbps during almost 15 minutes (this metrics are based into 30 minutes samples)
  • All need to have similar concurrent users connected (the average is 4610 Concurrent Users into this comparative)

We’ll use 5 metric to compare:

  • CPU Available (Average)
  • JVM Memory Used (Average)
  • Server RAM
  • Load (Average)
  • Connected Users (Average)

Working with Wowza logs

0


Improve logs management and rotation over linux, combining Log4j and LogRotate.

Modify the log4j.properties for using other Appender:

# Access appender
log4j.appender.serverAccess=org.apache.log4j.RollingFileAppender
log4j.appender.serverAccess.File=${com.wowza.wms.ConfigHome}/logs/wowzamediaserver_access.log
log4j.appender.serverAccess.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.serverAccess.layout.Fields=date,time,tz,x-event,x-category,x-severity,x-status,x-ctx,x-comment,x-vhost,x-app,x-appinst,x-duration,s-ip,s-port,s-uri,c-ip,c-proto,c-referrer,c-user-agent,c-client-id,cs-bytes,sc-bytes,x-stream-id,x-spos,cs-stream-bytes,sc-stream-bytes,x-sname,x-sname-query,x-file-name,x-file-ext,x-file-size,x-file-length,x-suri,x-suri-stem,x-suri-query,cs-uri-stem,cs-uri-query
log4j.appender.serverAccess.layout.OutputHeader=true
log4j.appender.serverAccess.layout.QuoteFields=false
log4j.appender.serverAccess.layout.Delimeter=tab

Create the logrotate configuration file into Centos/RedHat the path is:

nano /etc/logrotate.d/wowza
/usr/local/WowzaMediaServer/logs/*.log {
       rotate 3
       size=512M
       compress
       copytruncate
       missingok
}

And the last step is you have very big logs files like is my case, and into one day I can have 2 GB of logs or more, move the log rotate for check every hour instead of every day

mv /etc/cron.daily/logrotate /etc/cron.hourly/

Inspired in this post: http://nicerhttpd.blogspot.com/2008/11/logrotate-and-log4j.html

UPDATE:
Other way to work with Wowza logs, can be using external Jar for rotate the logs by Size and Time, this way if you are running into WinOS or don’t like to use LogRotate for this task can have similar functionality

For more details can check the Wowza Forum post: http://www.wowza.com/forums/content.php?263-Logging-How-to-use-TimeAndSizeRollingAppender

Live Publish from iPhone to Wowza

1

I’m really happy to be part of the Beta Testers of this new and amazing application.

With Livu you can stream from your iPhone 3GS or newest to any Wowza Media Server using RTMP protocol…

Features supported:

  • Adaptive bitrate streaming
  • Authentication support
  • Touch to focus
  • Double tap to expose
  • Torch mode (iPhone 4 only)
  • WIFI Video sizes (192×144, 480×360)
  • Cellular streaming is limited to a low bitrate stream.

Here some screenshots:

 

 

You don’t have one account in any Wowza Server for test contact me, I’ll provide one.

Thanks to Steve for this application

 

Awstats mod for process Wowza Stats

28

Download the package from this link:

AWStats Mod for WowzaMediaServer (3069)
Awstats mod conf file (2033)

Add this lines into your Apache configuration:

#
# Directives to allow use of AWStats as a CGI
#

Alias /awstatsclasses /usr/local/awstats-wmc/wwwroot/classes/
Alias /awstatscss /usr/local/awstats-wmc/wwwroot/css/
Alias /awstatsicons /usr/local/awstats-wmc/wwwroot/icon/
ScriptAlias /awstats/ /usr/local/awstats-wmc/wwwroot/cgi-bin/

#
# This is to permit URL access to scripts/files in AWStats directory.
#

AddHandler cgi-script cgi pl
Options ExecCGI
AllowOverride None
Order allow,deny
Allow from all

Setting up the Crontab for update the Stats automatically

*/5 * * * * /usr/local/awstats-wmc/tools/awstats_updateall.pl now > /var/log/awstats.log

Setting up the Wowza Log Appender for save the information in the format then Awstats will be understand

For doit this edit the log4j.properties normally into this path:

 /usr/local/WowzaMediaServer/conf/log4j.properties 

Add this appender:

 # Statistics appender (to use this appender add "serverAWStats" to the list of appenders in the first line of this file)

log4j.appender.serverAWStats=org.apache.log4j.DailyRollingFileAppender
log4j.appender.serverAWStats.DatePattern='.'yyyy-MM-dd
log4j.appender.serverAWStats.File=${com.wowza.wms.ConfigHome}/logs/wmconsulting_stats.log
log4j.appender.serverAWStats.layout=com.wowza.wms.logging.ECLFPatternLayout
log4j.appender.serverAWStats.layout.Fields=date,time,c-proto,c-ip,c-referrer,x-status,cs-uri-stem,sc-bytes,x-duration,x-sname,sc-stream-bytes,cs-stream-bytes,x-ctx,c-proto,x-appinst
log4j.appender.serverAWStats.layout.OutputHeader=true
log4j.appender.serverAWStats.layout.QuoteFields=false
log4j.appender.serverAWStats.layout.Delimeter=tab
log4j.appender.serverAWStats.layout.EventInclude=destroy

This way the logs will rotate automatically every day, and you can move this logs to the stats server… of curse you will have 1 day of delay with the stats… if the Awstats is on the same server than Wowza, you can update the logs every five minutes or less depend of your needs…

Test your configuration running the command manually

First run:

/usr/local/awstats-wmc/wwwroot/cgi-bin/awstats.pl -config=your.site.com

Update run:

/usr/local/awstats-wmc/wwwroot/cgi-bin/awstats.pl -update -config=site.com  -showdropped

Real Time demo:

VOD Files
Live Streams

Here a few ScreenShots:

How to know what is happening inside Wowza?

5

I have seen into the Blog stats than the most common question is How To Monitor Wowza Media Server… for this reason I’ll start to write a series of Posts about this topic.

In this first post I’ll request your help too, let me know what you would like to know about your Wowza Server.

Currently the information I consider useful, is:

  • Concurrent Connection (separated by Protocol –  HTTP Cupertino / HTTP Silverlight / RTSP / RTMP)
  • MemoryHeap (the memory used by the JVM)
  • Version (if you have multiple server,it is difficult remember what version you have installed)
  • Bandwidth Traffic (by Vhost, Application or ApplicationInstances)

And here the question… What info would you like to know about Wowza?

Please post into the comment what else you would like to know…

Into the next post, I’ll show examples, graphs and other tips to get this information from the Wowza Server, using JMX protocol.

But here I show a snake peak of the information and the Graph you can have.

Mobile Demos

6

For Watch this videos, and listen the Online Radio, please enter to this URL with your mobile device

Please after test the Streaming with your Mobile Device, take 1 minute to send your Device Model using our Contact form, this way we can load this in our database and make public this result.

IMPORTANT: The Mobile Device need support H264 codec

Videos
Android/Blackberry

Shoutcast Radio relay
Android/Blackberry

How to read VOD files from Amazon S3 on Standalone Wowza Server

11

I’ll explain how to is possible read the files from Amazon S3 Storage but without using a EC2 instance.
For example your “localhost” or your Standalone Wowza server running on your own datacenter.

I did this for working on my local development environment, I’ll describe the steps for a Linux environment.

1- Download from this URL the 2 Jars, and copy this in /lib directories.

wms-plugin-amazonaws.jar (1308)
wms-plugin-mediacache.jar (1307)

2- Create the MediaCache folder:

mkdir /mnt/mediacache

3- Insert into the conf/Server.xml this lines:

<ServerListeners>
      <ServerListener>
                <BaseClass>com.wowza.wms.plugin.amazonaws.ec2.mediacache.MediaCacheServerListenerAmazonEC2</BaseClass>
       </ServerListener>
</ServerListeners>

4- Insert into the conf/vods3/Application.xml (create this first) this lines:

                <MediaReader>
                        <Properties>
                                <Property>
                                        <Name>randomAccessReaderClass</Name>
                                        <Value>com.wowza.wms.plugin.amazonaws.ec2.mediacache.MediaCacheRandomAccessReaderAmazonEC2</Value>
                                </Property>
                                <Property>
                                        <Name>bufferSeekIO</Name>
                                        <Value>true</Value>
                                        <Type>Boolean</Type>
                                </Property>
                        </Properties>
                </MediaReader>

And in the end of the file, add this properties (I don’t know is needed this for read Public content):

                <Properties>
                        <!-- Set these two properties to do S3 authentication -->
                        <Property>
                                <Name>awsAccessKeyId</Name>
                                <Value>KEY</Value>
                        </Property>
                        <Property>
                                <Name>awsSecretAccessKey</Name>
                                <Value>KEY</Value>
                        </Property>
                </Properties>

5- Restart the Wowza Server and try reading from S3 Bucket.

The URL can be like this:
rtmp://localhost/vods3/_definst_/mp4:amazons3/wmconsulting.content/Extremists.m4v

Tell me is I missing some step or need some extra help.

Monitoring Wowza with the JMX interface

1

First we need Download CHECK_JMX from this link check_jmx

To see the option we have available, we can execute the plugin with the param “-help”

./check_jmx -help

Output:

Usage: check_jmx -U url -O object_name -A attribute [-K compound_key] [-I attribute_info] [-J attribute_info_key] -w warn_limit -c crit_limit [-v[vvv]] [-help]

where options are:

-help
Prints this page
-U
JMX URL, for example: service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi
-O
Object name to be checked, for example: java.lang:type=Memory
-A
Attribute of the object to be checked, for example: NonHeapMemoryUsage
-K
Attribute key for -A attribute compound data, for example, "used" (optional)
-I
Attribute of the object containing information for text output (optional)
-J
Attribute key for -I attribute compound data, for example, "used" (optional)
-v[vvv]
verbatim level controlled as a number of v (optional)
-w
warning integer value
-c
critical integer value

Note that if warning level > critical, system checks object attribute value to be LESS THAN OR EQUAL warning, critical
If warning level < critical, system checks object attribute value to be MORE THAN OR EQUAL warning, critical

Here had some generic examples
Show the Use of Memory Heap in to VM

./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 10000000 -c 100000000

How many users connected to this server:

./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O WowzaMediaServerPro:name=Connections -A current -vvvv -w 10 -c 20

El resultado es algo como esto:

JMX OK current=5

This is an special case:

./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O WowzaMediaServerPro:vHosts=VHosts,vHostName=_defaultVHost_,applications=Applications,
applicationName=rtplive,applicationInstances=ApplicationInstances,applicationInstanceName=_definst_,
name=Connections -A current -vvvv -c 1000 -w 750

Explain this parameters with more detail:

  • WowzaMediaServerPro:vHosts=VHosts <– No need more explanation
  • vHostName=_defaultVHost_ <– This are the Default VHosts
  • applications=Applications <– All the Application is inside of this Folder Name
  • applicationName=rtplive <– Application Name (In Version 2 of Wowza, is LIVE)
  • applicationInstances=ApplicationInstances <– can be multiple application instance in one application
  • applicationInstanceName=_definst_ <– Default application instance
  • name=Connections <– current connection to the Application Instance
  • Fell free to make any comment or question.

    VOD Demo (Amazon EC2 instance, S3 storage)

    0

    THIS DEMO IS NOT WORKING NOW, Whe are Updating the Server

    All this demos are Hosted in a Standalone Instance, and the files are serving from S3 Storage platform.

    Demo a 778k (650k video + 128k audio)

    [pro-player width='568' height='320'  type='rtmp' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoHi.mov[/pro-player]

    Demo a 578k (450k video + 128k audio)

    [pro-player width='568' height='320' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoMed.mov[/pro-player]

    Demo a 328k (200k video + 128k audio)

    [pro-player width='560' height='320' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoHi.mov[/pro-player]

    Demo a 328k (200k video + 128k audio) Tamaño Original

    [pro-player width='284' height='160' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoLo.mov[/pro-player]

    ShoutCast Relay Demo

    0

    Hi,

    In this first post I’ll show  a demo.

    About how can make a relay with Wowza Server from a Shoutcast server.

    Demo with JWPlayer

    [pro-player width='182' height='120' type='rtmp' fullscreen='false' autostart='false' streamer='rtmpt://ec2.wmconsulting.info/shoutcast']radiostation.stream[/pro-player]



    You can improve the user experience, provide a On-line Player in Flash, and running over the port 80 with RTMP protocol, this way don’t have problem with the firewall.

    Don’t hesitate to send your feedback or question about that.

    Go to Top