Wowza CPU Performance comparative for Live Streaming sites
4The idea of this post is share some technical details about different Wowza servers settings, running into Live Streaming sites.
This 4 years working into multiple projects allow to me recollect stats about performance, currently support more than 100 Wowza Server daily and this list of CPU’s is the common used.
One normal question from my customers are, what server need to buy for my streams?
For this reason into this post I’ll share some metrics and performance graph from multiple servers.
All the server run, with Wowza Media Server 2.2.3 with JDK 1.6.0_XX over Centos 5/6 64 bits.
The list of server for this comparative are: (benchmark from http://www.cpubenchmark.net):
- Intel Xeon X3210 @ 2.13GHz 2,622
- Intel Xeon X3220 @ 2.40GHz 3,046
- Intel Xeon X3430 @ 2.40GHz 3,951
- Intel Xeon X3470 @ 2.93GHz 5,987
- Intel Xeon E5-2620 0 @ 2.00GHz (Not listed into cpubenchmark.net) but is the most powerful server with 6 cores
Methodology of the metric recollections:
- At the moment to take the metrics all the server need to be using 1Gbps during almost 15 minutes (this metrics are based into 30 minutes samples)
- All need to have similar concurrent users connected (the average is 4610 Concurrent Users into this comparative)
We’ll use 5 metric to compare:
- CPU Available (Average)
- JVM Memory Used (Average)
- Server RAM
- Load (Average)
- Connected Users (Average)
Working with Wowza logs
0
Improve logs management and rotation over linux, combining Log4j and LogRotate.
Modify the log4j.properties for using other Appender:
# Access appender log4j.appender.serverAccess=org.apache.log4j.RollingFileAppender log4j.appender.serverAccess.File=${com.wowza.wms.ConfigHome}/logs/wowzamediaserver_access.log log4j.appender.serverAccess.layout=com.wowza.wms.logging.ECLFPatternLayout log4j.appender.serverAccess.layout.Fields=date,time,tz,x-event,x-category,x-severity,x-status,x-ctx,x-comment,x-vhost,x-app,x-appinst,x-duration,s-ip,s-port,s-uri,c-ip,c-proto,c-referrer,c-user-agent,c-client-id,cs-bytes,sc-bytes,x-stream-id,x-spos,cs-stream-bytes,sc-stream-bytes,x-sname,x-sname-query,x-file-name,x-file-ext,x-file-size,x-file-length,x-suri,x-suri-stem,x-suri-query,cs-uri-stem,cs-uri-query log4j.appender.serverAccess.layout.OutputHeader=true log4j.appender.serverAccess.layout.QuoteFields=false log4j.appender.serverAccess.layout.Delimeter=tab
Create the logrotate configuration file into Centos/RedHat the path is:
nano /etc/logrotate.d/wowza
/usr/local/WowzaMediaServer/logs/*.log { rotate 3 size=512M compress copytruncate missingok }
And the last step is you have very big logs files like is my case, and into one day I can have 2 GB of logs or more, move the log rotate for check every hour instead of every day
mv /etc/cron.daily/logrotate /etc/cron.hourly/
Inspired in this post: http://nicerhttpd.blogspot.com/2008/11/logrotate-and-log4j.html
UPDATE:
Other way to work with Wowza logs, can be using external Jar for rotate the logs by Size and Time, this way if you are running into WinOS or don’t like to use LogRotate for this task can have similar functionality
For more details can check the Wowza Forum post: http://www.wowza.com/forums/content.php?263-Logging-How-to-use-TimeAndSizeRollingAppender
Live Publish from iPhone to Wowza
1I’m really happy to be part of the Beta Testers of this new and amazing application.
With Livu you can stream from your iPhone 3GS or newest to any Wowza Media Server using RTMP protocol…
Features supported:
- Adaptive bitrate streaming
- Authentication support
- Touch to focus
- Double tap to expose
- Torch mode (iPhone 4 only)
- WIFI Video sizes (192×144, 480×360)
- Cellular streaming is limited to a low bitrate stream.
Here some screenshots:
You don’t have one account in any Wowza Server for test contact me, I’ll provide one.
Thanks to Steve for this application
Awstats mod for process Wowza Stats
28Download the package from this link:
AWStats Mod for WowzaMediaServer (2700)
Awstats mod conf file (1771)
Add this lines into your Apache configuration:
# # Directives to allow use of AWStats as a CGI # Alias /awstatsclasses /usr/local/awstats-wmc/wwwroot/classes/ Alias /awstatscss /usr/local/awstats-wmc/wwwroot/css/ Alias /awstatsicons /usr/local/awstats-wmc/wwwroot/icon/ ScriptAlias /awstats/ /usr/local/awstats-wmc/wwwroot/cgi-bin/ # # This is to permit URL access to scripts/files in AWStats directory. # AddHandler cgi-script cgi pl Options ExecCGI AllowOverride None Order allow,deny Allow from all
Setting up the Crontab for update the Stats automatically
*/5 * * * * /usr/local/awstats-wmc/tools/awstats_updateall.pl now > /var/log/awstats.log
Setting up the Wowza Log Appender for save the information in the format then Awstats will be understand
For doit this edit the log4j.properties normally into this path:
/usr/local/WowzaMediaServer/conf/log4j.properties
Add this appender:
# Statistics appender (to use this appender add "serverAWStats" to the list of appenders in the first line of this file) log4j.appender.serverAWStats=org.apache.log4j.DailyRollingFileAppender log4j.appender.serverAWStats.DatePattern='.'yyyy-MM-dd log4j.appender.serverAWStats.File=${com.wowza.wms.ConfigHome}/logs/wmconsulting_stats.log log4j.appender.serverAWStats.layout=com.wowza.wms.logging.ECLFPatternLayout log4j.appender.serverAWStats.layout.Fields=date,time,c-proto,c-ip,c-referrer,x-status,cs-uri-stem,sc-bytes,x-duration,x-sname,sc-stream-bytes,cs-stream-bytes,x-ctx,c-proto,x-appinst log4j.appender.serverAWStats.layout.OutputHeader=true log4j.appender.serverAWStats.layout.QuoteFields=false log4j.appender.serverAWStats.layout.Delimeter=tab log4j.appender.serverAWStats.layout.EventInclude=destroy
This way the logs will rotate automatically every day, and you can move this logs to the stats server… of curse you will have 1 day of delay with the stats… if the Awstats is on the same server than Wowza, you can update the logs every five minutes or less depend of your needs…
Test your configuration running the command manually
First run:
/usr/local/awstats-wmc/wwwroot/cgi-bin/awstats.pl -config=your.site.com
Update run:
/usr/local/awstats-wmc/wwwroot/cgi-bin/awstats.pl -update -config=site.com -showdropped
Real Time demo:
How to know what is happening inside Wowza?
5I have seen into the Blog stats than the most common question is How To Monitor Wowza Media Server… for this reason I’ll start to write a series of Posts about this topic.
In this first post I’ll request your help too, let me know what you would like to know about your Wowza Server.
Currently the information I consider useful, is:
- Concurrent Connection (separated by Protocol - HTTP Cupertino / HTTP Silverlight / RTSP / RTMP)
- MemoryHeap (the memory used by the JVM)
- Version (if you have multiple server,it is difficult remember what version you have installed)
- Bandwidth Traffic (by Vhost, Application or ApplicationInstances)
And here the question… What info would you like to know about Wowza?
Please post into the comment what else you would like to know…
Into the next post, I’ll show examples, graphs and other tips to get this information from the Wowza Server, using JMX protocol.
But here I show a snake peak of the information and the Graph you can have.
Mobile Demos
6For Watch this videos, and listen the Online Radio, please enter to this URL with your mobile device
Please after test the Streaming with your Mobile Device, take 1 minute to send your Device Model using our Contact form, this way we can load this in our database and make public this result.
IMPORTANT: The Mobile Device need support H264 codec
Shoutcast Radio relay
How to read VOD files from Amazon S3 on Standalone Wowza Server
11I’ll explain how to is possible read the files from Amazon S3 Storage but without using a EC2 instance.
For example your “localhost” or your Standalone Wowza server running on your own datacenter.
I did this for working on my local development environment, I’ll describe the steps for a Linux environment.
1- Download from this URL the 2 Jars, and copy this in /lib directories.
wms-plugin-amazonaws.jar (1115)
wms-plugin-mediacache.jar (1117)
2- Create the MediaCache folder:
mkdir /mnt/mediacache
3- Insert into the conf/Server.xml this lines:
<ServerListeners> <ServerListener> <BaseClass>com.wowza.wms.plugin.amazonaws.ec2.mediacache.MediaCacheServerListenerAmazonEC2</BaseClass> </ServerListener> </ServerListeners>
4- Insert into the conf/vods3/Application.xml (create this first) this lines:
<MediaReader> <Properties> <Property> <Name>randomAccessReaderClass</Name> <Value>com.wowza.wms.plugin.amazonaws.ec2.mediacache.MediaCacheRandomAccessReaderAmazonEC2</Value> </Property> <Property> <Name>bufferSeekIO</Name> <Value>true</Value> <Type>Boolean</Type> </Property> </Properties> </MediaReader>
And in the end of the file, add this properties (I don’t know is needed this for read Public content):
<Properties> <!-- Set these two properties to do S3 authentication --> <Property> <Name>awsAccessKeyId</Name> <Value>KEY</Value> </Property> <Property> <Name>awsSecretAccessKey</Name> <Value>KEY</Value> </Property> </Properties>
5- Restart the Wowza Server and try reading from S3 Bucket.
The URL can be like this:
rtmp://localhost/vods3/_definst_/mp4:amazons3/wmconsulting.content/Extremists.m4v
Tell me is I missing some step or need some extra help.
Monitoring Wowza with the JMX interface
1First we need Download CHECK_JMX from this link check_jmx
To see the option we have available, we can execute the plugin with the param “-help”
./check_jmx -help
Output:
Usage: check_jmx -U url -O object_name -A attribute [-K compound_key] [-I attribute_info] [-J attribute_info_key] -w warn_limit -c crit_limit [-v[vvv]] [-help]
where options are:
-help Prints this page -U JMX URL, for example: service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O Object name to be checked, for example: java.lang:type=Memory -A Attribute of the object to be checked, for example: NonHeapMemoryUsage -K Attribute key for -A attribute compound data, for example, "used" (optional) -I Attribute of the object containing information for text output (optional) -J Attribute key for -I attribute compound data, for example, "used" (optional) -v[vvv] verbatim level controlled as a number of v (optional) -w warning integer value -c critical integer value
Note that if warning level > critical, system checks object attribute value to be LESS THAN OR EQUAL warning, critical
If warning level < critical, system checks object attribute value to be MORE THAN OR EQUAL warning, critical
Here had some generic examples
Show the Use of Memory Heap in to VM
./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O java.lang:type=Memory -A HeapMemoryUsage -K used -I HeapMemoryUsage -J used -vvvv -w 10000000 -c 100000000
How many users connected to this server:
./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O WowzaMediaServerPro:name=Connections -A current -vvvv -w 10 -c 20
El resultado es algo como esto:
JMX OK current=5
This is an special case:
./check_jmx -U service:jmx:rmi://192.168.0.1:8084/jndi/rmi://192.168.0.1:8085/jmxrmi -O WowzaMediaServerPro:vHosts=VHosts,vHostName=_defaultVHost_,applications=Applications, applicationName=rtplive,applicationInstances=ApplicationInstances,applicationInstanceName=_definst_, name=Connections -A current -vvvv -c 1000 -w 750
Explain this parameters with more detail:
Fell free to make any comment or question.
VOD Demo (Amazon EC2 instance, S3 storage)
0THIS DEMO IS NOT WORKING NOW, Whe are Updating the Server
All this demos are Hosted in a Standalone Instance, and the files are serving from S3 Storage platform.
Demo a 778k (650k video + 128k audio)
[pro-player width='568' height='320' type='rtmp' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoHi.mov[/pro-player]
Demo a 578k (450k video + 128k audio)
[pro-player width='568' height='320' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoMed.mov[/pro-player]
Demo a 328k (200k video + 128k audio)
[pro-player width='560' height='320' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoHi.mov[/pro-player]
Demo a 328k (200k video + 128k audio) Tamaño Original
[pro-player width='284' height='160' type='rtmp' autostart='false' streamer='rtmp://wms1.wmconsulting.info/vods3/1001/']amazons3/wmconsulting.content/BigBuckCupertinoLo.mov[/pro-player]
ShoutCast Relay Demo
0Hi,
In this first post I’ll show a demo.
About how can make a relay with Wowza Server from a Shoutcast server.
Demo with JWPlayer
[pro-player width='182' height='120' type='rtmp' fullscreen='false' autostart='false' streamer='rtmpt://ec2.wmconsulting.info/shoutcast']radiostation.stream[/pro-player]
You can improve the user experience, provide a On-line Player in Flash, and running over the port 80 with RTMP protocol, this way don’t have problem with the firewall.
Don’t hesitate to send your feedback or question about that.