Bob's Adventures in Wireless and Video Headline Animator

Thursday, April 28, 2016

The Hidden Costs of Body Worn Cameras…

Citizens, politicians, civil rights advocates, prosecutors, and police officers want body worn cameras (BWC's). They are an effective tool for law enforcement, providing reliable evidence and deterring bad behavior by both the public and the police.  However, deploying cameras and implementing an effective policy can be difficult both politically and technologically.


A little background

The current generation of body worn cameras function basically like any video camera:  someone has to push the button to turn the camera on and off.  That someone is the police officer who wears the camera.  The video is stored on an SD Card or internal flash, and then downloaded by the officer at the end of his/her shift to a storage repository.  We refer to this as the “dockable” camera model.
In a later article we will address the benefits of next-generation wireless (LTE/Wi-Fi) cameras which are always connected and how they change everything, but for now we will focus on dockable cameras.

The difficulty with body worn cameras is striking the balance between privacy rights, cost of operations, and the collection of meaningful evidence.  Legislators, police departments, police officer unions, and other interest groups all try to draft policy that strike this balance. Generally the result is a long list of rules dictating when the officer needs to turn the camera on and off.
In most cases the resulting policy is a pragmatic balance that allows police to record most interactions with the public where evidence needs to be collected, and where the camera is off when privacy is a concern or when nothing relevant is occurring. 

A typical policy would have the officer activate the camera when they are dispatched for an incident, when they believe a crime or infraction may be in progress, or when they believe a life or property threatening incident is occurring.  It would also have the camera turned off when officers enter a private property without a warrant or without the prior permission of the resident or property owner; when a sexual crime has occurred and the victim requests privacy; when a minor is being interviewed in the absence of parental consent; and similar situations. The officer would also have discretion to turn the camera off for his/her personal privacy.  Effective policy implementation relies on officer discretion.

Then there is the issue of retention, which generally keeps all recorded video for a minimum 30-60 days, with incident-relevant video kept for longer periods, from 1 year to forever.

To oversimplify the positions of various interest groups1 :

  • Privacy advocates want the least amount of recording possible, only recording evidence of actual crimes where only the perpetrator and individuals directly involved are in the field of view. They want the video retained for the minimum time possible, with complete redaction before being released.
  • Social Justice groups want only recording of police actions, exclusive of the public, unless there is evidence which would exonerate someone.  They want evidence of police wrongdoing kept forever, and the rest immediately deleted.
  • Police officer unions want to record only criminal evidence, leaving out anything which might be taken out of context and result in disciplinary actions. Once the video is recorded, police officers  want the right to edit it, then determine how long to keep whatever they think is relevant.
  • Prosecutors would like everything recorded, giving them the ability to select what evidence becomes part of the record.  They would like to keep everything, but only release what they decide is relevant.
  • Courts prefer continuous recording of everything. And everything should be kept forever.
  • News media wants recording of all sensational events, with no redaction. They would keep the video themselves forever once it is released.
  • IT departments and city budget managers would like to record a fixed, predictable, and minimal amount of video, in order to reduce infrastructure costs. They actually drive the pragmatism of the policy.
  • Service providers want to record everything possible and store it forever.

The results of these divergent interests on policy vary dramatically, from a few minutes of video per shift on one extreme to hours of video on the other.


Mathematical Limitations

Let’s assume that all of the interest groups compromise and arrive at a policy right in the middle with the result:

  • 2 hours of video per officer per 10 hour shift is recorded.
  • Cameras record at the relatively low resolution of 720p (1280x720 pixels) at 30 frames per second.
  • The resulting video files are about 2.472 Gigabytes per hour, or 4.944 Gigabytes per shift.
  • All video is saved for 30 days.
  • 6% of all video is deemed to be potential evidence and is retained for 1 year.
  • 2% of all video becomes part of a case file and must be retained for 5 years.

Then let’s use an example of a mid-size agency with 60 officers who work 3 shifts per day with 20 officers per shift, each officer with his/her own camera. (Yes, the officers get no time off in my example):

  • 20 officers per shift X 2 hours of video per shift X 2.472GB per hour of video X 3 shifts per day = 297GB of video per day. This is how much needs to be stored and/or transferred to the cloud each day.
  • 30 days per month of all video = 8.7 Terabytes of video per month (rolling)
  • 6% of video per month is potential evidence = .06 X 8.7 TB = .522TB per month
  • 1 year Long term storage of potential evidence = .522TB x 12 months = 6.264TB per year (rolling)
  • 2% of video per month becomes evidence = .02 X 8.7TB = .174TB per month
  • 5 year Long term storage of evidence files = .174TB x 60 months = 10TB per five years (rolling)

Once you have recorded this video, you have to store it, protect it as evidence following chain of custody rules, make it searchable, and provide it to various constituencies in either raw or redacted from to protect privacy.

Regardless of how you store your video, you will need to purchase the cameras, which cost from $400 to $1,000, depending on the supplier. Then you will need a docking station, which can cost an additional $50 to $200 per port depending on the supplier.  These costs are fixed for both local storage or cloud storage architectures.

This is where a decision has to be made:  do you buy local storage, or do you use a vendor solution in the Cloud.


Local Storage

The math to determine the cost of local storage is pretty simple.  You need to buy a server, disks and tape, provide power, keep it maintained, and provide support.  It needs to be secured, following CJIS2 standards (for both physical and network security).   

While these may seem onerous at first, they are not unlike traditional evidence chain of custody management. Not transiting public networks or using a hosted facility means greatly simplifying the implementation of a complaint facility.

Moving the video from the camera to a dock to a storage system on site is straight forward. Everything can be on the same LAN and is local, so you have plenty of bandwidth.

This could be covered with a single PC Server with 12TB of RAID disk storage, an LTO (Long Term Offline) tape drive, 4-2TB tapes, and software supporting transfer, storage, search and redaction management.  Let’s say this whole system costs $40,000 in capital and about $5000 a year in recurring software and maintenance. Over 5 years the total cost of ownership is about $65,000. Allocating this to each camera it works out to be about $1083 per camera, or about $18 a month.

Of course you need an IT support engineer to maintain the system.  But this will not be a full time job and in most cases can be provided through existing resources.

You need to add in the cost of power and air conditioning as well, which could run about $100 a month.
No other infrastructure is required.



Cloud means using shared servers hosted at a remote facility.  This can reduce the labor and support requirements from IT, although not completely eliminate it, as the cloud service provider maintains the servers, storage, and support infrastructure at their facility.  For small agencies this may seem to be a very attractive option.

Your end to end network, reaching from your facility to the cloud hosting facility, must be CJIS compliant.  At present there are very few such facilities.  

Pricing for cloud storage solutions are advertised from about $79 per month per camera with unlimited storage.  But there are typically additional service fees which can add up to well over $100 per month per camera.  This adds up to $6,000 per camera over 5 years.  Compare this to buying your own storage solution at $1,083 per camera – what would be more affordable to you?

In some cases vendors set a maximum storage allocation per month which is covered by their base service cost.  Once this allocation is exceeded, additional costs (from 6.5 to 12 cents per gigabyte per month) are added.  A number of agencies have been surprised by these charges.  In some cases these charges are waived in the first year and then become payable after the agency has built a large repository of evidence.


How Secure Is your Data?

The CJIS guidelines provide great guidance on implementing security for video evidence which transits the public Internet, exists in shared storage environments and leaves the custody of the agency which created the video.  There is no doubt that these guidelines are well thought through.

However, as we all know, hackers seem to find ways into just about every facility, and CJIS data centers represent an attractive honeypot of data for them.  With all government agencies sharing one massive data center, it is incredibly attractive.  Once one server is compromised, the entire facility is at risk.

Further, the legal admissibility of cloud stored video evidence is still an open topic.  Often, the physical location where the data resides may not be known.  It may have been moved from one jurisdiction to another even though it never left the service provider's network.  Challenges to data access, data tampering, etc. have not yet been vetted through the courts. 

Keeping data local eliminates many of these issues, provided due care has been taken to physically and logically protect evidence files.  A well-engineered local storage system which follows CJIS guidelines and is kept off the Internet (air gapped) may avoid much of the current risk of cloud storage.

This is not to say that cloud storage is any more or less secure than local storage, but there are many unresolved issues with cloud that will only come to closure over time.  It may be better to wait on cloud.


Broadband Bottleneck

A basic assumption to use cloud services is that adequate broadband connectivity exists between the police facility and the cloud facility.  This is where police agencies can get bitten, hard.

Let’s go back to our storage requirement of 297GB of video per day.  A transfer of this video evenly throughout the day equals 12.375GB of video per hour.  To do this you will need a dedicated upload speed to the Cloud service provider of 33Mbps3.  That is UPLOAD speed.  It needs to be DEDICATED, not shared.  And that is persistent, running 24 hours a day, 7 days a week, 365 days a year.

Most broadband connectivity is asymmetric with the download speed being the focus. Upload speeds are typically much lower.  Higher upload speed requires the purchase of high cost business services such as 1Gbps Ethernet.

Based on the most recent FCC Measuring Broadband America4 report for 2015, the mean upload speed for consumer services is about 5Mbps:

Click to enlarge
So generally, you will need to have an Ethernet Virtual Private Line (EVPL) service or a dedicated Metro Ethernet connection between the police facility where the download docks are located and the cloud storage provider location. This type of service can cost thousands of dollars a month and can take months to years to install depending on your location.  This cost has to be added into the calculation of your cloud solution.

The other consideration is that you need to be transferring video continuously.  If there is an interruption in service, you will need to buffer your transfer until the service is restored, requiring temporary local storage, and more bandwidth to allow the additional buffer upload.  Most cloud service providers overlook this critical feature.


Inadequate Bandwidths Impact on Policy

Since most police agencies cannot afford the broadband connectivity required for the cloud, and they cannot afford to store everything in the cloud, they have to compromise policy.

Rather than recording all of the events that both the public and the police believe is appropriate, police opt to record less, record at lower resolution, record at lower frame rate, or store for a shorter period of time.

Using local storage (or maybe a hybrid model which only puts a subset of critical video files onto the cloud) allows you to store everything you should without compromise. Why bother with body worn cameras if you don't actually use them the way they are intended to be used?


Searching, Viewing, Redacting and Exporting the Data After Capture

Don’t forget that once your video is stored, you will need to quickly access it, search it, append it, redact it, and export it.  So, you should also evaluate how much bandwidth this will require, as well as what outbound data charges your service provider may impose.  Often vendors charge for outbound bandwidth.  Depending on your application architecture this can be significant.


End of Contract Blues

One of the unmentioned problems encountered by police agencies is moving their video data at the end of their service contract.  At the time that a contract is negotiated vendors will extoll the virtues of their proprietary video format for storage which improves compression, reduces storage requirements, etc. This may be true... until you need to extract your data from their repository.

The conversion of one file format and compression technology to another is called transcoding. Unfortunately this process can break the chain of custody for video files.  So, at the end of your contract you may need to keep using the vendors file tools in order to view the evidence.  Or you may be prevented from importing your video to a competing storage technology.  Of course you don’t find this out until after it is too late and all of your data is in a proprietary format.


Plan for Success

When planning a body camera architecture first agree on your policy.  Make sure you understand how it will be implemented as procedures.  Then determine how much storage will be required based on the policy requirements.  Look at the total long term costs over a three to five year period.  Be sure to allow for broadband connectivity costs, buffering hardware at your locations, cost of download and transcoding at the end of your contract.  Then make a decision whether you want to store your data locally or in the cloud.

Understanding how technology may constrain your policy will save you headaches now and in the future.


Shameless Plug

At HauteSpot Networks, our complete body worn video solution allows you to choose the right system architecture to meet your needs. Our HauteVIEW 100 camera is one of the best dockable cameras on the market today with industry leading features which are not found on many competing products.

The HauteSpot Dock Server is a simple, reliable and cost effective platform to which the HauteVIEW 100 cameras can download, process and store their video simply and easily. It can be ordered with hard disk storage, tape storage, or removable disk storage to accommodate whatever local storage requirements you may have. And the HauteSpot Dock Server can be used with either one of our two different evidence management software systems:

  • HauteVIEW Docking System software is a stand alone software solutions which runs on the HauteVIEW Dock Server and handles all of your basic transfer and storage functions right on the dock itself or to user provided NAS storage. It is a low cost solution which is perfect for smaller agencies on a limited budget.
  • HauteSpot Evidence Case Manager is a comprehensive evidence management system with is designed to work with multiple HauteVIEW Dock Servers, disk storage arrays, tape library systems and private or hosted cloud storage. Evidence Case Manager transfers video from cameras to the dock, from the dock to long term storage, provides robust searching, appending of additional meta data, and automatic redaction. It can be also used with third party systems like Computer Aided Dispatch, Interview Room Recording Systems, Records Management Systems and even Video Management Systems. Evidence Case Manager is your one comprehensive solution for all your evidence. Your officers will like the simple to use interface. Your IT manager will like the robust management features. And your Chief will like the low total cost of ownership.

And just wait until we release our new wireless, always connected HauteVIEW 200 cameras!

Friday, December 25, 2015

A Story of Giving

Back in the early 1990s I lived in Sri Lanka as the General Manager for American President Lines. It was an adventurous time for me and there were many things learned. But one of the most memorable was the friendship which I built with an unassuming man who worked at the Horton Plains National Park named Kulasuriya.

The Horton Plains is a truly amazing place of unbelievable beauty. It is filled with unusual animals such as white monkeys, an endless variety of colorful birds, deer, snakes, and an occasional leech.

On one trip up to see the park we happened to come across a short man sitting on the side of the trail watching monkeys in the trees and drawing with a number 2 pencil on some old scrap paper. We stopped to watch him work and he really was pretty good. His hands were hard and caked with dirt and his tools were really poor. His pencil was sharpened with a knife and was whittled down to not much more than a stub. And his paper was stained and torn. It was clear he was using the discarded trash from the rangers to make his art.

We stopped and sat down next to him and started to talk. We learned that he lived in the rain forest and that he liked to draw when he was not working. But there was more to him.

The park is guarded, and trails maintained by rangers who live year round in a small cabin near the entrance to the park. Of course there is someone who has to clean and maintain the cabin for the rangers. This job has the title of Peon, and is the lowest caste of jobs in Sri Lanka, which has a similar, although less rigidly enforced, caste system like India. Kulisuriya was the Peon of the ranger station. He was the lowest of the low economically. 

Kulasuriya was born and raised in Moratuwa, south of the capital of Sri Lanka, Colombo, where I lived. His father has passed away some time ago, so Kulisuriya was responsible for caring for his mother and younger sister. His salary of about $30 a month was what kept the family alive.

On a good day with light traffic it was about 90 minutes by car to drive there from my house. On the bus, it might take 2.5 hours. From Colombo up to Horton Plains was another story. By car it could take 4 to 5 hours. But by bus it could take 8 or 9 hours. Kulasuriya would travel back and forth from Moratuwa to Horton Plains and back once a month so he could check in on his mother and sister, make repairs to their meager home, and enjoy two days off, his only non-working time.

Kulasuriya and I talked for about an hour and he told me about his aspirations to become a wildlife artist. However he needed to care for his family and buying proper drawing materials was well beyond his means.

On my next trip home to California, I stopped in at an art supply store and purchased some colored drawing pencils, a clipboard, and a few packages of drawing paper. It was a small gesture, but I thought that maybe Kulasuriya might be able to get started with some proper supplies and maybe sell some of his drawings to other tourists in order to start making a living. When I returned to Colombo I sent the supplies up to the rangers station so Kulasuriya could use them.

I forgot about him for a few weeks.

Then one day Kulasuriya showed up at my house in Colombo. He had traveled all the way up from Moratuwa to Colombo by bus in order to thank me and deliver to me a small gift. I was very surprised, but delighted to see him. However my housekeeper and driver both looked at me like I was crazy for letting this poor little man in the garage, let alone into my house. They were shocked that I would allow a lowly peon anywhere near my home. I looked past this and invited Kulasuriya in.

He gave me a small package, wrapped in newsprint. He told me it was something for me. I unwrapped it and was surprised to see a small ceramic bunny rabbit. It was not particularly beautiful. There was nothing especially unique about it. It could be purchased at any shop for maybe 30 rupees ( $1). But coming from Kulasuriya, it was amazing. Here was a man who had to feed three people on $30 a month. A man who traveled 8 hours on a bus so he could see his mother and sister once a month. A man who had nothing.

I could not let this go, so I offered to give him a ride back to Moratuwa in my car, rather than have him endure another 2.5 hour bus trip home after coming all the way up to see me. Believe me, taking a bus in Sri Lanka is nothing like the air conditioned, comfortable metro transit that we have in the USA.

My driver, Fernando, was furious that I was going to make him drive Kulasuriya, a peon, and I all of that way. He tried to make excuses. I said I would drive him myself. Fernando relented, realizing that my driving in Sri Lanka, at night, was really dangerous. This was the time of the LTTE, military roadblocks everywhere, and extortion of foreign nationals who got into traffic accidents was very common.

So we set off into the dark wet night to deliver Kulisuriya home.

Home for Kulisuriya was a small, one room corrugated metal shack with a dirt floor, a single wood chair, a small table, three bamboo reed mats to sleep on, a dresser, a small cabinet, and a single light bulb hanging from the ceiling. There was no radio, no tv,  no wall coverings, no curtains. Plastic sheets hung over the windows. Cooking was done over a fire in the corner of the shack, ringed with stones. Outside was a round well, with a rope and a bucket. I didn't ask where the toilet was. Hopefully it was not next to the well.

Kulisuriya invited me in and asked me to sit in the chair. He introduced me to his mother and sister, neither of whom could speak English, and my Sinhalese was pretty bad. But I could see how excited they were that an American business man was in their house, invited by their brother. I also noticed that the neighbors were all watching and whispering amongst themselves.

Kulisuriya then opened the cabinet and I could see a single cup on the shelf. I don't know how the family drank otherwise. But it was a clear glass tea cup. And next to it was a small tea pot. A sense of dread came over me, as I realized that he was going to make a cup of tea for me. He asked me if I would like some tea, and of course I had to accept. His mother went outside to the well and got a bucket of water. She brought it in and filled up a pot on the fire and we waited, idly chatting, while the water boiled. At least I hoped it was boiling.

After what seemed like an eternity, Kulisuriya poured the water into the teapot and added some loose leaf tea. We waited a few more minutes for the tea to steep, and then he poured it into the glass cup on the table in front of me. The water was brown and cloudy, and I tried to see if there was anything floating in it that did not look like tea leaves. I smiled a weak smile and politely took a small sip. He asked me how it was, and I said "delicious".

He then went back to the cabinet and returned with a package of cookies. Vanilla cookies is what the label said. My guess is that the cookies had been purchased some years ago for an honored guest that maybe did not arrive. The label was old and faded, and the price tag on the package was peeling off, as the glue dried up. That package of cookies must have cost them another 30 or 40 rupees, and I really did not want them to squander their treasure on me. But they insisted. Kulasuriya opened the package and placed two cookies on a plate, also extracted from the cabinet. He set them down in front of me and suggested that I eat.

I ate both of the cookies. The dry, stale, hard, crumbly cookies that had been sitting on that shelf for years. The treasure of a family who had nothing. They were offering it to me, in this moment. It was everything they had. Everything they valued. Everything that they could offer to a friend from a distant land whom they might not every meet again.

I was stunned in the moment. I was saddened that all I had done was buy them some pencils and paper. I became keenly aware of my own life of relative ease and abundance. It was almost like I was at communion in church. It awakened something in me about thankfulness, about gratitude. But also about pride and self sufficiency. Kulasuriya was not asking for anything from me. He never did. He explained with pride what he did to support his family. He offered these gifts to me, not with the expectation of repayment. He was giving me the most valuable thing he could. He showed me kindness and gratitude.

On the ride back home I had to stop and vomit several times and I was as sick as a dog for the next week or so. The price of my experience with my friend.

Kulasuriya mailed me some of his first color drawings done with his new pencils and paper. I bought him a more extensive drawing kit which included more pencils, pens, and acrylics.

Hostilities in Sri Lanka heated up. My daughter Zoe was conceived, and it was time to leave Colombo. I lost touch with Kulasuriya, but I still have the little ceramic bunny on the mantel over our fireplace. Whenever I see that little bunny I think about was giving really means, and how a little man with nothing touched my life.

When I get home I will update this blog with a photo of the bunny.

Thursday, February 26, 2015

There is Nothing Neutral in Title II Regulation of the Internet

It has been a long time since I posted something to my blog. But Title II regulation of the Internet in the guise of implementing Net Neutrality is such as big issue I just had to write something.

A Little Background

I started working with FidoNet back in about 1979 in high school and was enamored with CompuServe throughout college. I joined the Well and used my Hayes 1200 baud modem to connect to others to discuss technology, social issues and the like. But I knew that there was going to be a commercial aspect to this that extended beyond just being a substitute for telephone party lines. 
In about 1988 I started a TCP/IP based BBS system in Oakland I called TransTech. Over time I hooked up to NSFNET. I was given a class C network (which I would later have to give back). And finally I found my calling when I built one of the first wireless ISPs in the Bay Area providing coverage off the top of the 1200 Broadway building in Oakland to the newly decommissioned Alameda Naval Air Station.

I never once had to get a license, ask permission, adhere to standards of decency, or worry about what my customers were using their connectivity for. I provided a pipe through which data flowed. I paid upstream providers for larger pipes, and I never had time to worry about metering content. I knew that if I did meter content, there were lots of competitors ready to take my place. Pricing was set by the market. Service was set by the market. Customers chose the best service for the best price.

Fast forward to present. Pretty much the model of operation for the Internet remains the same. 
However, fear of something that never even has been a real issue prompted cries for government intervention in order to enforce Net Neutrality. But people forget that we live in a Laissez-faire economic system in which transactions between private parties are relatively free from government interference such as these massive Title II regulations. The market will punish those that filter traffic through competition. 

Wireless and over the top/virtual ISPs exist as a locally owned bypass around monopolistic cable and telco companies. This keeps them in check. But not for much longer. We have now empowered the government to regulate from top to bottom a utility which was functioning perfectly well with no regulation. Beware unintended consequences.

Net neutrality and Title II FCC Internet regulation are not the same.

Net neutrality is the concept of treating all packets equally, regardless of the source or the content. But from a practical perspective this is not realistic. There is limited bandwidth on networks. Traffic needs to be managed, prioritized in order for it to all get through. Every network manager will tell you that they apply QoS (quality of service) prioritization on their network. You have to, unless you have limitless capacity. Network operators should be free to define these rules. And customers should be free to choose which networks they want to use based on the quality of service they receive. Networks will evolve based on the type of QoS prioritization they apply. Customers will have choice in a free market.

Net neutrality can and is achieved through competition. A free and open market is what is needed. Maybe there could be some work on local right of ways in order to spur competition. But Title II is like putting out a match with a fire hose.

Title II is an antiquated set of government restrictions designed for the bygone telephone monopoly of 1937. Title II sets up regulatory bureaucracies overseeing a utility that wants to be free, not controlled. 
The first casualties of this regulation will be the local ISPs who provide the very competition to the large monopolies who the advocates of Title II were trying to reign in. So unintended consequence #1 is elimination of competition and choice.

The second unintended consequence will be taxation. Now that the government has control of the Internet it will want to wring out of it every last penny it can. Expect fees, licenses, use taxes, excise taxes, and the like to start appearing on your ISP bill shortly. And they will grow.

The third unintended consequence will be increases in the barriers to entry for using the Internet as a vehicle for free expression. With no license fees, no bureaucracy, no regulation, today anyone can set up a blog, a web site, or chat at a cost approaching zero. But the government will make it increasingly harder for people to do this.

The fourth unintended consequence will be censorship. The fairness doctrine. The seven deadly words. Hate speech, and other arbitrary rules will be applied to curtail the use of the "public" (read government regulated) Internet. 

Beyond this, I am sure their will be many other unintended consequences. Curtailment of innovation. Decline in infrastructure investment. Who knows.

Pirate Internet

There is one thing everyone can start planning to do. Buy an outdoor wireless router. Load it up with some mesh software like Freifunk Wireless, and create a local community network which is apart from the government run Internet. Using currently unlicensed radio frequencies, you can bypass the regulation which is coming. You can maintain your freedom. And you can create competition for the new Ma Bell or whatever we will call the super monopolies which this is sure to produce.

Tuesday, December 3, 2013

Carriers Move to Limit Public Safety Cellular Data Transfer

A customer of ours told us, and this has not yet been confirmed, that Verizon is moving to follow the T-Mobile model of cellular data transfer throttling for public safety. Basically you will pay for a 5GB transfer cap at full LTE 4G speed (measured in Mbps) at a fixed fee, then once you hit the cap you can still transfer but your speed will be reduced to 256kbps or less. I assume that you can still purchase a plan with a higher transfer cap, but it will cost you more.

Actually I use T-Mobile 4G with such a plan for my cell phone and it is a great deal. But then I am only surfing the net, checking my emails and checking in on Facebook.

For police and other public safety personnel who have become accustomed to unlimited transfer at full speed, this presents a big problem. Many of our folks in blue use IP cameras to stream video from a remote site over cellular. It is easy to install, it is available everywhere, and it is compact.

Unfortunately streaming video requires a lot of bandwidth. A typical 720p at 15fps camera will use about 1.7Mbps. That is about 13MB per minute, or 765MB per hour. So you can easily reach your 5GB transfer limit in about 6 1/2 hours. So if you need 24x7 streaming you have to compromise.

Many IP cameras support multiple streams for the same video and some have SD cards that allow up to 64GB of storage (83 hours of 720p at 15fps). So you can send one small stream over cellular at a low resolution and slow frame rate while storing locally. If you were willing to run at VGA resolution (640x480) at 4fps you could get an video stream for "situational awareness" in 150kbps. But that is not a very good image quality, it is slow and jerky, and it still will burn through your data plan at 1MB per minute, or 1.6GB per day. So you could run for about 3 days a month before you run out of data transfer on your plan.

Another issue with this strategy is that you will need to determine what the minimum data rate that your cellular link supports is. If you are in an area with only 3G and are on the fringe of the coverage area or in an area that is highly congested, you may only achieve 100kbps. So you need to set your camera at this low rate in order to assure that your stream gets across. Your camera cannot adapt.

And then you have to think about retrieving your video evidence from your camera. You have your cellular connection, but if you try to transfer the locally stored high quality video from the camera back over that cell connection, you are again faced with your data cap.

And what if you want a 1080p camera or a 5MP camera, or a 20MP camera, or faster frame rate. What will you do?


What you really need is a video storage and transmission system that was designed for the limitations of a cellular world.

  • A system which allows you to use the camera you want, at the location you want, regardless of what kind of power is available or how much space there is.
  • A system which can store weeks or even months worth of full frame rate, full resolution video on site and yet allow you to easily retrieve it over the air.
  • A system that can accept any USB modem from any carrier.
  • A system which maintains an outbound connection to your HQ location, but does not send video when no one is watching.
  • A system which automatically adjusts it's frame rate, resolution, and compression for the best possible live viewing given the bandwidth available.
  • A system which supports remote viewing using any device you have, be it a pc, iPhone or Android device.
  • A system with wide operating temperature and choice of storage media up to 1TB internally.
  • A system that supports cryptographic tunneling for security.
  • A system that gives you a choice of using a simple Windows remote desktop or web interface for management.
  • A system that is backed by a team of experts in IP wireless video.
  • And a system that is reasonably priced.

You want a HauteSpot Networks MVE system.

I could go on about all of the benefits of the MVE system, but the best way to decide for yourself is to try the system out. Call HauteSpot Networks to order a demo system. +1 (800) 541-5589

Thursday, May 16, 2013

The Power Paradox

Three years ago HauteSpot Networks introduced the microNVR which was and remains a valuable tool for video security. The premise of the microNVR is to be a small (as in "micro"), power efficient, computing device that has all sorts of connectivity and the ability to store and process video at the edge. In this case the edge means on a pole, in a car, along a fence, on a roof, out in the woods...anywhere you just cannot place a big computer.

For some time now we have been working towards a upgrade replacement for the microNVR. We need to increase it's video processing capability and its storage, while retaining the same general low power consumption and size.

Intel left us without a good roadmap. The Atom processor family has basically gone without an upgrade since 2009, while Intel focused its resources on building their Mobile Phone System Platforms, they let the Atom family languish. And in the Core product line there really were no viable options presented either. Most of their chips started at 25 watts or about 10x the power consumption of the Atom.

The Z550 Atom that we use in the microNVR runs at 2GHz and is a single core processor with two threads. The GMA500 GPU is capable of encoding about 30fps of 1080p video into H.264 streams. If you have 3 two megapixel cameras running at 10fps each, and are encoding the video or transcoding it for transmission, then you have pretty much consumed all of the capacity of the processor and GPU.

CPU Boss Shootout review...
I have no idea where they got their power number.
The Atom is 1/10 the power of the Fusion.
So we went over to AMD and experimented with their Fusion product family, which was presented as having great video processing capabilities with low power. What we found was that AMD had no better story than Intel. Their embedded processors started at about 17 watts and their GPU was designed for better performance, at least in theory, than the Intel Atom. The AMD T40N processor, which runs at 1GHz does have a better GPU for decode (like watching a movie). But that GPU only decodes certain codecs well. H.264 encode requires third party libraries (from a third party called Main Concept) that when we tried them did not work. We contacted AMD and they said that the GPU did not support H.264 encode but that later versions of the Fusion family would. So that meant that any video processing on the platform other than decode had to be done on the CPU in software, which, because the CPU was clocked at only 1 GHz was even worse than the ATOM.

So we waited, and waited, and waited. Finally Intel released a low power 3rd generation i3/i5/i7 processor family. The really good news is that these processors have Quick Sync technology built into them as part of the HD Graphics 4000 GPU. Quick Sync accelerates both encode and decode of H.264 (for applications that are designed to use it).

So we were excited to take a look. There is a lot of supporting documentation regarding the performance that Quick Sync brings to H.264. We expect that it will mean about 3x performance over previous generations of Intel Core processors. Or about 98fps of 1080p H.264 transcode with almost any member of the i3/i5/i7 3rd generation family (Ivy Bridge). However, running at 17W means that our system will consume somewhere around 23W versus the 10W that we consume today with the Atom system. This much power means twice as many solar panels, twice as many batteries, etc.

Then we got some even better news that Intel was going to release a new Atom processor in the second quarter of 2013 and that this new Atom (Bay Trail) would have Quick Sync in it and have the same HD 4000 GPU. Great, this is the CPU we want...well don't get too excited...

CPU World reported that Bay Trail was going to be delayed until mid 2014. This would have been a 3W chip that had all of the performance we wanted in the exact package size we need. But who knows when or if this will get released.

So we are going to punt. This probably means building a system for release in the next quarter based on the low power i3 or i7 processor. The system will be a little bit bigger and take more power. Then when the new low power Bay Trail comes out, we will jump back to that.

Unless Intel jumps around some more. Keep your fingers crossed.

Sunday, April 14, 2013

Getting through the hype at ISC West

It was interesting to walk the ISC West 2013 show floor and speak with vendors. There were two wild claims being made at this years show and misrepresentations of what vendors were showing that I think integrators need to be aware of.

Video Streaming Over 4G

Example of a matrix view of thumbnails on a VMS
The most outlandish claim came from one VMS vendor who was saying that their product could transmit 16 megapixel camera streams at 30fps and full resolution in 1Mbps of bandwidth over 3G/4G. This was just a flat our misrepresentation of what they were doing.

There is a huge difference between transporting 16 streams as full frame rate and full resolution and sending thumbnail views of camera streams. Let me explain.

A typical H.264 IP camera with a 1.3 megapixel resolution will compress and stream video in 25.6KB frame size. At 30fps, that is 768KBps or 6144Kbps or 6.1Mbps. This assumes some motion, etc, but is a general ballpark. By turning up compression, you can get this down to around 2.5Mbps at the cost of image quality (artifacts).

In a typical security configuration the cameras send a primary stream over a high speed local area network to the VMS server without concern for bandwidth, recording at best quality and frame rate. Sending a 2.5 Mbps, or even a 6Mbps stream, between the camera and the server is not usually an issue. However, sending from the server out to a remote client such as a smartphone over a narrow cellular link can be an issue.

Example of how HauteSpot MVE Connects
If you are using a LTE phone you may be lucky and get 8 Mbps or more down, depending on network conditions, but rarely is this sustainable over long periods. A more likely down stream speed is going to be around 1 to 2 Mbps.

If you tried to stream at full resolution, full frame rate and full image quality even one H.264 1.3 megapixel stream the video will have little chance of getting through in real time. You will need to buffer a long time (aka YouTube, Hulu, etc). Remember that most broadcast streaming is non-realtime and they use multipass encoding techniques to improve the quality and reduce the size of the streams they send.

Further, a 1080p monitor can display 1920x1080 pixels or approximately 2 megapixels at any given time. So sending any more data than this from the server to your client is really useless.

So what most VMS systems do is set a second stream from the camera to a lower resolution for live viewing. They are still recording the primary live stream at full resolution. When someone goes to the VMS from their cell phone, they first get a thumbnail of each of the cameras using the small substream and then, when you click on a camera to look at in detail, the VMS will shift from the small secondary stream to the primary full resolution stream. But even this may be scaled and transcoded on the server to fit the size of the screen of your device.

So, really what you are getting from the VMS to the smartphone client is a 2 megapixel or smaller transcoded rendering of your cameras. When you are looking at 16 cameras, you are not receiving all 16 streams at full resolution, you are receiving either a bunch of small substreams or a transcoded single stream representing all of the substreams in 2 megapixels. This may then additionally be compressed to fit into a 1 Mbps pipe. Think of it as a "image of your streams".

There are many VMS who do this and the technique has existed for years. It is a commonly known method that is well documented in the public domain.

What is missing from most vendors implementations are several critical elements which are addressed by the HauteSpot MVE system:
  • Inbound Mobile Ingest of Cameras - It is all well and good to be able to view your cameras remotely over 4G, but connecting your cameras to the VMS over cellular is a different story. This requires uplink speed, which rarely exceeds 500kbps and can be extremely variable. MVE dynamically adjusts the uplink speed to fit the available bandwidth. It is not preset to a low "live view" rate and size. If you have good bandwidth, you will get a better image.
  • Persistence - Most VMS have no ability to deal with the constant disconnects of cellular. Roaming between cell towers, changing IP addresses, and lost signal all contribute to loss of connection. MVE makes and keeps persistent connections that have been proven to work on cellular networks in the worst conditions (hurricane Irene, tropical storm Sandy, floods and hot weather).
  • Remote Record with Transfer - The HauteSpot microNVR allows remote recording at full frame rate and resolution at the camera source. It then provides a persistent connection to the MVE server and the dynamic rate adaptation which adjusts to changing available bandwidth. It also allows remote, chain of evidence transfer of the high resolution video from the remote site in batch.
There is a big difference between transmitting full resolution, full frame rate, full quality video over cellular and sending a transcoded or down-res version of video from megapixel sources. Know what you are getting.

VMS IP Video Decode Capacity Vs Record Capacity

As we walked the show we asked a simple question of a number of VMS vendors "How many 1080p 30fps live streams can you simultaneously display (assuming that they had a multiple monitor video card on their system)?" The answers ranged from 4 to 64. And the answers were coming from field application engineers, not sales people.

A VMS provides a number functions including recording video streams to disk, displaying live and recorded streams, providing search capabilities, etc. The answers that we got indicated that the vendors really did not understand the difference between storing video and decoding video for display.

Again, IP VMS systems receive incoming video streams from cameras. These are typically either H.264 or MJPEG streams that have a fixed frame rate, resolution and compression profile as set on the camera. Most VMS systems will receive the video stream and write it to disk. This function does not require any significant CPU or GPU functions, since it is really just network IO. The system is limited only by the bandwidth of the network as to how many streams it can support. Disk IO is generally faster than network IO.

Tom's Hardware Benchmark for H.264 encoding
Displaying a live or recorded stream does require CPU and/or GPU resources. A stream must be read and decoded for display. This is significant work for a computer.

The chart on the left shows just how much time it takes to encode 1080i video using hardware and software. This is an example of when you are reading a raw source and sending it to a client to view. Performance is similar for decoding. As you can see, using hardware such as the Intel Quick Sync technology built into Ivy Bridge based systems can significantly improve performance. Without Quick Sync, software encoding/decoding consumes approximately 25-50% of a quad core CPU for just one 1080p 30fps stream. Using high end video cards helps some, but again, they are targeted at single or maybe dual display systems, so they really don't try to decode more than what they are capable of displaying.

If you were to use the hardware acceleration of Quick Sync, then you could get 120 fps of 1080p
video encoded or decoded for playback. So the best possible case for displaying full 1080p at 30fps is 4 simultaneous streams. But this is using hardware. If you are using software to decode/encode, as most VMS systems do, then you are really down to about 1 stream. Which makes sense given that most VMS systems only have one display attached.

So the vendors that said they are able to simultaneously decode 12 or 64 streams are clearly not correct.

If you are writing a file to disk or reading a file from disk to and from the network, then yes, you can support many cameras. If you are trying to actually display the video, then the capacity is much much lower.

It is interesting that only a couple of products like Network Optix HD Witness or HauteSpot's MVE system take advantage of Intel Quick Sync hardware acceleration in order to increase performance and are designed for high definition video processing.

This is actually a good thing for companies like RGB Spectrum who make display wall processors that allow you to aggregate multiple server video outputs into a single consolidated display. Solutions like their QuadView HDx allow you to consolidate multiple video sources into one screen, so even if you can only get a single 1080p stream out of your VMS, you can combine it with other servers to create faster, full frame rate, full resolution systems. Watch for more on high resolution display, particularly 4K, when I review NAB 2013 in my next blog.

Sunday, April 7, 2013

Last Chance for Free Exhibits Pass to ISC West 2013

It is that time of year again...ISC West 2013 in Las Vegas. If you have not already done so, there is still time to get your exhibits only pass. Visit our free exhibits only pass page by April 9th. Then come by our booth 9139 to see some of our cool new products including:

HauteMOBiLE - In-vehicle high performance multi technology multi function router 


HauteMESH - High availability wireless routing


Integrated 8MP 180 degree camera - NVR - wireless router 

HauteSpot and Sentry360 have collaborated to create a new, completely integrated remote surveillance solution which combines a 8MP 180 degree camera, the microNVR, and the HauteMOBiLE router into a single, easy to install, low power, rugged system. The new system allows situational awareness video surveillance to be placed anywhere and powered by just about anything (solar, wind, battery, ac). Video can be viewed live, in high frame rate and high resolution over HauteSpot private broadband or over 3G or 4G cellular. The system scales to hundreds of cameras. HauteSpot will be presenting this new product at the Sentry360 Partner Conference on April 9th at the Flamingo Hotel.

Collaborate and WIN

Come by the HauteSpot booth 9139 and meet all of the HauteSpot Team, discuss your upcoming projects.

We also are running a show special through the end of April on the HauteMOBiLE router. Only $299. See our booth for details and applicable limitations.

We are also raffling off products. Come by and say hi.