"Hoodoo Butte pictures" - How the page is created


Overview

This page describes how the "Hoodoo Butte pictures" page is created. Along the way we'll include some background to help explain why and how some things were done.

If you are inspired to create a similar web page, we can offer some additional advice. A lot has been learned along the way related to basic hardware operation, troubleshooting, and what it takes to keep something like this running. It's been an interesting education, and here's an opportunity to learn from our mistakes and experiences.

The weather station on Hoodoo Butte began operation in November 1998. Not long after its web page became available there were questions about "What's happening up there?". Interpreting numbers and trends for temperature, humidity, wind speed and direction wasn't satisfying. The saying that "A picture is worth a thousand words" suggested an answer: Provide a picture. A single camera was installed, and the adventure began.

In about 2004 the sponsoring organization (Santiam Pass Ski Patrol) expressed an interest in an experiment, to be able to see what was happening at various places around the ski area. "Situational awareness", or knowing what was happening, would be very useful to their radio dispatcher when things got busy. Cameras were gradually added, and these views were also made available on a web page.

The system developed prior to 2010 was completely "analog", similar to a closed circuit television (CCTV) system. Video signals from various cameras were brought to a central location, where they were digitized for the internet pictures. The signals arrived in several forms, including raw NTSC video on coaxial cable, modulated as channels on the local cable TV system, video modems using twisted pair ("phone grade") lines, and a 900 MHz radio link. Image quality varied, often suffering from problems related to the transmission channel.

In the Summer of 2010 we joined the digital age motivated by many factors, including image quality, flexibility, and cost. Most of the infrastructure for gathering and processing video signals was replaced with digital and wireless hardware. While the cameras mostly remained the same, where the video is digitized, and how information is moved, was radically changed.

Here's how we get the web camera views to you:

It should be noted that along the way, considerable help has been received from a wide variety of sources:

This has been an interesting example of synergy, where the result is so much better than what any one person could have created alone. My thanks to all, and hopefully it will continue to improve.


Analog versus Digital Video

Here's a quick comparison of advantages and disadvantages for these technologies as far as capturing and transporting images.

Consideration
Analog
Digital
Cameras CCTV or "Security" cameras are cheap and widely available
- Resolution limited to about 640x480 (0.3 Megapixel)
Network cameras (fixed & pan/tilt/zoom) are expensive
- Mutli-megapixel models are available
Video encoders are expensive (but offer flexibility)
- Works with analog cameras
Digitizing the Image Is done after the analog transmission path Is done before the image is transmitted
Image quality Degrades with every link
- Power line noise can warp the picture
- Static adds "snow" to the image
Once captured, digital information does not degrade
- Communication links have error detection & correction
Wire-based Signal Transmission Wire might be leveragable from existing infrastructure with modest image quality results
- Long runs of high quality coax are expensive
Wire (CAT5) is limited to ~100 meters (330 feet)
Wireless Signal Transmission Wireless hardware is specialized & uses expensive instrumentation to troubleshoot
- RF units get expensive quickly for ranges of hundreds and thousands of feet
- Licenses may be required with some units
Wireless range can exceed one mile with modest hardware
- Hardware is becoming easier to get, somewhat expensive, usually requires no license
Flexbility One view at a time per cable (or channel) One cable carries all traffic
Noisy & Weak Signals Can be viewed Doesn't work


Getting the Pictures - Cameras, lenses, and more

A pure digital video network would use a "network camera" to capture a view and share it over the network. These cameras contain the functionality of both a camera and web site file server, and can share either live video or static ".JPG" pictures over a network. These cameras tend to be somewhat expensive.

Once configured, these cameras can be connected to the network wherever there is an open connection. They can be remotely controlled for virtually everything but aiming, focusing, and cleaning the lens. We have used:

Pan-Tilt-Zoom (PTZ) network cameras are available. They can be remotely controlled so you aren't stuck with a single view. Unfortunately the price for a camera and housing which could survive and operate in our environment is beyond the current toy budget. We have looked at:

Ordinary "security cameras" can be used when combined with a "video encoder". The video encoder takes the camera's NTSC analog output, digitizes it, and provides a network interface equivalent to a network camera. Some video encoders can handle multiple video inputs, so adding another view can be as simple as installing another camera.

Most of our pictures come from the same cameras we used with the analog system. This made the shift from analog to digital much less expensive, and yields another benefit: Cameras occasionally have to be replaced. In an environment with a lot of bright daylight, some cameras can go "color blind" or get an image "burned" into their sensor. (A view of the sky looking toward the mid-day sun is especially hard.) It's a lot less traumatic to scrap a security camera than a Network or PTZ camera.

Most of our cameras are color CCTV or "security cameras", with a horizontal resolution of more than 400 lines. Image sensors are usually 1/3". The camera body accepts a "CS" mount lens.

Beware that cheap cameras are most likely to go "color blind". Many have been tried, and eventually replaced.

Lenses are "zoom" with an "auto-iris". The zoom lens allows you to fill the camera's field of view with the scene of interest. For a camera with a 1/3" sensor, a 5-40mm (or 5-50mm) lens is adequate for most views. Even a 4-12 mm Varifocal lens is preferrable to a fixed 4 mm lens.

The auto-iris feature allows the camera to handle a wide variety of scenes and lighting situations. In our area, over the course of a year, the view may range from dark volcanic soil on a rainy day to a snow-covered landscape on a sunny day.

Focus is manually set, as the views are usually of objects at a relatively long distance.

Cameras are often supported from above, especially when sharing a window with people. If hung from a shelf above the window, the camera and associated hardware are "out of the way". For aiming a camera, a "ball and socket" mount allows great flexibility to adjust in roll, pitch and yaw. This type of mount should have some form of protection against the camera from getting banged out-of-alignment by other users of the window.


Moving the Pictures - A wireless video network

In a digital system pictures are captured at the camera and then transferred (on demand) as files over a shared communication network. It's very similar to a wireless LAN installation in your home or business, except for the environment in which it works: Links between units are measured in fractions of a mile, and it's expected to survive if not continue operating through lightning storms, power outages, and winter weather in the mountains.

Our network is a chain with four links between five sites. This was determined by where we wanted cameras, available places to put hardware, and line-of-sight between locations. Wireless "bridges" provide the connectivity between sites.

One advantage of wireless is that the transmission medium requires no installation, it is free, and it tolerates the occasional lightning bolt or gnawing rodent more gracefully than copper wires. The "gotcha" of wireless is that there must be a clear line-of-sight for reliable communication: Snow-laden trees, people, machinery, rock outcrops, snow drifts and rime icing may put a communication link at-risk.

Most of the sites have two wireless links, which operate on different channels. This ensures that one link's transmitter does not deafen the other link's receiver. The sites are far enough apart, and transmission paths are spatially separated enough, that we only used two channels for the whole system. This conserves the radio spectrum for other users.

The wireless unit we use claims a maximum range of three miles. Our path lengths range between 0.4 and 0.8 miles. As a result our transmit power levels can be less than the maximum (e.g. 100 mW or +20 dBm), which should give the RF transmitter a longer service life.

Each bridge's link is encrypted and operates in "Point-to-Point" mode. This provides the equivalent of a very long and secure CAT5 cable connection between two sites, without cable's range limitation of ~100 meters (~330 feet).

At each site there is a multi-port "switch" which serves as a central point where everything on the video network makes its connection. Each unit, be it a "bridge" box, computer, network camera or video encoder connects to the "switch" with its own CAT5 cable. An open port is useful for a visiting laptop computer, as the whole network can be accessed and checked-out.

A five port switch is typically adequate for sites in the field. An eight port switch was installed at the lodge, where a network thermometer and several computers are permanently connected to this network.

The wireless video network is not connected to the public network. This simplifies security issues, and allows dedicating the network's capacity to handling video traffic. It also avoids impacting the public network's bandwidth with our traffic. One disadvantage of this arrangement is that no "live" video will be available to the public internet. (This could change, but it would require the ski area to handle the network traffic and security implications. It's not currently reasonable to ask for this.)

A "visiting laptop" can join the video network if it's given a suitable IP address for the network. Connect a CAT5 cable between the laptop's network connector and the video network's "switch". Using a standard web browser you can access any camera if you have its URL. With camera-specific syntax in the URL you can access either "live" video or "static" JPEG pictures. A local HTML page can simplify access, allowing users to access a view simply by clicking on a label or thumbnail picture.

A computer capable of "surfing the internet" can also have long-term access to the video network by adding a Network Interface Card ("NIC"). The NIC is connected to the video network's "switch" box. When the computer has been configured with an IP address on the video network, its web browser can seamlessly access URLs on either network.


Sharing the Pictures - Programs & Files

With the cameras, video encoders, wireless video network hardware and computers in-place, here's how we automate the system to gather, organize, and upload files (including the pictures) to the web site.

One program is used to coordinate the execution of a collection of other programs. These "other" programs each do a particular function, like fetching JPG files, renaming them, creating an HTML file, or uploading all of these files to the web site.

By keeping the programs small and simple they are easy to write and maintain. Improvements are easy, which is welcome when a program may go years between changes. Well-commented source code is also handy to help the infrequent editor understand how the program works.

cron.exe
The automation is choreographed by a program named "cron.exe". Inspired by a Unix program of the same name, it coordinates execution of other programs with one second resolution. It was locally written in the "C" language.

"Cron" runs continuously, while the programs it calls terminate when they are done. It's a way of coordinating a constellation of simple programs to accomplish a more complex mission. Each program is executed at a particular time, and in a specific order. The process is currently repeated every twelve minutes.

"Cron" sleeps in the background until it's time to start the next scheduled program, so the computer's memory usage and processor loading is minimized.

For more details see the weather station's description of "cron.exe".

cURL.exe
"cURL.exe" is a command line tool for transferring data with URL syntax. We use it to pull static ".JPG" files from each camera, and then save them in a local file. If executed manually, cURL is given the URL and local file name as command line arguments. A ".BAT" file can contain multiple cURL calls, allowing all views to be captured by the execution of one command.

"cURL.exe" is "freeware", available from a download site on the internet. Curl compiles and runs under a wide variety of operating systems.

On the video network each camera and video encoder has a unique IP address. With a little more information the syntax for a URL can be assembled to access each specific view.

The file name is where the contents of the URL will be saved on the computer. Once captured, these files can be operated upon by other programs.

"fetch_the_views.BAT"
This ".BAT" file contains multiple "cURL.exe" calls to capture a hardcopy of every available camera view. Each call is a single line command, to a specific camera (or video encoder and video channel), to acquire a specific picture. Our .BAT file contains lines like:

"web_camera.exe"
This program assembles the "Hoodoo Butte pictures" web page. The HTML-formatted page is assembled from a combination of hard-coded lines, interpreting configuration file information, and inclusion of a text file at the end. This allows the program to do the tedious HTML formatting, while allowing the site maintainer to edit a simple text file to pass along comments.

An interesting addition to this program was to quit updating all the views before it got too dark to see anything. Some views could have interesting activity at night (e.g. roads & parking areas), but most do not. This was done by always saving camera views to a temporary file name, and then renaming the file to the name used on the internet when appropriate. Updates stop when the program calculates that the sun is more than six degrees below the horizon, so a colorful sunset might be the picture retained overnight.

This program was locally written in the "C" language.

"weather4.htm"
This file is what "web_camera.exe" makes, and is the HTML page your web browser fetches when you click on the "Hoodoo Butte pictures" link. It is created anew for every upload since it contains a date/time stamp for when it was created. The pictures shown are reduced-size images of the individual JPG files. Each picture has a hyperlink, so you can click on it to see the full-sized image.

"FTP_frequently.BAT" and "FTP_frequently.TXT"
This pair of files work together to use File Transfer Protocal ("FTP") to upload "weather4.htm" and its JPG files to the web site. When they are done the web page has been successfully updated.

The ".BAT" file looks like:

The ".TXT" file referred to in the BAT file contains specific FTP commands, and looks like:

The web site we use is hosted by the ski area. One little gotcha we encountered is that the "Front Page" application used by the area to maintain their web pages does not share well. It changes ownership and file permissions for all files on the web site, thereby making it impossible for our FTP session (from a different login name) to overwrite our files. Establishing a separate area on the web site was necessary before we could upload new files.

It should be noted that having a high speed "T1" line has done wonders for the ability to upload files to the web site. In the days of a dial-up modem, sending the weather page and one picture update could take several minutes. That can now be done in seconds.

That would be the end of the story under normal circumstances. One example of the type of information in the "advice" web page is a work-around to an unexpected problem:

"FTP_watchdog.exe"
Experience showed that the FTP "BAT" file sessions would occasionally "hang", perhaps due to network loading or outages. When the hung sessions failed to terminate, control was not returned to the "cron" program and further data collection and processing was blocked. For a while these "hangs" were manually cleared when the system appeared to be frozen. Some blockages took hours before they were detected and fixed.

"FTP_watchdog.exe" is now started just before any FTP session is attempted. It first "sleeps" until the FTP session would be expected to be done, and then it wakes-up, looks for and "kills" any FTP session it finds running on the computer. It usually doesn't find an FTP session, but hung sessions are now found and cleared in less than a minute.

The program keeps a log file, noting when it ran and what it did. While usually boring, the log file has shown there to be times and days where the FTP watchdog has been very useful. It provided a major improvement in system reliability.


Feedback regarding this or related pages can be emailed to "".

2011 01/28 - First-pass edits almost complete. Some hyperlinks need testing & fixing.