Proprietary formats and compatibility

Most of the questions I receive about map making have to do with proprietary software. That is, while some software seems “universal,” the truth is that files produced in one version are not compatible in a competing version. The success, therefore, is learning to minimize proprietary formats.

The obvious…

The simplest solution is to limit products that only save data in proprietary formats. The government-funded GPS satellite cluster is normally interpreted in a standardized, public format established by the US Coast Guard called “NMEA.” The GPS receiver will read the satellite information and transmit NMEA lines as ASCII strings. Most important is the summary line, or ‘fix data’ line, known as the GGA line. This line transmits the time, latitude, longitude, quality of fix, number of satellites, horizontal dilution of precision, altitude, and some other information. Because it is a simple ASCII line, it is easy to write software that seeks out this transmitted line and parses out the needed information. The most common commercial GPS receivers are produced by Garmin and Magellan. However, to keep customer’s cash, Garmin and Magellan use mapping software that relies on proprietary transmitted information so you cannot use other mapping software and you cannot use other brands of receivers. In summary, make sure your software can read NMEA format. In fairness, both Garmin and Magellan produce excellent products and more and more third party software can allow for Garmin and Magellan input. Read the fine print!

In a similar circumstance, it seems that every GIS program uses its own proprietary format. Rather than regurgitate a litany of acronyms, let’s just assume any product you use will have its own file format. While ESRI shape files (explained in another blog) are the most commonly shared, not all GIS programs allow for shape files. My advice: if you are not sure, save your data as a text (.txt) or comma-separated values (.csv) format. These are more universally read.

The subtle…

Have you ever been frustrated because you thought you were using a standard file format and the program would not read the file? For starters, have you received a .docx yet? I recently worked on an interesting problem with a colleague of mine that was a model for subtlety. Todd wanted to use SOLO Field on a Nomad Computer for collecting GIS data. His background map was a trimmed image from a much larger air photo. Both the original and trimmed map were saved in the standard geotiff (.tif) format which is read by SOLO Field. However, every time he loaded the map, the program would crash. We thought the problem was that the file was too big (~30 MB), but that wasn’t the case. It ended up that he used an ArcGIS product to trim the image and for some reason, the new image could not be read in a non-ESRI GIS. He went back and trimmed the original image in SOLO Office and everything worked smooth. Moral of the lesson: just because the file format is on the software list, doesn’t mean it will work.

Is ESRI the only way to go for field GIS?

No. However, the ESRI shape-file format with the multitude of files is the standard for most industries and academia. So when you collect your data, make sure you can output as standard shape files, composed of the minimum triad of .shp, .sbx, and .dbf files. (The .prj is valuable if you have an assigned projection.) I should also chime in that ESRI continues to dominate the GIS market because they do make amazing products.

So why go with someone else? Try cost, ease of use, memory drain, and the fighting spirit to stick it to the man. Most software is essentially a trimmed down and specialized version of ArcGIS, tailored toward a specific industry. I have written previous blogs about the amazing and free MapWindow and also the very powerful Manifold GIS system. Of course, I continue to be amazed with the power of Google Earth…

Google Earth in the field? Yes!

…which leads me to today’s blog. Recently, I was doing field work in the Boundary Waters area of northern Minnesota. Without a doubt, the Boundary Waters is the most lovely mosquito and wood tick hatchery in the lower 48. But I digress. Our fearless field party got lost under my direction. After a while, all the ponds and mossy outcrops look the same. In desperation, I sent one of the folks (we will call him “CyberKnife”) back to the car to dig up my ToughBook and GPS receiver. I popped up Google Earth though we did not have an internet connection and was amazed by how much detail was stored in the cache. Years ago, I would use the program ExpertGPS for mapping because it had a deep cache, and I still find the program fantastic for flat, back maps and airphotos. I haven’t used Google Earth in the field because I didn’t think it would work. Well, it worked fantastic! The only issue was that the COM port assigned to the GPS was too high for Google Earth so that will need to be addressed.

OK, so what to use for collecting data in the field? The industry standard is still dominated by ESRI and ArcPad is there solution for data collecting. If you have a PC and want to plot lines or points on topo maps, I am still a fan of National Geographic TOPO. Need road, airphotos, and other layers? Well, jump on the Google Earth bandwagon. Once MapWindow allows for higher COM port numbers, I’ll recommend MapWindow for all ‘real’ GIS needs in the field.

(*As long as I can claim some bandwidth on the global web, I’d like to wish my pal Brian and the rest of the GIS crew on the edge of the northern California fires good luck. Soot-covered fireman get all the thanks on the local radio, but there are lots of other folks working long shifts behind the scenes.)

What is up with Bluetooth and why won’t my software run on new computers?

Recently, I have had several conversations with folks about the changes in Bluetooth. More to the point, Bluetooth technology has changed over the past few years so that software written for the older model is not compatible with the new model. That means, the software you used on, say, an Recon computer a few years ago might not work on a new Recon running Windows Mobile 6 with integrated Bluetooth.

The nitty-gritty

“Bluetooth” is simply a wire-free way to connect devices. While most folks are familiar with Bluetooth headsets or maybe printers, I mostly deal with GPS receivers. In most cases, you can think of Bluetooth as a replacement for a USB or serial cable. However, Bluetooth devices can communicate with several other devices simultaneously through a ‘piconet.’

The latest version is 2.1 with Enhanced Data Rate (EDR). This version came out in July, 2007 and superseded version 2.0 with EDR. The older version had a completely different construction (known as a stack) and, consequently, different transmission rates (though the same 2.45 GHz frequency) and protocol. That is why software written for the older Bluetooth won’t work on the new version.

Regardless of the Bluetooth version, there are three classes of Bluetooth that vary by strength of signal and power consumption. Class 1 is the high-end model and has a range of 100 meters. Class 2 is what is found in most computers and has a range of 10 meters. Class 3 is not common, but is the lowest power radio and has a range of 1 meter.

Was the change necessary?

Yes. I loved my Recon with Windows Mobile 2003 but I was frustrated with the Socket Bluetooth card. Socket is a great company, but the technology was tenuous to me, as a user. In fact, I wrote back then that Bluetooth would die soon. Once again, I was wrong. Various applets and updates came out to make Bluetooth more stable but I found myself appreciating the serial cable for my devices. With the latest operating system change, I find the Bluetooth very stable and have finally gone wireless with my GPS devices. What to do? Change the software. Kind of like when they ditched unleaded gas. You could keep adding additive to your fuel or buckle down and replace the valve seats.

What does the logo represent?

It made no sense to me until I dug deeper into the roots of Bluetooth. Harald Bluetooth was a 10^th century king of Denmark and Norwary who is noteworthy (at least in legend) for unifying tribes across Scandinavia. (Meanwhile, Hamlet whined about the castle and contemplated skulls while his sweetheart used poor judgement with herbs—but I digress.) The symbol is the Germanic runes for H and B forming a bind rune.

The Nomad goes on a fossil hunt!

Recently, three colleges converged in western Nevada for four days of field paleontology. Over thirty students participated in a series of field tasks that represented different kinds of field scientific work. The tasks ranged from classic description to modern collection techniques. For the latter, the students mapped a trilobite-rich area using a Nomad computer and SX Blue II sub-meter GPS from Walcott Scientific.

Students found the gear not too cumbersome to wear and use in the field.

Trilobites are extinct arthropods, related to modern insects, crabs, and lobsters. In the Cambrian period, about 500 million years ago, they ruled the seas. Deep water deposits of that age are rich in trilobite fossils. The field exercise was carried out in Esmeralda County, Nevada in glorious weather. The students divided into teams of four members and each team had a guide to the common trilobites. Then, they students scurried about the hills looking for fossil-rich zones. When they found a particularly rich area, the students logged their location with an SX Blue II sub-meter GPS that communicated to a TDS Nomad via Bluetooth.

Nevadia weeksi, common trilobite in the lower field area.

Data was collected as a point-based shape file in ArcPad, set against a topographic map downloaded from a state of Nevada GIS website. In addition to the location information, the students listed the dominant fossil genus and estimated the age. Both the Nomad and GPS worked without a hitch and there was plenty of battery life at the end of the project.

Back at the college, the data was transferred to MapWindow GIS and sent via Shape2Earth to GoogleEarth as a .kml file. This process took less then five minutes. Each student was then e-mailed the .kml file so they could see for themselves just how the fossil bugs were distributed in the field area.

This assignment, and similar ones I have done with my courses, really shows the students both how easy it is to collect data in the field, and how GIS adds value to a project. The Nomad proved an excellent tool in the Nevada desert with the bright screen and rugged design. While many researchers still use tablets and PCs in the field, the use of small PDA data collectors is more efficient on cost, weight, and battery life.

Quickie user guide to Windows Mobile

Your buddy / boyfriend / boss hands you a PDA for the first time and tells you to ‘figure it out.’ Where do you start? Can you understand your way around the computer? This short blog is meant to be a sort of triage to get you through a Windows Mobile device, like the Recon or Nomad.

It sure looks like Windows…

Windows Mobile look and feels like Windows XP even if the code below the surface is different. (Windows Mobile evolved from Win.CE so all the various operating systems share some similarities.) Many of the features, such as the Start button, Program list, menu buttons, etc., function in the same way so if you use a PC, it will be terra cognita. Here are the important differences:

1) When you click the X in the upper right corner, you usually only close the window, not the program like in Windows XP. You will need to close the program through a menu option or it will run in the background.

2) Use the File Explorer to find missing files, even when your files are on a CompactFlash or USB drive.

3) Most PDAs only have ‘USB out’ for synching to a desktop computer. That means you cannot plug in a USB device like a printer or flash (thumb) drive. The Nomad does have a ‘USB in’ option.

4) Can’t see the screen because it is dark? Try holding down the power button for a second or two and that should bring up the brightness. If you hold it down too long, you will enter a ‘soft reset mode’ which your probably don’t want to do.

My biggest advice…

Whenever you turn on a device, close a program, and just before turning off, go to Start>Settings>System>Memory>Running Programs to close all the programs running in the background. I just grabbed a Recon out of a drawer and checked and the following programs were running: ActiveSync, Internet Explorer, Wireless Manager, and File Explorer. Knowing our computers, I am surprised ‘Solitaire’ wasn’t running as well…

Finally, if you are using ActiveSync to send files back and forth, I recommend logging on as a “Guest” so you are not committed to a dedicated PC for file sharing.

Why can’t my Tablet-PC read my USB-connected GPS receiver?

I’ve written several times about COM ports and GPS receivers. Interestingly, in the last two days, I have spoken with four different folks about how this works, particular in regards to setting up a GPS receiver through a USB port on a tablet computer. It is valuable to understand how this works to avoid frustration in setting up software later.

What is a COM port?
A communication (COM) port streams data into the computer. Older computers had serial ports for this purpose. USB ports are not really COM ports, so you need to download a driver for the conversion. There are two facts about this process: you need a different driver for each different brand of GPS or other devices using the COM port, each USB port needs to be set-up as a COM port and each will be given a different port number. This COM port number will be used in any software that needs to connect to the GPS. (As a side note, you will also need to know the speed of the data stream from the GPS called the Baud rate.)

Setting up a COM port for an SX Blue II Sub-Meter GPS
The first step is to download the driver. I can’t post folders in the blog, but I put the driver on the Walcott Scientific website. This will comprise a folder with various .dll, security catalog, etc., files for the FTBI driver. The folder is zipped, so you need to unzip it. Then, place the folder somewhere you can find it, like the desktop. Now, power up the SX Blue II and plug it into the USB port you wish to use. The computer will open the “Found New Hardware Wizard.” No need for the internet, so click “No, not this time” and click “Next.” Then, choose “Install from a list or specific location (Advanced)” and “Next.” On this page, choose “Search for the best driver in these locations” and “Include this location in the search:” Use the Browse button to find the folder and click “OK” and “Next.” A warning box will appear that the USB Serial Converter has not passed the test. Click “Continue Anyway.” The driver is now installed and you will close the Wizard. But wait! Another Wizard will open and you will repeat the process. You’ve done it!

Finding the COM port
So, what is the COM port? You can use a ‘port sniffer’ like GPS Viewer which works pretty well for finding GPS devices. Alternatively, you can go to: Control Panel>Administrative Tools>Computer Management>Device Manager and click on the “Ports (COM & LPT)” to see the port number assigned to the SX Blue. This is the COM port you will use in your software. The SX Blue broadcasts at 9600 Baud, 8 stop bits, no parity.

Setting up an external Bluetooth GPS on a ToughBook

As I have written in a previous blog, I am not very pleased with the integrated GPS on the new Toughbooks. However, the Bluetooth is very stable and opens up a variety of GPS options. The following goes through the steps of setting up an external GPS on a Toughbook 19 running Windows XP Tablet. Other models should be similar; Vista has not been tested.

1) Turn on the GPS. As always, make sure the battery is charged!

2) On the Toughbook, open the SD card slot and flip on the Wireless toggle switch, Then, click on the Bluetooth logo on the bottom bar. A box will appear. Click “New Connection”. Here is an important step: use “Custom Mode” or else the computer will assign a COM port that may not work with some software! Click “Next.”

3) The computer will recognize the GPS. Highlight the name and click “Next.”

4) The computer will assign the GPS to a serial port. Click “Next.”

5) On the next screen, uncheck “Use default COM port.” Now, you can choose your own COM port. Many software packages only go up to COM 9 for GPS, so I recommend choosing COM 8 and “Auto Connect.” Click “Next.”

6) On the next screen you can select an icon if you wish. Click “Next” and you are done! The COM port you have chosen will be the same for all software you use. This also means that you cannot run two different software programs at the same time.

To save battery life in the field, you can turn off the WiFi while you are running Bluetooth. In the lower right corner of the bottom tool bar is an icon of a lightning bolt with a circle. Click it to bring up wireless options. Choose Wireless LAN OFF and Bluetooth ON. The 802.11a doesn’t matter.

The “Mystery of the Shifting COM Port”

I had an interesting case this past week that while unique, might shed light on some problems. A researcher has been using a CompactFlash GPS card in a Recon handheld computer without problems for nearly a year now. He e-mailed me that the GPS was working, but was coming in on a different COM port. Well….that can’t happen! COM ports, or Communication Ports are the electronic version of docks. They allow for data to be exchanged and, like a real dock, only one ‘vehicle’ can be there at a time. All computers use COM ports. Some have designated COM port numbers but allow you to create virtual COM ports. On a PC, it is typical for a serial port or an SD slot to have a unique COM port. If you have an integrated GPS that has a designated COM port. Because USB ports are wired differently, they require a driver to convert them to COM ports. Once a port is identified as a COM port, it stays a COM port. That is, the numbers don’t shift! On the Recon, the serial port is always COM1, the CompactFlash (either slot) is COM2, and Bluetooth devices come in on either COM8 or COM9. In the case of the “Mystery of the Shifting COM Port”, the CompactFlash jumped from COM2 to COM3.

Knowing the COM port for a GPS is necessary to run any software to read the data stream coming from the GPS. Unfortunately, the program I wrote for the researcher was set for COM2 so he could not use COM3. (As a quick fix, I re-wrote the program for COM3, but that didn’t address why the port should change.) Many programs allow you to figure out which COM port the GPS is connected to before you begin to collect data.

Back to the mystery, there are three scenarios: 1) The GPS tortured the computer into shifting COM ports, 2) there was a hardware malfunction in the computer, or 3) there was a gremlin in the operating system of the Recon. GPS receivers don’t have the capability to shift COM ports so that was ruled out. I was hoping the computer didn’t fail because the researcher was in the field in South Africa. So, I settled on the operating system. His Recon has Windows Mobile 5 which has a cool applet that allows you to re-route the GPS from its incoming hardware COM port (in this case, COM 2) to any outgoing COM port. The benefit is that many programs can read a virtual COM port. The analogy is that you phone can only take one call, but many receivers in your house can listen in. We tried changing the parameters on the GPS applet but that didn’t work. Dang. It seemed the most likely scenario (and the cheapest) was that the operating system flaked out. With Recons, it is really easy to restore the operating system. You do a hard reset by holding down the power button for a few seconds. When the yelp of a rapid double beep occurs, simultaneously press the flying windows and enter buttons. This will restore the factory settings. Of course the factory didn’t put on any extra programs or data so if you try this yourself, BACK UP ANY FILES OR PROGRAMS YOU INSTALLED!!

The result? One e-mail later, the researcher restore the factory settings and the CompactFlash GPS is back on COM2. The “Mystery of the Shifting COM Port” is closed. The lesson is, if it seems like the operating system is tweaked, return to the factory settings before you toss the computer. Oh, and back up the files!

Comment on ABC News: How accurate is your GPS?

Recently, ABC put out a news snippet about the “child killing” inaccuracies of GPS vehicle navigation units. Glossing over the semantically misleading title, the report did a decent job of quickly summarizing the obvious—navigational software is prone to errors. It is not that the receivers are inaccurate, but rather the GIS programming behind the directional software is not foolproof. Having driven sub-par vehicles all my life, I am well cautioned to not blindly trust ‘foolproof’ gauges like gas meters, speedometers, turn signal indicators, etc. I do trust billowing white steam as a statistically likely indicator that my car is overheating. So, where is the fault with the navigational systems and what can you do to lighten your chances of becoming a GPS child killer?

The various driving programs packaged by TomTom, Magellan, Garmin, DeLorme, and others, rely on a base map of roads. Already, there is going to be a spread of accuracy based on the detail of the map and the age of the data. Roads change; in some areas, a map can be outdated within a year. I live on a street that dead-ends against a creek and then shows up on the other side with the same name. It is not uncommon for folks to knock on our door wondering where the rest of the street lies. Some software allows for upgrades but it is up to you to stay on top of things. And, it is unlikely that daily, weekly, or monthly closures will appear. Secondly, in the better mapping packages, the roads are ‘weighted’ for traffic speed and other flow parameters. That way, you can mathematically deduce which is the best route even if it is not the shortest. We all know how this works whenever we try to explain to a friend or spouse that our way is “better.” Herein lies the problem. How many software packages are going to be more accurate than your knowledge, experience, and intuition for your neighborhood? Then why do you trust it for other neighborhoods you have never been to?

I blame the problem for the accidents and lost drivers on losing our ability to read a map and growing up in an increasingly virtual world. At some point in the Late Atarian, we evolved into a race that trusts virtual landscapes more than what is outside our window. Or, we just would rather inhabit that space. Why else would so many minivans and airplanes have TV sets instead of looking out the blessed window?

At this point, you may be wondering why I am writing about this on a fieldwork blog. Do you think the software for backcountry work doesn’t suffer the same shortcomings? In future blogs, I would like to explore how to improve the software we use for monitoring and recording our field work.

ToughBook computers and GPS receivers

I am not a big fan of the integrated GPS that one can purchase as an option on the ToughBook for about $300. In my experience, the GPS does not perform as well as cheaper add-on units. But, I will lay out my thoughts and you can make the call.

Benefits

The obvious benefit is that you have an integrated GPS so you will never lose it, forget to bring it to the field, or neglect to charge the battery. Also, there are no USB cables to get tangled in. If these are issues for you, and the extra one to two hundred dollars isn’t an impediment, than buy the integrated GPS. Why not? Regardless of the company’s claims, I would not rate this GPS for better than + / -15 meters accuracy. Is that important? For reference, remember that the standard USGS 7.5’ quadrangle maps that form the basis for most off-the-shelf high resolution software are at a scale of 1:24,000. Since a dot on that map is about 1 mm across, the dot represents 24,000 mm on the ground, or 24 meters. That is about the resolution of the integrated GPS. So, no problem. But, if you want to display your locations on a higher resolution map, it will be a problem.

Drawbacks

Beside the coarse accuracy, the biggest drawback is the price. It costs about $300 to add a GPS onto a computer from the factory. In comparison, I use a USB-driven device (GlobalSat) that costs about $70. I have been very, very pleased with the accuracy, quick start-up, and ruggedness of this unit. I have also used a Bluetooth-enabled device (GlobalSat) that costs about $115. Both of these options also allow for using the device with another computer—including a Macintosh or PDA! Folks who have read my other blogs will know I am a big fan of modularity. The GlobalSat units, in my testing, get closer to 3-5 meters of accuracy, or a dot on a 1:10,000 scale map.

For my ToughBook mapping, I use a USB-driven receiver and bundle the cord and tie it to the handle of the tablet. As a geologist, I have worked in many terrains with this set-up and have never felt burdened or been trapped on bushes. However, if I was to buy a ToughBook on a grant and could spare the few extra hundred dollars in the budget, I would likely get the GPS for when I was doing coarse mapping, or forgot the USB GPS, or forgot to recharge the Bluetooth GPS…