ESRI’s brilliant marketing move and what Open Source GIS can learn from it

ESRI has announcedESRI Maps for Office”, an Excel add-in which will be available in public beta the first half of 2012. So imagine: You have all your Excel data, all the power of Excel for analysis and now, you can also display this data on a map WITHIN Excel to explore and analyze it in a whole new way.Fantastic – no?

It probably is. ESRI hasn’t released any information about pricing or licensing (e.g. would you need to have ArcGIS installed on the desktop to use the add-in?) but this is beside the point. In this post, I would argue that the real story here goes far beyond of having another tool in Excel. It brings “GIS” or at least thematic mapping, to thousands (millions?) of new users who till then probably didn’t have anything to do with maps or “GIS” (btw, the quotes are because we still don’t know what kind of spatial/mapping functionality will be available).

This move by ESRI, penetrates -in a big way- a previously untapped market. And this is a lesson Open Source GIS companies can learn a lot from.

In a recent blog post, Sofia Parafina, argues that the release of TileMill for Windows will be the game changer for the adoption of open source mapping. Why? Because, she says 84% of the computers in the world run some variant of Windows (actually in the link she provides this is higher-85%). And also because till now, there was a lack of good cartographic tools for open source programs running on Windows.

I beg to differ. In her own post she states that:

…it’s rather bare bones in that there is no tool to select a feature and set the line style, color, or fill in a menu. You have to create each style using MSS in a TileMill editor, …

So here we go. Another glorified notepad. Don’t get me wrong. I have played with it, its great and I am sure it will help. You can now create thematic maps more easily. But I very much doubt that will be a game changer and people will suddenly queue to download open source GIS.

BUT, providing a –say- QGIS add-in for Excel might do the trick. Do you want some more stats? According to Microsoft, 500 million people worldwide use Microsoft Office. Although this claim may be rather excessive, even empirically, if 85% of computers run windows, most new computers come with some version of Microsoft office, and even WITHOUT including pirated copies, well… you get the picture.

So to finish the same way I started, imagine: You have all your Excel data, all the power of Excel for analysis and now, you can also display this data on a map WITHIN Excel to explore and analyze it in a whole new way by downloading an add-in for Excel for free! .

Now, this IS a game-changer!


Setting codepages in shapefiles to display in ArcMap 10 – A saga

I spent a good few hours last week trying to get ArcMap 10 to display a shapefile with Greek data in. Not an easy job. The shapefile was Open Street Map data including all place names in Greece, downloaded from Cloudmade. When I opened the attribute table in ArcMap it looked something like this:


On the other hand, the shapefile was displaying fine in QGIS IF you opened it by setting the Encoding to UTF-8. So I set forth trying to find a way of specifying the encoding in ArcMap. To skip right to the bottom, I can tell you now that nothing I tried using ESRI-only tools and suggestions worked. But during my quest I found some useful bits of information which may be prove helpful for someone in the future: So this is my story:

I first made sure that my Regional Settings was set to Greek/Greece and the same was true for the Language for Non-Unicode programs- just in case. (this used to work for Arcview 3.x – obviously ArcMap IS a Unicode program):


Then I tried adding the *.cpg file. This is basically a single-line text file with the same name as the shapefile which stores codepage information for the attribute data (if the shapefile does not carry this information). Tried various codepage codes. It didn’t work.

I then move on to add some registry values according to this little gem: HowTo:  Read and write shapefile and dBASE files encoded in various code pages. Nope.

Getting desparate, I then tried the 30th byte way from ESRI which tells you how to determine whether a shapefile has a code page or not (apparently this information is stored in the 30th byte)::

To find the 30th byte, you count the sets of characters in the center. In this example, you start with 03, which counts as 1. Count over 30, counting only the character sets shown in blue in this example. If the set is 00, the code page is not set. The 30th character in this example is 0E. Therefore, the code page is set.

0B8D:0100 03 64 02 07 01 00 00 00-A1 00 41 00 00 00 00 00 .d……..A…..
1489:0110 00 00 00 00 00 00 00 00-00 00 00 00 00 0E 00 00 …………….

If the code page is not set in the .dbf header, you can create a code page file (.cpg) to store the code page. To create a code page file, you use a text editor, such as vi or Notepad, add the code page identifier for the shapefile to the file, and save it with a .cpg extension in the same location as the other files that make up the shapefile. You have to be sure you know the encoding used for the shapefile so you place the correct code page in the .cpg file.

If for some reason you have a .cpg file and the code page is set in the .dbf, the information in the .dbf header takes priority when importing a shapefile. If no code page is set in the .dbf, the code page is read from the .cpg file. If the code page is not set in the .dbf and no .cpg file is present, the code page of the current locale of the operating system from which shp2sde is being run (the server where ArcSDE is installed) will be used.

So I opened the dbf in a HexEditor. It did have a value on the 30th byte. I put it to 00. Re-opened shapefile in ArcMap. Same rubbish characters.

By that time I had enough. I opened the file in QGIS (with a UTF-8 encoding) and then saved it under a new name with a CP-1253 encoding. And of course it worked. Which was what I thought to try in the beginning but my masochist self wanted to find a solution without using any 3rd party tools.

I mean, admire the simplicity of it all:

Step 1. Add the shapefile and set encoding


Step 2. Save it under a new name


Now, how difficult it would be for ESRI to have an encoding option when opening a shapefile?

If you think I missed something and there is indeed a much simpler way (using ESRI tools) to view attribute data in ArcMap I would be more than happy to learn about it!

GIS Certified–To Be or Not to Be?

Ελληνική Έκδοση

There has been a lot of talk recently about the relative virtues (or not) of GIS Certification. Especially after the GIS Certification Institute announced they are considering to add a testing component to their certification process and invited comments on their proposal. A few weeks back, ESRI announced the launch of their technical certification program. The GIS Certification topic was always a controversial one, sure to create a heated debate amongst GIS professionals.

Closer to home, the Surveying Engineers Association of Northern Greece (ΣΔΑΤΜΒΕ), recently signed an agreement with the ACTA Certification Center on the joint issue of GIS Professional Certifications. Having taken part in this effort from the Surveying Engineers side, I would like to share some thoughts on this whole Certification issue leaving aside product-specific certifications like ESRI’s.

From one hand, I understand the views saying that it is very difficult to create a one-fits-all  “GIS Certification”. The GIS domain  is so vast, with different applications, uses and technologies that change and evolve at a breaking pace, that the idea of certifying someone as an expert or a professional is likely to change as each year passes.

Having said that, I believe that some sort of certification is very important for individuals who initially started working in non-GIS related industries but their job function evolved, demanding more and more ‘spatial’ skills. An example would be people in marketing research who would now need to use geodemographics and spatial analysis.

A GIS Certification in those cases would be very useful for establishing a base line of skills. GIS certified individuals should be able to have a thorough understanding of spatial issues both in theoretical and practical level and be able to manage spatial projects. This was also the thinking behind the GIS Certification by the Surveying Engineer Association and ACTA as mentioned above.

For people with formal GIS qualifications and substantial work experience, I would be more in favor of specific industry related certifications or -more correctly-accreditations, such as a “GIS in Transportation” or “Environmental GIS”.

And then of course there is the issue of the assessment process.  Portfolio review only or including an exam component as well? Again, my opinion is that it depends on the kind of certification/accreditation. When talking about  general GIS Certification I would argue that an exam component is not only useful but required. But for a specialist certification, a peer review and portfolio assessment would be the way to go as a much more objective and straight-forward process.

Want to (just) display data from Oracle or SQLServer in ArcMap? You have to pay for the privilege!

I was just reading Abe Gillespie’s post about the decision to release zigGIS, the Arcmap plugin to access PostGIS data  as open source (again) and it got me thinking:- If you want to display oracle or sql server spatial data in ArcMap you have to also pay for ArcSDE. And it doesn’t come cheap. Bit cheeky this from ESRI I think. Especially since you did NOT have to do this back in ArcGIS 8.x days. The sde libraries were part of the ArcMap product and you could use “direct connect” to simply display the data. (You still had to create the sde schema and owner in your database which was another major pain, but will leave that for now…)

Compare this with the $279 you have to pay now for ziGIS to access PostGIS data (and soon to be $0) and the $0 you pay to display Oracle or SQLServer data using MapServer or QGIS. And none of them requires you to tinker with your database!

Is it just me who thinks that ESRI (“ezree”) got it wrong somewhere along the line? Why do I have to go for a fully-blown ESRI solution? Why should my customers pay for something they will never use? Still though, since ESRI’s ArcMap IS a good product and it IS used by a lot of people, are there  any other brave souls out there who are working for Oracle or SQLServer plugins?