“Your wife might leave you if you work with open source” – No more

pros_cons_programmer

pros_cons_programmerThis is what an ESRI distributor told me (more or less) referring to the amount of time it takes someone to set up an interactive map web site  built around open source software.

Was meant in jest of course. He did have a point though. For anyone that is not very technical savvy, trying to create a simple website with a map and some interactivity was… let’s say, challenging the least. Compare having to install various components, editing Apache configs, copy/paste files around, searching for plugins, writing code for simple map interactions like zoom in/out, with the clean, drag & drop way of Web AppBuilder for ArcGIS.

To me, this was always the big gap in the FOSS4G ecosystem. Desktop GIS like QGIS, were not a problem any more. Download and install via a wizard. Easy. But there was never an quick (or painless) way of creating a map website.

Not until now, that is. Boundless has announced the release of their QGIS 2.14.1 distribution (1) which includes, among other goodies, the Web App Builder (currently in Beta) plugin.

Web App Builder allows you to create web apps directly from QGIS. Without writing a single line of code.

You can add different base maps or overlays:

wab_base_maps

And select the tools/functionality you want available on your site:

wab_controls

There are lots of other customization settings but those were the most interesting ones in my opinion. There are a few bugs (the tabbed interface is not working, doesn’t work well with different SRIDs) which result in blank pages, but all in all, I had a fully functional web site in less than 5 mins. That by itself is a huge improvement from the past!

You can download the Boundless QGIS distribution here. There is a form where you have to enter your details, etc but its definitely worthwhile.

Lots of kudos to the guys in Boundless and  I am sure lots of families will be thanking them too😉

(1) To avoid confusion or misunderstandings as per @Mr Fahrenheit’s comment: QGIS is an open source project created and still actively been developed by an international team of contributors and not by just Boundless. This post is referring to a specific distribution/branch of QGIS which includes the WAB plugin

Display oracle spatial data on a leaflet map

I wrote previously about how to convert oracle spatial data into GeoJSON. In this post I will take this  a step further and display that GeoJSONdata onto a leaflet map. A note before that though: The main sdo2geojson function return every record on the spatial table as a valid GeoJSON. That’s great, but not enough if you want to display the WHOLE table as a GeoJSON Feature collection. So I created a new sdo2geojson_partial function which will return an…erm… partial (and therefore not valid) GeoJSON strings which I could then concatenate through code to a valid feature collection object. You can download the updated oracle package here

The end result of this exercise would be to create a simple web site with a leaflet map that will display an oracle spatial layer at the time of load.

So I created a simple ASP.NET web site to demonstrate this. The main.aspx page is shown below.

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <meta charset="utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Test</title>

    <!-- Bootstrap -->
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css" />
    <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.css" />

    <style type="text/css">
        #map {
            height: 600px;
            width: 100%;
        }
    </style>

    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/js/bootstrap.min.js"></script>
    <script type="text/javascript" src="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.js"></script>
    <script src="scripts/map_functions.js"></script>
    <script type="text/javascript">
        $(document).ready(function () {
            map_props = init_map(40.587914266, 22.953796219, 13);
            map = map_props[0];
        });
    </script>
</head>
<body>
    <form>
        <div class="container-fluid">
            <div class="row">
                <div class="col-md-6">
                    <div class="panel panel-default">
                        <div class="panel-heading">Map</div>
                        <div class="panel-body">
                            <div id="map"></div>
                        </div>
                    </div>
                </div>
                <div class="col-md-6">
                    <div class="panel panel-default">
                        <div class="panel-heading">Another panel</div>
                        <div class="panel-body">
                            Content
                        </div>
                    </div>
                </div>
            </div>
        </div>
    </form>
</body>
</html>

The bulk of the above markup has to do with the layout of the page using bootstrap and the setting of the leaflet map. I won’t go into any details about them, you can find out how to create the maps through their excellent tutorials or creating web sites using bootstrap in w3schools

Notice however the link to scripts/map_functions.js. This is the bit of javascript that makes a jQuery AJAX call, to a web method which in turns call the oracle function to create the GeoJSON objects.

function init_map(lon, lat, zoomlevel) {
    map = L.map('map').setView([lon, lat], zoomlevel);
    var mapbox = L.tileLayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', {
        attribution: '&copy; <a href="http://osm.org/copyright">OpenStreetMap</a> contributors',
        id: 'examples.map-i875mjb7'
    }).addTo(map);

    var url = "../Services/GetGeoJsonData.asmx/getJsonFromOra";

    $.ajax({
        type: "POST",
        url: url,
        contentType: "application/json; charset=utf-8",
        dataType: "json",
        data: "{'table_name':'v_sdo_ksim_wgs84'}",
        success: function (geodata) {
            var obj = jQuery.parseJSON(geodata.d);
            var smallIcon = L.Icon.extend({
                options: {
                    iconSize: [29, 25],
                    iconUrl: '../img/sign1.png'
                }
            });
            geojson = L.geoJson(obj, {
                pointToLayer: function (feature, latlng) {
                    return L.marker(latlng, { icon: new smallIcon({ iconUrl: '../img/sign.png' }) });
                },
                onEachFeature: onEachFeature
            });
            geojson.addTo(map);
        },
        error: function (xmlHttpRequest, textStatus, errorThrown) {
            console.log(xmlHttpRequest.responseText);
            console.log(textStatus);
            console.log(errorThrown);
        }
    })

    return [map];
};
function onEachFeature(feature, layer) {
    layer.on({});
    layer.bindPopup('<a href="#" class="speciallink">' + feature.properties.EVA_EVENT_DESCRIPTION) + '</a.';
};
function zoomToFeature(e) {
    map.fitBounds(e.target.getBounds());
}

In a nutshell, the script calls through GetGeoJsonData.asmx web service, the getJsonFromOra method method (line 8) using a spatial table name as a parameter called v_sdo_ksim_wgs84 (line 15). In turn the getJsonFromOra method calls the oracle function using a spatial table name as a parameter and returns a valid GeoJSON feature collection string.

    [WebMethod]
    [ScriptMethod(UseHttpGet = false, ResponseFormat = ResponseFormat.Json)]
    public string getJsonFromOra(string table_name)
    {
        string json = string.Empty;
        string connectionString = "User Id=user;Password=pwd;Data Source=xe";
        string jprefix = "{\"type\": \"FeatureCollection\",  \"features\": [";
        OracleConnection oraCon = new OracleConnection();
        oraCon.ConnectionString = connectionString;
        oraCon.Open();
        string select = "select ora2geojson.sdo2geojson_partial('select *  from " + table_name + "',ROWID, shape) FROM " + table_name;
        OracleCommand oraCommand = new OracleCommand(select, oraCon);
        try
        {
            OracleDataReader Orareader = oraCommand.ExecuteReader();

            while (Orareader.Read())
            {
                json = json + "{" + Orareader[0].ToString() + ",";
            }
            json = json.TrimEnd(',');
            return jprefix + json + "]}";
        }
        catch (Exception)
        {
            throw;
        }
        finally
        {
            oraCon.Close();
            oraCon.Dispose();
        }
    }

The end results is a map like the one below. The traffic signs you see are oracle spatial features.

image

You can download the whole solution here. Remember to change the spatial table name to suit your settings.

Happy coding!

The 10 commandments of Egoless Programming

Came across these on various sites and thought I should re-publish. The concept of “Egoless programming” was introduced by Jerry Weinberg in his 1971 seminal book “The Psychology of Computer programming”10commandment. All very useful and clever, but I doubt ALL of them doable ALL the time.  Or, are they?

  1. Understand and accept that you will make mistakes. The point is to find them early, before they make it into production. Fortunately, except for the few of us developing rocket guidance software at JPL, mistakes are rarely fatal in our industry, so we can, and should, learn, laugh, and move on.
  2. You are not your code. Remember that the entire point of a review is to find problems, and problems will be found. Don’t take it personally when one is uncovered.
  3. No matter how much “karate” you know, someone else will always know more. Such an individual can teach you some new moves if you ask. Seek and accept input from others, especially when you think it’s not needed.
  4. Don’t rewrite code without consultation. There’s a fine line between “fixing code” and “rewriting code.” Know the difference, and pursue stylistic changes within the framework of a code review, not as a lone enforcer.
  5. Treat people who know less than you with respect, deference, and patience. Nontechnical people who deal with developers on a regular basis almost universally hold the opinion that we are prima donnas at best and crybabies at worst. Don’t reinforce this stereotype with anger and impatience.
  6. The only constant in the world is change. Be open to it and accept it with a smile. Look at each change to your requirements, platform, or tool as a new challenge, not as some serious inconvenience to be fought.
  7. The only true authority stems from knowledge, not from position. Knowledge engenders authority, and authority engenders respect – so if you want respect in an egoless environment, cultivate knowledge.
  8. Fight for what you believe, but gracefully accept defeat. Understand that sometimes your ideas will be overruled. Even if you do turn out to be right, don’t take revenge or say, “I told you so” more than a few times at most, and don’t make your dearly departed idea a martyr or rallying cry.
  9. Don’t be “the guy in the room.” Don’t be the guy coding in the dark office emerging only to buy cola. The guy in the room is out of touch, out of sight, and out of control and has no place in an open, collaborative environment.
  10. Critique code instead of people – be kind to the coder, not to the code. As much as possible, make all of your comments positive and oriented to improving the code. Relate comments to local standards, program specs, increased performance, etc.

Convert Oracle Spatial data to geojson

So at some point I needed to convert Oracle Spatial data (SDO) into geojson format. I quickly came across this excellent post from SpatialDB Advisor. Which basically does exactly that. Converts SDO data into geojson. However -as he already states in his post- the function does not support attributes. So based on this original function, I created a package that will create a FULL geojson including attributes using a bit of dynamic SQL (DBMS_SQL package). And in fact, you can select which attributes by providing a select statement as a parameter. I also *think* I fixed a couple of minor bugs in the original sdo2geojson function.

You can find the package (ora2geojson.rar) here. Unzip the file and run the package header  (.pkh) and body (.pkb) files e.g.

SQL>@ora2geojson.pkh

and

SQL>@ora2geojson.pkb

Example:

SELECT ora2geojson.sdo2geojson('select * from v_segments_all_sdo',ROWID, shape) FROM v_segments_all_sdo;

Where v_segments_all is the spatial table or view and shape is the name of the spatial column. The ROWID is needed to collect the attributes. I am sure there are more sufficient ways to do this but this one seemed to work fine for me.

A sample usage and result is shown below:

sdo2geojson

 

Land Surveyors vs GIS Professionals: What’s all the fuss about?

Drama? What Drama?
Drama? What Drama?

I was reading the other day the Land Surveying and GIS Revisited: An Unnecessary Drama post in GIS Lounge and I am sorry but I don’t get it. I mean, I agree with what the article says, that it IS an unnecessary drama but I don’t understand why there was a drama in the first place. This of course may be down to my ignorance of how the Land Surveying and GIS professions work in the US so please bear with me.

This is not the first time I hear about this rivalry. Similar posts have appeared here, here and here  There is also this image from the “GIS-Analyst: What my friends think I do” meme:

what_surveyors_think_i_do

So Land Surveyors think that GIS people play with coloring books… Right….

I still don’t get it.

You see, what I don’t understand is WHY all this press seems to point to a hard line distinction between a “Land Surveyor” and a “GIS Professional”. I am not saying that ALL Land Surveyors “do” GIS, or that everyone that works with GIS is a Land Surveyor, but isn’t GIS a tool that different professions -from Surveyors to Environmental Engineers to Marketing professionals use to help them in their day-to-day jobs? A tool rather than a profession on its own? And isn’t a “GIS Professional” someone that uses GIS within a specific realm? And yes, of course there are those who started with a undergrad degree in GIS and they are the ‘vanilla’ GIS professionals if you like, but I still can’t see why Land Surveyors will look down on them so to speak.

On the posts I mentioned above, the grudge seems to focus around cadastral issues and how a GIS-only person cannot show someone’s legal boundary data on a GIS map. What if the ‘GIS Map’ and the parcel layer is actually constructed from ground topographic measurements? Why is that not accurate? And what if that GIS person is also a Surveyor?

Obviously, if I show someone a Google map and draw on top his/her property boundary by just following the lines I see on the map and then tell him he/she can use it as evidence on a court case, I should be shot on site. This is just ignorance.  But if the GIS I am using includes cadastre data which by definition is accurate, then what’s the problem?

This is something that is used quite a bit here in Greece; Public-facing GIS ran by the City Councils allow users to download boundary data for planning applications. And data IS accurate. And in the majority of cases, these systems are ran by Surveyors.

So as I said in the beginning, the GIS and Land Surveying professions might work somewhat different between the various countries and these discussions are US-focused for which I now very little about but I think there are some fixed constants.

I can’t see the reason of any dramas or “epic battles”. I would be grateful if someone would enlighten me

Disclosure: I have a 5-yr Surveying Engineer’s degree and an M.Sc in GIS. I would call myself a GIS professional rather than a Surveyor.

So how much does Greek Local Government really spend? Find out on an interactive map

diavgeia_maps_enAnd where else would a better place be, you may ask. If you head over to Diavgeia Spending Maps and after you hit the “It’s all Greek to me” button (Aren’t I a clever boy – get the pun?) you should be able to query spending data for every municipality in Greece for a time period of your choosing. Make sure to check the FAQ first.

Data is coming from the Greek Government’s “Transparency  Portal” (“Diavgeia” stands for transparency in Greek)

This has been my toy project for a bit now trying to work on it on my spare time. This is the first release. Do have a play and let me know what you think. If you want more details on how it was done, read on.

The background

Under the Greek Transparency Program initiative, beginning October 1st, 2010, all government institutions are obliged to upload their acts and decisions on the “Transparency Portal”. In fact, NO administrative act or decision is valid unless published on this portal. More info here.

challenge_accepted_barneyThe good folks at the Ministry of Administrative Reform and e-Governance who run this portal, have also provided an API so external apps can use their data. Now, although there were some external sites utilizing the API, none of them had any maps. So I raised to the challenge. In a nutshell, the app makes subsequent calls to the API, one for each municipality, for decisions and acts relating to spending data and simply adds them together. The result is the total amount spent and its the value used to color the map.

There are two -or three very important caveats here – at least in this first version:

The API will only return a maximum of 500 results per query. This means that if -for the date range you selected- the municipality has issued more than 500 decisions relating to spending, the total amount shown for this municipality will be incorrect. However, one can get around it by selecting shorter time periods. In any case the application will report on the number of results returned as well as the number of actual results for each query so you should be able to know if all data was included.

The second is more serious, and in all probability due to my ignorance of the inner workings and processes within the municipalities and their interaction with the Transparency Portal.

To begin with, some of the decisions have a zero amount next to them. And that’s despite the fact that in some cases, even the title of the particular decision states the amount (which of course is not zero). At first I thought it was some sort of bug in my code, but no, it WAS the data stupid. I haven’t got the foggiest why this is.

On the other hand, I noticed that some decisions are duplicated. Exact same title and amount but different decision id. Again, not a clue.

I do intend to try and contact the Diavgeia people to find out but till then -and its plastered all over the site- do use the results with EXTREME caution. Results and total amounts are indicative and should be used for informational purposes only!

The technical bits

Diavgeia Spending Maps was developed using Leaflet (based on the Interactive Choropleth Map example) and jQuery. Controls were based on the JQuery UI tools. Data was returned in Json format. Administrative boundaries were downloaded from geodata.gov.gr as a single shapefile and converted to GeoJson after some generalization to reduce in size.

Greek Version