Saturday, 8 June 2019

How to set up a leaflet map which connects to your Postgres/PostGIS layers using 'RStudio-Server' and 'Shiny-Server'

For those potential GI users who are too busy to learn GIS skills but need to know where those boundaries are, this could be the answer for you. Through a simple leaflet map we can connect directly into our spatial data in PostgreSQL/PostGIS data with 'R' and 'Shiny'. For more information click the link.

In the example below we are going to install 'RStudio-Server' on an Ubuntu 18.04 LTS server in order  to run an 'R' script although I could just as easily have created this script using any other IDE or text editor if preferred. 'RStudio' is a complete environment for prototyping web apps (known as 'Shiny' apps) which allow the user to interact with the data imported from various sources for example to look into changes over a ten year timescale interactively from an excel spreadsheet.

'RStudio-Server' is a free and open-source integrated development environment (IDE) for R, a programming language for primarily statistical computing, graphics and mapping analysis (it is a full gis with input, analysis and output 'packages' utilising many of our familiar OSGEO libraries used in QGIS, SAGA, GRASS, i.e. GDAL, PROJ4, GEOS etc but more focused on statistical and web output).

The 'RStudio' Interface, 'script' top left, 'console' lower left and 'output' lower right.

There are minor similarities to the 'qgis2web' plugin (for QGIS) with some 'Power BI' style tools thrown in the mix with more functionality to 'fine-tune' your data output whether an excel spreadsheet or a database table or from any other source. There are other important differences between 'R' compared to standard HTML/CSS/PHP/Javascript web frameworks in that there is only one script containing all the above tools including the UI and Server functionality.

I admit here that I am not a developer or coder but it is really easy to pick up once you get started. A great tool for fast prototyping. There is also an interesting discussion around 'R Shiny' v 'Power BI' if anyone is interested.

I'd been looking for a tool to get map data stored in a Postgres/PostGIS tables on the web quickly for non traditional GIS users to be able to view the data and for me to be able to modify the code changing features on the fly without having to restart the server.

Let us begin and get 'RStudio-Server' installed.

Some text conventions before we start:-


$ red instructions = manually enter terminal command.
Red italics with underline = add your own info/config here.
# Instruction = non-parsing instructions or comments preceding the command.



#If you don't have 'R' installed already (on UBUNTU 18.04 SERVER in this case).

$ sudo apt-get install r-base r-base-dev 

#(R 3.6) (if package incompatibilities arise or above packages won't install then delete:- /usr/local/lib/R) then (remove --purge, autoremove) and install again as above.

#install dependencies

$ sudo apt-get install gdebi-core

#download the shiny server packages


#install shiny-server-

$ sudo gdebi shiny-server-

#To access rstudio-server v1.2.1.1335 thereafter

http://XXX.XX.X.XX:8787 user:your_user pass:your_password (DEFAULTS TO PORT '8787' ON YOUR SERVER)

#main project location



#Install the 'cran' 'R' packages (in a 'Global' library so all users can access them) used in your app starting with 'devtools'

$ sudo su - your_user

$ sudo apt-get -y install libcurl4-gnutls-dev libxml2-dev libssl-dev

$ sudo su - -c "R -e \"install.packages('devtools', repos='')\""
$ sudo su - -c "R -e \"install.packages('shiny', repos='')\""your_user
$ sudo su - -c "R -e \"install.packages('shinythemes', repos='')\""
$ sudo su - -c "R -e \"install.packages('rsconnect', repos='')\""
$ sudo su - -c "R -e \"install.packages('RPostgreSQL', repos='')\""
$ sudo su - -c "R -e \"install.packages('rpostgis', repos='')\""
$ sudo su - -c "R -e \"install.packages('sp', repos='')\""
$ sudo su - -c "R -e \"install.packages('rgdal', repos='')\"" (**requires 'libgdal-dev' on sudo apt install** ln -s  )
$ sudo su - -c "R -e \"install.packages('leaflet', repos='')\""
$ sudo su - -c "R -e \"install.packages('leaflet.extras', repos='')\""

Once this has been done we can remove the 'install.packages("")' parts of the 'app.R' script



#Upload folders/files with (use winSCP on Windows Servers) the directory (your_shiny_app_directory_here) to /srv/shiny-server/ then move it to the main server directory:-

$ sudo mv /srv/shiny-server/your_shiny_app_directory_here/ /opt/shiny-server/

#Then 'symlink' back to /srv/shiny-server along with 'sample-apps' folder and 'index.html' (used for testing).

$ sudo ln -s /opt/shiny-server/your_shiny_app_directory_here/srv/shiny-server/your_shiny_app_directory_here
$ sudo ln -s /opt/shiny-server/samples/welcome.html /srv/shiny-server/index.html
$ sudo ln -s /opt/shiny-server/samples/sample-apps /srv/shiny-server/sample-apps



A default 'shiny-server.conf' configuration file looks like this:-


preserve_logs true

# Instruct Shiny Server to run applications as the user "shiny"

run_as your_user;
# Define a server that listens on port '3838'
server {
  listen 3838;

  # Define a location at the base URL
  location / {

    # Host the directory of Shiny Apps stored in this directory
    site_dir /srv/shiny-server;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;



$ sudo systemctl restart shiny-server


/var/log/shiny-server (shiny-server specific)

leaflet zoom levels:-

5km = Zoom 10
3km = Zoom 11
2km = Zoom 12

1km = Zoom 13


The 'R' Script (download here)

Red italics with underline = add your own info here

Before copying this script into the script editor if you are using any images within the page please ensure they are situated within the 'www' directory.

# To create and test this script in RStudio you'll need to install the packages first if you've installed them 'Globally' on your server then remove them here in the script


# Install the libraries


# Specify Driver

pg <- dbDriver("PostgreSQL")

# Connection Details

con <- dbConnect(pg, user ='your_user', password ="your_password_here", host = "your_host_here" , port = 5432, dbname="your_db_here")

# List all Tables in database (to test the connection is working)


# Read the PostGIS table "layer_1" into a 'data.frame' for the dynamic labels

dbReadTable(con, "layer_1", schema = "layer_1_schema_here") %>% head(3)

# list the Fields in the table 

dbListFields(con, "layer_1", schema = "layer_1_schema_here")

# Query the tables for each layer you require

query <- "SELECT * FROM layer_1"
layer_1 <- pgGetGeom(con, c("layer_1_schema_here","layer_1"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

query <- "SELECT * FROM layer_2"
layer_2 <- pgGetGeom(con, c("layer_2_schema_here","layer_2"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

# Plot the layer in Viewer


# Change the projection from OSGB36 (epsg:27700) to WGS 84 (epsg:4326)

layer_1 <- spTransform(layer_1, CRS("+init=epsg:4326"))
layer_2 <- spTransform(layer_2, CRS("+init=epsg:4326"))

# UI Function

ui <- fluidPage(theme = shinytheme("cerulean"),
                tags$head(HTML("<title>myTitle</title> <link rel='icon'type='image/png' href='myImage.png'>")),
                titlePanel(title=div(img(src="myImage.png", height = 100, width = 100, align = "left","myTitle"))),
                h4("My h4 text here."),
                p("My paragraph text here"),
                leafletOutput("layer_1", height = 500))

# Server Function

server <- function(input, output, session)
output$layer_1 <- renderLeaflet ({
    map <- leaflet(layer_1) %>%
      flyTo(-0.0000, 00.0000, zoom = 10) %>%
      addTiles(group="OpenStreetMap.Mapnik") %>% 
      addProviderTiles("Esri.WorldImagery", group = "Satellite") %>%
      addProviderTiles("OpenStreetMap.HOT", group = "OpenStreetMap.HOT") %>%
      addProviderTiles("Stamen.TerrainBackground", group = "Stamen.TerrainBackground", options = providerTileOptions(minzoom = 1, maxzoom = 13)) %>%
      addMiniMap(tiles = "OpenStreetMap.HOT", toggleDisplay = TRUE) %>%
      addSearchOSM() %>%
      addEasyButton(easyButton(icon="fa-globe", title="Zoom to Level 1", onClick=JS("function(btn, map){ map.setZoom(1); }"))) %>%
      addEasyButton(easyButton(icon="fa-crosshairs", title="Locate Me", onClick=JS("function(btn, map){ map.locate({setView: true}); }"))) %>%
                  fillColor = "#e41a1c", 
                  fillOpacity = 0.4, 
                  stroke = "#690E0E",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~name_of_your_table_attribute_column_to_label_here,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_1",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
                  fillColor = "#377eb8", 
                  fillOpacity = 0.4, 
                  stroke = "#20445F",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~yourLabel,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_2",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
      addLegend("bottomleft", title = "Map Legend", colors = c("#e41a1c","#377eb8"), labels = c("layer_1","layer_2")) %>%
      addFullscreenControl() %>%
      addMeasure() %>%
      addScaleBar(position = c("bottomleft")) %>%
      addLayersControl(baseGroups = c("OpenStreetMap.Mapnik","Satellite","OpenStreetMap.HOT","Stamen.TerrainBackground"),
                       overlayGroups = c("layer_1","layer_2"),
                       options = layersControlOptions(collapsed = TRUE))

# ShinyApp Function

shinyApp(ui = ui, server = server)

Friday, 15 February 2019

Joplin - An Open Source alternative to Evernote

I never thought I would need a 'to-do' list or a 'note taking' application before. I turn up at meetings with my trusty black pocket book and a pen. How 'old school' I hear you say but I am always late to the party. I signed up to Evernote some years ago but that is as far as I got. Now I cannot stop adding stuff to remember in 'Joplin'. So what is so different about this syncing 'to-do' app (compared to the Evernote free version)?

Evernote has no 'markdown' support.
Evernote has no 'end to end' encryption.
Evernote has a two device limit.
Evernote cannot work offline.
Evernote isn't open source.

The name 'Joplin' comes from composer Scott Joplin

"Joplin is a free, open source note taking and to-do application, which can handle a large number of notes organised into notebooks. The notes are searchable, can be copied, tagged and modified either from the applications directly or from your own text editor. The notes are in Markdown format.
Notes exported from Evernote via .enex files can be imported into Joplin, including the formatted content (which is converted to Markdown), resources (images, attachments, etc.) and complete metadata (geolocation, updated time, created time, etc.). Plain Markdown files can also be imported.
The notes can be synchronised with various cloud services including Nextcloud, Dropbox, OneDrive, WebDAV or the file system (for example with a network directory). When synchronising the notes, notebooks, tags and other metadata are saved to plain text files which can be easily inspected, backed up and moved around.
The application is available for Windows, Linux, macOS, Android and iOS. A Web Clipper, to save web pages and screenshots from your browser, is also available for Firefox and Chrome". Source:-

Installation on Ubuntu (copy and paste) in terminal and press 'return'

wget -O - | bash

Thanks to my colleague Phil Barlow for introducing me to it.

Tuesday, 18 December 2018

Proud to be a sponsor of UBports

I'm extremely proud that I can play a very small part in this amazing project to get Ubuntu Touch on Mobile devices.

Monday, 26 November 2018

How to install Postgresql-10 and PostGIS-2.4 on Ubuntu 18.04 LTS (Bionic Beaver) in six easy steps (updated 26-11-2018).

Install Ubuntu 18.04 LTS (Bionic Beaver)

1. check your Ubuntu version:-

lsb_release -a

Install Postgresql

2. On the server or your PC (Terminal):-

sudo apt-get update
sudo apt-get install -y postgresql postgresql-contrib

Create a database and a user for access

Replace DATABASE_NAME_HERE and USER_NAME_HERE with the values you want to use.
# this will prompt you for a database password...also note the capital letter 'O' not number '0' (zero) below:-

3. sudo -u postgres createuser -P USER_NAME_HERE
sudo -u postgres createdb -O USER_NAME_HERE DATABASE_NAME_HERE

Test Connection to Postgresql


Postgresql will ask you for your password. Then you should see the following:


To exit type:-


To add PostGIS-2.4 support to the database

5. sudo apt-get install -y postgis postgresql-10-postgis-2.4

sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;" DATABASE_NAME_HERE

6. Finally... install PgAdmin3 and set your 'service' connections

 sudo apt-get install pgadmin3

My thanks to Jon Saints for his original post dated from 13th August 2014 (for Ubuntu 14.04) can check your terminal output with the one below:-

:~$ sudo apt install -y postgresql postgresql-contrib
[sudo] password for xxxx:
Reading package lists... Done
Building dependency tree     
Reading state information... Done
Suggested packages:
The following NEW packages will be installed
  postgresql postgresql-contrib
0 to upgrade, 2 to newly install, 0 to remove and 27 not to upgrade.
Need to get 11.6 kB of archives.
After this operation, 127 kB of additional disk space will be used.
Get:1 bionic/main amd64 postgresql all 10+190 [5,784 B]
Get:2 bionic/main amd64 postgresql-contrib all 10+190 [5,796B]
Fetched 11.6 kB in 0s (101 kB/s)             
Selecting previously unselected package postgresql.
(Reading database ... 399415 files and directories currently installed.)
Preparing to unpack .../postgresql_10+190_all.deb ...
Unpacking postgresql (10+190) ...
Selecting previously unselected package postgresql-contrib.
Preparing to unpack .../postgresql-contrib_10+190_all.deb ...
Unpacking postgresql-contrib (10+190) ...
Setting up postgresql-contrib (10+190) ...
Setting up postgresql (10+190) ...
:~$ sudo -u postgres createuser -P postgis
Enter password for new role: xxxx (your password)
Enter it again: xxxx (your password)
:~$ sudo -u postgres createdb -O postgis webgis
createdb: database creation failed: ERROR:  database "webgis" already exists
:~$ sudo -u postgres createdb -O postgis polybase
:~$ psql -h localhost -U postgis polybase
Password for user postgis:
psql (10.6 (Ubuntu 10.6-0ubuntu0.18.04.1), server 9.6.8)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.



:~$ sudo apt install -y postgis postgresql-10-postgis-2.4
Reading package lists... Done
Building dependency tree     
Reading state information... Done
postgis is already the newest version (2.4.3+dfsg-4).
The following NEW packages will be installed
  libprotobuf-c1 postgresql-10-postgis-2.4
0 to upgrade, 2 to newly install, 0 to remove and 27 not to upgrade.
Need to get 578 kB of archives.
After this operation, 1,748 kB of additional disk space will be used.
Get:1 bionic/universe amd64 libprotobuf-c1 amd64 1.2.1-2[19.0 kB]
Get:2 bionic/universe amd64 postgresql-10-postgis-2.4 amd64 2.4.3+dfsg-4 [559 kB]
Fetched 578 kB in 0s (2,861 kB/s)               
Selecting previously unselected package libprotobuf-c1:amd64.
(Reading database ... 399421 files and directories currently installed.)
Preparing to unpack .../libprotobuf-c1_1.2.1-2_amd64.deb ...
Unpacking libprotobuf-c1:amd64 (1.2.1-2) ...
Selecting previously unselected package postgresql-10-postgis-2.4.
Preparing to unpack .../postgresql-10-postgis-2.4_2.4.3+dfsg-4_amd64.deb ...
Unpacking postgresql-10-postgis-2.4 (2.4.3+dfsg-4) ...
Setting up libprotobuf-c1:amd64 (1.2.1-2) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up postgresql-10-postgis-2.4 (2.4.3+dfsg-4) ...
:~$ sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;"polybase
ERROR:  syntax error at or near "polybase"
LINE 1: ...XTENSION postgis; CREATE EXTENSION postgis_topology;polybase
:~$ sudo apt install pgadmin4
Reading package lists... Done
Building dependency tree     
Reading state information... Done
Package pgadmin4 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'pgadmin4' has no installation candidate

:~$ sudo apt install pgadmin3
Reading package lists... Done
Building dependency tree     
Reading state information... Done
pgadmin3 is already the newest version (1.22.2-4).
0 to upgrade, 0 to newly install, 0 to remove and 27 not to upgrade.

Featured post

Qgis-server...Installing the QGIS Lizmap Plugin & Lizmap Web Client

This post follows on from my previous three (most recent first in list) linked below. There is no doubt that in just a short time from no...