Wednesday, 11 December 2019

rollApp rollApp...

Ever get fed up with installing software then upgrading every so often? 'rollApp' is another example of 'SaaS' but larger government departments and companies may find the cost of doing this work often more than the cost of the pricing polices of running the software in the 'cloud'. In the UK this amounts to £71.88 or £5.32 a month (at time of writing) for individual accounts. There is a discount if your on the 'premium for groups' level which is £471.59 a month for 100 users.

A quick search on the apps reveals that there appear to be all of the FOSS applications you would need including spatial and mapping ( 'RStudio'). I've tried the 'free' tier and the performance is very good and usable as long you don't 'save' anything to local disk or cloud. Worth a try even just for the convenience of being able to use your applications on any device!

Saturday, 8 June 2019

How to set up a leaflet map which connects to your Postgres/PostGIS layers using 'RStudio-Server' and 'Shiny-Server'

For those potential GI users who are too busy to learn GIS skills but need to know where those boundaries are, this could be the answer for you. Through a simple leaflet map we can connect directly into our spatial data in PostgreSQL/PostGIS data with 'R' and 'Shiny'. For more information click the link.

In the example below we are going to install 'RStudio-Server' on an Ubuntu 18.04 LTS server in order  to run an 'R' script although I could just as easily have created this script using any other IDE or text editor if preferred. 'RStudio' is a complete environment for prototyping web apps (known as 'Shiny' apps) which allow the user to interact with the data imported from various sources for example to look into changes over a ten year timescale interactively from an excel spreadsheet.

'RStudio-Server' is a free and open-source integrated development environment (IDE) for R, a programming language for primarily statistical computing, graphics and mapping analysis (it is a full gis with input, analysis and output 'packages' utilising many of our familiar OSGEO libraries used in QGIS, SAGA, GRASS, i.e. GDAL, PROJ4, GEOS etc but more focused on statistical and web output).

The 'RStudio' Interface, 'script' top left, 'console' lower left and 'output' lower right.

There are minor similarities to the 'qgis2web' plugin (for QGIS) with some 'Power BI' style tools thrown in the mix with more functionality to 'fine-tune' your data output whether an excel spreadsheet or a database table or from any other source. There are other important differences between 'R' compared to standard HTML/CSS/PHP/Javascript web frameworks in that there is only one script containing all the above tools including the UI and Server functionality.

I admit here that I am not a developer or coder but it is really easy to pick up once you get started. A great tool for fast prototyping. There is also an interesting discussion around 'R Shiny' v 'Power BI' if anyone is interested.

I'd been looking for a tool to get map data stored in a Postgres/PostGIS tables on the web quickly for non traditional GIS users to be able to view the data and for me to be able to modify the code changing features on the fly without having to restart the server.

Let us begin and get 'RStudio-Server' installed.

Some text conventions before we start:-


$ red instructions = manually enter terminal command.
Red italics with underline = add your own info/config here.
# Instruction = non-parsing instructions or comments preceding the command.



#If you don't have 'R' installed already (on UBUNTU 18.04 SERVER in this case).

$ sudo apt-get install r-base r-base-dev 

#(R 3.6) (if package incompatibilities arise or above packages won't install then delete:- /usr/local/lib/R) then (remove --purge, autoremove) and install again as above.

#install dependencies

$ sudo apt-get install gdebi-core

#download the shiny server packages


#install shiny-server-

$ sudo gdebi shiny-server-

#To access rstudio-server v1.2.1.1335 thereafter

http://XXX.XX.X.XX:8787 user:your_user pass:your_password (DEFAULTS TO PORT '8787' ON YOUR SERVER)

#main project location



#Install the 'cran' 'R' packages (in a 'Global' library so all users can access them) used in your app starting with 'devtools'

$ sudo su - your_user

$ sudo apt-get -y install libcurl4-gnutls-dev libxml2-dev libssl-dev

$ sudo su - -c "R -e \"install.packages('devtools', repos='')\""
$ sudo su - -c "R -e \"install.packages('shiny', repos='')\""your_user
$ sudo su - -c "R -e \"install.packages('shinythemes', repos='')\""
$ sudo su - -c "R -e \"install.packages('rsconnect', repos='')\""
$ sudo su - -c "R -e \"install.packages('RPostgreSQL', repos='')\""
$ sudo su - -c "R -e \"install.packages('rpostgis', repos='')\""
$ sudo su - -c "R -e \"install.packages('sp', repos='')\""
$ sudo su - -c "R -e \"install.packages('rgdal', repos='')\"" (**requires 'libgdal-dev' on sudo apt install** ln -s  )
$ sudo su - -c "R -e \"install.packages('leaflet', repos='')\""
$ sudo su - -c "R -e \"install.packages('leaflet.extras', repos='')\""

Once this has been done we can remove the 'install.packages("")' parts of the 'app.R' script



#Upload folders/files with (use winSCP on Windows Servers) the directory (your_shiny_app_directory_here) to /srv/shiny-server/ then move it to the main server directory:-

$ sudo mv /srv/shiny-server/your_shiny_app_directory_here/ /opt/shiny-server/

#Then 'symlink' back to /srv/shiny-server along with 'sample-apps' folder and 'index.html' (used for testing).

$ sudo ln -s /opt/shiny-server/your_shiny_app_directory_here/srv/shiny-server/your_shiny_app_directory_here
$ sudo ln -s /opt/shiny-server/samples/welcome.html /srv/shiny-server/index.html
$ sudo ln -s /opt/shiny-server/samples/sample-apps /srv/shiny-server/sample-apps



A default 'shiny-server.conf' configuration file looks like this:-


preserve_logs true

# Instruct Shiny Server to run applications as the user "shiny"

run_as your_user;
# Define a server that listens on port '3838'
server {
  listen 3838;

  # Define a location at the base URL
  location / {

    # Host the directory of Shiny Apps stored in this directory
    site_dir /srv/shiny-server;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;



$ sudo systemctl restart shiny-server


/var/log/shiny-server (shiny-server specific)

leaflet zoom levels:-

5km = Zoom 10
3km = Zoom 11
2km = Zoom 12

1km = Zoom 13


The 'R' Script (download here)

Red italics with underline = add your own info here

Before copying this script into the script editor if you are using any images within the page please ensure they are situated within the 'www' directory.

# To create and test this script in RStudio you'll need to install the packages first if you've installed them 'Globally' on your server then remove them here in the script


# Install the libraries


# Specify Driver

pg <- dbDriver("PostgreSQL")

# Connection Details

con <- dbConnect(pg, user ='your_user', password ="your_password_here", host = "your_host_here" , port = 5432, dbname="your_db_here")

# List all Tables in database (to test the connection is working)


# Read the PostGIS table "layer_1" into a 'data.frame' for the dynamic labels

dbReadTable(con, "layer_1", schema = "layer_1_schema_here") %>% head(3)

# list the Fields in the table 

dbListFields(con, "layer_1", schema = "layer_1_schema_here")

# Query the tables for each layer you require

query <- "SELECT * FROM layer_1"
layer_1 <- pgGetGeom(con, c("layer_1_schema_here","layer_1"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

query <- "SELECT * FROM layer_2"
layer_2 <- pgGetGeom(con, c("layer_2_schema_here","layer_2"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

# Plot the layer in Viewer


# Change the projection from OSGB36 (epsg:27700) to WGS 84 (epsg:4326)

layer_1 <- spTransform(layer_1, CRS("+init=epsg:4326"))
layer_2 <- spTransform(layer_2, CRS("+init=epsg:4326"))

# UI Function

ui <- fluidPage(theme = shinytheme("cerulean"),
                tags$head(HTML("<title>myTitle</title> <link rel='icon'type='image/png' href='myImage.png'>")),
                titlePanel(title=div(img(src="myImage.png", height = 100, width = 100, align = "left","myTitle"))),
                h4("My h4 text here."),
                p("My paragraph text here"),
                leafletOutput("layer_1", height = 500))

# Server Function

server <- function(input, output, session)
output$layer_1 <- renderLeaflet ({
    map <- leaflet(layer_1) %>%
      flyTo(-0.0000, 00.0000, zoom = 10) %>%
      addTiles(group="OpenStreetMap.Mapnik") %>% 
      addProviderTiles("Esri.WorldImagery", group = "Satellite") %>%
      addProviderTiles("OpenStreetMap.HOT", group = "OpenStreetMap.HOT") %>%
      addProviderTiles("Stamen.TerrainBackground", group = "Stamen.TerrainBackground", options = providerTileOptions(minzoom = 1, maxzoom = 13)) %>%
      addMiniMap(tiles = "OpenStreetMap.HOT", toggleDisplay = TRUE) %>%
      addSearchOSM() %>%
      addEasyButton(easyButton(icon="fa-globe", title="Zoom to Level 1", onClick=JS("function(btn, map){ map.setZoom(1); }"))) %>%
      addEasyButton(easyButton(icon="fa-crosshairs", title="Locate Me", onClick=JS("function(btn, map){ map.locate({setView: true}); }"))) %>%
                  fillColor = "#e41a1c", 
                  fillOpacity = 0.4, 
                  stroke = "#690E0E",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~name_of_your_table_attribute_column_to_label_here,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_1",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
                  fillColor = "#377eb8", 
                  fillOpacity = 0.4, 
                  stroke = "#20445F",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~yourLabel,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_2",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
      addLegend("bottomleft", title = "Map Legend", colors = c("#e41a1c","#377eb8"), labels = c("layer_1","layer_2")) %>%
      addFullscreenControl() %>%
      addMeasure() %>%
      addScaleBar(position = c("bottomleft")) %>%
      addLayersControl(baseGroups = c("OpenStreetMap.Mapnik","Satellite","OpenStreetMap.HOT","Stamen.TerrainBackground"),
                       overlayGroups = c("layer_1","layer_2"),
                       options = layersControlOptions(collapsed = TRUE))

# ShinyApp Function

shinyApp(ui = ui, server = server)

Friday, 15 February 2019

Joplin - An Open Source alternative to Evernote

I never thought I would need a 'to-do' list or a 'note taking' application before. I turn up at meetings with my trusty black pocket book and a pen. How 'old school' I hear you say but I am always late to the party. I signed up to Evernote some years ago but that is as far as I got. Now I cannot stop adding stuff to remember in 'Joplin'. So what is so different about this syncing 'to-do' app (compared to the Evernote free version)?

Evernote has no 'markdown' support.
Evernote has no 'end to end' encryption.
Evernote has a two device limit.
Evernote cannot work offline.
Evernote isn't open source.

The name 'Joplin' comes from composer Scott Joplin

"Joplin is a free, open source note taking and to-do application, which can handle a large number of notes organised into notebooks. The notes are searchable, can be copied, tagged and modified either from the applications directly or from your own text editor. The notes are in Markdown format.
Notes exported from Evernote via .enex files can be imported into Joplin, including the formatted content (which is converted to Markdown), resources (images, attachments, etc.) and complete metadata (geolocation, updated time, created time, etc.). Plain Markdown files can also be imported.
The notes can be synchronised with various cloud services including Nextcloud, Dropbox, OneDrive, WebDAV or the file system (for example with a network directory). When synchronising the notes, notebooks, tags and other metadata are saved to plain text files which can be easily inspected, backed up and moved around.
The application is available for Windows, Linux, macOS, Android and iOS. A Web Clipper, to save web pages and screenshots from your browser, is also available for Firefox and Chrome". Source:-

Installation on Ubuntu (copy and paste) in terminal and press 'return'

wget -O - | bash

Thanks to my colleague Phil Barlow for introducing me to it.

Featured post

Qgis-server...Installing the QGIS Lizmap Plugin & Lizmap Web Client

This post follows on from my previous three (most recent first in list) linked below. There is no doubt that in just a short time from no...