Philly, Beyond the Grid

A short list of a city’s physical elements may include: buildings, streets, public spaces, and the natural landscape. All contribute something to a place’s geographical character, but, in urban environments the scale, concentration, and sheer number of buildings allow the architectural element to achieve visual and spatial dominance, a condition that imbues architecture with particular power over the physical orientation of individuals moving through the cityscape. However, once urban entities are translated into cartographic form, the street grid supersedes as the prominent navigational element—an argument supported by the default zoom level of both Google Maps and OpenStreetMap. Building polygons are not yet visible, space is explained with densely nested hierarchical ribbons of asphalt. But what happens when these familiar elements are stripped from the urban map? Are there other physical elements that can express the presence and characteristics of a metropolis? Natural features like mountains or rivers may be enough to orient the acquainted viewer, but for the unfamiliar, for example, the confluence of the Schuylkill and Delaware Rivers does little to impart that there are 1.5 million people living in between.

The industrial elements of Philadelphia—power lines, railways, power plants—were mapped using data from OpenStreetMap (download from Metro Extracts, http://metro.teczno.com/) and referenced against the accompanying OSM wiki. OpenStreetMap is entirely volunteered geographic information, yielding some interesting insights about what exactly the public feels compelled to map. Features were styled using CartoCSS in TileMill. The terms industrial and utility were taken to mean any structures or conduits that help facilitate general commercial activity, but are generally not directly accessed by the general public. For example, individuals among the general public access electricity but do not come into direct contact with power lines, thus power lines are an industrial/utility feature. An exception to this rule was the inclusion of railways, which serve to move both freight and the general public. They were included because 1.) While both Google Maps and OpenStreetMap demarcate railways, they are clearly secondary to roadways in visual hierarchy, 2.) Because unlike roadways, the general public is unable to make navigational choices or even see an unobstructed view along these routes and so are less likely to be able to conceive railways in their entirety, and 3.) The great dependence of manufacturing and industrial activity on rail systems. Though freight often moves along highways, roads were not included for the first two reasons listed in the last sentence; this is a cartographic exercise in removing the two most familiar means of urban cartographic orientation. Four categories of features were styled relating to general industry (factories, industrial land use, storage tanks), railways (railroads, railyards, train stations, and excepting subway or monorail lines which move people not freight), power (power cables, power lines, electrical poles, portals), and water (pipelines, canals, piers, drains, ditches).

Mapped, the city’s industrial underbelly clearly offers suggestions of Philadelphia’s scale and regional relationships. Cities are intense agglomerations of human effort and material exchange. These industrial underpinnings are the conduit of that interchange.  Rail lines radiate from the city’s center, but also show a distinct southwest to northeast direction, suggestive of Philadelphia’s regional context in the country’s northwest urban corridor from Washington, D.C. to Boston. Industrial buildings and land uses also follow the same diagonal pattern, reiterating the interstates that connect Philadelphia with the other regional metros. Railways visually dominate the map, perhaps because the public, who volunteered the data, interacts most frequently with this industrial feature. Interestingly, power lines and poles radiate into Philadelphia, but disappear in the city’s actual confines, their ubiquity is superseded by the cacophony of other forms.

Creating Web Maps in Python Using Folium – First Impressions

Folium is a Python package built to bridge the data wrangling muscle of Python with Leaflet’s easy-to-use JavaScript library for creating attractive, interactive web maps. The open source Leaflet is a highly popular web mapping tool due to its flexibility, with a healthy number of community-developed plug-ins further expanding its native capabilities. While Python is a robust programming language, with many packages contributing to geospatial analysis– Pandas, GeoPandas, Fiona, Shapely, Matplotlib, and Descartes to name a few– Folium differentiates itself through ease of use and the interactive potential of the final product. After some experimentation with the library, it did not take very long to produce a functional, albeit simple, web map with clustered point data, accompanied by popup windows. However, it was obvious that there is more to explore with Folium, as it plays well with many types of geospatial data, includes built-in functions and methods for producing choropleths, temporal visualizations, and allows for the marriage of the best of Python and Leaflet.

The code and resulting maps show a straightforward exercise in extracting the geographic coordinates (already matching Leaflet’s default web-mercator projection) and a few attribute values corresponding to warehouse/distribution centers in Pennsylvania’s Lehigh Valley from an excel spreadsheet. The Pandas library was used to read the excel document and convert the desired information to a dataframe. Folium was used to initialize a Leaflet map, add records as points with some stylization applied. This is brief code that could easily be added at the end of a more intensive spatial analysis using Python. It can provide a quick way to publish results in an interactive format without necessitating the use of JavaScript/html/CSS, or could serve as a jump start on more elaborate styling.

Map #1 – Mostly default styling

import pandas as pd
import folium

#converting the imported data to a pandas dataframe object

df = pd.read_excel(“Warehouses.xlsx”)
#identifying the long, lat columns, and other properties for the popup window
df.head()
pdlat='Latitude'
pdlon='Longitude'
#removing unwanted columns
avail =df[3:]
#initalizing folium map object as m and using the geographic mean of the data points to center the viewpoint; basemap defaults to OSM
m=folium.Map(location=[df[pdlat].mean(), df[pdlon].mean()], zoom_start=12)
#Iterate through edited dataframe to extract coordinates and property name for each record
for row in avail.iterrows():
prop = str(row[1]['Property'])
lat = row[1][pdlat]
lon = row[1][pdlon]
#attach each record to a default marker style based on coordinates and add property name to popup window
m.simple_marker(location=[lat, lon], marker_color='red', popup=prop)
#outputting html document with code for an interactive map to working directory
m.create_map('map.html')

Map #2 – Custom MapBox Tiles, Clustered Markers, Different Icon Imagery

Only showing code that differs
#Generate map using custom Mapbox tiles
m=folium.Map(location=[df[pdlat].mean(),
df[pdlon].mean()], zoom_start=9,
tiles='https://api.mapbox.com/styles/v1/username/yourstyle/tiles/256/{z}/{x}/{y}?
access_token=pk.yourtokenhere',attr='My data attribution')

#Iterate through edited dataframe to extract coordinates and property name for each record

for row in df.iterrows():
prop = str(row[1][‘Property’])
lat = row[1][pdlat]
lon = row[1][pdlon]
#used the marker_icon argument to select from natively supported bootstrap supported icons and added clustering affect to markers
m.simple_marker(location=[lat, lon], marker_color='red', marker_icon='flag', clustered_marker=True, popup=prop)

Using FFmpeg as an Animation Conversion Command Line Tool

Editing maps and doing data analysis, that’s my thing! But when it comes to video editing, I’m a bit out of my league. In addition to that, I have no expensive software that will make things look cool and fun. Or at least I thought I didn’t.

FFmpeg is “a complete, cross-platform solution to record, convert and stream audio and video.” You can download it here. This tool is limitless when it comes to production.

So, working with the time series images that I created from the precipitation data and working with FFmpeg, I was able to create an animation!

You’ll be working in command line window so you need to know where that lives. For me and my Windows 10 computer, it lives here … “C:\Windows\WinSxS\amd64_microsoft-windows-explorer-shortcuts_31bf3856ad364e35_10.0.10586.0_none_443d824ebb4341e2\01 – Command Prompt.lnk” but it may be easier just hitting the windows start button and typing command prompt.

ffmpeg -framerate 24 -i Rplot_%04d.png output.mp4

My images were named Rplot_0001.png. If your image was named img_001.jpeg you would refer to your image as img%03d.jpeg. That took me a few minutes to figure out but after I hit enter, I had a time series precipitation video. I wasn’t very specific in my frame rate but you can be. I received a lot of help from this FFmpeg wiki.

imagestovideo

Plotting Time Series Graphs for Animation with MapMate Package in R

Animations can be a little tough to create but luckily, Matthew Leonawicz has created a package for R that plots and saves images easily called MapMate. The map portion of the package I have yet to master, as it is globe based and my project was to map a static city. But the time series climate data of precipitation that I collected from The National Oceanic and Atmospheric Administration Department of Commerce was a great project to produce an animation.

Needed for the images to save is ImageMagick which is a free download.

The first step to creating an animation was to make a sequential numeric column named FrameID for the order of plotting. The next step was to identify the x limits and the y limits.

climate1 <- climate
climate1$FrameID <- 1:nrow(climate1)                  
xlim <- range(climate1$Date)
ylim <- range(climate1$Percip)
save_ts(climate1, x = "Date", y = "Percip", id = "FrameID", col = "grey",
         xlm = xlim, ylm = ylim, dir = "C:/Users/tuf29742/Documents/Vector Control/Animation/")

With x as the date and the y as the precipitation, and a specified location for the images to be saved, this image creator took roughly 30 minutes to produce 3,159 images for animation. Below are just 5 images from the package capture.

These images could now be used in a video editor to create an animation. I would check out Matt’s MapMate Instructional Page for more fun with animation!

Working with Time Series Data in R

Time series analysis is very useful in economics but I have been using it in my studies of mosquitoes and climate change in environmental health. The relationship that I am examining is whether or not the climate change had any effect on increased mosquito quantity within trapping areas over 17+ years. The first step of that process is to see if there was a change in climate over the 17+ years that match the mosquito collection data.

The mosquito season starts in May and ends in October, and it is common for field workers to set traps every Monday through Thursday, leave them for 24 hours and then collect the traps to have the mosquitoes tested. So, all precipitation and temperature data was collected from The National Oceanic and Atmospheric Administration Department of Commerce.

Working within R and using the Stats package, I was able to transform the dataframe into a time series class and visualize over time.

library(stats)print(climate)
class(climate)
length(climate)

Results :

print(climate)
Temp Percip       Date       time
1      55   0.00   5/1/2000 2000-05-01
2      59   0.09   5/2/2000 2000-05-02
3      60   0.00   5/3/2000 2000-05-03
...
class(climate)
[1] "tbl_df"     "tbl"        "data.frame"
length(climate)
[1] 4

Now, we transform the dataframe to a ts object

climatets <- ts(climate, start = 1, frequency = 1)
print(climatets)
length(climatets)

Results :

> climatets <- ts(climate, start = 1, frequency = 1) Warning message: In data.matrix(data) : NAs introduced by coercion
print(climatets)
Time Series:
Start = 1
End = 3159
Frequency = 1
Temp Percip Date
1   55   0.00   NA
2   59   0.09   NA
3   60   0.00   NA
...
length(climatets)
[1] 12636
plot(climatets)


It is common for string dates to be eliminated in the time series transformation. If you would like to keep them for labeling purposes, I would investigate ?as.POSIXlt() for date time numeric conversion.

You can see that there is a slight upward trend in temperature over the 17+ years. But for the most part, the climate over the 17 years doesn’t change much in trend. Which tells me that if there has been an increase in mosquito quantity at trapping sites, it may not be connected to temperature.

This way of visualizing data is essential in making predictive modeling and recognizing trends.

Alternative Graphics Editor

As someone that is about to lose access to my Temple sponsored software one of the things I’m going to miss the most is Adobe Illustrator (AI). The Cartographic Design requirement for the PSM in GIS really made it clear how a few simple image adjustments using a program like Illustrator can make a map product exported from ArcGIS or QGIS look thousands of times better. Luckily while attending a recent Technical Workshop at Azavea one of the lightning talks was about improving product visualization. The speaker Jeff Frankl, User Experience Designer at Azavea demonstrated Figma.com, an online alternative to AI. Currently Figma is free to use as an individual, with the ability to have three active projects stored for 30 days. The user interface is very similar to AI, but if you’re new to both programs there are plenty of instructional videos, the major difference being that Figma projects can be shared online and worked on collaboratively.

There are also plenty of other free or low cost design products like Inkspace and GIMP there is a pretty thorough list available at this link.

 

Adding Citations to R Markdown

Versions of this blog entry are available as an HTML page, PDF and a text file containing the info from the Rmd file.  If you download the text file and save it with an .Rmd extension you will be able to create this document in R Markdown and edit it to fit your report requirements.

Abstract

Putting this section in between asterisks prints it out in italics. This blog goes shows you how to add citations to an Rmarkdown document.

Introduction

Adding citations is an important part of any complete document, this blog in addition to the previous entry about setting up an R markdown document should hopefully get you started.

Methods

  • Create a new R markdown document and save it with the file extension .Rmd to your working directory, which should be set somewhere convenient as you need to save other files to this location. Delete everything in that file but the info at the top in between the set of three dashes. Add a new line to that section ‘bibliography: bibliography.bib’.
  • You will need to save the references as a .bib file. Luckily for most items available from the Temple University Library system most journals and other items have an ‘Export citation’ look for something that says .bib or BibTex.
  • Save those files and open a new text file named bibliography.bib in your working directory, copy the data from the library file to your bibliography file. Example here
  • The format to create your own .bib file is written up very clearly here  if you can’t get the completed file from your source library.
  • In your text if you want to cite something just use the @ sign and brackets to wrap the name of the reference from the first line of the .bib data. This usually is just the author’s name underscore year.
  • For example writing [see @R_Core_Team_2017] produces (see R Core Team 2017)
  • Or just write @R_Core_Team_2017 so the text appears as part of the docume nt like this: as noted in R Core Team (2017)
  • References will show up at the bottom of your document after last text entry, so add a header and you’re done!

As an extra bonus R Studio has a spell-checking function- just hit F7

References

Osborne, Martin J. 2008. “Using Bibtex: A Short Guide.” https://www.economics.utoronto.ca/osborne/latex/BIBTEX.HTM.

R Core Team. 2017. R: A Language and Environment for Statistical Computing. Vienna, Austria: R
Foundation for Statistical Computing. https://www.R-project.org/.

Using R Markdown to Create Reports

Intro

Using R and Rstudio along with the R Markdown package allows you to produce documents in markdown that can be easily converted to MS Word, PDF, or saved as an HTML file that can be hosted as a website. The R Markdown file can contain just text like simple written report or a much more complex document with embedded R code to create charts, graphs, maps, or any other plot that can be produced using R. The text plots and formatting can all be created in a single document with a streamlined workflow. Instead of producing charts, tables, and graphs in MS Excel, and then placing those items in the text of a Word document, and then adding any other images (maps etc.) all that work can be completed in one place. If the report is required to be replicated with new data, it is simple to update the data rerun the code and produce a new version of the document.

Important links

The following preliminary steps should be taken before proceeding.
Download and install:
R https://cran.r-project.org/bin/windows/base/.
RSTUDIO https://www.rstudio.com/products/rstudio/download/#download.
To produce PDF documents you may also have to download and install pandoc, which is available here.

Package Installation

In the tools menu of RStudio click ‘Install packages’ and type Rmarkdown, make sure that the ‘install dependencies’ box is checked, that will also install some other important packages, including knitr.

For reference you can download the most recent Rmarkdown cheat sheet by navigating to the help menu and selecting the appropriate file under cheatsheets.

Once all of that is completed go to the file tab in RStudio and select new file -> R Markdown fill out the boxes and click okay on the window that pops up, that info can be updated later.

Save the new document that R Studio opens somewhere convenient and make sure the file ends in “.Rmd” (the case is very important here).

See the slightly darker bands? Those are called ‘code chunks’ within each chunk you can select various options via the gear symbol such as: to display the R code, to run and display the code, to not run or display code (this is used when you want to set up for an action completed later -like call a library), show or hide warnings in your final document etc.

The other sections are where you place your text. The text is formatted using the Pandoc format (summarized on the cheatsheet) which shows up after create the in your desired format. A # means Header size 1 and two spaces creates a paragraph break etc. You can play with the formatting of the sample page, add text and format a few lines to see how they come out.

Then select the small pull down at the top of the Source panel (where your new document appeared) that says Knitr and looks like a ball of yarn -there are options to Knit to HTML, PDF or Word,

From the Rmarkdown cheatsheet:
When you render, R Markdown
1. runs the R code, embeds results and text into .md file with knitr
2. then converts the .md file into the finished format with pandoc

Running R Code

Here is a flow chart of the R Markdown process which was included to show how to include plot made by the R code:

```{r, echo=TRUE}
DiagrammeR::DiagrammeR("graph LR;
 A(Rmd)-->B((knitr)); 
 B-->C{pandoc};
 C-->D>HTML];
 C-->E>PDF];
 C-->F>Word];")
 C-->F>Word];")
```

As far as I can tell WordPress does not allow for directly loading a .md file so versions of this blog entry are available as an HTML page, PDF and a text file containing the info from the Rmd file.

Flow charts in R using DiagrammeR

This is really short but I was really excited to share this info. When I was trying to find a way to create a flow chart in R I started with a few different packages that required a huge amount of prep work  before plotting a flow chart, and then I found the magical DiagrammeR  package.

It really should be more difficult:

library(DiagrammeR)
DiagrammeR("graph LR;
           A-->B;
           B-->C;
           B-->D")

Creates this:

Pretty cool, Right?

By assigning each letter you can update the flow direction, text and shape

DiagrammeR("graph TB;
    A(Rounded)-->B[Squared];
    B---C{Rhombus!};
    C-->D>flag shape];
    C-->E((Circle));")

Creates this:

Notice that after graph it says TB (top bottom) vs LR, Text can be added, type of brackets dictate the output shape and — vs –> makes a line instead of an arrow.

This package is really impressive and the documentation has some  very interesting images of very complicated graphs and flow charts, this image was made using the example provided in the package instructions.