SnapLoc — Places of Interest in a City, From Geo-tagged Photos
SnapLoc is a product I have created as a final project of a data science bootcamp, Metis. It does automatic image classification and spatio-temporal analysis in order to recommend the places of interest for traveling in a new city. The idea came out of a need, as I find most of the current point of interest (POI) recommendation services very painful to use. I love to travel and whenever I have been to a new city and felt a need to know about the most interesting places to explore near me, I generally looked up interesting photos on various photo services like Instagram or 500px.
I mean the choice is between -
“Hawk Hill is a 923-foot peak in the Marin Headlands, just north of the Golden Gate Bridge and across the Golden Gate strait from San Francisco, California. The hill is within the Golden Gate National Recreation Area.”
If you are around Golden Gate, I highly recommend going to Hawk Hill at sunrise, as you would see something similar to the view in figure 1.
How did it all start?
I started looking at the pictures that people were posting on various services and saw that most of these pictures convey our interests, and can be put into categories such as food, natural scenes/scapes, urban scenes, wildlife, birds, etc. Further, I could see a pattern of places where there were more pictures taken than others and there were different kinds of pictures taken at different locations and different times of the day. The question that I wanted to explore using these spatio-temporal patterns was, how to use this data to build an application that could figure out how to parse them and make recommendations based on the user preferences. I also referred this paper to check the feasibility of the idea.
How is this application useful ?
- To recommend the most interesting places based on user preferences, for example a small cupcake shop that everyone has been posting about but unless you search for it, you would never know about it.
- To the growing number of photo enthusiasts, I want to provide a way to see some great pictures of well known places. For instance, there is an awesome view of Golden Gate from Hawk’s point but many do not know that.
- And finally I want to take the location recommender a step further to suggest a travel itinerary for a new place that user wants to travel to. It is also a photo opportunity recommender and a travel itinerary recommender based on hotspots and landmarks.
Damn, how do you implement this awesome idea?
To accomplish this, I looked at some 1 million geo-tagged images on Flickr from Flickr API and classified them in common categories of interest. For the first part of my pipeline, I trained a deep neural net of images, which would be able to classify any new image with high accuracy. Here the use of a convolutional neural network (CNN) also means that you can add more categories anytime in the future and it becomes an automatic classifier.
CNN Classification: I did a convolutional neural network with Inception V3 and got 0.81 accuracy on classifying these six classes. The benefit of doing the CNN here is I could add onto these categories, like what if I want to explore the location for adventures/activities that are going on around me and use transfer learning to be able to classify that category to make recommendations for that category to users. For classification, I referred to this notebook.
I popped the last layer of trained inception V3 model and added my labels instead:
For the second part, I took the location data of these images and did density-based clustering, which gave me hotspots in terms of geolocations. These are places, along with their categories, that people are talking about a lot. Then I ranked these images based on the number of user preference signals, popularity of photo, distance from user, as well as time of day. That’s how I created a personalized recommender of images based on category, location and time.
Spatial Clustering: For clustering, I used DBSCAN to come up with density based clusters with the help of latitude and longitude. Also, I had to convert these clusters into polygons to see the varying shapes and sizes of all my clusters, for which I used shapely and convex hulls. Finally, I did querying with KD-tree to get the nearest clusters and ranked these clusters based on user preferences, distance and time. This way I was able to provide personalized locations to users, relevant at that time. I referred the work for spatial cluster analysis from this github notebook.
I ran the DBSCAN clustering algorithm (DBSCAN helps removing outliers and bad clusters) for density clusters of image locations; you can see the result in the map in figure 2.
How will the application look ?
I also made a demo of my application using Flask where you enter the latitude, longitude and time. On the first page you get to select the cluster of photos that you want to look at, then clicking on that cluster you can see the bunch of images in that category. Here's another example we can look at near Golden Gate.
How could I miss out on these?
- If I'd had more time, I could have made the application more personalized based on user profiling and interests and given suggestions based on maybe just the photography of wildlife.
- I could complete the itinerary creation for a new place — for instance, create a map of landmarks and hotspots that could be covered in a day based on current location and time a person wants to spend in that area.
- And who says that only Flickr’s geotagged images can be used? The whole world is open and I could take up tagged images from Instagram, Facebook and many other sites to do the same.
Here is the link to Github Repo for this project: https://github.com/kalgishah02/SnapLoc
Kalgi Shah is a Data Scientist with a background in architecture, transportation, and GIS data modeling. This story originally appeared in Kalgi's Medium blogpost January 29, 2018 and was featured in Data Science Weekly Newsletter Issue #224.