Replies: 13 comments 16 replies
-
|
The problem is mainly that the location string is rather unstructured. Generally the country comes at the end but the middle parts are not so reliable and probably vary from country to country. I tried doing something similar to start with but gave up. You could try (in PictureFrame2020geo.py): geolocator = Nominatim(user_agent=config.GEO_KEY)
location = geolocator.reverse(geo_key, language=language).address.split(",")
location = [location[i] for i in [1, 2, -1]] ### <<<<<<<<<<<<<<<<<<<<<<<< 2nd, 3rd and last
location_split = [loc.strip() for loc in location]
formatted_address = "" |
Beta Was this translation helpful? Give feedback.
-
|
Thanks but I gave also up on yours. From Nominatim manual:
Maybe its possible to make the |
Beta Was this translation helpful? Give feedback.
-
|
Well done for looking further into that. It would actually be relatively trivial to not bother with geopy and just use nominatim directly. i.e. import json
import urllib.request
url = "https://nominatim.openstreetmap.org/reverse?format=geojson&lat={}&lon={}&zoom={}"
with urllib.request.urlopen(url.format(53.8, -2.8, 14)) as url: # lat, lon, zoom but other options available
data = json.loads(url.read().decode())
address = data['features'][0]['properties']['address']
print(address['city'])
print(address['suburb'])
# etc etcand supplying an email address if you are making a lot of requests. I will play around with this a bit more and maybe switch or provide an alternative version of Thanks again, I will let you know what I come up with Paddy |
Beta Was this translation helpful? Give feedback.
-
|
That sounds good. |
Beta Was this translation helpful? Give feedback.
-
|
Firstly, is there any progress with this? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, PictureFrame2020geo.py on the develop branch now includes a commented out section as per # alternative using OSM tags directly. NB you should comment out the section
# above from `geolocator = ..`, also your config.GEO_KEY should be your email address
import json
import urllib.request
URL = "https://nominatim.openstreetmap.org/reverse?format=geojson&lat={}&lon={}&zoom={}&email={}&accept-language={}"
with urllib.request.urlopen(URL.format(decimal_lat, decimal_lon, config.GEO_ZOOM, config.GEO_KEY, language)) as req:
data = json.loads(req.read().decode())
adr = data['features'][0]['properties']['address']
# change the line below to include the details you want. See nominatim.org/release-docs/develop/api/Reverse/
# because OSM doesn't use fixed tags you probably have to experiment!
# i.e. `suburb` will be invalid for rural images, `village` invalid for urban ones
if 'suburb' in adr and 'city' in adr:
formatted_address = "{}, {}, {}".format(adr['country'], adr['city'], adr['suburb'])
elif 'village' in adr and 'city' in adr:
formatted_address = "{}, {}, {}".format(adr['country'], adr['city'], adr['village'])
else:
formatted_address = ", ".join(adr.values())As mentioned in the comments the main issue with this is the non-regular tags used in OSM - not trivial to solve. You could add the file name to the address stored but, because the lat/lon used as key is accurate to 4dp you will often get several images all at the same location, only the first will be written to the file. You also have to decide what to do with the info, as well as strip it from the location text. Obviously you would also need to pass the file name to the |
Beta Was this translation helpful? Give feedback.
-
|
Hi, tested this and got it working except this part here. It always shows the complete geo location. I thought this part would "filter" the location (only show country, city and suburb/village) or did I misunderstand something?
|
Beta Was this translation helpful? Give feedback.
-
|
Are you sure there are the relevant tags "suburb" or "village" and "city" in the address info? Quite a few of my pictures don't have that. You could check by printing out a list of keys, here's some example code import json
import urllib.request
URL = "https://nominatim.openstreetmap.org/reverse?format=geojson&lat={}&lon={}"
with urllib.request.urlopen(URL.format(13.8, 44.8)) as req: # lat, lon
data = json.loads(req.read().decode())
address = data['features'][0]['properties']['address']
print(address.keys())which gives, for various random locations (That last is an iconic 29,029 peak on the border of Nepal and Tibet) |
Beta Was this translation helpful? Give feedback.
-
|
@vinyli85 In the restructured version of PictureFrame that I'm working on I've done something like this to get round the problem. key_list = [
['tourism','amenity','isolated_dwelling'],
['suburb','village'],
['city','county'],
['region','state','province'],
['country']]
...
adr = data['features'][0]['properties']['address']
# some experimentation might be needed to get a good set of alternatives in key_list
formatted_address = ""
if key_list is not None:
comma = ""
for part in self.__key_list:
for option in part:
if option in adr:
formatted_address = "{}{}{}".format(formatted_address, comma, adr[option])
comma = ", "
break # add just the first one from the options
else:
formatted_address = ", ".join(adr.values())so when applied to the examples I listed above would give
|
Beta Was this translation helpful? Give feedback.
-
|
So, it looks like each sub-list in The only thing that seems a bit odd to me is the multiple methods of CSV string creation ( vals = []
if key_list is not None:
for part in key_list:
for option in part:
if option in adr:
vals.append(adr[option])
break # add just the first one from the options
else:
vals = adr.values()
formatted_address = ", ".join(vals) |
Beta Was this translation helpful? Give feedback.
-
|
Clearly I am not understanding this properly (this is actually no surprise to me!). I assumed that, when an image is displayed, or, possibly, when the shuffled slideshow is created, the gps data would be picked up from the image's exif data. Hence why couldn't the filename be recorded at the same time and written to the text file along with the associated gps data? Re. the detail, perhaps I should say that I have not yet had the chance to try your new alternative using OSM tags. Martin |
Beta Was this translation helpful? Give feedback.
-
|
Martin, as the system is set up now you couldn't do that, to amend 'records' as you suggest would need more of a database setup. The system now is intended to create as small a location file as possible assuming that quite a few images will be taken in the same approximate location. The location data is just stored as a simple text file i.e. When the slideshow starts all the file is read into a dictionary object then, as each picture is loaded the lat,lon key is checked in the dictionary, if it's there the address is used, if it's not the address is looked up and added to the dictionary as well as appended to the txt file. The file name could be added to the record, say using additional '=' or '|' characters but that would require the whole file to be re-written for all cases apart from adding a file name to the very last record. As well as requiring lots of processing, writing many times to the SD card is generally a bad idea so an alternative method would have to be used. The obvious route would be to create a second file where each line is for each image and, to keep it simple, have the lat,lon and address on each line. To use that you would simply have to duplicate the code for the lat,lon keyed file and change it to be keyed on file name (NB you might need to use full file path as image file names are sometimes duplicated, on older digital cameras) |
Beta Was this translation helpful? Give feedback.
-
|
Martin, I've started the process of at least thinking about re-structuring the picture frame code (maybe using some ideas from https://github.com/helgeerbe/picture_frame or just adopt that and modify it) and it occurred to me that using a 'proper' database i.e. sqlite3 for the file_list which with a linked gps table and possibly others. That would allow things like switching from random order to date order to location order to directory/file_name order etc instantly even when the total number of images was very large. Issues to think about would be having the database in memory but saving to disk before shutdown (I think sqlite can do that, a while since I used it. But maybe it could be written to disk as it should be a write once operation) and it wouldn't be human readable so you would need a tiny program to view the data (either a few lines of python or, even easier just get one like https://sqlitebrowser.org/) There would also be the option of changing the description of the location i.e. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @paddywwoof, you did a great job with with viewer. If you like, I could offer you my support, if you going to refactor the code. Never tested it, how much time it takes so resort List with up to 1000 images. Does this take so much time? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I will definitely think twice (and do a few trials) before opting for a database. sqlite is already built into python core modules so there's very little to worry about on the installation side. I've not really tested many scenarios for timing but I have noticed that if you specify a fairly tight start and end date (maybe just including a single day for birthday pictures) it can take a noticeable time. This is especially true if there are no images at all matching the query. Recent issue here #44 was talking about a 2TB samba drive - that sounds like a lot of photos! I'm leaning towards getting people to use your picture_frame and the preferred option. I suppose it would need to include all the functionality that has accumulated in PictureFrame2020, but there's no point trying to develop two versions in parallel. Paddy |
Beta Was this translation helpful? Give feedback.
-
|
OK start dreaming right now. I had in mind to integrate a small webserver to act as a camera for homeassistant, so one can see which image is actual displayed. If we use a database we could get rid of the huge config file. Storing all config data in the database and do the settings through a webinterface. On could also change the metadata of the images in the webinterface. If we implement a service-facade to the controller. It could be controlled by mqqt, keyboard or webinterface and ..... |
Beta Was this translation helpful? Give feedback.
-
|
There are some really nice ideas. Looking forward to this. |
Beta Was this translation helpful? Give feedback.
-
|
I was actually going to add a web interface on an informal basis to the frame I've just put in here for my wife. Setting up MQTT apps on the phone is easy enough for me but for others it's much simpler to just open a web page! So long as it's not accessible from outside the lan and speed and isn't important then adding a web server in python is pretty trivial. As you say, much better to split the functionality out into a multipurpose interface. Maybe running in its own process - I had problems previously as the web server wanted to be in the main thread and pi3d has to run in the main thread. Easy enough to use different processes and queues. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds great Paddy. |
Beta Was this translation helpful? Give feedback.
-
|
Just a note asking - is it really the case that you will need a database?
Could you not store the data as JSON? Then anyone could read and write it
with any tools
…On Sat, Jan 9, 2021 at 9:26 PM paddywwoof ***@***.***> wrote:
I was actually going to add a web interface on an informal basis to the
frame I've just put in here for my wife. Setting up MQTT apps on the phone
is easy enough for me but for others it's much simpler to just open a web
page! So long as it's not accessible from outside the lan and speed and
isn't important then adding a web server in python is pretty trivial.
As you say, much better to split the functionality out into a multipurpose
interface. Maybe running in its own process - I had problems previously as
the web server wanted to be in the main thread and pi3d has to run in the
main thread. Easy enough to use different processes and queues.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#41 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB53MQJFHD4WJUJHJVFUUDSZC3YJANCNFSM4VCFVVXQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Tom, good to hear from you. Yes, I'm not really sure about the db but the kind of thing like I think, even with JSON to hold dictionary info the files wouldn't necessarily be easy to use. In order to not re-write the whole structure to SD every time a new image was loaded the images would need to be appended to one file while gps locations where append to another (and possibly other info added to other 'tables'). I will do some poking around and see what I find. |
Beta Was this translation helpful? Give feedback.
-
|
OK, I've nearly convinced myself that normal python iteration over dictionaries will be fine for any reasonable number of images, and is pretty easy to do (just a function for each type of query and img_list = list(filter(lambda img: func_q(img, from, to), img_table))type of thing). On RPi3 it seems to generate a filtered list from just the images (i.e. assuming that contains file_name, date, lat, lon, etc) in 20ms for 10,000 images and a lookup select (i.e. checking if 'Germany' is in the geo_table) in 60ms for 10,000 images. And they seem to be O(n) as expected, so 6s for 1,000,000 images. I think the saved form of the tables needs to be plain CVS style rather than JSON as records need to be added on the end as soon as a new image file or geo location is opened. |
Beta Was this translation helpful? Give feedback.
-
|
Heya, hope you're doing OK in this crazy, crazy world! One day I'll come
visit.
CSV is fast, fairly small, but doesn't deal with heterogeneous rows, or
entries that aren't either a string or a number.
What about line delimited JSON:
https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON ?
for line in open(file):
yield json.loads(line)
or even
return [json.loads(i) for i in open(file)]
…On Sun, Jan 10, 2021 at 11:04 PM paddywwoof ***@***.***> wrote:
OK, I've nearly convinced myself that normal python iteration over
dictionaries will be fine for any reasonable number of images, and is
pretty easy to do (just a function for each type of query and
img_list = list(filter(lambda img: func_q(img, from, to), img_table))
type of thing). On RPi3 it seems to generate a filtered list from just the
images (i.e. assuming that contains file_name, date, lat, lon, etc) in 20ms
for 10,000 images and a lookup select (i.e. checking if 'Germany' is in the
geo_table) in 60ms for 10,000 images. And they seem to be O(n) as expected,
so 6s for 1,000,000 images.
I think the saved form of the tables needs to be plain CVS style rather
than JSON as records need to be added on the end as soon as a new image
file or geo location is opened.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#41 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB53MWMSAP6KSSW7BHJ7Y3SZIP7LANCNFSM4VCFVVXQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Interesting discussion. But file manipulation is not an easy task. I would indeed tend to use sqlite as the data store (used for create, update, delete). As far what I have read you can also store Json objects and query its elements. So why don’t we use the database?
|
Beta Was this translation helpful? Give feedback.
-
|
Helge, you are right that our first steps should be to think what the application of the information should be (what filtering or sorting as well as caching to avoid reprocessing), then think about the scope of the information (what 'fields'), then think about structure of the information (what 'tables'), then think about the mutability (enable changes of info within a 'table' or always append revisions). Only at the end of that should we decide on the db, CSV, JSON etc type questions. Tom, for a while it seems slightly less crazy on this side of the Atlantic! Are you still in NL? At some stage it will become feasible to travel more than walking distance from home, then you should visit the land of warm beer and driving on the wrong side of the road. |
Beta Was this translation helpful? Give feedback.
-
|
I've only been poking around with PictureFrame2020.py for less than a week now, but my first inclination was to put a SQLite-based cache behind it too - so I was encouraged to see that being discussed here. I've created a number of commercial software packages backed by SQLite and have never been sorry. Once it's in place, it's just so useful - both for things you're thinking about now and things you won't think about until later. That said, my SQLite experience is via Tcl/Tk and C# - never Python. I'd also agree with the sentiment that adding meta-data management / editing to a "photo frame" app seems like a stretch. Photo meta-data is a highly-complex topic (EXIF, IPTC, XMP, ...) and non-trivial to get right. Anyway, it's great to see all the discussions and activity. I'm interested to see where this goes. |
Beta Was this translation helpful? Give feedback.
-
|
Why not just use the Python csv module?
…On Thu, Jan 28, 2021 at 7:58 PM Jeff Godfrey ***@***.***> wrote:
So, it looks like each sub-list in key_list is a collection of *similar*
location data, in order of usage-preference. That looks like a really slick
solution - nice! So, a user could define their preferences there if
desired, or just leave it blank to get everything that was returned...
The only thing that seems a bit odd to me is the multiple methods of CSV
string creation (join always seems cleaner to me). Maybe that could be
normalized to something like this?
vals = []
if key_list is not None:
for part in key_list:
for option in part:
if option in adr:
vals.append(adr[option])
break # add just the first one from the options
else:
vals = adr.values()
formatted_address = ", ".join(vals)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#41 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB53MXCDTUK4CNHOJB74WTS4GXXBANCNFSM4VCFVVXQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
I've had a go at adding the existing PictureFrame functionality to github.com/helgeerbe/picture_frame and while I was at it added a basic cache system https://github.com/paddywwoof/picture_frame/blob/develop/picframe/model.py#L117 and https://github.com/paddywwoof/picture_frame/blob/develop/picframe/model.py#L335 it's pretty simple and the advantage over plain csv is that the more complicated record structure (json below) can be dumpsed and loadsed v. easily. But it's rather clunky 'linking' the lists and dicts. I will try to delay making a final choice for the cache system until the last minute but an advantage of sqlite is that the 'duplicated' info can be split out into different tables very easily and, as you can see, there is a lot of repeated info cluttering up the text file. i.e. field names (which csv would improve), file path and floats as text (which is always going to happen with serialized data) Anyway there's nothing like trying it with some real pictures to realize all the subtleties (aka bugs) Paddy {"fname": "/home/pi/Pictures/general_photos/BA4D0FD6-9C11-4FE6-9651-6A3FFC8148E8.jpg", "orientation": 1, "mtime": 1599558601.0, "dt": 1599558601.0, "fdt": "Sep 08, 2020", "lat": 575321, "lon": -61420, "aspect": 1.334, "shown_with": null, "image_attr": {"latitude": 57.532066666666665, "longitude": -6.142108333333334, "EXIF FNumber": 1.8, "EXIF ExposureTime": "1/1316", "EXIF ISOSpeedRatings": "32", "EXIF FocalLength": "17/4", "EXIF DateTimeOriginal": "2020:09:08 10:50:01", "Image Model": "iPhone 11 Pro"}, "location": null} |
Beta Was this translation helpful? Give feedback.
-
First, I'll start by saying that I'm no Python expert and I'm not familiar with Python's
|
Beta Was this translation helpful? Give feedback.
-
Is it possible to customize the geolocation text?
Currently there is a very long output: e.g.: street, city, post code, district, state, country
Depending on display size and text size this output is up to 3 lines (in my case)
I would like to have just the city, state and country as text. Is there a way to make this customizable?
Thanks,
Timo
Beta Was this translation helpful? Give feedback.
All reactions