site banner
naraburns  nihil supernum  1yr ago (text post)   3961 thread views

Friday Fun Thread for March 3, 2023

Be advised: this thread is not for serious in-depth discussion of weighty topics (we have a link for that), this thread is not for anything Culture War related. This thread is for Fun. You got jokes? Share 'em. You got silly questions? Ask 'em.

4
Top Bottom Old Controversial Comments
Jump in the discussion.

No email address required.

I've spent the weekend playing around with geodata. My original idea was simple: I wanted to draw a fantasy map of Russia. For that I needed a vector map of population density. Russian federal subjects are huge, so I needed a more granular map. I couldn't find one I could load into QGIS, so I thought to myself: how hard can it be?

  1. I downloaded a data dump for Russia from OSM.

  2. I converted it into Spatialite to query for admin boundaries.

  3. Turns out the JDBC driver for SQLite doesn't work in DBeaver when you want to load a native extension. Like Spatialite. Or rather, it works in the paid version of DBeaver. DBeaver itself recognizes the geometry field, but I can't run any geo functions.

  4. While I was screwing around with the data in DBeaver I realized it contained pieces of other countries. No, I am not talking about Crimea.

  5. I gave up on Spatialite and installed PostGIS. This allowed me to check for weird intersections between admin boundaries. None found. Good job, OSM mappers!

  6. I also realized I needed to somehow link the data to the census data to get the population. My only connection was going to be Wikidata.

  7. Turns out 12 admin boundaries in OSM don't have a Wikidata link. I actually went and fixed them in OSM.

  8. I then queried Wikidata for each boundary and got a bunch of JSON I parsed.

  9. Then I finally got around to the big problem of Russia having two types of boundaries: admin (OKATO) and municipal (OKTMO). They match, but not quite. Oh well, that's a problem for future orthoxerox to solve.

  10. But at the very least I now had a list of (presumably admin) boundaries from OSM I could join with Wikidata data, netting me a bunch of OKATO and OKTMO codes.

  11. I downloaded the list of OKATO codes and joined it with my boundaries for a sanity check. Found a few more admin boundaries from China and Japan and a few Wikidata entries without OKATO codes. Something for me to fix later.

  12. What's worse, a few small pieces of Russia were missing in the OSM dump. I missed them because I loaded the vector layer in QGIS, took a look at it and thought: "yep, looks like Russia to me".

  13. Now I am trying to find out what went wrong with the dump, but OSM forum authentication decided to die as soon as I registered there.

All in all, pretty entertaining. Now that I'm blocked from proceeding further, I can go look at the way DuckDB reads legacy Parquet timestamps.

I've never worked with GIS software before, so I don't know what they are usually capable of.

But I did work on a nasty boundary polygon problem in the past and the turfpy (I think the original package was made for node) package along with the geojson package in python was tremendously useful for sorting out some of the issues that were similar to yours.

Thank you, I'm pretty happy with the tools I have. PostGIS and QGIS have got me covered, it's the data quality that got me this time.

What is this place?

This website is a place for people who want to move past shady thinking and test their ideas in a court of people who don't all share the same biases. Our goal is to optimize for light, not heat; this is a group effort, and all commentators are asked to do their part.

The weekly Culture War threads host the most controversial topics and are the most visible aspect of The Motte. However, many other topics are appropriate here. We encourage people to post anything related to science, politics, or philosophy; if in doubt, post!

Check out The Vault for an archive of old quality posts. You are encouraged to crosspost these elsewhere.


Why are you called The Motte?

A motte is a stone keep on a raised earthwork common in early medieval fortifications. More pertinently, it's an element in a rhetorical move called a "Motte-and-Bailey", originally identified by philosopher Nicholas Shackel. It describes the tendency in discourse for people to move from a controversial but high value claim to a defensible but less exciting one upon any resistance to the former. He likens this to the medieval fortification, where a desirable land (the bailey) is abandoned when in danger for the more easily defended motte. In Shackel's words, "The Motte represents the defensible but undesired propositions to which one retreats when hard pressed."

On The Motte, always attempt to remain inside your defensible territory, even if you are not being pressed.


New post guidelines

If you're posting something that isn't related to the culture war, we encourage you to post a thread for it. A submission statement is highly appreciated, but isn't necessary for text posts or links to largely-text posts such as blogs or news articles; if we're unsure of the value of your post, we might remove it until you add a submission statement. A submission statement is required for non-text sources (videos, podcasts, images).

Culture war posts go in the culture war thread; all links must either include a submission statement or significant commentary. Bare links without those will be removed.

If in doubt, please post it!


Rules


Recommended Posts And Communities

Recommended Realtime Chats