Jekyll2024-03-18T16:59:52+00:00https://blog.k-nut.eu/k-nut — BlogThoughts and insights by Knut HühneA microdata enhanced HTML Webcomponent for Leaflet2024-03-18T13:30:00+00:002024-03-18T13:30:00+00:00https://blog.k-nut.eu/leaflet-html-webcomponent<p>The people in my RSS reader have been talking some more about Web Components in the
past couple of months. Jim Nielsen had <a href="https://blog.jim-nielsen.com/2023/html-web-components/">a post</a>
referring to Jeremy Keith coining the term <a href="https://adactio.com/journal/20618">HTML Web Components</a>
for a special kind of Web Components. The idea essentially is that you have regular HTML markup that is wrapped
by a custom Web Component which enhances the user experience.</p>
<p>I thought that this was quite an interesting idea and thought about situations in which
it could be applied. Working with open data quite a bit, we often find ourselves in
situations where we build maps. Additionally, the German open data scene has been lobbying for
more linked open data in the past couple of months. I think I came up with an example which
combines these three things - HTML Web Components, maps and Linked Data (or a flavor
thereof) quite nicely.</p>
<p>As an example, assume that we want to render schools on a map. The data in
our example is taken from our <a href="https://jedeschule.codefor.de/docs">jedeschule.de API</a>
which has the goal of making all primary and secondary schools in Germany
searchable and queryable.</p>
<p>With the approach I’m proposing, we can author markup that looks like this:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><leaflet-map></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"map"</span><span class="nt">></div></span>
<span class="nt"><ul</span> <span class="na">class=</span><span class="s">"locations"</span><span class="nt">></span>
<span class="nt"><li</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/School"</span> <span class="na">data-fresh-key=</span><span class="s">"NW-153710"</span><span class="nt">></span>
<span class="nt"><h2</span> <span class="na">itemprop=</span><span class="s">"name"</span><span class="nt">></span>Dahlingschule, Städt. Förderschule im integr. Verbund, FSP Lernen u. Emot. und soziale Entwicklung,-Primarstufe u. SekI<span class="nt"></h2></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"address"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/PostalAddress"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"streetAddress"</span><span class="nt">></span>Dahlingstr. 40<span class="nt"></div></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"postalCode"</span><span class="nt">></span>47229<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"addressLocality"</span><span class="nt">></span>Duisburg<span class="nt"></span></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"geo"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/GeoCoordinates"</span><span class="nt">></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"latitude"</span> <span class="na">content=</span><span class="s">"51.38331874331818"</span><span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"longitude"</span> <span class="na">content=</span><span class="s">"6.700606101666003"</span><span class="nt">/></span>
<span class="nt"></div></span>
<span class="nt"></li></span>
<span class="nt"><li</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/School"</span> <span class="na">data-fresh-key=</span><span class="s">"NW-166480"</span><span class="nt">></span>
<span class="nt"><h2</span> <span class="na">itemprop=</span><span class="s">"name"</span><span class="nt">></span>Montessori-Gymnasium Städt. Gymnasium für Jungen und Mädchen<span class="nt"></h2></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"address"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/PostalAddress"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"streetAddress"</span><span class="nt">></span>Rochusstr. 145<span class="nt"></div></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"postalCode"</span><span class="nt">></span>50827<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"addressLocality"</span><span class="nt">></span>Köln<span class="nt"></span></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"geo"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/GeoCoordinates"</span><span class="nt">></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"latitude"</span> <span class="na">content=</span><span class="s">"50.963204818343755"</span><span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"longitude"</span> <span class="na">content=</span><span class="s">"6.904840537750957"</span><span class="nt">/></span>
<span class="nt"></div></span>
<span class="nt"></li></span>
<span class="nt"><li</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/School"</span> <span class="na">data-fresh-key=</span><span class="s">"NW-167782"</span><span class="nt">></span>
<span class="nt"><h2</span> <span class="na">itemprop=</span><span class="s">"name"</span><span class="nt">></span>Städt. Grillo-Gymnasium<span class="nt"></h2></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"address"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/PostalAddress"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"streetAddress"</span><span class="nt">></span>Hauptstr. 60<span class="nt"></div></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"postalCode"</span><span class="nt">></span>45879<span class="nt"></span></span>
<span class="nt"><span</span> <span class="na">itemprop=</span><span class="s">"addressLocality"</span><span class="nt">></span>Gelsenkirchen<span class="nt"></span></span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">itemprop=</span><span class="s">"geo"</span> <span class="na">itemscope</span> <span class="na">itemtype=</span><span class="s">"https://schema.org/GeoCoordinates"</span><span class="nt">></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"latitude"</span> <span class="na">content=</span><span class="s">"51.5130837882258"</span><span class="nt">/></span>
<span class="nt"><meta</span> <span class="na">itemprop=</span><span class="s">"longitude"</span> <span class="na">content=</span><span class="s">"7.099798427442939"</span><span class="nt">/></span>
<span class="nt"></div></span>
<span class="nt"></li></span>
<span class="nt"></ul></span>
<span class="nt"></leaflet-map></span>
<span class="nt"><style></span>
<span class="nf">#map</span> <span class="p">{</span>
<span class="nl">width</span><span class="p">:</span> <span class="m">800px</span><span class="p">;</span>
<span class="nl">height</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span>
<span class="py">aspect-ratio</span><span class="p">:</span> <span class="m">16</span><span class="p">/</span><span class="m">9</span><span class="p">;</span>
<span class="nl">background-image</span><span class="p">:</span> <span class="sx">url("/map.png")</span><span class="p">;</span>
<span class="nl">background-size</span><span class="p">:</span> <span class="n">contain</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt"></style></span>
</code></pre></div></div>
<p>In this form, this isn’t interactive yet but it can already be easily consumed by both humans
and computers. Humans will see a static map (I simply took a screenshot and set is as a
background image for the <code class="highlighter-rouge">#map</code> node) and a list of schools with their addresses.
Computers will be able to also parse the schema.org annotations to extract structured data out of the list.</p>
<p>Since the data is semi-structured and machine readable, we can now also consume it from
a Web Component to add interactivity with <a href="https://leafletjs.com">Leaflet</a>. The code to do so
looks like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nx">LeafletMap</span> <span class="kd">extends</span> <span class="nx">HTMLElement</span> <span class="p">{</span>
<span class="nx">connectedCallback</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">mapElement</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s2">"#map"</span><span class="p">);</span>
<span class="nx">mapElement</span><span class="p">.</span><span class="nx">style</span><span class="p">.</span><span class="nx">backgroundImage</span> <span class="o">=</span> <span class="s2">"none"</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">schools</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">querySelectorAll</span><span class="p">(</span>
<span class="s1">'[itemType="https://schema.org/School"]'</span><span class="p">,</span>
<span class="p">);</span>
<span class="kd">var</span> <span class="nx">map</span> <span class="o">=</span> <span class="nx">L</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">mapElement</span><span class="p">).</span><span class="nx">setView</span><span class="p">([</span><span class="mf">51.505</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.09</span><span class="p">],</span> <span class="mi">13</span><span class="p">);</span>
<span class="nx">L</span><span class="p">.</span><span class="nx">tileLayer</span><span class="p">(</span><span class="s2">"https://tile.openstreetmap.org/{z}/{x}/{y}.png"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">attribution</span><span class="p">:</span>
<span class="s1">'&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors'</span><span class="p">,</span>
<span class="p">}).</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">markers</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">L</span><span class="p">.</span><span class="nx">featureGroup</span><span class="p">();</span>
<span class="nx">schools</span><span class="p">.</span><span class="nx">forEach</span><span class="p">((</span><span class="nx">school</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">latitude</span> <span class="o">=</span> <span class="nx">school</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'[itemProp="latitude"]'</span><span class="p">).</span><span class="nx">content</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">longitude</span> <span class="o">=</span> <span class="nx">school</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'[itemProp="longitude"]'</span><span class="p">).</span><span class="nx">content</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">name</span> <span class="o">=</span> <span class="nx">school</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'[itemProp="name"]'</span><span class="p">).</span><span class="nx">textContent</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">address</span> <span class="o">=</span> <span class="nx">school</span><span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="s1">'[itemProp="address"]'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">h2</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s2">"h2"</span><span class="p">);</span>
<span class="nx">h2</span><span class="p">.</span><span class="nx">innerText</span> <span class="o">=</span> <span class="nx">name</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">popup</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">createElement</span><span class="p">(</span><span class="s2">"div"</span><span class="p">);</span>
<span class="nx">popup</span><span class="p">.</span><span class="nx">classList</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="s2">"popup"</span><span class="p">);</span>
<span class="nx">popup</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">h2</span><span class="p">);</span>
<span class="nx">popup</span><span class="p">.</span><span class="nx">appendChild</span><span class="p">(</span><span class="nx">address</span><span class="p">.</span><span class="nx">cloneNode</span><span class="p">(</span><span class="kc">true</span><span class="p">));</span>
<span class="kd">const</span> <span class="nx">marker</span> <span class="o">=</span> <span class="nx">L</span><span class="p">.</span><span class="nx">marker</span><span class="p">([</span><span class="nx">latitude</span><span class="p">,</span> <span class="nx">longitude</span><span class="p">]).</span><span class="nx">bindPopup</span><span class="p">(</span><span class="nx">popup</span><span class="p">);</span>
<span class="nx">marker</span><span class="p">.</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">markers</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">markers</span><span class="p">.</span><span class="nx">addTo</span><span class="p">(</span><span class="nx">map</span><span class="p">);</span>
<span class="nx">map</span><span class="p">.</span><span class="nx">fitBounds</span><span class="p">(</span><span class="nx">markers</span><span class="p">.</span><span class="nx">getBounds</span><span class="p">());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nb">window</span><span class="p">.</span><span class="nx">customElements</span><span class="p">.</span><span class="nx">define</span><span class="p">(</span><span class="s2">"leaflet-map"</span><span class="p">,</span> <span class="nx">LeafletMap</span><span class="p">);</span>
</code></pre></div></div>
<p>I quite like the way that this reads. The <code class="highlighter-rouge">itemProp</code>s make for very nice selectors and actually
allow developers to completely change the structure of the HTML. As long as the annotations
are kept, the Web Component will be able to still extract the locations and to load them into
leaflet.</p>
<p>You can try this on CodePen:</p>
<iframe height="800" style="width: 100%; height: 800px;" scrolling="no" title="Leaflet Microformat HTML Webcomponent" src="https://codepen.io/k-nut/embed/ZEZLQZw?default-tab=result" frameborder="no" loading="lazy" allowtransparency="true" allowfullscreen="true">
See the Pen <a href="https://codepen.io/k-nut/pen/ZEZLQZw">
Leaflet Microformat HTML Webcomponent</a> by Knut Hühne (<a href="https://codepen.io/k-nut">@k-nut</a>)
on <a href="https://codepen.io">CodePen</a>.
</iframe>
<p>This of course is nowhere from production ready but shall only serve as a proof of concept.
In a more advanced version, we wouldn’t limit our initial query selector to <code class="highlighter-rouge">[itemType="https://schema.org/School]</code>
but would probably want to find a way to query for everything that is a child of <code class="highlighter-rouge">https://schema.org/Place</code>.
We would also need to adapt the static image if we wanted to change the underlying data. If we have server
side rendering at our disposal, we could use something like the <a href="https://docs.mapbox.com/api/maps/static-images/">Mapbox Static Images
API</a> to dynamically create the fallback
images for clients that do not support JavaScript or Web Components.</p>
<p>All in all, this feels like a nice approach to me though. It provides content
that can be consumed by computers and humans with up to date or legacy (or privacy concious)
devices easily. Using the <code class="highlighter-rouge">itemProp</code>s for query selectors also makes for a nice
authoring experience with good separation of concerns.</p>
<p>What do you think? Let me know on <a href="https://berlin.social/@knut">mastodon</a>.</p>The people in my RSS reader have been talking some more about Web Components in the past couple of months. Jim Nielsen had a post referring to Jeremy Keith coining the term HTML Web Components for a special kind of Web Components. The idea essentially is that you have regular HTML markup that is wrapped by a custom Web Component which enhances the user experience.TIL: Timers in systemd are an interesting alternative to cron2023-10-16T19:30:00+00:002023-10-16T19:30:00+00:00https://blog.k-nut.eu/systemd-timers<p>I have been tracking the rides of the Berlin Critical Mass to create small visualisations
of where the route was taking people for a while now. To do so, I wrote a little script
that uses the <a href="https://www.criticalmaps.net">Critical Maps</a> API and takes bi-minutely snapshots.</p>
<p>To make sure that I had the data to visualise, I needed to remember to start the script on
every last Friday of the month. Sometimes this was not possible since I was out of town
with no access to my server where I kept the script.</p>
<p>I wanted to automate this and planned to set up a small cron job to start the script for me.
After a little research I learned that the cron syntax does not allow for “every last Friday of the month”.
Instead, what people would mostly do as a workaround is to set up cron to run their scripts <em>every</em>
Friday and then to exit early in the script if it was not in fact the last Friday of the month.</p>
<p>This did not seem so nice to me, so I kept looking for better options.</p>
<p>I read somewhere that systemd’s timer feature allowed one to define jobs in intervals like the one I was looking for.
I initially felt like the setup would be too complicated but then realised that the system is quite nice.
I’m going to share what I learned in the rest of this aritcle (this is mostly from the <a href="https://documentation.suse.com/smart/systems-management/html/systemd-working-with-timers/index.html">documentation on suse.com</a>).</p>
<p>Setting up a timer in systemd actually requires one to set up a <em>service</em> first. Mine looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Critical Tracks Recorder
[Service]
ExecStart=python3 /home/knut/critical-tracks/main.py
</code></pre></div></div>
<p>This file is called <code class="highlighter-rouge">critical-tracks.service</code> and resides in <code class="highlighter-rouge">/etc/systemd/system/critical-tracks.service</code>.</p>
<p>Along with the service, I also needed to set up a <em>timer</em>. The configuration is not much more complex and looks like this.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=Critical Tracks monthly timer
[Timer]
OnCalendar=Fri *-*~07/1 18:00:00
[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<p>This file is called <code class="highlighter-rouge">critical-tracks.timer</code> and resides in <code class="highlighter-rouge">/etc/systemd/system/critical-tracks.timer</code>.</p>
<p>The matching between service and timer is achieved by them sharing the same name so the only complicated part of the config
is the <code class="highlighter-rouge">OnCalendar</code> section. Details on the setup can be found by running <code class="highlighter-rouge">man 7 systemd.time</code> which includes this section:</p>
<blockquote>
<p>A date specification may use “~” to indicate the last day(s) in a month. For example, “*-02~03” means “the third last day in February,” and “Mon *-05~07/1” means “the last Monday in May.”</p>
</blockquote>
<p>This looked quite like what I wanted already and just required some more tweaking to match my exact use case. Luckily, there
also is a handy executable called <code class="highlighter-rouge">sytemd-analyze</code> which lets one test the calendar pattern.</p>
<p>Running <code class="highlighter-rouge">systemd-analyze calendar --iterations 12 "Fri *-*~7/1 18:00:00"</code>
Gave me:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Original form: Fri *-*~7/1 18:00:00
Normalized form: Fri *-*~07/1 18:00:00
Next elapse: Fri 2023-10-27 18:00:00 CEST
(in UTC): Fri 2023-10-27 16:00:00 UTC
From now: 1 week 3 days left
Iter. #2: Fri 2023-11-24 18:00:00 CET
(in UTC): Fri 2023-11-24 17:00:00 UTC
From now: 1 month 8 days left
Iter. #3: Fri 2023-12-29 18:00:00 CET
(in UTC): Fri 2023-12-29 17:00:00 UTC
From now: 2 months 13 days left
Iter. #4: Fri 2024-01-26 18:00:00 CET
(in UTC): Fri 2024-01-26 17:00:00 UTC
From now: 3 months 10 days left
Iter. #5: Fri 2024-02-23 18:00:00 CET
(in UTC): Fri 2024-02-23 17:00:00 UTC
From now: 4 months 8 days left
Iter. #6: Fri 2024-03-29 18:00:00 CET
(in UTC): Fri 2024-03-29 17:00:00 UTC
From now: 5 months 12 days left
Iter. #7: Fri 2024-04-26 18:00:00 CEST
(in UTC): Fri 2024-04-26 16:00:00 UTC
From now: 6 months 10 days left
Iter. #8: Fri 2024-05-31 18:00:00 CEST
(in UTC): Fri 2024-05-31 16:00:00 UTC
From now: 7 months 14 days left
Iter. #9: Fri 2024-06-28 18:00:00 CEST
(in UTC): Fri 2024-06-28 16:00:00 UTC
From now: 8 months 12 days left
Iter. #10: Fri 2024-07-26 18:00:00 CEST
(in UTC): Fri 2024-07-26 16:00:00 UTC
From now: 9 months 10 days left
Iter. #11: Fri 2024-08-30 18:00:00 CEST
(in UTC): Fri 2024-08-30 16:00:00 UTC
From now: 10 months 14 days left
Iter. #12: Fri 2024-09-27 18:00:00 CEST
(in UTC): Fri 2024-09-27 16:00:00 UTC
From now: 11 months 12 days left
</code></pre></div></div>
<p>which I confirmed where the dates that I wanted.</p>
<p>I wanted to fully understand the syntax though and it took me a bit to understand the last part. The documentation says:</p>
<blockquote>
<p>In the date and time specifications, any component may be specified as “*” in which case any value will match. Alternatively, each component can be
specified as a list of values separated by commas. Values may be suffixed with “/” and a repetition value, which indicates that the value itself and
the value plus all multiples of the repetition value are matched.</p>
</blockquote>
<p>I tried around a bit and finally understood that the <code class="highlighter-rouge">~</code> made it so that we were <em>counting from the end of the month</em>. That is, instead of adding, the <code class="highlighter-rouge">/</code>
would subtract from the n-th last day in multiples of the value after the slash.</p>
<p>To start easy, <code class="highlighter-rouge">systemd-analyze calendar --iterations 3 "Fri *-*~1 18:00:00"</code> for example would only give us Fridays that were <em>exactly</em> the last day of the month:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Original form: Fri *-*~1 18:00:00
Normalized form: Fri *-*~01 18:00:00
Next elapse: Fri 2024-05-31 18:00:00 CEST
(in UTC): Fri 2024-05-31 16:00:00 UTC
From now: 7 months 14 days left
Iter. #2: Fri 2025-01-31 18:00:00 CET
(in UTC): Fri 2025-01-31 17:00:00 UTC
From now: 1 year 3 months left
Iter. #3: Fri 2025-02-28 18:00:00 CET
(in UTC): Fri 2025-02-28 17:00:00 UTC
From now: 1 year 4 months left
</code></pre></div></div>
<p>Similarly, <code class="highlighter-rouge">"Fri *-*~2 18:00:00"</code> would give us Fridays on the second last day of the month.</p>
<p>Adding the <code class="highlighter-rouge">/1</code> suffix made it so that we could also change the last value by increments of that value. <code class="highlighter-rouge">"Fri *-*~2/1 18:00:00"</code>
would now give us Fridays on the <em>last</em> or the <em>second-last</em> day of the month. One could also choose non-1 increments for really complicated setups
such as <code class="highlighter-rouge">"Fri *-*~03/2 18:00:00"</code> which would translate to “Fridays that are on the third (3) last or the last (3 - 2*1) day of the month”.
This would be much more than I needed though. I could get just what I wanted from <code class="highlighter-rouge">Fri *-*~7/1 18:00:00</code> that is the last Friday on the seventh or fewer
last of day of the month.</p>
<p>I would now only have to enable the timer by running:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl enable critical-tracks.timer
systemctl start critical-tracks.timer
</code></pre></div></div>
<p>I could check that the timer was registered for the correct day by listing all timers:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NEXT LEFT LAST PASSED UNIT ACTIVATES
Tue 2023-10-17 00:00:00 CEST 4h 42min left Mon 2023-10-16 00:00:01 CEST 19h ago dpkg-db-backup.timer dpkg-db-backup.service
<snip>
Fri 2023-10-27 18:00:00 CEST 1 week 3 days left n/a n/a critical-tracks.timer critical-tracks.service
</code></pre></div></div>
<p>And that was it I think. I’ll see at the end of the month but my learning of the day was that setting up timers
in systemd might be much easier than I had initially feared.</p>I have been tracking the rides of the Berlin Critical Mass to create small visualisations of where the route was taking people for a while now. To do so, I wrote a little script that uses the Critical Maps API and takes bi-minutely snapshots.An introduction to TransformStreams2022-12-05T14:50:00+00:002022-12-05T14:50:00+00:00https://blog.k-nut.eu/transform-streams<p>I have been really enjoying <a href="https://protohackers.com">protohackers</a> recently. Protohackers feels quite similar
to Advent of Code but on a less rigid schedule and with a focus on low level networking code (TCP & UDP). I have been
doing the challenges with <a href="https://deno.land">Deno</a> and have learned a lot both about networking, binary protocols and about Deno.</p>
<p>In the beginning, my code was very imperative but for one of the more recent challenges, I started looking into Deno’s
streaming APIs some more. Deno uses the newly standardised <a href="https://developer.mozilla.org/en-US/docs/Web/API/Streams_API">Web Streams API</a>
which attempt to make streaming APIs available in a variety of JavaScript runtimes.</p>
<p>For the challenge I was attempting, I needed to parse binary data and pass it on as objects on a higher abstraction
level. I quickly found that a <code class="highlighter-rouge">TransformStream</code> would be what I wanted for converting the data but I could not find a lot of detailed documentation
on them. The Deno blog contains a <a href="https://deno.com/blog/deploy-streams#http-proxy-with-transform">basic example</a> and I also found it
helpful to read the <a href="https://deno.land/std@0.157.0/streams/delimiter.ts?source#L152">source code of the <code class="highlighter-rouge">DelimiterStream</code> class in Deno’s <code class="highlighter-rouge">std</code>-lib </a>.</p>
<p>I’m going to try to summarise what I have learned and will show another couple examples here.</p>
<p>Let’s start with the most basic way to use streams. We’ll create a stream that is both writable and readable and will pass some data though it:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">writableStreamFromWriter</span> <span class="p">}</span> <span class="k">from</span> <span class="s2">"https://deno.land/std@0.157.0/streams/conversion.ts"</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">stream</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">TransformStream</span><span class="p">();</span>
<span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span>
<span class="p">.</span><span class="nx">pipeThrough</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoderStream</span><span class="p">())</span>
<span class="p">.</span><span class="nx">pipeTo</span><span class="p">(</span><span class="nx">writableStreamFromWriter</span><span class="p">(</span><span class="nx">Deno</span><span class="p">.</span><span class="nx">stdout</span><span class="p">))</span>
<span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">texts</span><span class="p">:</span> <span class="nx">string</span><span class="p">[]</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"message 1</span><span class="err">\</span><span class="s2">n"</span><span class="p">,</span> <span class="s2">"message 2</span><span class="err">\</span><span class="s2">n"</span><span class="p">];</span>
<span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">();</span>
<span class="k">for</span> <span class="p">(</span><span class="kd">const</span> <span class="nx">part</span> <span class="k">of</span> <span class="nx">texts</span><span class="p">)</span> <span class="p">{</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">ready</span><span class="p">;</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="nx">part</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we run this, we will get the following output:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message 1
message 2
</code></pre></div></div>
<p>Not very exciting yet. We create a <code class="highlighter-rouge">TransformStream</code> - that is a stream that has both a readable and a writable side. In reality, you would probably have a readable stream here, something like a <code class="highlighter-rouge">Deno.conn</code> or the result of a <code class="highlighter-rouge">fetch</code> call. We then connect a new output to the stream by wrapping <code class="highlighter-rouge">Deno.stdout</code> in a <code class="highlighter-rouge">writableStreamFromWriter</code>.</p>
<p>We could do something quite similar to the example from the Deno blog above and do:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span>
<span class="p">.</span><span class="nx">pipeThrough</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">TransformStream</span><span class="p">({</span>
<span class="na">transform</span><span class="p">:</span> <span class="p">(</span><span class="nx">chunk</span><span class="p">,</span> <span class="nx">controller</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">controller</span><span class="p">.</span><span class="nx">enqueue</span><span class="p">(</span><span class="nx">chunk</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">());</span>
<span class="p">},</span>
<span class="p">})</span>
<span class="p">)</span>
<span class="p">.</span><span class="nx">pipeThrough</span><span class="p">(</span><span class="k">new</span> <span class="nx">TextEncoderStream</span><span class="p">())</span>
<span class="p">.</span><span class="nx">pipeTo</span><span class="p">(</span><span class="nx">writableStreamFromWriter</span><span class="p">(</span><span class="nx">Deno</span><span class="p">.</span><span class="nx">stdout</span><span class="p">))</span>
<span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">);</span>
</code></pre></div></div>
<p>We now get:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MESSAGE 1
MESSAGE 2
</code></pre></div></div>
<p>What I struggled with though and what I think is even more interesting, is not passing on data chunk by chunk but instead changing when it gets forwarded.
Let’s change our writer to write a new message every second:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">writer</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">writable</span><span class="p">.</span><span class="nx">getWriter</span><span class="p">();</span>
<span class="kd">let</span> <span class="nx">count</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">setInterval</span><span class="p">(</span><span class="k">async</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">ready</span><span class="p">;</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span><span class="s2">`message </span><span class="p">${</span><span class="nx">count</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
<span class="nx">count</span><span class="o">++</span><span class="p">;</span>
<span class="p">},</span> <span class="mi">1</span><span class="nx">_000</span><span class="p">);</span>
</code></pre></div></div>
<p>We are now going to add a custom transformer which only ever forwards messages in chunks of 50 characters:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kd">class</span> <span class="nx">StringLengthTransformer</span> <span class="kd">extends</span> <span class="nx">TransformStream</span><span class="o"><</span><span class="nx">string</span><span class="p">,</span> <span class="nx">string</span><span class="o">></span> <span class="p">{</span>
<span class="na">data</span><span class="p">:</span> <span class="nx">string</span> <span class="o">=</span> <span class="s2">""</span><span class="p">;</span>
<span class="nl">LIMIT</span><span class="p">:</span> <span class="nx">number</span><span class="p">;</span>
<span class="kd">constructor</span><span class="p">(</span><span class="na">limit</span><span class="p">:</span> <span class="nx">number</span><span class="p">)</span> <span class="p">{</span>
<span class="k">super</span><span class="p">({</span>
<span class="na">transform</span><span class="p">:</span> <span class="p">(</span><span class="nx">chunk</span><span class="p">,</span> <span class="nx">controller</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="err">#</span><span class="nx">handle</span><span class="p">(</span><span class="nx">chunk</span><span class="p">,</span> <span class="nx">controller</span><span class="p">);</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span> <span class="o">=</span> <span class="nx">limit</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">#</span><span class="nx">handle</span><span class="p">(</span><span class="na">chunk</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span> <span class="na">controller</span><span class="p">:</span> <span class="nx">TransformStreamDefaultController</span><span class="o"><</span><span class="nx">string</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">data</span> <span class="o">+=</span> <span class="nx">chunk</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">length</span> <span class="o">>=</span> <span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">controller</span><span class="p">.</span><span class="nx">enqueue</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">));</span>
<span class="k">this</span><span class="p">.</span><span class="nx">data</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As you can see, we create a new class which subclasses <code class="highlighter-rouge">TranformStream</code>. It holds a <code class="highlighter-rouge">data</code> field in which it stores the data. For every chunk that it gets, it appends it to <code class="highlighter-rouge">data</code> and checks if the limit is reached yet. If so, it forwards the message.</p>
<p>We can also slighlty refactor, how we construct our stream and how we log the result:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">chunkedStream</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span>
<span class="p">.</span><span class="nx">pipeThrough</span><span class="p">(</span><span class="k">new</span> <span class="nx">StringLengthTransformer</span><span class="p">(</span><span class="mi">20</span><span class="p">));</span>
<span class="k">for</span> <span class="kr">await</span> <span class="p">(</span><span class="kd">const</span> <span class="nx">message</span> <span class="k">of</span> <span class="nx">chunkedStream</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">message</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>On execution we now see (after a while):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>message 0message 1me
ssage 2message 3mess
age 4message 5messag
e 6message 7message
8message 9message 10
</code></pre></div></div>
<p>Note that there is one edge case the we have not handled yet. If were to change our writer to:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">count</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="nx">setInterval</span><span class="p">(</span><span class="k">async</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">ready</span><span class="p">;</span>
<span class="kr">await</span> <span class="nx">writer</span><span class="p">.</span><span class="nx">write</span><span class="p">(</span>
<span class="s2">`This is a really long message. It is message number </span><span class="p">${</span><span class="nx">count</span><span class="p">}</span><span class="s2">`</span>
<span class="p">);</span>
<span class="nx">count</span><span class="o">++</span><span class="p">;</span>
<span class="p">},</span> <span class="mi">1</span><span class="nx">_000</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">chunkedStream</span> <span class="o">=</span> <span class="nx">stream</span><span class="p">.</span><span class="nx">readable</span><span class="p">.</span><span class="nx">pipeThrough</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">StringLengthTransformer</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="p">);</span>
<span class="k">for</span> <span class="kr">await</span> <span class="p">(</span><span class="kd">const</span> <span class="nx">message</span> <span class="k">of</span> <span class="nx">chunkedStream</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">timeStamp</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Intl</span><span class="p">.</span><span class="nx">DateTimeFormat</span><span class="p">(</span><span class="s2">"de-DE"</span><span class="p">,</span> <span class="p">{</span>
<span class="na">timeStyle</span><span class="p">:</span> <span class="s2">"long"</span><span class="p">,</span>
<span class="p">}).</span><span class="nx">format</span><span class="p">(</span><span class="k">new</span> <span class="nb">Date</span><span class="p">());</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`[</span><span class="p">${</span><span class="nx">timeStamp</span><span class="p">}</span><span class="s2">]: </span><span class="p">${</span><span class="nx">message</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We would get:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[15:21:35 MEZ]: This is a really lon
[15:21:36 MEZ]: g message. It is mes
[15:21:37 MEZ]: sage number 0This is
</code></pre></div></div>
<p>Our chunk now gets split into three parts. The problem is that we only enqueue a message whenever a new one arrives. This means that by the time the third message arrives, we just get done with enqueuing the first one. In order to handle messages that are longer than the limit already, we can swap out the <code class="highlighter-rouge">if</code> for a <code class="highlighter-rouge">while</code> in our Transformer:</p>
<div class="language-typescript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="err">#</span><span class="nx">handle</span><span class="p">(</span><span class="nx">chunk</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span> <span class="nx">controller</span><span class="p">:</span> <span class="nx">TransformStreamDefaultController</span><span class="o"><</span><span class="nx">string</span><span class="o">></span><span class="p">)</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">data</span> <span class="o">+=</span> <span class="nx">chunk</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">length</span> <span class="o">>=</span> <span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">controller</span><span class="p">.</span><span class="nx">enqueue</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">));</span>
<span class="k">this</span><span class="p">.</span><span class="nx">data</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">slice</span><span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="nx">LIMIT</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now, the output looks much closer to what we expect. We get multiple logs per second so that all data can be written:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[15:24:29 MEZ]: This is a really lon
[15:24:29 MEZ]: g message. It is mes
[15:24:30 MEZ]: sage number 0This is
[15:24:30 MEZ]: a really long messa
[15:24:30 MEZ]: ge. It is message nu
</code></pre></div></div>
<p>There are many more things that one could do with <code class="highlighter-rouge">TransformStreams</code> and I’m quite happy to have learned about them. I’m already looking forward to the next time I get to use them.</p>I have been really enjoying protohackers recently. Protohackers feels quite similar to Advent of Code but on a less rigid schedule and with a focus on low level networking code (TCP & UDP). I have been doing the challenges with Deno and have learned a lot both about networking, binary protocols and about Deno.Downloading videos for later on macOS with Play, yt-dlp and Shortcuts2022-11-07T09:00:00+00:002022-11-07T09:00:00+00:00https://blog.k-nut.eu/downloading-youtube-with-play<p>I was traveling recently and wanted to watch some videos while not overusing the
shared wifi connection on the train. I had been marking my videos as to watch with the
iOS/macOS app <a href="https://apps.apple.com/de/app/play-save-videos-watch-later/id1596506190?l=en">Play</a>.
Since Play has shortcuts support, I figured it should not be too hard to download the videos
to my Mac. It did take me longer than I hoped but I eventually found a solution that works well enough
for me. It starts with this very simple shortcut:
<img src="/static/play-shortcuts.png" alt="A screenshot of the Shortcuts app. Two steps: Play and Text" />
The first steps gets the videos from Play (filtered by those that have not been watched yet) and the second
step gets the <code class="highlighter-rouge">Url</code> field for every video, turning it into a list of YouTube URLs.</p>
<p>I originally tried to also include the actual downloading in the shortcut but I couldn’t figure out
how to properly pass variables to Terminal actions or how to truncate a list while retaining the
PlayVideo->Url extraction.</p>
<p>So instead, I now use this shortcut from the command line where it is wrapped in a tiny <a href="https://fishshell.com">fish</a> command substitution
and then forwarded to <a href="https://github.com/yt-dlp/yt-dlp">yt-dlp</a> like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yt-dlp (shortcuts run Play | head -n 5)
</code></pre></div></div>
<p>Note that by default, the <code class="highlighter-rouge">shortcuts</code> command line app does not print to stdout. You need to pipe it to <code class="highlighter-rouge">cat</code>, <code class="highlighter-rouge">head</code> or a similar
tool if you want to see the output. I wanted to limit my output anyway (as not to download all videos I had marked as to watch) so this worked well for me.</p>
<p>So that’s my setup now, a shortcut with two steps and a super simple shell script. I like how easy this was in the end but it took me
quite a while to get there and the split between Shortcuts and shell seems quite arbitrary to me. But it does work. Hopefully, this
also helps some of you.</p>I was traveling recently and wanted to watch some videos while not overusing the shared wifi connection on the train. I had been marking my videos as to watch with the iOS/macOS app Play. Since Play has shortcuts support, I figured it should not be too hard to download the videos to my Mac. It did take me longer than I hoped but I eventually found a solution that works well enough for me. It starts with this very simple shortcut: The first steps gets the videos from Play (filtered by those that have not been watched yet) and the second step gets the Url field for every video, turning it into a list of YouTube URLs.Giving nice names to tests in Python2022-10-22T18:00:00+00:002022-10-22T18:00:00+00:00https://blog.k-nut.eu/naming-tests-in-python<p>I originally come from a Python background but I have been doing mostly TypeScript for the last couple of years
in my day job.</p>
<p>One of the things that I really enjoy about the JS/TS ecosystem is <a href="https://jestjs.io">jest</a>. I found it to be a really nice way of
writing and running tests. A very small detail that I always enjoyed when coming from pytest was that you pass the test name
as the first argument to your test function. In jest, you might for example write a test like this:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">describe</span><span class="p">(</span><span class="s2">"capitalize"</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">it</span><span class="p">(</span><span class="s2">"should capitalise a string"</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">capitalize</span><span class="p">(</span><span class="s2">"hello"</span><span class="p">)).</span><span class="nx">toEqual</span><span class="p">(</span><span class="s2">"Hello"</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Running the tests gives us this output:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> PASS ./main.test.js
capitalize
✓ should capitalise a string (1 ms)
Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total
Snapshots: 0 total
Time: 0.165 s
</code></pre></div></div>
<p>In Python’s <a href="https://docs.python.org/3/library/unittest.html">unittest</a>, tests are discovered by being methods in a class that extends <code class="highlighter-rouge">unittest.TestCase</code> and starting
with <code class="highlighter-rouge">test</code> as per <a href="https://docs.python.org/3/library/unittest.html#basic-example">the documentation</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">unittest</span>
<span class="k">class</span> <span class="nc">CapitalizeTests</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_should_capitalise_a_string</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">capitalize</span><span class="p">(</span><span class="s">"hello"</span><span class="p">),</span> <span class="s">"Hello"</span><span class="p">)</span>
</code></pre></div></div>
<p>Running the tests gives us this output:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test_should_capitalise_a_string (__main__.CapitalizeTests) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
</code></pre></div></div>
<p>I think that one of the main jobs of a test is to communicate expected behaviour to future readers. One way to achieve this is to use good test names. To me, it always felt a big awkward to use snake_cased_test_names to communicate what I wanted my tests to cover in Python. I just <a href="https://stackoverflow.com/questions/22781716/python-unittest-how-to-display-a-better-group-test-name/22781860#22781860">recently learned via stackoverflow</a> that unittest will take into account the methods’ docstrings and display those as method names. I think that this is a nice compromise and lets us communicate more naturally.</p>
<p>This is how you can add more explicit test descriptions as prose via docstrings:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">CapitalizeTests</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">test_should_capitalise_a_string</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" Should capitalise the string """</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">capitalize</span><span class="p">(</span><span class="s">"hello"</span><span class="p">),</span> <span class="s">"Hello"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_should_handle_none</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s">""" Should return an empty string if None is given"""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">capitalize</span><span class="p">(</span><span class="bp">None</span><span class="p">),</span> <span class="s">""</span><span class="p">)</span>
</code></pre></div></div>
<p>Running the tests now gives us this output:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test_should_capitalise_a_string (__main__.CapitalizeTests)
Should capitalise the string ... ok
test_should_handle_none (__main__.CapitalizeTests)
Should return an empty string if None is given ... ERROR
======================================================================
ERROR: test_should_handle_none (__main__.CapitalizeTests)
Should return an empty string if None is given
----------------------------------------------------------------------
Traceback (most recent call last):
<...abbreviated>
----------------------------------------------------------------------
Ran 2 tests in 0.000s
FAILED (errors=1)
</code></pre></div></div>
<p>I think this is nice as it might give future readers some more context for what behavior a test expected to cover.</p>
<p>As an additional benefit, those test names are also nicely displayed in PyCharm, my Python IDE of choice:
<img src="/static/pycharm-unittest-results.png" alt="A screenshot of PyCharm showing the test names in the test summary" /></p>
<p>I still think that jest’s way is a bit more straightforward to use but I quite like this as a small improvement to the Python test that I write.</p>
<p>So there you go, happy testing!</p>I originally come from a Python background but I have been doing mostly TypeScript for the last couple of years in my day job.This Month in Open: January 20212021-02-08T13:15:00+00:002021-02-08T13:15:00+00:00https://blog.k-nut.eu/this-month-in-open<p>Welcome to the third edition of <em>This Month in Open</em>, my Berlin-focused look at the latest
developments in the Open Data and Civic Tech scene.</p>
<h1 id="supaplex-publishes-a-beautiful-visualisation-of-parking-spaces-in-neukölln">Supaplex publishes a beautiful visualisation of parking spaces in Neukölln</h1>
<p><img src="/static/tmio-2021-01/parkraumkarte.png" alt="Screenshot of parking spaces map" />
The Berlin based OpenStreetMap <em>Verkehrswende</em> community keeps doing great work. This month,
user Supaplex published <a href="https://supaplex.uber.space/map/neukoelln.html#19/52.47699/13.42210">a really nice map of parking spaces</a>.
Apart from showing just how much space is used for cars it is also an excellent example for what
kind of beautiful ans specialised maps one can create with data from OpenStreetMap and official public data sources.</p>
<h1 id="thomas-publishes-an-analysis-of-internet-speed-in-berlins-schools">Thomas publishes an analysis of internet speed in Berlin’s schools</h1>
<p><img src="/static/tmio-2021-01/internet-at-schule.png" alt="Screenshot of Internet@Schule Website" />
<a href="http://twitter.com/tursics/">Thomas</a> also keeps publishing great work. This month, he brought
together his two interests of Open Data and education again to create a little
<a href="http://tursics.de/story/schule-breitband-2020/">scrollytelling piece about the speed of internet in Berlin’s shools</a>.
I have to say that I was very surprised to learn just how many schools still only have a
16 Mbit/s connection. Thomas also shows what alternative top speed the schools could get
but knowing that they currently have to share a connection with their whole school that I
would consider terribly slow just for me was unexpected.</p>
<h1 id="odis-publishes-a-tool-to-optimise-electoral-districts">ODIS publishes a tool to optimise electoral districts</h1>
<p>ODIS, the Open Data branch of the <a href="https://www.technologiestiftung-berlin.de/en/home/">Technologiestiftung Berlin</a>
published <a href="https://lab.technologiestiftung-berlin.de/projects/wahlbezirke/en/">a blog post describing a tool that they developed for the administration to
optimise election districts</a>.
This is required since people move around a city while election districts should remain
consistent in size. The blog post describes how the tool was built and also lets users try it
out in their browsers.</p>
<h1 id="berlin-publishes-data-on-water-measurements">Berlin publishes data on water measurements</h1>
<p>The city of Berlin now <a href="https://wasserportal.berlin.de/stationen_start.php">publishes data on it’s ground water measurements</a>. This includes water levels
as well as water quality for some stations. By just skimming the data, I was able to find water
level measurements dating back all the way to 1932 as well as very recent measurements on the levels
of certain metals in the water. Hopefully all of this data will also be coming to <a href="https://daten.berlin.de">Berlin’s Open Data
portal</a> soon. You can find more information in the official (German)
<a href="https://www.berlin.de/sen/uvk/presse/pressemitteilungen/2021/pressemitteilung.1045131.php">press release</a>.</p>
<h1 id="a-look-back-at-the-wikipaka-talks-at-rc3">A look back at the Wikipaka talks at rc3</h1>
<p>The Open Knowledge Foundation Germany published <a href="https://okfn.de/blog/2021/01/nachlese-wikipaka-rc3/">a blogpost</a> (in German) looking back at all
of the interesting things that happened at the Wikipaka community at least year’s Remote
Chaos Experience (rc3). If you speak German, you should check it out.</p>
<h1 id="the-south-african-government-publishes-a-new-browser-to-bring-back-flash">The South African government publishes a new browser to bring back Flash</h1>
<p>There isn’t a lot to say about this: The South African Government <a href="https://www.zdnet.com/article/south-african-government-releases-its-own-browser-just-to-re-enable-flash-support/">now publishes its own browser</a> to re-enable flash support which is needed for some official
web forms. This goes to show that lobbying for open standards in government is important for both
future compatibility as well as general accessibility.</p>
<h2 id="thats-it">That’s it</h2>
<p>That’s it for this month. If you found an article that should be included in next month’s issue, let me
know <a href="https://twitter.com/notknut">on twitter</a> or via <a href="mailto:this-month-in-open@k-nut">e-mail</a>.</p>Welcome to the third edition of This Month in Open, my Berlin-focused look at the latest developments in the Open Data and Civic Tech scene.This Month in Open: December 20202021-01-20T07:00:00+00:002021-01-20T07:00:00+00:00https://blog.k-nut.eu/this-month-in-open<p>This is the second edition of <em>This Month in Open</em> in which I try to summarise interesting articles
and events from the Open Data and Civic Tech scene in Germany and the world.</p>
<h2 id="code-for-münster-plays-with-ab-street">Code for Münster plays with A/B Street</h2>
<p><a href="https://github.com/dabreegster/abstreet/#ab-street">A/B Street</a> is a really interesting project
from the US. The web application allows users to load data from OpenStreetMap and
then runs traffic simulations on it. Users get a nice visualisation but are also encourage to play
with different street layouts. This allows users to be little city planners. This is a lot of fun
to play around with - especially in the German <em>Verkehrswende</em> community. The folks from the
OK Lab Münster <a href="https://twitter.com/codeformuenster/status/1336421089153609728">tweeted</a> about loading
the data for their city into the tool. In the thread there are also people from Berlin saying that
they are investigating a possible deployment of the tool. Exciting times!</p>
<h2 id="the-state-of-the-german-onlinezugangsgesetz">The state of the German Onlinezugangsgesetz</h2>
<p>The <a href="https://de.wikipedia.org/wiki/Onlinezugangsgesetz">Onlinezugangsgesetz</a> is a German law
which requires public administration to make their processes available online. There is an
<a href="https://www.onlinezugangsgesetz.de/Webs/OZG/DE/umsetzung/dashboard/ozg-dashboard/ozg-dashboard-node.html">official dashboard</a>
by the German government in which the current implementation state is tracked. By design this
dashboard is of course rather biased to paint a positive picture. Community member Lilith built
<a href="https://ozg.verdrusssache.de">an alternative dashboard</a> which also explains the law and
implementation state in more detail. If you speak German, I highly encourage that you check it out.</p>
<h2 id="italy-is-on-github">Italy is on GitHub</h2>
<p>Just last month I learned that the country of Italy is on github at github.com/italia. Pretty
progressive! They also publish a range of interesting projects such as
<a href="https://github.com/italia/publiccode.yml">publiccode.yml</a> which is standard for marking up
publicly funded open source software. It is great to see a government go where the community
is already. I encourage you to browse through all of Italia’s repositories.</p>
<h2 id="problems-with-digital-transformation">Problems with digital transformation</h2>
<p>Mark Headd published an <a href="https://mheadd.medium.com/filling-up-the-civic-tech-toolbox-a976161a84be">interesting article</a> about
what he has learned with trying to bring digital transformation to government and the problems that
one might encounter in this process. Rebecca Williams published a <a href="https://twitter.com/internetrebecca/status/1325829059155202049">twitter thread</a>
on it that starts with the beautiful quote</p>
<blockquote>
<p>I like this because once you realize tech isn’t the problem, technologists stop being the solution.</p>
</blockquote>
<p>Good food for thought!</p>
<h2 id="kleineanfragende-is-going-read-only-">kleineanfragen.de is going read only 😢</h2>
<p>The Open Knowledge Foundation Germany published <a href="https://okfn.de/blog/2021/01/zur-abschaltung-von-kleine-anfragen/">a blogpost</a>
to announce that kleineanfragen.de is officially going into read only mode. <a href="https://twitter.com/robbi5">Maxi</a>
started the project a long time ago which aggregates and publishes so called <em>Kleine Anfragen</em>. These
<em>Anfragen</em> are formal inquiries by parliament to the government. As the federal states largely do not
provide formalised APIs for this, kleineanfragen.de always had to resort to webscraping. This is of
course not well maintainable in the long term as websites will change and scrapers will have to be updated.
So this great project of the community is shutting down. This is really sad because it was used by community,
journalists but also by members of the parliament themselves. Again, if you speak German, you should check out the
original article to read about changes that would need to be made for a tool like this to be possible in the
long run.</p>
<h2 id="german-vaccination-status-in-city-sizes">German vaccination status in city sizes</h2>
<p>Johl made a really small and playful <a href="https://www.johl.io/impfdaten/">website</a> to track the status
of German vaccinations. It takes the official RKI data about the total number of people that have been
vaccinated and then queries Wikidata to find a city that is comparable in size. I think this is a nice
example of what one can do when there is an easily queryable repository of the world’s knowledge.</p>
<h2 id="thats-it">That’s it</h2>
<p>That’s it for this month. If you found an article that should be included in next month’s issue, let me
know <a href="https://twitter.com/notknut">on twitter</a> or via <a href="mailto:this-month-in-open@k-nut">e-mail</a>.</p>
<ul>
<li></li>
<li></li>
</ul>This is the second edition of This Month in Open in which I try to summarise interesting articles and events from the Open Data and Civic Tech scene in Germany and the world.This Month in Open: October 20202020-11-12T10:45:00+00:002020-11-12T10:45:00+00:00https://blog.k-nut.eu/this-month-in-open<p>At the <a href="https://codefor.de/berlin/">Open Knowledge Lab Berlin</a> we run a monthly event where we
invite expert speakers to tell us about their branch of Open Data or Civic Tech. A while ago
we got inspired by <a href="https://berlinhackandtell.rocks">Berlin Hack and Tell</a> where they started
doing a news section to talk about interesting things that happened in the Open Source world since
the last event. We decided to do the same for our events and call the section “This Month in Open”.
I realised that instead of just sharing the news there, I could also share it on this blog so here
we go.</p>
<p>This is This Month In Open for November 2020.</p>
<h2 id="wikidata-is-getting-a-rest-api">Wikidata is getting a REST API</h2>
<p>Wikibase - the software behind Wikidata is getting a REST API. The idea here is to make it easier
for new developers to add tooling to query and - more importantly - add and modify data in Wikidata.
Even better: the developers are currently asking for feedback. They provide an OpenAPI document and
some of their design ideas as a base for discussion. You can find more information on the
<a href="https://www.wikidata.org/wiki/Wikidata:REST_API_feedback_round">project page</a>.</p>
<h2 id="tobi-inquires-about-berlindes-use-of-open-source">Tobi inquires about berlin.de’s use of Open Source</h2>
<p>Inspired by <a href="https://digitales.wien.gv.at/site/projekt/open-source-software-oss/">this nice page by the city of Vienna which lists the Open Source software used by the city</a>,
Tobi <a href="https://twitter.com/tbsprs/status/1323002066911039489?s=19">asked the city of Berlin on twitter</a> if they did something
comparable. The city did not respond yet but our fingers are crossed.</p>
<h2 id="the-city-of-berlin-is-asked-about-open-source-some-more">The city of Berlin is asked about Open Source some more</h2>
<p>There was a <a href="https://pardok.parlament-berlin.de/starweb/adis/citat/VT/18/SchrAnfr/S18-25055.pdf">written inquiry to the Berlin senate</a> to ask
about Open Source software being used for processes in the administration. The reply to the inquiry contains a rather long
list of software and its use cases. Unfortunately it neither contains information about the license nor about a
location where the source code could be found. My personal highlight though: The software with the id V0415 has this note attached to it:</p>
<blockquote>
<p>Auf DVD im Tresor hinterlegt</p>
</blockquote>
<blockquote>
<p>On a DVD in the safe</p>
</blockquote>
<p>This is the funniest form of Open Source that I have ever encountered. I kindly asked one branch of the administration
about a particular piece of software that they seem to use but I have not heard back from the yet. I’m looking forward to it though.</p>
<p>That’s it for this month. If you find interesting news from the world of Open Data or Civic Tech, let me know <a href="https://twitter.com/notknut">on twitter</a>
or <a href="mailto:this-month-in-open@k-nut.eu">via e-mail</a>.</p>At the Open Knowledge Lab Berlin we run a monthly event where we invite expert speakers to tell us about their branch of Open Data or Civic Tech. A while ago we got inspired by Berlin Hack and Tell where they started doing a news section to talk about interesting things that happened in the Open Source world since the last event. We decided to do the same for our events and call the section “This Month in Open”. I realised that instead of just sharing the news there, I could also share it on this blog so here we go.Hackathons miss the fundamental point of software development2020-09-28T09:25:00+00:002020-09-28T09:25:00+00:00https://blog.k-nut.eu/hackathons<p>I participated in a range of hackathons over the last years. Just this weekend, I took part
in the latest <a href="http://energyhack.de">energyhack</a> which made me realise how much my opinion
of hackathons has changed. I now think that hackathons, as I have experienced them, set the
wrong expectation for how software development works and miss the most fundamental point
of software development: We write software to provide tools to users. In order to know what
tools to build, we first need to talk to them.</p>
<p>In a traditional hackathon format (especially one that is somehow based on new data being
provided), the setup usually follows an established pattern. People come together and
get to know each other, the organisers present their problem description / challenge
and demonstrate the data or API that has been newly made available for this hackathon.
The participants are then expected to think about ways to “solve this problem”, form groups
and spend the next hours working on their prototypes. They go away into their little chambers,
write lots of code and eventually return with a presentation which the original organisers
then judge and possibly pick a “winner”.</p>
<p>Note the absence of the organiser in this setup for the complete time of the development effort.
This is just not, how software development should work. It is, in its essence everything that is
wrong with software development and I was hoping that we as the software development community
had slowly found our way out of it throughout the last years. In the traditional format,
developers - mostly privileged dudes with their <a href="https://smartenoughcity.mitpress.mit.edu/pub/8dthlkrx/release/1?from=10136&to=11026">tech goggles</a> on, start working on software that they think will help
with the problem. Could it actually help though? In order to figure this out, they should talk
to the domain experts - the very organisers of the hackathon. After all, those people though that
they had a challenge important enough to invite people to work on it. They probably know a thing
or two about the domain that is being worked on.</p>
<p>In the professional context, agile methodologies have now found their way to most teams because
they deliver better results. I think that a big part of the reason why agile works is not because
of daily standups but because of the idea that every sprint, developers, designers and product
people come together to discuss new features. In this process, the product owners can present the
issues that they would like to see solved and propose solutions. The tech people can then make a
call on the feasibility from a design and tech perspective. They can propose alternative implementations
for which the product people (the domain experts) can then check if they still help with the
original task. These dialogues make sure that the software that is being written actually helps.</p>
<p>It has been noted in other articles already (e.g. in <a href="https://stefan.bloggt.es/2020/09/der-antiberblingercontest/">this German article by stk</a>)
that we need a more long term setup instead of the expectation that an organisation can simply
invite a couple of tech people and then walk away with a perfect solution to their problem.</p>
<p>So, imagine you are a German governmental body and you would like to run a hackathon. What should you do?
First of all, you should make sure that you are set up for long term collaboration with the
community.<sup><a href="#fn1" id="ref1">1</a></sup> You can then invite people to your hackathon and bring your smartest
people yourself. You want to bring the people that have worked in the field for a while and know
all of the potential problems and pitfalls <sup><a href="#fn2" id="ref2">2</a></sup>. You might even
want to bring some lawyers to make sure that they don’t ruin the fun at the very end, telling you
that your idea is not possible given the governing laws. Is the goal of your hackathon to help a third
party? If you want to for example “solve homelessness”, you might want to invite a couple of homeless
people to hear what they think about the ideas that you are proposing for them. Once you got all
of these people together, you can start forming groups where all parties are represented. This allows
you to <a href="http://civicpatterns.org/patterns/build-with-not-for/">“build with, not for”</a>. The newly formed
groups can then take the time of the hackathon to talk about potential approaches and to discuss
upsides and downsides. They might even be able to start developing some clickdummies already
or to check if something is technically possible, but they should not be expected to go out of the
event with something that works. This will have to come at a later point through long term collaboration.</p>
<p>All of this probably does not sound as exciting as “24 hour of intense coding” but I believe
that it is a much more sustainable solution for all parties and might actually help in delivering
something that is of value in the end. Lets not forget: We can never develop “solutions”, if we
can do anything, we can build tools. And in order to build the right tools, we need to do one
thing first: talk.</p>
<hr />
<p><sup id="fn1">1. From working in this environment for a while now, I have learned that this is not trivial:
If the government is available for collaboration at all, they are available during the day job hours of the community.
When the community meets, they meet outside of working hours of the government. This makes it difficult
enough already to meet at all.<a href="#ref1" title="Jump back to footnote 1 in the text.">↩</a>
</sup></p>
<p><sup id="fn2">2. At the energyhack I have for example talked to the Berlin fire brigade which has lots of interesting data about the calls that they get. They do
not want to publish this data as they fear that one could use the addresses to draw conclusions
on the people that live there. This made immediate sense to me. When I proposed that they could publish
data on an aggregate - e.g. neighborhood - level, they told me that they had also thought about this
already but that they feared that insurance companies would use this to increase rates for high
risk neighborhoods. These are the kinds of disussions that we should be having.
<a href="#ref2" title="Jump back to footnote 2 in the text.">↩</a>
</sup></p>I participated in a range of hackathons over the last years. Just this weekend, I took part in the latest energyhack which made me realise how much my opinion of hackathons has changed. I now think that hackathons, as I have experienced them, set the wrong expectation for how software development works and miss the most fundamental point of software development: We write software to provide tools to users. In order to know what tools to build, we first need to talk to them.Postgres json aggregate functions are pretty cool2020-08-08T08:00:00+00:002020-08-08T08:00:00+00:00https://blog.k-nut.eu/postgres-json-functions<p>In my current project at work, we have a TypeScript backend that is backed by a Postgrses database. Our
usual approach would be to load the data from Postgres and then transform it into the desired shape for
our api with Typescript. We did some experimentation and realised that we could actually use Postgres
for many of the transformations itself which I thought was quite cool so I will demonstrate how this can
be done in this post.</p>
<p>Imagine we have a set of users (with an additional <code class="highlighter-rouge">age</code> column) that can be in different groups (n:m). Our data could look like this:</p>
<table>
<thead>
<tr>
<th>user_name</th>
<th>age</th>
<th>group_name</th>
</tr>
</thead>
<tbody>
<tr>
<td>User 2</td>
<td>32</td>
<td>Group 2</td>
</tr>
<tr>
<td>User 1</td>
<td>21</td>
<td>Group 2</td>
</tr>
<tr>
<td>User 2</td>
<td>32</td>
<td>Group 1</td>
</tr>
<tr>
<td>User 3</td>
<td>43</td>
<td>Group 1</td>
</tr>
<tr>
<td>User 1</td>
<td>21</td>
<td>Group 1</td>
</tr>
</tbody>
</table>
<p>Our goal is to transform this for our api to return this data keyed by the group_name, with each group having a collection of user objects.
For the table above, it would look like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"Group 2"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"age"</span><span class="p">:</span><span class="w"> </span><span class="mi">32</span><span class="p">,</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"User 2"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"age"</span><span class="p">:</span><span class="w"> </span><span class="mi">21</span><span class="p">,</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"User 1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"Group 1"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"age"</span><span class="p">:</span><span class="w"> </span><span class="mi">32</span><span class="p">,</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"User 2"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"age"</span><span class="p">:</span><span class="w"> </span><span class="mi">21</span><span class="p">,</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"User 1"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"age"</span><span class="p">:</span><span class="w"> </span><span class="mi">43</span><span class="p">,</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"User 3"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>We can create the data that is used for this example like this with a fresh Postgres:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="n">EXTENSION</span> <span class="n">IF</span> <span class="k">NOT</span> <span class="k">EXISTS</span> <span class="nv">"uuid-ossp"</span><span class="p">;</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">user_groups</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">TEMP</span> <span class="k">TABLE</span> <span class="n">user_groups</span> <span class="p">(</span><span class="n">id</span> <span class="n">uuid</span> <span class="k">default</span> <span class="n">uuid_generate_v4</span><span class="p">(),</span> <span class="n">name</span> <span class="n">text</span><span class="p">);</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">users</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">TEMP</span> <span class="k">TABLE</span> <span class="n">users</span> <span class="p">(</span><span class="n">id</span> <span class="n">uuid</span> <span class="k">default</span> <span class="n">uuid_generate_v4</span><span class="p">(),</span> <span class="n">name</span> <span class="n">text</span><span class="p">,</span> <span class="n">age</span> <span class="n">INT</span><span class="p">);</span>
<span class="k">DROP</span> <span class="k">TABLE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">user_group_map</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">TEMP</span> <span class="k">TABLE</span> <span class="n">user_group_map</span> <span class="p">(</span><span class="n">user_id</span> <span class="n">uuid</span><span class="p">,</span> <span class="n">group_id</span> <span class="n">uuid</span><span class="p">);</span>
<span class="k">delete</span> <span class="k">from</span> <span class="n">users</span><span class="p">;</span>
<span class="k">insert</span> <span class="k">into</span> <span class="n">users</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">age</span><span class="p">)</span> <span class="k">values</span>
<span class="p">(</span><span class="s1">'User 1'</span><span class="p">,</span> <span class="mi">21</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'User 2'</span><span class="p">,</span> <span class="mi">32</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'User 3'</span><span class="p">,</span> <span class="mi">43</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'User 4'</span><span class="p">,</span> <span class="mi">54</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'User 5'</span><span class="p">,</span> <span class="mi">65</span><span class="p">);</span>
<span class="k">DELETE</span> <span class="k">FROM</span> <span class="n">user_groups</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">into</span> <span class="n">user_groups</span> <span class="p">(</span><span class="n">NAME</span><span class="p">)</span> <span class="k">values</span> <span class="p">(</span><span class="s1">'Group 1'</span><span class="p">),</span> <span class="p">(</span><span class="s1">'Group 2'</span><span class="p">);</span>
<span class="k">DELETE</span> <span class="k">from</span> <span class="n">user_group_map</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">user_group_map</span> <span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">group_id</span><span class="p">)</span> <span class="k">values</span>
<span class="p">((</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">users</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">),</span> <span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">user_groups</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">)),</span>
<span class="p">((</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">users</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">user_groups</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">)),</span>
<span class="p">((</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">users</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">user_groups</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">)),</span>
<span class="p">((</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">users</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">0</span><span class="p">),</span> <span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">user_groups</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">1</span><span class="p">)),</span>
<span class="p">((</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">users</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="k">select</span> <span class="n">id</span> <span class="k">from</span> <span class="n">user_groups</span> <span class="k">limit</span> <span class="mi">1</span> <span class="k">OFFSET</span> <span class="mi">1</span><span class="p">));</span>
</code></pre></div></div>
<p>Let’s see how we can build the query that is required for our expected result step by step.</p>
<p>To start, the table that I showed above was created like this:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="n">users</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">users</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">name</span> <span class="k">from</span> <span class="n">users</span>
<span class="k">join</span> <span class="n">user_group_map</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">users</span><span class="p">.</span><span class="n">id</span>
<span class="k">join</span> <span class="n">user_groups</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">group_id</span> <span class="o">=</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">id</span>
</code></pre></div></div>
<p>This gives us:</p>
<table>
<thead>
<tr>
<th>user_name</th>
<th>age</th>
<th>group_name</th>
</tr>
</thead>
<tbody>
<tr>
<td>User 2</td>
<td>32</td>
<td>Group 2</td>
</tr>
<tr>
<td>User 1</td>
<td>21</td>
<td>Group 2</td>
</tr>
<tr>
<td>User 2</td>
<td>32</td>
<td>Group 1</td>
</tr>
<tr>
<td>User 3</td>
<td>43</td>
<td>Group 1</td>
</tr>
<tr>
<td>User 1</td>
<td>21</td>
<td>Group 1</td>
</tr>
</tbody>
</table>
<p>We can start by combining the user related data into one column like this:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">group_name</span><span class="p">,</span> <span class="n">to_jsonb</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="o">*</span><span class="p">)</span> <span class="k">as</span> <span class="n">user_data</span> <span class="k">from</span> <span class="n">users</span>
<span class="k">join</span> <span class="n">user_group_map</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">users</span><span class="p">.</span><span class="n">id</span>
<span class="k">join</span> <span class="n">user_groups</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">group_id</span> <span class="o">=</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">id</span>
</code></pre></div></div>
<p>This gives us:</p>
<table>
<thead>
<tr>
<th>group_name</th>
<th>user_data</th>
</tr>
</thead>
<tbody>
<tr>
<td>Group 2</td>
<td><code class="highlighter-rouge">{"id": "8f60793b-1849-4ba7-809f-b4ee3df5402b", "age": 32, "name": "User 2"}</code></td>
</tr>
<tr>
<td>Group 2</td>
<td><code class="highlighter-rouge">{"id": "c5b5d707-8556-4563-816f-e51e8e1eb5cf", "age": 21, "name": "User 1"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"id": "8f60793b-1849-4ba7-809f-b4ee3df5402b", "age": 32, "name": "User 2"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"id": "c5b5d707-8556-4563-816f-e51e8e1eb5cf", "age": 21, "name": "User 1"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"id": "ccf85355-27e8-4002-8562-72a3b285bcec", "age": 43, "name": "User 3"}</code></td>
</tr>
</tbody>
</table>
<p>For our api we aren’t actually interested in the <code class="highlighter-rouge">id</code> column though so we can drop it like this
(using the fact that this is now a jsonb column):</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">group_name</span><span class="p">,</span> <span class="n">to_jsonb</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="o">*</span><span class="p">)</span> <span class="o">-</span> <span class="s1">'id'</span> <span class="k">as</span> <span class="n">user_data</span> <span class="k">from</span> <span class="n">users</span>
<span class="k">join</span> <span class="n">user_group_map</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">users</span><span class="p">.</span><span class="n">id</span>
<span class="k">join</span> <span class="n">user_groups</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">group_id</span> <span class="o">=</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">id</span><span class="p">;</span>
</code></pre></div></div>
<p>This gives us:</p>
<table>
<thead>
<tr>
<th>group_name</th>
<th>user_data</th>
</tr>
</thead>
<tbody>
<tr>
<td>Group 2</td>
<td><code class="highlighter-rouge">{"age": 32, "name": "User 2"}</code></td>
</tr>
<tr>
<td>Group 2</td>
<td><code class="highlighter-rouge">{"age": 21, "name": "User 1"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"age": 32, "name": "User 2"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"age": 21, "name": "User 1"}</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">{"age": 43, "name": "User 3"}</code></td>
</tr>
</tbody>
</table>
<p>Now, all of this data is still split into multiple lines, lets group it by group_name using the <code class="highlighter-rouge">json_agg</code> function:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">group_name</span><span class="p">,</span> <span class="n">json_agg</span><span class="p">(</span><span class="n">to_jsonb</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="o">*</span><span class="p">)</span> <span class="o">-</span> <span class="s1">'id'</span><span class="p">)</span> <span class="k">as</span> <span class="n">user_data</span> <span class="k">from</span> <span class="n">users</span>
<span class="k">join</span> <span class="n">user_group_map</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">users</span><span class="p">.</span><span class="n">id</span>
<span class="k">join</span> <span class="n">user_groups</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">group_id</span> <span class="o">=</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">id</span>
<span class="k">group</span> <span class="k">by</span> <span class="n">group_name</span><span class="p">;</span>
</code></pre></div></div>
<p>This gives us the following table:</p>
<table>
<thead>
<tr>
<th>group_name</th>
<th>user_data</th>
</tr>
</thead>
<tbody>
<tr>
<td>Group 2</td>
<td><code class="highlighter-rouge">[{"age": 32, "name": "User 2"}, {"age": 21, "name": "User 1"}]</code></td>
</tr>
<tr>
<td>Group 1</td>
<td><code class="highlighter-rouge">[{"age": 32, "name": "User 2"}, {"age": 21, "name": "User 1"}, {"age": 43, "name": "User 3"}]</code></td>
</tr>
</tbody>
</table>
<p>As a final step, we can use Postgres’ <code class="highlighter-rouge">json_object_agg</code> function to now merge the data into our expected output.
To do so, we first wrap our old query in a <code class="highlighter-rouge">with</code> clause and then select from there:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">with</span> <span class="n">groups_with_users</span> <span class="k">as</span> <span class="p">(</span>
<span class="k">select</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">name</span> <span class="k">as</span> <span class="n">group_name</span><span class="p">,</span> <span class="n">json_agg</span><span class="p">(</span><span class="n">to_jsonb</span><span class="p">(</span><span class="n">users</span><span class="p">.</span><span class="o">*</span><span class="p">)</span> <span class="o">-</span> <span class="s1">'id'</span><span class="p">)</span> <span class="k">as</span> <span class="n">user_data</span> <span class="k">from</span> <span class="n">users</span>
<span class="k">join</span> <span class="n">user_group_map</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">users</span><span class="p">.</span><span class="n">id</span>
<span class="k">join</span> <span class="n">user_groups</span> <span class="k">on</span> <span class="n">user_group_map</span><span class="p">.</span><span class="n">group_id</span> <span class="o">=</span> <span class="n">user_groups</span><span class="p">.</span><span class="n">id</span>
<span class="k">group</span> <span class="k">by</span> <span class="n">group_name</span><span class="p">)</span>
<span class="k">select</span> <span class="n">json_object_agg</span><span class="p">(</span><span class="n">group_name</span><span class="p">,</span> <span class="n">groups_with_users</span><span class="p">.</span><span class="n">user_data</span><span class="p">)</span> <span class="k">from</span> <span class="n">groups_with_users</span><span class="p">;</span>
</code></pre></div></div>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">{</span>
<span class="nv">"Group 2"</span><span class="p">:</span> <span class="p">[</span>
<span class="err">{</span>
<span class="nv">"age"</span><span class="p">:</span> <span class="mi">32</span><span class="p">,</span>
<span class="nv">"name"</span><span class="p">:</span> <span class="nv">"User 2"</span>
<span class="err">}</span><span class="p">,</span>
<span class="err">{</span>
<span class="nv">"age"</span><span class="p">:</span> <span class="mi">21</span><span class="p">,</span>
<span class="nv">"name"</span><span class="p">:</span> <span class="nv">"User 1"</span>
<span class="err">}</span>
<span class="p">],</span>
<span class="nv">"Group 1"</span><span class="p">:</span> <span class="p">[</span>
<span class="err">{</span>
<span class="nv">"age"</span><span class="p">:</span> <span class="mi">32</span><span class="p">,</span>
<span class="nv">"name"</span><span class="p">:</span> <span class="nv">"User 2"</span>
<span class="err">}</span><span class="p">,</span>
<span class="err">{</span>
<span class="nv">"age"</span><span class="p">:</span> <span class="mi">21</span><span class="p">,</span>
<span class="nv">"name"</span><span class="p">:</span> <span class="nv">"User 1"</span>
<span class="err">}</span><span class="p">,</span>
<span class="err">{</span>
<span class="nv">"age"</span><span class="p">:</span> <span class="mi">43</span><span class="p">,</span>
<span class="nv">"name"</span><span class="p">:</span> <span class="nv">"User 3"</span>
<span class="err">}</span>
<span class="p">]</span>
<span class="err">}</span>
</code></pre></div></div>
<p>The <a href="https://www.postgresql.org/docs/9.5/functions-aggregate.html">Postgres documentation</a>
contains much more information about all of the different JSON aggregate functions so make sure to check that out
to learn more.</p>
<p>I think that this is pretty cool as it allows us to get the data from the database exactly as we want it instead of mixing
the select/transform between Postgres and TypeScript. In our project, we actually decided to not make use of this too
much though as this is probably quite difficult to understand for people that have not worked much with Postgres before
and we did not want to scare away new developers. I hope that by spreading this knowledge we can use this more in production
eventually though.</p>
<p>If you want to follow along, I also created a <a href="https://franchise.cloud/">Franchise</a> notebook that you can use to play around
with the examples <a href="https://static.k-nut.eu/franchise-postgres-json-aggregate-functions.html">here</a>.</p>In my current project at work, we have a TypeScript backend that is backed by a Postgrses database. Our usual approach would be to load the data from Postgres and then transform it into the desired shape for our api with Typescript. We did some experimentation and realised that we could actually use Postgres for many of the transformations itself which I thought was quite cool so I will demonstrate how this can be done in this post.