grahl/ch

Tolino needs better notes support

Mon, 02 Jan 2023 00:00:00 +0000

A while ago I traded my Kindle in for a Tolino Vision 6 to no longer give money to that awful company and I’m overall satisfied.

The hardware is excellent, importing books worked fine, and I haven’t found a book yet via Orell Füssli that they did not have.

The only bad aftertaste is that international titles often have family sharing restrictions, which means I cannot share them with my partner, and they would have to re-buy (booh). But that’s a manageable trade-off compared to notes management.

Notes and highlights

Notes, or much more often just highlights, are a function that basically any e-reader supports and that should be a core use-case with a robust implementation similar to page navigation, book purchase or backlight adjustment.

Tolino supports highlights and collects them in a „My notes“ book on your e-reader. This book is not available in the „webreader“. However, when you open a book in the webreader, in which you highlighted passages (on the e-reader), you see the same highlights and can open and edit them one by one. You do not have a download or export button, but you can see them. This means that they have full cloud syncing of your notes, they exist in their database and Tolino can make them available how they choose. They chose to make them inaccessible.

Exporting the notes.txt

Okay, but how about that „My notes“ book? Could I get that from the e-reader and avoid manual work that way? Actually, yes, the device contains a notes.txt file with a well formatted reference to the title a predictable delimiter, nice.

So let’s open it on a computer and attach the reader via USB like it’s 2005. And then nothing happens on a Mac since they chose MTP, for heaven’s sake. Their own documentation says that „This enables users to continue to use the tolino e-reader while it is connected to the computer.“ Cool, nobody needs that and they are not offering a legacy option. So let’s install unwanted third-party software for which they so eloquently point out that they cannot take responsibility for.

And there is a notes.txt file and there are…not all highlights in there. It turns out that the device file only contains notes made on the device since the last factory reset and does not sync from other devices including reading in the iOS app. So the book I have, which has 12 highlights, now has 2 entries in notes.txt. Fantastic. So, basically, my only option is to script something in the webreader to get the information out.

Feature requests

It would be a fairly simply feature to add a button in the webreader and let users get their notes. They already have it in their database and a TXT file would suffice.

JSON or CSV would be nice but plain text would work just fine. It would be cool if they’d offer me the option for a webhook to integrate a highlight in real-time into my notes application of choice, but I’m really just looking to get a few lines of text here without copy and pasting every single item.

With the current implementation it’s faster for me to photograph the page and use OCR on my phone to get a quote out of a book for reference. That can’t be it.

Comparing Lando performance on M1 Pro

Sun, 21 Nov 2021 00:00:00 +0000

This post was updated 2021-11-27 with additional numbers from the experimental feature “Use the new Virtualization framework.”

In the previous post we saw how the file system mounts can have a huge impact on overall Docker performance. Since then Apple has introduced their new Macbook Pro on their M1 architecture and significant work has happened so that there are now images available for Lando.

Apples and oranges

When running Docker Desktop images they can actually be built for an Intel architecture but performance issues are likely to arise there. For the time being this includes the default Lando database backend from bitnami/mysql.

You can see this in Docker Desktop with the “amd64” warning label or follow this post from Stack Overflow to find the image architecture.

Unfortunately the official mariadb is not a drop-in replacement for bitnami/mariadb so it can’t be used a service override, or if you do know what it would need consider adding it to The Great Armification #2688.

Speed comparison

As previously we use PHPUnit to run a test suite, this time a bit smaller with 19 tests and 227 assertions on PHPUnit 9.5 with PHP 7.4. Numbers are averaged across 2-3 tests. (Since some suggest PHP 8 might make a difference: Numbers seemed identical on M1.)

A notable difference from the last comparison is that we are using SQLite in memory as our test backend for the database to avoid any measurement error since the I/O bottleneck is the largest contributor to any performance issues for Docker on macOS.

System	Test execution
i5 4 core Linux Docker	47s
i9 8 core macOS native	24s
i9 8 core Docker*	32s
i9 8 core Docker†	1m54s
M1 Pro 10 core macOS native	28s
M1 Pro 10 core Docker*	59s
M1 Pro 10 core Docker†	3m54s
M1 Pro 10 core Docker experimental*	36s
M1 Pro 10 core Docker experimental†	1m46s

*) with Drupal core/contrib/vendor in exclude †) without exclude

Of course this one testsuite is highly subjective, not generally applicable and only looks at burst performance but that’s also something quite common for developers to need. Iterate, rerun, and not wanting to wait.

It’s noteworthy that on M1 nearly all tests ran slower, even native, so there is likely some optimization in the runtime still possible in future versions of PHP on ARM, since from the commonly published single-core scores the M1 Pro should be significantly faster.

Conclusion

Lots of improvements have gone into Docker on ARM and it’s generally usable now with Lando but Docker I/O performance is still a hurdle.

I recommend upgrading to at least Docker 4.2.0 and using the experimental virtualization framework, otherwise you have significant and unnecessary overhead from qemu-system-aarch64. Others have reported slow I/O with it but at least in this version it’s the opposite for me.

Working with the M1 Pro is much nicer given how much the fan kicked on in Intel models, let alone running on battery, but it’s not the leapfrogging that other disciplines are seeing in their workloads with PHP.

One last thing to note is that these tests ran sequentially. The tests above can be processed in parallel for example with ParaTest but in my tests the differences between ARM and Intel only amplify there with 338 tests natively coming in at 1m59s on Intel with 16 threads while AARCH64 comes in at around 4m30s (experimental 03m03s).

Tweaking a slow Lando site

Thu, 13 May 2021 00:00:00 +0000

Docker for Mac is notoriously slow, especially in terms of file syncing. It has gotten better in the past but it’s still the platform underperforming the most.

Recent versions of Lando have introduced some optimizations we can make use of to improve performance. To know if that actually benefits us, let’s measure.

Establishing a baseline

First, we need a fairly stable scenario, I choose a module in a project with a decent amount of Drupal kernel tests (70 tests, 1685 assertions). The number of tests should iron out any variations of other things on the host influencing the test.

Executing that on a mediocre i7 running Linux gives us a baseline number without any filesystem overhead:

Time: 5.11 minutes, Memory: 8.00 MB

Beefy i9 with lots of RAM but macOS:

Time: 18.71 minutes, Memory: 8.00 MB

Ouch. So let’s try to use excludes.

Excludes

We can tell Lando to sync certain directories only once, they will then no longer be copied to and from the container. If you are working in a team, where others don’t have this bottleneck you can put that into .lando.local.yml.

excludes:
  - docroot/core
  - docroot/libraries
  - docroot/modules/contrib
  - docroot/themes/contrib
  - docroot/themes/custom/your_theme/node_modules
  - vendor

Note that changes in these folders require a rebuild every time you change something in there, or you execute the command locally as well as in the container for consistency.

Time: 8.34 minutes, Memory: 8.00 MB

It turns out that this particular project consisted of 238k files due to multiple node_modules folders and that 221k of those files could be excluded with little effort.

Just to see what would happen I added the entire repository to the container in an exclude and reran the test, it came in at 3.76 minutes. Not practical for everyday use but showing that with that bottleneck removed, the i9 is now outperforming the other system.

Disabling the home mount

I’m not a fan of the fact that Lando mounts my home directory into the container, apart from performance, that just doesn’t belong there and I’m much happier not having my SSH keys in there.

I have made a fairly trivial change to the CLI you can use to disable this: Remove home mount by grahl · Pull Request #1 · grahl/cli · GitHub

This should help with sluggishness and system load overall. I did not expect it to have a measurable impact when PHPUnit wasn’t looking in /user but it seems to have helped:

Time: 6.16 minutes, Memory: 8.00 MB

Removing unnecessary app mounts

I have not done extensive tests on this but by default every service gets the “app” mount. That’s unnecessary, you can disable it with: app_mount: disabled

Daily shutdown script for Timingapp

Wed, 24 Mar 2021 00:00:00 +0000

When I’m in the middle of work I often tend to overlook how long I’ve already worked on any given day, especially when working from home during Covid where I mostly don’t have fixed start or end dates, like so many of us.

To avoid having my overtime balance increase unintentionally by working half an hour here and there I’ve created a script for myself based on the excellent Timing to see when I’ve worked about 7.5h to allow me to begin to shut down my work for the day.

You can do that with a small Applescript by looking for the total time tracked today in seconds and then creating a notification. There’s probably a more efficient way than running this command every minute and just comparing number of seconds, but it seems to do the trick.

In addition to the notification I also force dark mode to nudge myself a bit more proactively.

tell application "TimingHelper"
	if not scripting support available then
		error "Scripting support requires a Timing Expert license. Please contact support via https://timingapp.com/contact to upgrade."
	end if
end tell

tell application "TimingHelper"
	set usageData to get time summary between current date and current date
	set total to overall total of usageData
	 # work day	8 * 60 * 60: 	28'800s
	 # threshold 7.5 * 60 * 60:	27'000s
	if total ≥ 27000 and total < 27060 then
		display notification "You have worked 7.5h, begin shutting down." with title "Timing alert" sound name "Frog"
		tell application "System Events"
			tell appearance preferences
				set dark mode to true
			end tell
		end tell
	end if
	delete usageData
end tell

Now add an agent to ~/Library/LaunchAgent such as the one shown below like com.example.timing_monitor.plist and enable it with launchctl load com.example.timing_monitor.plist and you should be good to go.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.timing_monitor</string>
<key>Program</key>
<string>/usr/bin/osascript</string>
<key>ProgramArguments</key>
<array>
    <string>osascript</string>
    <string>/your/path/your script.scpt</string>
</array>
<key>StartInterval</key>
<integer>60</integer>
</dict>
</plist>

Sharing Drupal's public file storage

Fri, 20 Dec 2019 00:00:00 +0000

In a client project we recently had the challenge of not having persistent file storage available for public/private files across multiple instances.

The only realistic solution, at the moment, is using a module such as S3 File System to employ a different storage backend. The S3 module for Drupal isn’t perfect but let’s look at more of why this is even difficult.

Drupal’s public storage structure

ls -l sites/default/files
drwxrwxr-x 2 www-data www-data 20480 Dec 20 13:57 css
drwxrwxr-x 2 www-data www-data 16384 Dec 20 13:56 js
drwxrwxr-x 3 www-data www-data  4096 Jul 28  2018 media
drwxrwxr-x 3 www-data www-data  4096 Dec 20 13:56 php
drwxrwxr-x 9 www-data www-data  4096 Jun 24 21:29 styles

This minimal example already gives us four distinct behaviors which should really be addressed through distinct storage solutions. Unfortunately we are currently stuck with one for historic reasons. I’ll address css/js at the end, since it’s the tricky one.

files/media (or any upload folder)

You can define arbitrary folders in Drupal’s Field UI to manage uploads coming in from end users (even though that’s risky and should probably be in private), as well as images, videos, etc. from content editors.

We need this to be synced across all instances so this is something where we absolutely need something like an S3 bucket or NFS storage.

files/php

I’m assuming that you are not using config folders in here (keep them in your repo) but rather have Twig generating caches in php/cache.

This is a pure cache which could just as well live in a temp directory. End users do not receive its data. If files are deleted in that folder they get regenerated on-demand without a manual cache clear. Nothing to do here, it’s correct on all instances.

files/styles

This folder is basically a cache, one we want to persist if at all possible, but if it were to vanish and be empty, we would have no problem. It would simply take a bit more processing when incoming requests to them are generating assets on the fly (like php/twig does).

It would be nice if this were available on a shared storage but it’s not absolutely necessary. It should persist in some form since always recomputing all images is not really acceptable but that’s going to vary depending on what your infrastructure makes available on instantiation of additional instances.

files/css & files/js

During cache rebuilding (i.e. drush cr) the asset dumper writes the relevant assets to disk. Afterwards they can be served directly by the web server without involving PHP.

This process makes sense insofar as that we need everything bootstrapped to know which assets to build. It works mostly fine with the S3 module unless you change buckets while having to regenerate caches and have messy metadata (e.g. prod ➡ dev).

Unfortunately these files cannot be regenerated on-demand. This means that if I deploy and the cache is rebuild on instance 1 (assuming a change in assets) the end user will get a 404 if it’s requested from instance 3.

At this point there does not seem to be a separate shorthand available to just dump the assets, so per instance one would need to drush cr, which is not an acceptable penalty.

Thus, we need this on a persistent storage as well. Or do we?

There has been some work on aggregation in core: Race conditions, stampedes and cold cache performance issues with css/js aggregation (#1014086). However, advagg promises us to do just that in terms of lazy-loading. Without file I/O so it might even be worthwhile when you do have a shared storage but no CDN.

I’ve used advagg in the past just to get a bit better of an asset preprocessor but clearly this module provides much more essential features and should probably be on your list of standard performance optimizations.

Linkdump July 2019

Wed, 31 Jul 2019 00:00:00 +0000

Noteworthy links, I probably posted them on Twitter, too.

Can I use prefers-color-scheme? Yes you can, and you should. (Just added it to this blog ✌)
Choose boring technology «We should generally pick the smallest set of tech that covers our problem domain, and lets us get the job done»
B612 is an highly legible open source font family designed and tested to be used on aircraft cockpit screens.
Popups: 10 Problematic Trends and Alternatives

Linkdump May 2019

Fri, 31 May 2019 00:00:00 +0000

Noteworthy links, I probably posted them on Twitter, too.

Sending lots of mail with Drupal My post on our company’s blog
Renders links to Drupal.org issues based on issue status (Chrome extension)
The Drupal 8 render pipeline (in all its glory)
Packagist semver checker
The Post-Meritocracy Manifesto
Building an Intuitive Admin: Usability for the Forgotten End-User Very good overview on backend UX and what the site builder needs to do to make the customer happy.
How to prepare for Drupal 9 You can tell d.o to check your project for deprecated code automatically.
Drupal Admin UI See where Drupal’s UI is going and what Claro will bring.
Search Overrides A Search API module to override specific terms
PHPStorm 2019.1 Twig breakpoints (!)
Writing better Drupal code with static analysis using PHPStan
Links to cross-origin destinations are unsafe
Reducing MTTR the Right Way
Footers 101: Design Patterns and When to Use Each
Fifty Fizzbuzzes
Splitting user stories

Linkdump December 2018

Mon, 31 Dec 2018 00:00:00 +0000

Noteworthy links, I probably posted them on Twitter, too.

The Container Operator’s Manual - Velocity NY 2018 Video
Entity Explorer, your trusty helper 🐕 I wrote a small debugging module, check it out on our company blog.
On Slow Software
Shopping Cart or Wishlist? Saving Products for Later in Ecommerce You should persist carts.
bat is a great way to improve your cat
Smallest possible syntactically valid files of different types
How to keep your Drupal 8 contributed modules ready for Drupal 9
How content drives conversion
Technology Myths and Urban Legends «When users don’t clearly understand how systems function, they develop unique (and often incorrect) theories to explain their experiences.»
Revisions tab is now visible with one draft d.o Change record
What’s The Forecast? OmniFocus 3 fixes some long outstanding issues such as having to use fake due dates.
Drupal 8.6.0 is available
Developer Tools Initiative - Part 5: Announcing our Migration Drupal’s getting Gitlab.
PHP 7.4 erhält Typed Properties: Auf dem Weg zum typsicheren PHP? Typed properties in PHP 7.4 (German).
The Top 3 Ways to Fail at Interviewing Software Developers
How to take full page screenshot in Chrome without extensions

Linkdump June 2018

Sat, 30 Jun 2018 00:00:00 +0000

Noteworthy links, I probably posted them on Twitter, too.

When should we release Drupal 9?
Execute SQL against structured text like CSV or TSV
Moderation Note Module to provide annotations in Drupal Content Moderation.
Tweet The senior dev opens a usability bug.
5 examples of web animation done right
GDPR consent design: how granular must adtech opt-ins be?
When Memcached Slows Your Drupal Site’s Performance
Slack is the opposite of organizational memory
Drupal 8 Release Planning in the Enterprise
Be Promiscuous with Drush’s core-quick-drupal
Plain Language Is for Everyone, Even Experts
Ansible for DevOps Book
New option to hide untranslatable field widgets Drupal change record
Developer-driven development

Demystifying Apple's USB-C charging

Sat, 10 Jun 2017 00:00:00 +0000

Apple sells three different models of USB-C chargers and includes them respectively with their three Macs:

Model	Charger
Macbook	29W through 14.5V at 2A
Macbook Pro 13''	61W through 20.2V at 3A (and fallback 9V at 3A)
Macbook Pro 15''	87W through 20.2V at 4.3A (and fallback 9V at 3A)

So there are now several USB-PD (Power Delivery) available from Apple, at three different voltages. In nearly all cases the higher wattage should be better but there are interesting differences.

Macbook (2016)

The Macbook supports charging at 20.2V and 14.5V respectively. However, the larger charger doesn’t actually help this model, since it draws less power at the higher voltage to equal the ~28W of the stock charger:

Charger	Measurement
29W model
61W model

I’ve also tested a 5V 3A power bank and the Macbook can also charge at that voltage, though only giving it 15W.

Macbook Pro 13’’ (2016)

No surprise here that the Macbook Pro charges faster with the charger that was delivered with it but it’s important to note that you can charge it with the smaller one just fine and there is a significant difference in size and weight (approx. 120g vs 200g), which makes the 29W model a great alternative for business travel.

However, if the battery isn’t fully charged and you are putting the machine under heavy load, it will complain constantly that there isn’t enough to charge the laptop (charging mode can toggle on and off).

Charger	Measurement
29W model
61W model

iPad Pro 12.9’’ (2015)

The iPad Pro is an interesting outlier since not only does it support the 5.2V profile but also the 9V and 14.5V profile but it cannot do the 20.2V.

This means that the iPad can actually be charged faster with the smaller charger:

Charger	Measurement
29W model
61W model

Summary

I did not have a 87W charger at hand to play around with and presumably it could still charge at least the Macbook Pro faster but I don’t think the additional cost will actually have much of an impact.

My takeaway from the comparison is that the cross-device compatibility is great and I’ll likely only travel with the 29W model and use the 61W when stationary for the Macbook Pro.

Migrating to BLT - Part 2

Mon, 05 Jun 2017 00:00:00 +0000

OK, so you have your site locally running after Part 1, now it’s time to deploy it. As before I’m basing this on BLT version 8.8.3.

Deployment basics

As shown in the last post, you don’t need much else besides blt deploy and as the manual states you can combine multiple commands as such to push directly to the remote repository defined in your project.yml:

blt deploy -Ddeploy.branch="master" -Ddeploy.commitMsg="Your message"

You can run this locally and the target repository now should contain your built site.

I personally prefer to push a tagged branch but as of now one can only push tags without branches, meaning you’d be in detached head. Check this issue for more information and choose either branch or tag for now.

Tip: Make sure that your CI server has a recent Git version, or force pushing might give interesting results. (It’s automatically checked in the next release 😎)

Automating it

I have all my projects on a Gitlab server with Gitlab CI, so naturally I was looking for a way to do the CI integration without the already supported platforms, such as Travis. If you run blt ci:travis:init the resulting Travis-file contains basically all the information you need.

Here for example I’ve taken some setup information from the before_install phase and adapted it to for a minimal example, which is triggered on certain tags.

stages:
  - build
  - deploy

make-release:
  stage: build
  script:
    - composer self-update
    - composer validate --no-check-all --ansi
    - composer install
    - vendor/bin/blt validate:all
    - vendor/bin/blt deploy -Ddeploy.branch="master-build" -Ddeploy.commitMsg="Release $CI_COMMIT_REF_NAME"
  only:
    - /^release/.*$/

prod-deployment:
  stage: deploy
  variables:
    LOGIN: 'user@host'
    APP_PATH: '/var/www/directory'
  script:
    - ssh -T $LOGIN "cd $APP_PATH && git pull"
    - ssh -T $LOGIN "cd $APP_PATH && vendor/bin/blt setup:update"
  only:
    - /^release/.*$/
  dependencies:
    - make-release
  environment:
      name: production

Caveats and tips

I like to make a database update at the beginning of the deploy stage and usually set the Environment Indicator to the release number via_ drush sset environment_indicator.current_release $VAR.
Have a look at vendor/acquia/blt/scripts/travis to see see what would be required to run all tests, which is somewhat out of scope of this tutorial but definitely recommended.
This example assumes that you’d initially set up your prod environment manually and have added the relevant settings in local.project.yml and local.settings.php.

Using configuration split

If you run local|setup|deploy:update you might have noticed that the rules under “modules” in project.yml activate and deactivate modules after everything has run, which is not ideal. So let’s just empty all those rules with { }. But I’d be really nice to have environment-specific configuration as the example shows for devel, views_ui and so on, which is where config_split comes in.

Out-of-the-box BLT provides folders for ci and default but otherwise hasn’t enabled these. The blog post by Jeff Geerling provides general information but isn’t quite up-to-date.

However, you hardly need to do anything: Just enable the profiles in the backend with the common names such local, ci, test, prod and start using them. If you have config_filter enabled, you can just use cim/cex without any additional parameters and can ignore csex/csim.

The thing which confused me the longest time is the wording of configuration split, where blacklisting actually means that I want to keep something only within this environment (e.g. devel locally) and graylisting means I don’t want to overwrite something (such as webform forms on production). Thankfully, people are working on that.

So there you go, deployment on BLT with a minimum of moving parts on your own stack.

Migrating to BLT - Part 1

Fri, 26 May 2017 00:00:00 +0000

Photo CC-BY by amagill

While I’m a fan of CI/CD automation it took me a while to reinvent a few wheels until I realized that Acquia’s BLT for Drupal 8 is the template that I want. In the next few posts I will provide you some tips and tricks, for the things which aren’t self-explanatory, yet.

Prerequisites

Make sure you read over the BLT documentation once fully to understand certain general concepts. For example, it’s imperative to understand that the project repository is not what gets deployed but rather a deployment artifact is pushed to a separate repository that also includes the blt tools but no development dependencies.

Migrating an existing project

The documentation recommends starting with a fresh site but I find it much easier to see the differences in setups when I have a working real-life site to do the comparison on. The documentation says it’s possible to add BLT to an existing project but I find it unhelpful for beginners to figure out if the migration actually was complete and ideal, or not. Plus, you likely have idiosyncrasies in your setup. Here’s how you can get there (based on v8.8.3) through manual merging.:

Create a new site with BLT. Follow all steps including setting up the VM and blt local:setup.
1. Don’t forget to set the installation profile to what you had before in project.yml.
2. Remember to customize the vagrant_ip (and of course set php_version to "70").
3. Don’t try moving the site to a subdirectory of the repo, weird stuff happens with Vagrant and other scripts.
Delete the git directory that was created for you and move your existing git folder over. You can now add items selectively and git add them until you have everything. A good start would be your “custom” folders and the require entries in composer.json, minus core et al. At the end your staged changes should mostly show renames.
Add a useful target to the site aliases to sync from and set it in project.yml. For example, to sync directly from production set remote under drush.aliases to '${project.machine_name}.prod' and execute blt local:refresh afterwards.
1. BLT is actually smart enough that it can reset your site uuid and install from the config directory from scratch but you’re going to need relevant data anyway, so why not fetch it now.
2. The refresh will sanitize user accounts, so you might want to use drush upwd or drush uli afterwards.
You have essentially removed the installed site now but left all supporting files intact and it’s now easy to move over the yml configuration into config/default and you can locally cim & cex or use blt commands afterwards.
1. Note that the site hash is in /salt.txt and needs to be updated, too, from the existing site.
Done! Your site should now be locally available with everything you had before but under BLT.

Commands to try out

I’m far from an expert yet but just the pure joy of use of everyday usage is really worth the effort. For example:

blt local:refresh Already mentioned above, it syncs your environment from a remote destination and does all the stuff you might forget to do. It can also fetch the site files, or do that directly with blt sync:files. To think I used to write drush sql-sync commands by hand…
blt local:update Clearly a lot nicer than chaining drush commands, e.g.: drush cim -y && drush updb -y && drush cr -y
blt fix Just run all the fixes. No weird issues with the phpcs config path, if not installed globally, or having to manually specify paths. It just works. 🙌
blt deploy Details on that in a later post but one of the key features of BLT which really improves on simpler scripts, which most of us use to just automate copying files and calling composer. I’m extremely satisfied with the output of this command and the abstraction it provides.

In a few days I’ll post more about how to work with config_split and the relevant deployment commands, when you are not using Travis/Acquia Cloud.

Refactoring complex Drupal modules

Sun, 01 Jan 2017 00:00:00 +0000

In 2016 I spent a significant amount of time on porting and refactoring the LDAP module for Drupal 8. This work is ongoing and lots of work still needs to be done for a proper 1.0 release.

Here’s what I encountered along the way and why Drupal 8 is such a massive step forward for complex modules like LDAP.

Good intentions

If you look at the 7.x codebase you can clearly see that the authors wanted to have a well-tested and supported module which worked as expected even though it contains a cornucopia of features. This was not easy in 7 and it shows.

It’s visible in the fact that unit tests are really integration tests (due to SimpleTest). It’s visible in the fact that integration tests use complex bootstrapping mechanisms which switch out significant amount of code to substitute it with test code in huge, monolithic test classes. Plus, there are multiple lines of test architectures and manual test instructions.

The most interesting part is that you see that the authors wanted to include object-oriented code but Drupal’s structure and the code’s legacy architecture prevented them from doing so.

First obstacle: Class loading

In 7 there is no real autoloading. There are solutions now but those were not available to the authors. What did they do?

They wrote a custom class loader in each module to fetch their helper classes.

function ldap_user_conf($type = NULL, $reset = FALSE) {
  static $ldap_user_conf;
  static $ldap_user_conf_admin;

  if ($type == 'admin' && ($reset || !is_object($ldap_user_conf_admin))) {
    ldap_servers_module_load_include('php', 'ldap_user', 'LdapUserConfAdmin.class');
    $ldap_user_conf_admin = new LdapUserConfAdmin();
  }
  elseif ($type != 'admin' && ($reset || !is_object($ldap_user_conf))) {
    ldap_servers_module_load_include('php', 'ldap_user', 'LdapUserConf.class');
    $ldap_user_conf = new LdapUserConf();
  }

  return ($type == 'admin') ? $ldap_user_conf_admin : $ldap_user_conf;
}

Note that this function not only switches between loading a subclass and its parent class but also contains a custom caching mechanism.

An additional problem was that these classes could not declare their dependencies so that the following was not uncommon and created code which is tied together only by wishful thinking and manual testing:

module_load_include('php', 'ldap_user', 'LdapUserConf.class');
module_load_include('inc', 'user', 'user.pages');

class LdapUserConfAdmin extends LdapUserConf {

The refactoring in the D8 branch allowed us to get rid of LdapUserConfAdmin and handle its tasks directly in a standard configuration form class. While we have not yet gotten rid of LdapUserConf, the beginning of the class exemplifies how autoloading has already made a huge difference:

namespace Drupal\ldap_user;

use Drupal\Component\Utility\Unicode;
use Drupal\ldap_servers\Entity\Server;
use Drupal\ldap_servers\TokenFunctions;
use Drupal\ldap_user\Exception\LdapBadParamsException;
use Drupal\user\Entity\User;
use Drupal\user\UserInterface;

class LdapUserConf {

Second obstacle: Class and configuration structure

Due to the inefficient mechanisms above it’s no wonder that classes weren’t separated into functional groupings but rather stuck to one huge class to handle everything, with a subclass for the administration form.

Additionally, the existing classes tried to provide an interface to the complex attributes required to configure a site with LDAP authentication, user mappings, etc., and thus, we see the following:

// Creation & default value for saveable data.
public $userConflictResolve = LDAP_USER_CONFLICT_RESOLVE_DEFAULT; 
 
// Complex data structure without saveable data.
public $provisionSidFromDirection = array(
  LDAP_USER_PROV_DIRECTION_TO_DRUPAL_USER => LDAP_USER_NO_SERVER_SID,
  LDAP_USER_PROV_DIRECTION_TO_LDAP_ENTRY => LDAP_USER_NO_SERVER_SID,
); 

function __construct() {
  // Loads all saveable fields via variable_get().
  $this->load();

Obviously, Drupal’s configuration storage can and should take of at least all saveable values. That conversion has already made a huge difference in terms of code complexity but required painstaking manual conversions due to subtle logic differences in many places.

Also note that once we no longer have to manually load the configuration values from the database the cache for LdapUserConf is not needed anymore. We could then remove a significant amount of code dealing with setting and resetting it.

Third obstacle: Global constants

While constants can in general be useful to denote, well, constant values, their usage in the LDAP module is often problematic:

define('LDAP_USER_EVENT_SYNCH_TO_DRUPAL_USER', 1);
define('LDAP_USER_EVENT_CREATE_DRUPAL_USER', 2);
define('LDAP_USER_EVENT_SYNCH_TO_LDAP_ENTRY', 3);
define('LDAP_USER_EVENT_CREATE_LDAP_ENTRY', 4);
define('LDAP_USER_EVENT_LDAP_ASSOCIATE_DRUPAL_ACCT', 5);

For one, their numeric values are a problem since it makes reading the configuration settings difficult and cryptic:

ldap_user_conf:
   drupalAcctProvisionTriggers:
    2: '2'
    1: '1'
  ldapEntryProvisionTriggers:
    6: 0
    7: 0
    8: 0
    3: 0
  manualAccountConflict: 3
  acctCreation: 4

More importantly though they are global in scope and need to be declared at the right time, which is also why they are not normally found in context but just in bulk in the your_module.module file. Thus, they also need to be manually loaded in some cases, e.g. LdapUserConf:

require_once('ldap_user.module');

For now we have stashed these constants as static values in a trait, that makes them much easier to read and maintain:

namespace Drupal\ldap_user;

trait LdapUserConfigurationValues {
  public static $eventCreateDrupalUser = 1;
  public static $eventSyncToDrupalUser = 2;
  public static $eventCreateLdapEntry = 3;
  public static $eventSyncToLdapEntry = 4;
  public static $eventLdapAssociateDrupalAccount = 5;

Now we can just call them with either depending on the context:

LdapUserConfigurationValues::$eventCreateDrupalUser
self::$eventCreateDrupalUser

These magic values are now put somewhere safe and core’s configuration storage does the rest. Ideally, we would not need even a fraction of these magic values, but it’s a decent interim solution which avoids complex update hooks to transform legacy states into new states.

Fourth obstacle: Unit testing

Once we have had those taken care of and all those other dependencies cleanly resolved, running unit tests became feasible and lots of pseudo-code for integration tests could be removed.

For example, here is TokenTests, where we now can easily mock a function buried deep within the LDAP module:

protected function setUp() {
  parent::setUp();
  /* Mocks the Server due to wrapper for ldap_explode_dn(). */
  $this->serverFactory = $this->getMockBuilder('\Drupal\ldap_servers\Entity\Server')
  ->disableOriginalConstructor()
  ->getMock();

  $this->serverFactory->expects($this->any())
  ->method('ldapExplodeDn')
  ->willReturn($this->data);

  $this->container = new ContainerBuilder();
  $this->container->set('ldap.servers', $this->serverFactory);
  \Drupal::setContainer($this->container);
  // ...
public function testTokenReplacement() {
  // ...
  $nameReplacement = $tokenHelper->tokenReplace($account->reveal(), '[property.name]', 'user_account'); // Test calling ldapExplodeDN()
  $this->assertEquals($ldap_entry['sAMAccountName'][0], $nameReplacement);

The process of writing that test also made the code itself more robust since it was necessary to encapsulate the call to the PHP function ldap_explode_dn() within a wrapper. Due to this wrapper we can now run the test even if the LDAP extension is not installed (such as on drupal.org).

Looking back

Knowing what I know now would I do it the same way again? Definitely not.

It’s important to note that a huge amount of work was already done by larowlan, queenvictoria, and many more to get a first port successfully running. With that already in place, the existing state required a step-by-step refactoring and that is why this approach was needed. It’s also why this journey and its results are likely relevant to other module maintainers with partial ports as well.

Here is how I would now approach a complex module without an existing Drupal 8 port:

Start fresh. Drupal Console makes it so easy to set up the skeletons, just use those and put all the existing code into a legacy folder.
Make a plan. Make a list of each file and function and what it should do and what it currently does. Then question each if it’s still useful and needed.
Migrate functions not structure. Take the useful logic from the legacy folder but question whether the architecture of your files and classes really couldn’t be structured better.
Profit.

If you are still unsure of how good code should look and how to get started with refactoring, I’d recommend Clean Code for an excellent introduction.

GTD: Processing email on iOS

Thu, 22 Dec 2016 00:00:00 +0000

If you are a GTD’ler and using a Mac there is a good chance that you are using OmniFocus. On that machine processing your email and throwing it into the OmniFocus inbox is a great way to getting to Inbox Zero and still responding quickly and easily right within the message thread.

This is made possible because OmniFocus uses message:// links to open the archived message in the note, They are embedded when the message is dragged into an OmniFocus window or if you used the Clip-o-Tron.

In an ideal world the same would happen on iOS as well with the share sheet. It doesn’t. However, there are third-party solutions to make that possible.

I’ve been using Dispatch for the last few months and can definitively recommend that to get that setup working. Just make sure that you select the proper link protocol in the settings (i.e. message://).

I’ve rarely had issues with Dispatch but I recommend keeping the stock email client set up as well and just choosing notifications for your preferred one. My personal preference is no notifications except for a very short list of VIP accounts from the stock accounts. The stock client is also helpful if the rendering of Dispatch isn’t perfect or if an attachment fails.

Now you don’t have an excuse anymore for scrolling through the same dozen messages on your mobile and not doing anything about it. ✉️🔨

Bulk generating calendar events with Applescript

Sat, 17 Dec 2016 00:00:00 +0000

Sometimes you want to create calendar events which differ in predictable ways such as “Read/do chapter/session/workout x”.

A trivial problem for a programmer but basically impossible with the stock macOS calendar or third-party tools such as Fantastical as far as I can tell. They can have complex recurrence rules but all deviations per event are a manual process it seems.

So here is an Applescript script I quickly googled together for your convenience to create thirty events each containing an integer reference. Adjust as necessary.

tell application "Calendar"
	activate
	--- date() uses your localized date formatting.
	set theCurrentDate to date ("17.12.2016 09:00")
	set i to 1
	repeat while i < 31
		tell calendar "Home"
			make new event at end with properties {description:"", summary:"My task - Day " & i, location:"", start date:theCurrentDate, end date:theCurrentDate + 60 * minutes}
		end tell
		set i to i + 1
		set theCurrentDate to theCurrentDate + (1 * days)
	end repeat
	reload calendars
end tell

The script has obvious limitations with the hardcoded values it contains. A more flexible solution would use dialogs to ascertain the number of events, starting date, calendar to use, or event details.

I’m just content I don’t have to keep adding individual events by hand to get this done.

Export cookies from Safari in Sierra

Sun, 09 Oct 2016 00:00:00 +0000

I wanted to automate a batch processing job and was too lazy to programmatically fetch the cookies by emulating a login form so I looked for a way to just reuse my cookies.

That should be pretty easy in general, for example with the Python bindings on macOS, but the syntax changed a bit between 10.10 and 10.12 respectively, so I fired up Xcode and the following ten-ish lines of Swift work nicely in generating output which is fit for wget and other tools using the ancient cookie file format.

import Foundation

print("The url to fetch:", terminator: " ")
let response = readLine(strippingNewline: true)

let storage = HTTPCookieStorage.sharedCookieStorage(forGroupContainerIdentifier: "Cookies")
let cookies = storage.cookies(for: URL(string: response!)!)

print("# HTTP Cookie File\n")

for cookie in cookies! {
    var secure = "FALSE"
    if (cookie.isSecure) {
        secure = "TRUE"
    }
    var expires:Int = Int(cookie.expiresDate!.timeIntervalSince1970)
    print("\(cookie.domain)\tTRUE\t\(cookie.path)\t\(secure)\t\(expires)\t\(cookie.name)\t\(cookie.value)")
}

I don’t get points for exception handling or elegance but I have my 🍪.

I haven’t yet figured out a good workflow of generating such CLI binaries without hunting for them in the build folder and copying them over, input on that is very welcome.

First look at Acquia Lightning

Thu, 15 Sep 2016 00:00:00 +0000

![Some modules in lightning](/img/lightning-modules.png)

Lightning is a distribution of Drupal 8 developed by Acquia which attempts to deliver a preconfigured and optimized out-of-the box experience.

First impressions

The installation via Composer is smooth and setting up the site with the default profile is all that is necessary. Nice.

You get a site which provides you with Panels via Panelizer, Media management via media and entity browser as well as moderation via Scheduled Content and Workbench Moderation.

On the other hand, I’m surprised not to see Paragraphs here and the reliance of media embedding in rich-text components seems inelegant. Also power-user essentials, such as Admin Toolbar, are lacking.

Media

Lightning provides a very nice showcase on how you can easily upload and reuse assets with the entity browser.

I look forward to using that in future projects, though its still odd that there is barely any documentation on organizing or segmenting asset management there.

Apparently, Views will need to fill that gap, but I hope future improvements here will set a greater focus on image pools, as easily accessible as now but with more fine-grained controls, when you can’t trust your editors to do the right thing.

Moderation

Until the 8.2 merge of Workbench moderation as “Content Moderation” I was not certain which access control suite might come out on top for 8 but Lightning seems to have had the right idea.

What is surprising is that the experience of scheduled content is rather grim in terms of workflow states and publishing processes. The user does not have any good feedback what version is currently published, how to go about modifying them, nor do additional pending states seem to work at all out of the box. That needs lots of work and is a major weak point compared to the usual enterprise CMS offerings, which I had hoped Lightning would have a better answer to.

Conclusion

Lightning is a great demo case to show how Drupal 8 is in many ways a different ecosystem than 7 and what it can do but I would not necessarily choose it as a base distribution since it’s not hard to take a look at their example and tailor your site to your actual needs.

Spress on AWS with HTTPS

Sun, 11 Sep 2016 00:00:00 +0000

When I got started on my migration to Spress I figured that I would just run the site on a generic hoster that also offers letsencrypt certificates. Turns out, there really wasn’t any hoster who came close to the $7 DigitalOcean VM. So, AWS it is.

Specifically, deploying to S3, letting it do the static site hosting and putting CloudFront in front (haha) of it to handle HTTPS.

Amazon has an excellent guide to get you started setting that up. Just follow that and when it comes to the certificates, use AWS Certificate Manager or letsencrypt-s3front to handle the certificates.

Caveats

I had a terrible time getting letsencrypt-s3front running on macOS, mostly due to problems with Python and build dependencies. When I ran it on a regular CentOS VM with a working letsencrypt setup, everything worked fine. When Docker finally runs half-way stable on my machine I plan on handing that off to an image to rerun without effort when I have to renew the certificate… Update 2016-12-30: Consider using AWS Certificate Manager instead of Let’s Encrypt, it’s much less hassle.

Apparently, CloudFront does not support HSTS in any way. While that’s not perfect, it’s a reasonable compromise for everything else this AWS setup can do.

If your homepage works without index.html but your posts do not, see this stackoverflow post and use the S3 endpoint URL (HTTP only) to avoid issues.

Spress integration

You don’t have to upload your build manually to S3, the AWS deployment plugin will do that for you. The Github documentation should suffice for setting it up (thanks to the maintainer merging my v2 support and documentation updates already).

D8 PSA: Don’t use Multiversion (yet)

Wed, 07 Sep 2016 00:00:00 +0000

The deploy suite is an interesting suite of modules, which makes it possible to revision content in such a way that it becomes deployable across different sites or workspaces.

Also, it makes it possible to move the CRUD model Drupal runs under into a CRAP model, which anyone will appreciate who has ever gotten a call from a customer who just deleted a dozen nodes.

To make all that possible, basically all entities need adjustments to their storage base. That is not a trivial task and especially with complex contrib modules such as ECK, Field Collections, etc. the list of problematic and complex cases rises.

Looking at the current dev state and the last release as of this writing (alpha10), many of those complex cases are not yet supported. Field collections seems broken, content translation has problems with taxonomy terms and many other minor and major bugs are present.

Also, you should beware from using multiversion since not only is enabling it tricky (migrate all content to a temp storage and back), the same applies to uninstalling and it can leave you stranded with a broken system.

So, wait at least until the first beta is out.

Switching from Drupal to Spress

Sun, 28 Aug 2016 00:00:00 +0000

This site was powered by Drupal for a full decade. It went through Drupal 5, 6, 7 and 8, but it’s outlived its usefulness in that form. I think it might have even begun as a 4.7 site.

Why

My reasons for switching to a static site generator are quite plain but highlight my shifting needs:

Over 3/4 of posts here contain a code snippet, this is vastly easier to handle with Markdown or similar tools, than CKEditor.
The content is by all accounts static. The end-user is not rewarded through interactive features but rather burneded by the wait time of the CMS overhead. Even with the dramatically improved page cache in D8, it doesn’t come close to plain HTML.
I don’t have nontechnical users to support here.
I have enough other Drupal sites in my life to use for trying out new modules and experimenting.
I can stop maintaing a server to have things as flexible and speedy as I like, for a minor site.
And finally: All posts are revisioned in Git, without any intermediate systems.

How

My choice fell on Spress even though it’s not that prevalent (yet), since it uses the stack I’ve come to appreciate everywhere else: Symfony, Twig, Composer.

Converting data is straightfoward and easy. Just write a quick script to fetch all nodes and output their fields as desired.

League’s HTML-to-Markdown library easily converted CKEditor HTML with only minor corrections necessary afterwards.

Then just write it to a file with fopen/fwrite/fclose. Yes, of course that could be done more elegantly, but for a single migration it works fine.

Along the way I pruned the content back to what could still be considered tangentially relevant. I look forward to seeing if this will make posts here more frequent, as I expect, the opposite, or neither.

Continuous integration with Gitlab CI & Drupal

Sat, 06 Aug 2016 00:00:00 +0000

Today I want to highlight a solution that has become one of my favourite tools in web development over the summer: Gitlab with Gitlab CI.

I used to be a bit skeptical if a continuous integration solution would not be overkill for small projects but with configuration management in Drupal 8 and composer & composer-patches I’m a dedicated convert and would never want to go back to doing it by hand. You might think it’s just a git pull here or a cache-clear there but I’m certain that once you’ve tried it you don’t want to go back.

Gitlab setup

Fire up a VPN and install the omnibus edition, then follow the instructions. Just make sure that you have at least 1GB of RAM and several GB of swap. I tried 512MB, you will run into issues that are not worth your time and effort.

Next you’ll need to actually define a runner, again, see the documentation for that. For my case it works perfectly fine to have the runner be a second user on that server. Since I’m using SSH later I’ll need to provide that user with a key-pair, since we need to add that to the authorized_keys of the production server.

Project setup

First, you’ll need a .gitlab-ci.yml file in the repository root and that can be quite minimalistic. You can execute commands directly there but since most of what I’m doing for automated deployments is happening in an ssh session, I just wrap that in a deploy script:

.gitlabl-ci.yml

stages:
  - deploy

prod-deployment:
  stage: deploy
  script:
    -  ssh user@server.ccom 'bash -s' < prod-deployment.sh
  only:
    - master

prod-deployment.sh

#!/bin/bash
set -e # We want to fail at each command, to stop execution
cd /var/www/project/drupal/web
drush sql-dump --gzip > ../../somewhere-safe/db/`date +%Y-%m-%d-%H%M`.sql.gz
cd ..
git pull
~/bin/composer install
cd web
~/bin/drush updb -y
~/bin/drush cim -y
~/bin/drush cr

When you commit that CI file you will likely get a notification that the build is stuck. Just follow the prompts until you get to the list of runners so that you can enable it for the specific project.

Of course that could be done more elegantly in a variety of ways (maintenance mode, aliases instead of directories, etc.) but it actually works extremely well. I rarely have to touch the server anymore and adding additional steps such as unit testing and code standard validation is extremely easy with such a lightweight CI solution.

Wemos D1 Mini DHT HomeKit setup

Sat, 09 Jul 2016 00:00:00 +0000

The tutorial below is based primarily on work done by Aditya Tannu (see here and here) and tries to make things easier for beginners by reducing the setup to the bare essentials and providing clean and simple examples.

So, you might have gotten the iOS 10 beta and seen the Home app but don’t wan’t to spend an inordinate amount on commercial solutions and nowadays ESP8266 solutions are ubiquitous and cheap and those can be made to work with HomeKit.

You’ll need three things running to actually make that work: a bridge for your sensors, an MQTT broker and the sensor itself. While you can certainly use a Raspberry Pi for the first two I chose a Mac mini since I have that already running anyway. Sadly, that solution is Node-based but I haven’t seen any other solution that works out there.

The server

HAP (bridge)
- Clone the HAP-NodeJS repository
- Install with npm install (just Google if you don’t have node yet)
- Set a custom pincode in BridgedCore.js
- Run with DEBUG=\* node BridgedCore.js
Mosca (broker)
- Install with sudo npm install -g mosca bunyan (bunyan is only needed for output)
- Run with mosca -v | bunyan

You should now be able to add the_Node bridge_ in your HomeKit management application and see the fake sensors which are present in accessories. Once everything is running smoothly, you can run these as system services with an unprivileged user. See the project root for plist examples, just add your user name where necessary. I had some issues with calling node directly for HAP so I just made a one-liner bash script which then runs ’node BridgedCore.js’ in the relevant directory.

Problems

The pairing fails after a longish timeout. Your firewall settings on the server are the likely cause turn if off, if that helps, adjust as necessary.

The sensor

We can program the sensor with the NodeMCU stack but in my experience the Arduino one is far less volatile (for now).

Get the Arduino IDE
Add the board definition and ESP8266 package
Add pubsub and DHT libaries (the latter is also available in the IDE library manager)
Don’t forget to install the CH340 driver if you use a Wemos

You can now flash something onto the Wemos which will periodically send your data to the server. You can use my ino templates as a starting point. You still need set your Wifi SSID, password and the IP of the server you are sending data to.

Now that you are sending your results to the broker you still need the bridge to tell your HomeKit application about the accessory you wish to use and update its data when an update comes along. You can start with my accessory templates for humidity and temperature. You will only need to update those with the server IP, if you did not change anything else in the Arduino templates.

Problems

Uploading fails. Check the baud rate, the definition might not have set it to the recommended 115200.
I see the data in my serial monitor, the broker gets a connection but the values are still zero. Make sure that that the channel you are pushing to is the same as in the accessory file.
My accessory doesn’t show when I start the bridge. Make sure all files end in _accessory.js.

Caveats

Hardcoding the channels for each accessory in the js template and the Arduino sketch is inefficient at best. If someone has a good tutorial for HAP-Nodejs that does that right, let me know.

I’ve noticed that the default of 10 minutes until the next update is a bit much and the accessory can be shown as timed out or zero out. Also let me know if you have a good default here or recommendations on power optimisation.

Quick tip: Working with facets directly in D8

Thu, 02 Jun 2016 00:00:00 +0000

There are some cases where you want to interact with Facets on a Search API page, apart from just rendering them.

The API for facets in D8 provides you with services to make that easily possible, however, there aren’t many examples out there yet for cases such as these where you work outside the primary rendering pipeline. The trick is to process the facet not build it, to get at the actual data:

/** @var \Drupal\facets\FacetManager\DefaultFacetManager $manager */
$manager = Drupal::service('facets.manager');
/** @var \Drupal\Core\Entity\EntityStorageInterface $storage */
$storage = Drupal::service('entity_type.manager')
  ->getStorage('facets_facet');
/** @var \Drupal\facets\Entity\facet $facet */
$facet = $storage->load('my_facet');
$processed_facet = $manager->returnProcessedFacet($facet);

From here on out working with the facet is trivial and straightfoward, e.g.:

results = $processed_facet->getResults();

$topics = array();
foreach ($results as $result) {
  if ($result->isActive()) {
    $topics[] = $result->getRawValue();
  }
}

AwayTracker

Sun, 13 Mar 2016 00:00:00 +0000

AwayTracker is a small app for your Mac to display when you locked your screen, to more accurately track time for time management.

Support

While the app itself is provided as-is without any guarantees. Feel free to send feedback via email.

Getting Linux and OS X to play nice together: OmniFocus

Sun, 07 Feb 2016 00:00:00 +0000

Nowadays I primarily use OS X and rely on several commercial tools there to work effectively, specifically, 1Password, TextExpander, OmniFocus and PHPStorm. I still like to play with Linux from time to time though and being able to use it effectively is being able to access their data in similar tools there, where they are not available natively, such as PHPStorm.

Here’s how I created a working solution for me with an xmonad based desktop (since I don’t want to waste my life looking at Gnome & KDE apps), starting with the most difficult one, OmniFocus.

Interacting with OmniFocus

The bad news up front: Unless you are willing to invest significant time into reverse-engineering a full client, true read/write access is not possible. My goal was to be able to add items and read items in some form or another and that is possible.

Collecting actions

Collecting actions is actually fairly easy, just set up a Mail Drop adress and send yourself an email. A trivial script such as the following one makes it easy to send actions from the command line.

#!/bin/bash
echo -n "Your task: "
read title
echo -n "Optional description: "
read desc
echo $desc | mutt -s "$title" yourcustomaddress@sync.omnigroup.com 
echo "Task created"

You also have the option of extending the syntax of that mail, if you install an AppleScript extensions.

Accessing your existing data

Getting access to your OmniFocus database isn’t as easy, but it isn’t that difficult either. Generally, the data sits in an sqlite database on the client. So the first step is to have access to that data.

I use a LaunchAgent to periodically copy that database to a personal server. From there I can easily run a cron task such as the one below to fetch the database on Linux. I recommend always only working on a copy which isn’t written to by any client, to avoid conflicts.

*/10 * * * * rsync -u remote-server:~/OmniFocusDatabase2 ~

Now you have the database, so just write some SQL when you want to know what’s due. However, you also have option of using the PHP-based tool OFReader I wrote, which gives you access to the most common queries, such as outputting tasks due:

See the Github page for all the details on installing it.

Diving into Swift Part 2

Thu, 14 Jan 2016 00:00:00 +0000

This is the second post on Diving into Swift, if you haven’t read the first part, consider reading it for a general overview of the Xcode/Swift ecosystem.

Here are now the three areas in which I struggled the most and want to give advice to newcomers to Swift, who are used to scripted languages.

Optionals

An optional value either contains a value or contains nil to indicate that a value is missing. From “The Swift Programming Language”

Understanding optionals is critical to writing Swift code, and I can only echo what Touch Code Magazine describes as “the Swift code I see published on the Internet tends to rather randomly unwrap Optionals”. Xcode will complain if you haven’t marked an optional as such or force-unwrapped, so you can stumble towards runnable code with the help of the IDE. However, I highly recommend reading their article, to understand when you need to do which and why.

Especially understanding the if-let pattern is required if you have any complex interactions which depend on optionals.

Asynchronicity

In the PHP world you execute all as fast as you can and if something slows down your page load you do it in a batch job or via Ajax. Thread-what? That won’t work here for two reasons 1) don’t block the main thread, or your UI will become non-responsive and 2) the core libraries are all optimized for asynchronous operation.

A good example for this is fetching data from the web and responding to it. If you use NSURLSession in a function, this will background that operation. Should you use data from that asynchronous callback in an if-let-construct (i.e. change a variable in the function’s scope) and at the end of a function containing it return said data, then don’t be surprised if the breakpoint of the function return is hit before the network request has occured. This is fun to debug.

You could force this request to become synchronous with a semaphore but the better solution is figure out a way to react dynamically to the request once it comes back or put the network operation into a place before you need to depend on that data (Easily possible in my case, not always possible, of course.)

Understanding & debugging storyboards

While it is quite nice that a GUI editor makes drag-and-drop for creating screens possible, it also comes with many frustrations. You should keep the following in mind:

Base values are good for adding elements and connecting them.
Final values are required for the actual positioning.
Even though you can do portrait and landscape for iPhone and iPad in one app. Don’t (for now). It reduces unnecessary complexity, since each element on each screen variant is likely to require subtle different constraints.
Also, don’t assume that just because you removed an IBOAction or Outlet, that the link was dropped. You have to find the element in the storyboard and unlink them (via right-click) or you will get a build or runtime error.

I don’t actually have any good solutions or tutorials for working with storyboards and their constraints. My recommendation would be to use a layouting tool such as Sketch first, so that you can start cleanly in Xcode and only add what constraints you actually want to set and then add required missing constraints. Feel free to tweet me recommendations for better storyboard work though (@grahl).

Diving into Swift Part 1

Sun, 03 Jan 2016 00:00:00 +0000

While I mostly do PHP, I wanted to dive deeper into a more complex software stack and Swift was a good reason to do so: Get into desktop and app development with a modern, readable language and do things that not only live in a browser.

Turns out, the learning cliff is much steeper than in the usual web stack, at least if you want to build something which doesn’t fall neatly into the handful of online tutorial use-cases. Not to put too fine a point on it but a certain CMS with a tad higher learning curve than similar ones is extremely simple in comparison.

Getting started

You need a Mac, otherwise this is pointless. Then you need Xcode from the Mac App Store. Be prepared to shell out $100 each year, if you want to get your code on anything useful, since you’ll need to sign up for the developer program to get your builds signed.

Not essential but pretty nice is the integration OS X Server provides (another $20). If you have that running on, say, a Mac Mini, you can assign build Bots to automatically build and test your application, offloading some work from your local machine, on which you’ll be waiting to compile all the time during development anyway.

I began with a discounted legacy course from Bitfountain, which was fine in general, but I have not come across any material which blew me away, so consider researching and trying out a few budget alternatives before committing to a pricier course. I do think just reading the documentation is insufficent, unless you have a CS degree and a background in comparable languages and software stacks.

First impressions

In a nutshell: It’s challenging, but it’s fun.

Some things are a bit special, such as discovering how Storyboards are connected via drag-and-drop with the code. Tutorials are definitely helpful for things such as this. Nonetheless, it’s pretty straightforward to get a demo application up and running with the boilerplate code Xcode provides and start experimenting from there.

Consider doing the following if you also want to start with this:

Have a look into the official Swift book to discover some general principles behind the language and usage patterns.
Consider starting with an iOS application instead of OS X, in my experience the documentation is vastly greater for the former than the latter.
Have a look at Wenderlich’s tutorial, I’ve found them to be a bit better than the rest. StackOverflow is of course also often helpful / necessary.

Swift 1.2, 2.0, 2.1, …

I initially started experimenting with a version of Swift before 2.0 had come out. This was not perfect timing, since 2.0 broke a lot of things. While Xcode made a valiant attempt to update code which no longer complied with the spec (println is print, etc.), manual intervention is often necessary, especially in terms of error handling and unwrapping optionals (more on that in part 2).

Those changes by themselves are not dealbreakers but it makes googling a solution quite cumbersome. The majority of tutorials and how-tos use Xcode 6.2 or 6.3 (major changes already between those) and earlier Swift versions, which means it will look different and it won’t compile. Xcode does help you in fixing your non-compiling code but it isn’t really fun if you don’t have good examples to learn from and I have often tried three different solution from the web until I came close enough to something which worked after massaging it. Try your search with “$YOURISSUE swift 2”.

The reliance on tutorials / help from the IDE is also one of the reasons why I have not yet given AppCode a real try, before I am minimally capable in this language, though I’d love to use JetBrains better autocompletion support and other niceties.

Alternative uses

Finally, I’m hopeful that open sourcing Swift will bring the language further into other venues, though I haven’t tried it out on Linux, yet. Without good new cross-platform UI libraries I’m skeptical though, if that will ever catch on. However, you aren’t limited to running GUI applications on OS X and iOS, you can also use Swift to generate classic binaries akin to javac, as well as executing it as if it were a script.

In the next post I’ll delve deeper into the specific hurdles I faced when trying to begin building apps.

AppleScript: How long is my reading list?

Sun, 06 Dec 2015 00:00:00 +0000

If you are using Safari it’s easy and powerful to interact with your browser via AppleScript. For example, trying to figure out how many items there are in that reading list can be done as follows:

set tfile to (path to library folder from user domain as text) & "Safari:bookmarks.plist"
set counter to 0
tell application "System Events"
    repeat with i in (property list items of property list item "Children" of property list file tfile)
        tell i to try
            if value of property list item "Title" = "com.apple.ReadingList" then
                repeat with thisDict in (get value of property list item "Children")
                    set counter to counter + 1
                end repeat
                exit repeat
            end if
        end try
    end repeat
end tell
return counter

If you want to do more, such as extract the URL from that list or something similar, check out this thread from which I started.

Now running Drupal 8

Fri, 13 Nov 2015 00:00:00 +0000

Over the past year I have tried numerous times to upgrade my personal blog to Drupal 8, after it has been running for the last 9 years on Drupal 5, 6 and 7. Now the migration tasks are finally stable enough that migration is possible, after helpful people found the remaining issue for my case, which was file and picture assignment to nodes.

And you should absolutey update if you can. The system is not only so much cleaner it also solves long-standing issues in file and image handling, internationalization and much more, it’s a pleasure to use, even with several contrib modules still missing.

Getting started

If you want to try a migration, a good starting point is this Drupal.org page. This assumes that you already have a Drupal 8 site running and I highly recommend using the Composer template referenced in the excellent Lullabot article on the topic to do so, you can easily manager your core and contrib modules there already.

Here are my tips for you:

Don’t use the GUI for migrations, calling it via drush is faster and easier. You are likely to rerun your migration many times, until everything fits with migrate-upgrade, migrate-rollback, migrate-status and migrate-reset-status.
Note that if drush is telling you a migration is still running for something because an error was thrown, you still have to stop it. Rollback will not be enough, you need to reset the task as well.
Before you start migrating, create a clean database dump, so you can easily verify that your configuration isn’t at fault.
Be careful with disabling extraneous text formats, you might have content which was using it and getting that transferred is incredibly cumbersome (I migrated again, when I noticed that).

What’s missing

Contrib module are very much hit and miss with 8.0 RC 3, there are some which work perfectly and others where work hasn’t really started yet. Here’s what you can try to manage things:

Always google the module name with “drupal 8”, an issue is likely to exist and in many cases a third-party repository contains a newer state than drupal.org / composer.
Consider using cweagans/composer-patches to track patches you need to make a particular version run.
Write the port yourself.

Using Textexpander with Drupal

Sat, 24 Oct 2015 00:00:00 +0000

I’ve been using Textexpander for a while and recommend using a snippet tool very highly, especially if you have to deal with a significant amount of email, it’s a lifesaver. It’s also quite useful for development.

On the one hand you can save yourself time by creating shortcuts to heavily used commands. For most tools, such as Git, there are already existing collections out there but you can also easily and quickly create them with a few clicks yourself. Here’s one of my most used ones, which provides me with the cursor exactly where it needs to be to type in a commit message. (When I don’t want to auto-add, I just go to then end of the line and delete what’s there.)

git commit -m "%|" -a

One of my favorite ones is setting up a database. I do not want to deal with a third-party tool just to manage what amounts to creating a database a month and also not have to waste 2 minutes to get the syntax of this command right:

CREATE DATABASE %fill:database%; GRANT ALL PRIVILEGES ON %fill:database%.* TO '%fill:username%'@'%filltext:name=hostname:default=localhost%' IDENTIFIED BY '%fill:password%'; FLUSH PRIVILEGES;

Which then provides me with this nice dialog to focus on what’s important. It even automatically syncs the fields used twice (see the screenshot above).

Drupal development

When you are working on a Drupal project, the most automation often comes from abbreviating Drush even further and combining it with Git automation. Clear the cache, update, commit, etc.

For example, I use a snippet for “drush up –security-only -y” to update a repository, often calling that in several tabs at once for different projects. If it’s a contrib module, I have a wrapper script for automating this even further (I will provide that in a later post), however, core updates are seldom without minor reviews (did robots.txt or .htaccess change or override something, etc.). Thus, the following snippet does the stuff which should be uncritical, i.e. add files, remove versioning info, show me what’s left:

git add includes misc modules profiles scripts themes%key:enter%
rm CHANGELOG.txt COPYRIGHT.txt INSTALL.mysql.txt INSTALL.pgsql.txt INSTALL.sqlite.txt INSTALL.txt LICENSE.txt MAINTAINERS.txt README.txt UPGRADE.txt%key:enter%
git status%key:enter%

Textexpander is also quite useful, when you need to write PHPTemplate code. Since it’s inline PHP, even great IDE such as PHPStorm can’t autocomplete well and you are typing the same basic wrappers again and again. That’s not necessary. So, next time you want to print out a variable, a t() or maybe render an element, just use a snippet:

<?php print t('%|'); ?>

Should you have any problems with Textexpander behaving erratically in PHPStorm when you use cursor positioning or other modifications, have a look at this issue and set your clipboard retention to something such as 1.

Here are my Drush/Drupal snippets as of now.

Working with navbar and ember

Sun, 15 Mar 2015 00:00:00 +0000

On two sites, including this one, I have switched from Admin Menu combined with Adminimal Theme / Admin Menu to Navbar and Ember. The reasons for this should be pretty obvious:

Once I start using 7 and 8 sites simultaneously, having a very similar UI reduces mental load since I do not have to keep two ways of interacting in mind.
I can evaluate how robust and power-user friendly Navbar actually is and if it is likely to need a lot of tweaks such as Admin menu was needed for Drupal 7 to work efficently.

First impressions

Both modules are still rather beta and have their minor issues but overall it’s a decent look-and-feel with what’s possible in D7. The most annoying thing is not being able to create a node directly from the main menu, I have now actually started using shortcuts for this (which I previously just deactivated entirely).

So, as expected, it is a bit slower since the menu obviously relies on clicks but for me the responsive layout is already enough justification to relearn one or two muscle memories.

To use Devel you’ll also need the sandbox Navbar Devel plus my patch for it makes it look decent. Environment Indicator also works right away with Navbar, so the only thing really missing from admin_menu is the fast user switching and command search but since I hardly use those, I don’t have any major impediments currently.

I’m hoping some things like the floating settings block on the right might make it into Ember in the future but for now I have a good working alternative.

If you are in any way interested in Drupal 8 and are also maintaing any Drupal 7 sites, I highly recommend you check out “Drupal 8 Has All the Hotness, but So Can Drupal 7” as well as “Future-Proof your Drupal 7 site” for more ways to prepare for the future.

Getting Drupal's mail out of the spam folder

Mon, 02 Mar 2015 00:00:00 +0000

When your Drupal CMS sends emails, for example to deliver module update mails, password resets and so on, and they just won’t arrive as you’d like them to, you can safe yourself a lot of time by checking these items first. Gmail will give you hints where to look but it’s often unclear if you are actually on the right track.

#1 Is the email getting delivered?

Check the mail log of your server, typically /var/log/mail.log or /var/log/mail/something. This should tell you whether the mail gets delivered. If it doesn’t google the error message pertaining to your MTA and go from there.

If that email never arrives, not even in your spam folder, consider using another email account from a different provider to get a copy to start debugging or reduce -all to ~all in your SPF records to get a processed copy, though if you had that set, you will have to wait for the TTL set in your DNS record to time out.

#2 Did you look at the email?

The headers of the email often contain sufficient information to diagnose the problem. Had I done this more thoroughly, I would have found the culprit in no time. For example, there was this:

Received-SPF: none (google.com: useraccount@www.example.com does not designate permitted sender hosts) client-ip=2600:3c02::ffff:ffff:ffff:ffff;

This means, that Google cannot find my SPF record. This is odd, since I am publishing one. It turns out, that my application settings were incorrect and weren’t able to set the proper from address and thus localuser@www.example.com was used instead of myapplication@example.com.

This normally doesn’t happen with sendmail but you can change the behavior either with -f in the sendmail path in php.ini or via this solution.

I would not have run into this problem had myhostname not referenced the www subdomain but in this case I need it to also forward local mail without conflicting with local user accounts.

#3 Is your SPF record OK?

If you still have problems and are publishing an SPF record the headers again should tell you if your SPF record could be found and why it’s not being matched. Most likely, your IP address is missing, so verify with the SPF records checking tools.

Also, be on the lookout for IPv6, if you have it on your machine’s network, even if you are not serving an AAAA record, you need to include such an address, because you might be connecting to it via Google and they want to verify the address they receive content from, e.g.:

Received-SPF: pass (google.com: domain of myapplication@example.com designates 2600:3c02::ffff:ffff:ffff:ffff as permitted sender) client-ip= 2600:3c02::ffff:ffff:ffff:ffff; Authentication-Results: mx.google.com ... )

If that still doesn’t help I’d suggest moving on to burned offerings to the cloud gods.

Shutting down stgallenwiki.ch

Sun, 26 Oct 2014 00:00:00 +0000

Localwiki was one of the very early projects on Kickstarter and succeeded in their funding way back in 2010. Since then they have created several great model communities to show off how (hyper) local wikis can be a community enabler. The idea still seems so obvious to me, like Wikipedia is in hindsight, that I am still confused why the idea has not taken off exponentially.

Running a localwiki

Creating a localwiki was really easy with the documentation provided by the maintainers, it amounted to spinning up a virtual machine with Ubuntu on one’s preferred hosting plattform, adding one repository and installing one package, to get up and running. Which is just what I did under stgallenwiki.ch to get to know the platform itself (not having any Django knowledge) and to get to know my current city better.

In the beginning of 2014 the service providing free mapping tiles went away and individual localwikis could choose to either migrate to the centrally managed architecture of the core development team or to acquire a license key to continue receiving tiles for maps. Since I have not been able to find enough people in St. Gallen to work on this project and am alone not motivated enough to work on it further I have now taken the site down.

I am writing this post to document this canceled project and provide an archive of the few pages which were created should anyone care to try something similar in the future.

First look at CentOS 7

Sun, 07 Sep 2014 00:00:00 +0000

In July, a month after the release of RHEL7, the CentOS team had released their clone of the 7 series. Time to take a look, since this will likely be the standard version for new projects I will be building, at the very latest by the middle of next year when a minor release has further stabilized the series.

New and noteworthy

The installer now aligns with what we have become familiar with based on the Gnome 3 desktop in Fedora, which is overall a good experience.
XFS is now the default root file-system including boot. That decision seems fairly bizarre when EXT4 is the default everywhere else and has an upstream path towards Btrfs.
Systemd is now in full effect, so I will have to take some time to finally learn the startup script changes properly.
Network interfaces are now carrying “Predictable Network Interface names”. While that’s a nice idea it also means that old common tricks such as “dhclient eth0” no longer work. Side note: Minimal install does not even contain ifconfig, so find your interfaces in /etc/sysconfig/network-scripts/ifcfg*, if you went with minimal.
PHP 5.4.16, this is the part I’m most excited about, since this was the primary reason in my work to install third-party repositories such as IUS to get above ancient 5.3 versions present in the RHEL6 series.
You will look for the package mysql-server in vain, MySQL is now to be found via the fork mariadb-server.

Three lessons from amateur PHP code

Sun, 11 May 2014 00:00:00 +0000

Photo CC-BY by tnarik 

I did not get a CS degree and thus learned web development as a self-taught amateur. While that was fun, it also meant that in the process of building things without a curriculum or a mentor to provide a foundation I wrote pretty bad code in the beginning.

I have dug up the PHP code for a site which has been superseded with a Drupal site since 2009 and it’s a great way to show failure by example without anyone getting hurt.

The following shows common mistakes made by amateur PHP programmers and a few pointers on how to fix them and become a better developer.

Mistake #1: Repeating yourself

Every time you see redundant code it should give you pause. Even a decade ago I recognized this and tried to move the page header into a separate file. Here is the start of item.php and dozens of other files like it:

<?php
$pagetitle="Item Details";
include "main.php";
echo "<div id=\"content\">";
// [...]

Viewing an item would thus be the URL /item.php?id=123. This structure exemplifies that the author has not thought enough about inheritance and overall structure. Furthermore, we can see in main.php that the file is just dumping content to the output like the all other PHP files in that application:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<?php
include "settings.php";

It’s clear that the author has not even fully internalized procedural programming and is using includes simply to reduce redundancy in a static, linear script.

Fixing the mistakes

Point all requests to index.php as the sole entry point and use parameters to call the respective logic. Not only will you reduce errors but you can also centralize parameter checking and sanitization, as well as authentication and authorization. You can also easily add nice, readable URLs with a one-liner in htaccess.

Convert the static pages such as item.php into functions. Which is basically a given, if you restructure everything around your index.php as recommended above. In this case if we had all item.php logic wrapped in a function (or more likely a collection of them), we could call it where required through our routing logic and not attempt to build a linear script for each page. Ideally, you would follow established patterns like MVC for your program structure when doing so.

**Don’t solve this yourself. **Seriously consider if your homebrew application should continue to exist. The above points explain how one would fix something like it if you cannot ditch it but consider carefully if a framework such as Symfony2 or a CMS like Drupal is not the better solution to start from scratch, since they provide all that essential work of handling I/O, routing pages, abstracting templating, etc. for you, if you invest the time to learn how these systems work.

Mistake #2: Mixing logic, data and display

Here is an additional excerpt from the item.php shown above:

if ($row[isbn] != NULL)
echo "<tr><td>ISBN/ISSN #:</td><td> $row[isbn] </td></tr>";
echo "<tr><td>Section:</td><td> $row[section]</td></tr>";
// [...] 20 more <tr> like that
$commentquery = mysql_query("SELECT comment FROM comments WHERE item='$row[item_id]'");
if (!mysql_num_rows($commentquery)==0)
echo "<br /><br />Comments by members about this item:";

while($row = mysql_fetch_array( $commentquery )) {
    echo "<div id=\"comdiv\">";
    echo nl2br($row['comment']);
    echo "</div>";
}

Thankfully, the query parameter was properly sanitized, so it is not a case of little Bobby Tables but it is a Smörgåsbord of bad patterns. Not only is redundancy a concern but the indiscriminate mix of data, html markup and query logic shows missing levels of abstraction. The weird string double escaping we’ll just ignore.

Fixing the mistakes

Use PDO and prepared statements. If you write your SQL by hand and work with user input, you are asking for trouble. PDO tutorials abound.

Functions, functions, functions. The author should have seen the redundancy in his or her code and concluded that a helper function should deal with creating the table (let’s ignore whether a table is the right solution here). The block on comments could equally as well be encapsulated in a function. However, that function alone would not add much readability (at least as long as it is as short as it is now), unless one takes out the markup, which the following recommendation encompasses.

Create templates. Even if you do not follow the general advice from mistake #1 on using an established base to build upon, you can improve legacy code by stripping out the markup into templates. Even if you don’t want to fully commit to Symfony, you could use their Twig engine to do so and achieve much more readable and maintainable code. Again, consider not reinventing the wheel.

Mistake #3: Sloppy code conventions

if (!mysql_num_rows($commentquery)==0)
echo "<br /><br />Comments by members about this item:";

The line above we already saw in mistake #2. The problem here is that through missing brackets, spaces, and indentations the code becomes significantly harder to read. It’s easy to overlook the scope of the if-loop and for example incorrectly assume that the while-loop following it is still in it’s scope. The many formatting problems in the code were mainly due to the fact that I didn’t use software to help me stick to a standard, since I wrote it all by hand.

Fixing the mistakes

Use an IDE. A text editor with syntax highlighting is just not enough, it’s too easy to make mistakes. Your environment, whichever you decide upon, should do auto-formatting and code checking as-you-type. I use PHPStorm but the same can be achieved with free tools such as Eclipse.

Follow PSR-1/2. If you have until now followed your personal preferences, just get over yourself and standardize to PSR-2, that way all your code and all modern libraries look the same. There are also tools to help you convert your existing code.

All of the above will not make you into a great developer but I hope to have provided helpful advice to others that I wish I had been given at the time.

Goodbye comments

Sat, 03 May 2014 00:00:00 +0000

Photo CC-BY by arndog 

Today I removed this blog’s comment functionality after having it for over seven years. I do not host debates and do not have content which regularly inspires comments apart from “thanks” so I don’t think that this will in any way diminish this blog. I’m sure if someone actually needs help or has information they want to give me, they will find the about page.

And on the other hand, comment spam is simply out of control. At first I had Drupal’s image captcha which worked well for a while but had to be exchanged for Recaptcha but when that let too many spam posts through I switched to Mollom and they have provided an excellent service but still let about one false negative through every other day for the 500+ attempts every day.

So, to follow the spirit of @AvoidComments, no more more comments here.

Using a Kindle as a monitoring tool (or not)

Sun, 20 Apr 2014 00:00:00 +0000

I had an old Kindle 3 lying around and really wanted to do something fun with the e-ink display in it, especially since its Linux underneath anyway.

If you browse around the web you’ll find plenty of information to get you started with the prerequisite steps of rooting the device, enabling network and ssh. Since a lot of the information is hidden in forums (ugh), I’ll point to Liraz Siri’s tutorial which covers all important points.

Developing a kindlet

Unless you want to directly work on the framebuffer you will want to create a java application which the Kindle can run, called a kindlet. I strongly suggest you follow David Given’s guide. Getting his Hello World to run consistently is a good starting point for everyone like me who only dabbles with Java. Additionally, the MobileRead wiki has additional information.

Since developing a kindlet without any resources from Amazon basically amounts to compiling, copying, opening and repeating that process it’s essential to automate that process, thus I recommend ammending Given’s excellent makekindlet script with the following two lines (for OS X):

[...]
jarsigner -keystore $KEYSTORE -storepass password $JAR dn$USER

cp $JAR /Volumes/Kindle/documents/
diskutil eject /Volumes/Kindle

Network communication in a Kindlet

To do anything useful in a Kindlet apparently requires one to have the ability to work with sockets, so the security settings need to be adjusted. A blog post by Alex Hutter explains how, i.e. add the following in external.policy:

grant signedBy "Kindlet" {
 permission java.net.SocketPermission "*:443-", "accept,connect,listen,resolve";
 permission java.net.SocketPermission "*:80-", "accept,connect,listen,resolve";

If you are stuck at such a point it’s often helpful to visit the crash.log in developer/APPNAME/work to point you in the right direction of the problem.

Rendering content

Now, the rest should be easy, right? Well, rendering local images worked fine with this example snippet, however any permutation of loading network resources into images went exactly nowhere, getWidth() returns 0 and no image can be rendered and that’s where I’m currently stuck and discontinuing this project.

Summary

The Kindle is NOT intended as a general use computing device.

The quote above is from Hutter’s blog post, I don’t have much to add to that. This was a fun project but it became clear that without getting to know Java a lot better this would not be worth the effort, compared to solving the problem with an Android app, no matter how nice the display is.

Track when you lock your desktop

Sun, 02 Feb 2014 00:00:00 +0000

Photo CC-BY by way2go 

If you do any sort of work where you have to report your activities in 15-minute increments, you have probably experienced that it is very easy to miss things, especially if you often have to deal with interruptions.

There are numerous solutions to parts of this problem such as apps which continuously poke you to update a tracker, to ones which simply log everything and sum it up. I use some of them but I also add another check to my routine to have more accurate timing information when I wasn’t paying attention. What I’m doing is writing timestamps to a file which register screen locks/unlocks as well as suspend and it has in the past year helped me to significantly become more accurate in my tracking by having reference timestamps such as the following excerpt:

Thu Jan 23 09:53:48 CET 2014: SCREEN_UNLOCKED
Thu Jan 23 10:02:21 CET 2014: SCREEN_LOCKED
Thu Jan 23 10:18:16 CET 2014: SCREEN_UNLOCKED
Thu Jan 23 10:27:02 CET 2014: SCREEN_LOCKED

If you want to do this, too, just follow one of the examples below. If you want to do something similar and customize it, the problem is rather trivial if you break it down into a trigger and an action. The first is heavily dependent on your environment, the latter could be anything.

Linux/GNOME

To react to events you’ll have to run a command at login which then continuously monitors DBUS to see if the relevant event has fired. Here is a working example:

$ #!/bin/bash

function write_log {
  if [ -z $1 ]; then
    1="unspecified"
  fi
  echo -e "$1\t$(date)" >> "/home/youruser/time.log"
}

write_log "LOGGED_IN"

# Monitor gnome for screen locking. Log these events.
dbus-monitor --session "type='signal',interface='org.gnome.ScreenSaver'" | \
  (
    while true; do
      read X;
      if echo $X | grep "boolean true" &> /dev/null; then
        write_log "SCREEN_LOCKED"
      elif echo $X | grep "boolean false" &> /dev/null; then
        write_log "SCREEN_UNLOCKED"
      fi
    done
  )

OS X

The only decent trigger method I found (albeit after only 15 minutes of googling) is the paid app Scenario. The app itself should be self-explanatory and allows you to put one or several scripts into the respective folders for sleep, lock, log out, etc. Since I’m much more familiar with shell scripting than Applescript, I’m just using the later to call the former. Obviously, this could be simplified.

Applescript in Scenario:

do shell script "~/.bin/screenlock.sh unlock"

Shell script:

#!/bin/bash

if [ $1 = "lock" ]
then
  echo -e "`date`: SCREEN_LOCKED" >> ~/time.log
elif [ "$1" = "unlock" ]
then
  echo -e "`date`: SCREEN_UNLOCKED" >> ~/time.log
else
  echo -e "`date`: UNKNOWN_ACTION" >> ~/time.log
fi

Running Drupal on a proper PHP version with IUS

Sat, 23 Nov 2013 00:00:00 +0000

I recently read Thijs Feryn post on the maturity of PHP and he makes a convincing argument for actually using what PHP provides with 5.5 now being stable.

Drupal 7 has seen many improvements over the past year in regards to 5.4 compatibility and as it turns out, it now runs pretty smoothly, though many modules still create E_NOTICEs due to legacy code. The breakage which 5.4 caused in many places when Drupal 7 (and let’s not even talk about a 6 site) came out is not comparable with upgrading from 5.4 to 5.5, which is mostly harmless.

Managing PHP versions

You have basically two options for upgrading PHP on your server: find some pre-compiled packages or build from source. The latter should be avoided by nearly everyone, unless you’re willing to diligently subscribe to a release mailing list and compile again and again.

On CentOS/RHEL, which is my preferred hosting platform at the moment one can rely on the excellent IUS repository. It makes switching between 5.3, 5.4 and 5.5 extremely easy. Once you have IUS installed, you can select your packages via the name php5Xu. Which means you can switch between 5.3 and 5.5 simply by removing the packages installed from php53u and installing again with php55u. All I needed afterwards to get PHP running again was copying the /etc/php-fpm.d/www. conf.rpmsave over www. conf and so 15 minutes later this blog is running on 5.5:

# yum list installed | grep php
php55u-cli.x86_64 5.5.5-2.ius.centos6 @ius
php55u-common.x86_64 5.5.5-2.ius.centos6 @ius
php55u-fpm.x86_64 5.5.5-2.ius.centos6 @ius
php55u-gd.x86_64 5.5.5-2.ius.centos6 @ius
...

An additional benefit is that IUS provides packages in sync with the latest released package. This is very helpful in determining what you are actually running. For example, your server might report that you are running 5.3.3 (as stock CentOS 6.4 does), even if critical fixes have been backported by RedHat, Fedora, et al from later versions, with IUS you’ll get a proper 5.3.27.

Caveat: Of course, don’t try this on production if you don’t know what you’re doing.

A commercial marketing database lets you peak inside

Sun, 06 Oct 2013 00:00:00 +0000

Array

There is an old entry in my backlog of ideas for the blog which went along the lines of try to get a copy of what data mining marketing firms have on file for you. Nothing ever came of that until I saw Ed Felten’s post on one of those companies doing just that, with a spiffy site to boot, and I’m really surprised about this.

A quick primer

Most everyone knows that companies as a whole are tracking purchasing habits and customer preferences and correlating them with socio-demographic information to sell you more stuff. The who and what is much less widely know though.

Most often these collection companies are specialized businesses, which sell subscriptions to their databases, which they feed from multiple sources, especially in the U.S. private as well as public sources. At the time I jotted down a few notes on the topic ChoicePoint was one of them.

Most of these companies do not simply stop at providing information on marketing data to business and government (yes, the U.S. government is not prohibited from using private databases, even if they are not allowed to create those on citizens in the first place). They often have a close relationship to data based on or provided by credit scoring agencies, such as Altegrity, TransUnion or ISU; which are listed as direct competitors of ChoicePoint by Hoovers. The differences in their aggregate datasets are probably small, the impact of the result on individuals often not. ChoicePoint was later acquired by LexisNexis, which primarily focuses on publication databases, and some divisions by Acxiom. It shows the further agglomeration of databases of different types within these data brokers.

Why aboutthedata.com?

If I had been asked a week ago, I would have concluded that such data brokers have no interest in providing a site such as aboutthedata.com. It seems intuitive that such brokers would want to collect as much data as possible on their subjects without giving them reason to share less data with them. Ideally, to not be noticed at all.

This could be an initiative of Acxiom to improve the public perception of data mining firms such as themselves. It could be an attempt to preempt any negative coverage on data brokers due to the continuing public discourse on surveillance et al. Or it could just be an attempt to further improve their data.

Let’s try it out

Since I did spent several years in the U.S. I figured that they should have at least something on me. I entered my personal data from my last residence and was surprised to find that in the categories shown above they had basically nothing, except for an inferred marital status and income bracket, which matched, but that would have been possible to extrapolate from the address and age itself I entered to register. Ed Felten was more successful but also apparently underwhelmed by the level of detail shown.

My speculations on why this is so fall into two basic groups:

1. They are only showing a minimal set

It’s possible that Acxiom is only providing their basic result data and keeping any other further (and more creepy) analysis results to themselves and their clients, but since that’s pure speculation I’m going to ignore this avenue for now.

Also, I have become in general skeptical of the predictive power of big data in shaping or determining customer behavior, which is in my opinion often at best on par with a trained sales representative.

2. I didn’t provide enough data

Two of the primary categories provided by Acxiom are basically public databases with vehicle and house ownership, neither applied to me at the time. Also, I generally did not sign up to loyalty cards. It’s possible and likely that Acxiom then either really does not have much more information on myself or that they are unable to merge incomplete datasets on my person into a consistent record.

The latter case highlights the main problem with abouthedata.com: Basically, I can be pretty certain that I’m not seeing the full list of entries Acxiom has on me, simply due to the fact that a human operator would have to make a judgment call, whether a fragment refers to the same canonical person or not. They are unlikely to be able to manually merge a significant number of entries, which would mean that there are have to be significant numbers of entries in their database which they cannot bring together in this web application by algorithm alone. They cannot ask me “is this you as well” without accidentally disclosing data from someone else in many cases, they have to err on the side of caution, more so than is probably necessary for most of their customers.

Thus, I’m still left wondering what the site is supposed to accomplish. Calm me? Improve the paper spam selection? Not sure.

Bulk create searchable PDF from paper documents in Linux

Sun, 29 Sep 2013 00:00:00 +0000

For some documents you have to retain the original in dead tree storage format. For most documents which arrive in the mail, however, a digital copy is just fine and there really isn’t any need to retain the paper version, especially if your computer can store millions of them in the space needed for one paper binder.

To archive such documents one could now buy a scanner, maybe even an ADF office appliance which spits out a searchable PDF and all that and let the device collect dust for the next few months until another batch is processed. However, you can also achieve nearly the same result with your phone and a Linux desktop.

Step 1: Capture

Firstly you will need to actually digitize your documents, you can of course use any scanner for this but a phone can be the perfect device to quickly capture dozens of documents, often vastly faster than with a flatbed scanner optimized for photos.

I personally am using the Scanbox to do this but any contraption which can hold your phone or digital camera steady such as a tripod mounted at similar distance to the document should do the trick. Speed is the important factor in my solution, not accurately capturing 6pt legalese in the footer.

Step 2: Format

After capturing you might need to rotate your photos first to get the page into portrait mode. Watch out though that the generic image preview in Gnome might rotate on-the-fly from EXIF data and not tell you. If you were to OCR those files, you would not get any text from it. You can check if your files still need to be rotated by opening them in GIMP, which will ask you if you want to rotate. You can bulk-rotate according to the camera setting with:

mogrify -auto-orient *.jpg

If your camera orientation did not match your document orientation, you’ll have to convert by hand, the latter will do so 90 degress clockwise:

mogrify -rotate 90 *.jpg

Step 3: Process

Now you are ready to convert your images to PDF with text in them, basically, all you need to do is to call hocr2pdf and tesseract, the rest of the script below is only concerned with naming things and cleaning up. Thus the packages tesseract-ocr-eng, imagemagick and exactimage should be all that’s needed on Debian-based systems, it worked flawlessly for myself with Ubuntu 13.04. Essentially, it’s a cruder version of Konrad Voelkel’s solution.

#!/bin/bash

for f in *.jpg
do
    localname=$(basename "$f")
    filename="${localname%.*}"

    tesseract $f $filename -l eng hocr
    hocr2pdf -i $filename.jpg -s -o $filename.pdf < $filename.html

    rm $filename.html
    # I wouldn't do this, but you could...
    # rm $filename.jpg
done

Et voilà, you have a searchable PDF which you can locate with the desktop search of your choice, for example Recoll.

Exporting Remember the Milk data to CSV

Sat, 22 Jun 2013 00:00:00 +0000

Photo CC-BY by johanl  

Remember the Milk, RTM for short, is a popular web-based todo list manager. You can access your tasks in your browser, through an app and they provide a print template. Exporting the actual data is a bit more cumbersome: They consider iCalender to be the primary backup mechanism and the only other thing they provide is an Atom feed but the content is a bunch of tags. Nothing you’d want to parse.

However, they also provide an API and David Waring has made use of it with his rtm-cli python script. With very little effort I was able to amend his script to include a function to create a csv file for all general RTM fields without notes. Since it’s based on his ls function, you can use rtm-cli’s general filtering functions. If you wanted to export all unfinished tasks you would execute the following:

$ python rtm -c csv

It would create a UTF-8 file output.csv in your working directory with a structure as such:

ID;List;Title;Completed;Priority;Due date;Tags;Link;
123;"Inbox";"Error reporting broken";;N;;;"http://www.example.com";
124;"Inbox";"Login problem";;N;2013-06-14T00:00:00+02:00;"mysql,apache";;
125;"Inbox";"Add web service";;N;2013-07-21T00:00:00+02:00;;;
126;"Inbox";"Fix IE8";;N;;"ie";"http://www.example.com";

You can get the code through a forked repository. If this exporter worked for you, you could let the maintainer of rtm-cli know by choosing Approve in the pull request

2013-06-24 Update: The maintainer has accepted the pull request.

2013-07-09 Update: If you encouter problems like Tyler with a missing rtm module, make sure you have pyrtm installed (e.g. pip install pyrtm)

Let me know if notes is something you’d need to export, too.

Load testing a Drupal site with Locust

Sun, 09 Jun 2013 00:00:00 +0000

What is Locust?

Locust is an exciting new framework to do load testing on a site. Test scenarios are written in Python and are easily customizable. I used it to create benchmarks for a performance comparison of several cloud hosting providers.

There are several alternatives to Locust, such as the humble apache benchmark (ab), Siege, Proxysniffer and many others. Though, in my opinion, none of those offer the ability to realistically mimic site specific usage patterns, while still being incredibly easy to set up, providing a great interface for using it, and being open source.

It also offers features such as distributed load testing through a master/client setup as well as “ramping up” load testing to determine stability limitations, though I did not make use of those features.

Writing scripts

Since test scenarios are basically just python functions which are extending Locust’s base functions, one has to first outline the tasks a simulated user should make.

There is nothing wrong with simply specifying a list of urls (possibly parsing a sitemap.xml) and then relying on Locust to let requests randomly hit those URLs within predefined, randomized intervals. Based on the base example on Locust’s front page, the following works for a sitemap generated by Drupal’s XML Sitemap module:

#!/usr/bin/python

import random

from locust import Locust, TaskSet, task
from pyquery import PyQuery

class RandomSitemapWalk(TaskSet):
    def on_start(self):
        r = self.client.get("/sitemap.xml?page=1")
        pq = PyQuery(r.content, parser='html')
        self.sitemap_links = []
        for loc in pq.find('loc'):
            self.sitemap_links.append(PyQuery(loc).text())

    @task(10)
    def load_page(self):
        url = random.choice(self.sitemap_links)
        r = self.client.get(url)

class AwesomeUser(Locust):
    task_set = RandomSitemapWalk
    host = "http://www.example.com"
    
    min_wait = 1  * 1000
    max_wait = 10 * 1000

Now you can start Locust with that, set the number of users you want to be spawned and start testing your application.

Those tasks by themselves aren’t that special, one could as easily parse the sitemap with awk/sed, pass it to Sieve and get a similar results. As the example on their front page shows, procedures such as logging in are trivial and there we can build complex workflows.

Profiling logged in users

Most of the time, anonymous requests aren’t really the optimization problem, at least if your node-to-hit ratio isn’t so low that Varnish isn’t already serving those users quickly. So, to simulate logged in users (as their primary example shows), all that’s needed is to POST to the login form, adapted to Drupal 6 this would be done as follows:

def on_start(self):
    self.client.post("/user/login", {
        "name": "loadtest@example.com",
        "pass": "loadtest",
        "form_id": "user_login",
        "op": "Log in"
    })

From there on we could either reuse our sitemap and let users randomly hit pages or do something with their second base tutorial “Example with HTML parsing” and start responding to the content actually delivered to a logged in user. The following would log in, read the front page, index all links, and choose a target at random. By itself that is rather limited compared to the sitemap set but it should be trivial to work out how to extend this to walk through a varying number of link depths:

class WalkPages(TaskSet):
    def on_start(self):
        self.client.post("/user/login", {
            "name": "loadtest@example.com",
            "pass": "loadtest",
            "form_id": "user_login",
            "op": "Log in"
        })
        # assume all users arrive at the index page
        self.index_page()
    

    @task(10)
    def index_page(self):
        r = self.client.get("/")
        pq = PyQuery(r.content)
        link_elements = pq("a")
        self.urls_on_current_page = []
        for l in link_elements:
          if "href" in l.attrib:
            self.urls_on_current_page.append(l.attrib["href"])

    @task(30)
    def load_page(self):
        url = random.choice(self.urls_on_current_page)
        r = self.client.get(url)

You can now download the results of these tests as CSV and analyze in depth on the basis of average page performance or fulfillment rate per percentage range per page, or simply store those results for future comparison.

Where to go from here

First of all, I think that defining use cases for a web project at the outset and defining a load testing script along those expected behaviours is a great step in modelling a platform before launch. Furthermore, it makes it easier to evaluate how additional features might impact a live site and its users before deployment. So, just writing a task list and then using Locust’s task priorities would be a great way to more closely emulate realistic user navigation paths.

What these examples don’t do is act on the Javascript on the pages retrieved. I was considering to attempt this with the pyv8 wrapper but haven’t gotten around to it yet.

Cloud performance compared with New Relic

Sat, 01 Jun 2013 00:00:00 +0000

Recently I ran into some performance problems with a moderately complex Drupal 6 site running on EC2. New Relic reported an average application time around 1000-1200ms, clearly too high. Since the site was already rather optimized with MySQL being tuned, using Nginx and Varnish, as well as APC and Memcache I started to suspect the instance itself.

I came across an interesting post from 2009 which did seem to indicate that processing performance, especially in regards to I/O, varied widely across cloud hosting providers. This led me to switch to Linode with that site but I made some tests along the way to show how instances differ.

Test setup

To make relevant comparisons between the individual offerings I created a Locust script (more on that in a later post) to log into the site and crawl a random but limited set of pages. The test ran with an average of 10 concurrent users for around half an hour each.

Disclaimer

The tests wer run on a single instance, they are thus not statistically sound and your mileage wiIl vary; though I do think that the comparisons give valuable insight.
The EC2 instances were all run from the same EBS volume in US East and should be directly comparable, for Linode the same OS and setup was undertaken but details do differ.

Results

My laptop was used to establish a baseline performance with real hardware, which came in at 272ms. To get below this it would be necessary to look at individual features and see if they can optimized or removed, which isn’t really an option for me.

The EC2 micro instance was basically unsable and ramped up to an application time of 4030ms.

The small instance, which was also used in production, performed similarly to the usage I saw on the live site and clocked in at 1180ms.

When switching to an EC2 medium instance, performance was finally at a decent average with 533ms.

In contrast, the entry-level from Linode “Linode 1GB” averaged 841ms.

Comparison

The conclusion I drew from this is that the performance-to-price ratio is too low for using EC2 when running such a Drupal site with a single host and hosting costs are an issue, such as for non-profit organzisations or private sites.

These tests are relevant for quickly delivering a site with a predictable and limited usage pattern: speed, not scalibility. If I had a highly fluctuating user load (e.g. mass media events) the situation might be quite different. In those cases, launching a few more instances and paying spot prices for medium or larger instances for the time needed is probably still cheaper than paying a monthly fee for those instances at a hoster like Linode.

I do want to highlight that this is highly dependent on your application and its users. For example, I’m running a Django-based site on a micro instance in EC2 Ireland and it comes in at an impressive 92ms at a very good price:

TinyPrinter in a box

Sun, 12 May 2013 00:00:00 +0000

After coming across a great tutorial on pikiosk on how to interface a thermal printer with a Raspberry Pi I knew that I had try building one of those. I wasn’t as interested in printing graphics rather than making it easily accessible, so I chose to use their code as a basic service for printing text and doing everything else in a small web frontend; which gave me an opportunity to delve into Symfony 2.

Assembly

With the tutorial above, connecting the Pi via GPIO was a piece of cake and most of the time consisted of finding a stable pin connection without soldering it into place (male pin header casing without pin worked very will together with breadboard cabling). The Pi’s GPIO connector is really the key element for its usefulness and fills a great niche there. Without a full Linux doing anything non-trivial with networking isn’t really fun on. This is especially true for Arduino and Android is still to cumbersome as an intermediary there (but that’s a different post).

The casing itself is a pretty durable spare tupperware container I had lying around, a bit too durable to work smoothly with it. Should I work on another enclosure I’ll likely choose a thinner plastic or similar material. Nonetheless, a Dremel with a cut-off wheel and a drill was sufficient to cut a square hole in the top and a hole to feed the power cabling through. A spare wifi adapter provides connectivity.

Symfony 2

I really enjoyed working with the structure and tools that Symfony provides and how the documentation makes it easy to create a clean and structured project from scratch. The web debug toolbar (see here) in particular is a nice tool to efficiently debug your application. Having some of those elements such as routing and Twig in Drupal is going to be a great advancement.

The only thing which was annyoing here was the template inheritance structure, the documentation could be a bit more detailed on this topic. In particular how and where Twig uses extend to inherit elements and which are automatically aggregated, which are manually deployed and so on.

Making printouts with TinyPrinter is surprisingly addictive. The code is on Github.

Quick ad-hoc file transfer with IPv6

Sun, 28 Apr 2013 00:00:00 +0000

Nowadays I rarely use wired networking at home. However, when you want to transfer 50GB or more, the round-trip over the wireless router can take several hours to half a day.

The easiest solution is of course to attach two ethernet cables to your home router and be done with it. Depending on your setup, though, that might be cumbersome with a desktop in another room or not having 10m or more of ethernet cabling available. So, in those instances it makes sense to dust off a crossover cable or a switch to connect the devices together.

In IPv4-land one would now have to configure a local network with static addresses in the 192.168.0.0/16 or 10.0.0.0/8 range. With IPv6 enabled devices, however, such an address is already present in the form of a link-local address. The following example shows how to discover devices on the local network with a broadcast ping and then how to connect as usual via interface em1:

$ ping6 -I em1 ff02::1
PING ff02::1(ff02::1) from fe80::f2de:f1ff:f1ff:f1ff em1: 56 data bytes
64 bytes from fe80::f2de:f1ff:f1ff:f1ff: icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from fe80::250:8dff:8dff:8dff: icmp_seq=1 ttl=64 time=0.393 ms (DUP!)
$ scp myfile user@[fe80::250:8dff:8dff:8dff%em1]:/home/user/target

Drupal quick tip: Ignore custom modules when checking for updates

Tue, 22 Jan 2013 00:00:00 +0000

When you are doing Drupal development and have complex deployment scenarios (I blogged about that on the Namics blog [German]), it’s not unusual to have several modules containing custom functionality, as well as several Features modules. In a large site this can quickly become a dozen or more, depending on the aggregation and splitting of various functionality across modules. All of those Features will be checked for updates, every time the Update status module is run and will not find any releases for it.

Now that’s annoying and helpful Stack Overflow/Drupal Answers people have already found the hook you need to fix it. Of course there is a hook for that. The code snippet taken from there is reproduced below. I amended it to also contain a second hook, which then also excludes these modules from being checked when translations are checked by the Localization update module.

function mymodule_update_projects_alter(&$projects) {
  $blacklist = array(
    'collection_feature',
    //etc.
  );

  foreach ($blacklist as $module) {
    unset($projects[$module]);
  }
}

function mymodule_l10n_update_projects_alter(&$projects) {
  mymodule_update_projects_alter($projects);
}

Quick tip: Fixing common name / server name warning messages

Thu, 01 Nov 2012 00:00:00 +0000

When you configure Apache httpd to server SSL you’ll most likely define a VirtualHost, give it a ServerName, possibly a ServerAlias and various other settings. If you did everything correctly, the site will come up with the familiar green/gold/blue https markers, instead of a big red warning page. If you got to this point but you are still seeing the following error message in your logs, most likely, you have flipped your ServerName and the certificate’s alternative name.

[warn] RSA server certificate CommonName (CN) `example.com' does NOT match server name!?

At least this turned out to be the mistake I made when configuring one server with a certificate which had example.com as the common name and www.example.com as the alternative name. You can find what your certificate delivers (in Chrome) under “Certificate Fields”, “Extensions”, “Certificate Subject Alternative Name”. Your VirtualHost has to match the certificate’s preference to not throw the warning. So simply change the former to the latter, even if you are primarily (or only) serving from www.example.com.

Logs an error in my case:

ServerName www.example.com
ServerAlias example.com

Does not log an error:

ServerName example.com
ServerAlias www.example.com

Migrating Firefox's password storage to Loxodo

Fri, 06 Apr 2012 00:00:00 +0000

Photo CC-BY by myklroventine

Passwords are inevitable. At least for the next few years. So it makes sense to find a reasonably secure but efficient way of managing them.

For the last few years, I’ve used a combination of CLI password generator, Firefox’s password manager with a master password, a manually encrypted reference and my memory. This was an adequate solution but has certain limitations in terms of dependence on a specific browser profile and manual workarounds when a specific site misbehaves. So I looked around.

Choosing a password manager

There are hunders of password managers available today, so I’m not going to attempt to summarize even a portion of them here. Password Safe stands out from the rest, though. First, it helps that its original author is a credible source. More importantly, though, the file format is documented and alternative clients and libraries are available. Also, there is an Android app available, too, and it comes with sensible permissions. I look forward to using this when away from my laptop.

A Linux client (Beta) now exists of the original Password Manager and I might try it out sometime regarding auto-input, but for now the simplicity of Loxodo appealed to me. I’m unsure whether I’ll invest the time to learn wxWidgets to add a grouping pane or whether I’ll switch clients at some time. In either case, the underlying file format gives me an easy way to switch back and forth between v3 clients.

Exporting from Firefox

Getting the list of logins in Firefox Password Manager exported requires a separate extension, Password Exporter worked well for me. I based my work on the CSV output. I suggest cleaning the file up a bit before importing. For example, consider removing superfluous headers. Since there isn’t an exact match between the title field in Password Safe and the export from Firefox, I used the host entry as title and formSubmitURL as url. The former really doesn’t need the http, so stripping that made sense for me:

:%s/^"https:\/\//"/g
:%s/^"http:\/\//"/g

Importing to Loxodo

Armed with a CSV file, getting the data into Loxodo was a straighforward process and was made extremely easy through the existing CLI components. Iterating through the CSV file and calling the relevant vault function for adding an entry can be done in Python with a handful of lines, you can find my importer commit on Github. Working with PHP most of the time, I find the elegance, efficiency and fun of Python still suprising every time I have a chance to use it.

Drupal Quick Tip: Overriding strings

Sun, 22 Jan 2012 00:00:00 +0000

Photo CC-BY by smithser

Drupal comes with its own terminology and sometimes the strings returned to the user are not quite the vocabulary he or she uses. For example, a small library might refer to the people who have their library card as patrons or members. If a module to manage those users uses a different term this then can cause inconsistency, confusion, etc. One can, of course, just change the strings within the given contrib module but then upgrading becomes a tedious process of merging changes by hand. There is a much nicer solution available.

String Overrides

Within settings.php one can specify strings which should be overriden, so that a consistent word choice can be implemented without having to mess up core or contrib. The OpenAtrium documentation has a nice explanation of how to use this.

If you don’t mind adding one more contrib module to your installation, there is also a gui solution to do this for you, called the String Overrides module.

Drupal7: Empty nodes or comments after upgrade

Mon, 15 Aug 2011 00:00:00 +0000

If you are trying to upgrade a site running Drupal 6 to 7 and have messed around with your language settings or input formats, you might be in for a surprise. So, if you are seeing nodes or comments which have a title but an empty body on view and edit, it makes sense to check for the following.

If you have turned the locale module off entirely, but you still have content which is not classified as language neutral, it’s likely that you will now encounter problems. This could probably also happen if you have locale enabled but your specific language is not configured. Let’s presume you do not want to enable the locale module. Missing input formats for content which still has it assigned can be similarly problematic.

In either case, the problem can be solved quickly by adjusting a few rows by hand. If you are familiar with the Drupal 6 table layout and, for example, expect text bodies to show up in node_revisions, you’ll be in for a surprise, since the database schemas have changed in several important areas.

Therefore, you’ll find a primer below the warning on where to find text body specific definitions for input formats and languages for nodes and comments in Drupal 7.

Big fat warning: Of course you need an sql-dump so you can restore your database when you make a typo.

Locale

Reset comments to language neutral and set nodes with language ‘de’ to ‘und’ (i.e. language neutral):

UPDATE comment SET language="und";
UPDATE field_data_comment_body SET language="und";
UPDATE node SET language="und" WHERE language="de";
UPDATE field_data_body SET language="und" WHERE language="de";

Input format

Switches input format from 3 to 1 for all nodes and comments:

UPDATE field_data_comment_body SET comment_body_format=1 WHERE comment_body_format=3;
UPDATE field_data_body SET body_format=1 WHERE body_format=3;

A quick drush cc all is definitely encouraged since the menu cache is at the very least affected when it comes to localized url aliases. Case in point: the language specific alias vanished for content which I switched back to ‘und’ and the page could now be found under node/123.

Minutiae: Return the content of an element with xmlstarlet

Sun, 27 Feb 2011 00:00:00 +0000

It’s not uncommon to find desktop applications storing some information in XML files. While it’s nice that it isn’t binary and it might be technically human readable, it’s hardly as easy to read or edit as a CSV or plaintext file.

So, when you want to export data from XML, xmlstarlet is a parser that can be a quick solution to the problem. It’s documentation, though, is a bit arcane. Every six months or so, I start googling the same problem for xmlstarlet: How to get the content of an element, not just the value of an attribute. I might be using the wrong terms to find what I’m looking for, since even old posts of mine show up in Google for it.

Either way, the solution is simply a . and here is an example that uses the XML of KDE’s Wallet Manager:

xmlstarlet sel -t -m //map -v @name -n -m mapentry -o " " -v @name -o " : " -v . -n wallet.xml

The code above will iterate through all map elements, write the name attribute and a newline, iterate through all mapentries of the map and write the value of the subsequent name value indented by two spaces and then finally the actual content of the mapentry element after a colon separated by whitespaces. So there you go, an entire post because I don’t think the dot-for-content is documented well. The above example then converts takes XML structured as in the following example:

<map name="http-www.example.com">
  <mapentry name="login">a@example.com</mapentry>
  <mapentry name="login-2">b@example.com</mapentry>
  <mapentry name="password">12345</mapentry>
  <mapentry name="password-2">54321</mapentry>
</map>

This is then turned into:

 http-www.example.com
  login : a@example.com
  login-2 : b@example.com
  password : 12345
  password-2 : 54321

LibreOffice

Sun, 10 Oct 2010 00:00:00 +0000

Starting Over

A week ago Twitter and Identi.ca were abuzz with news that the Document Foundation had been founded and that they had released a first beta of their OpenOffice.org fork, LibreOffice. Some voiced cautious optimism about the project but were not convinced that it was a true, full fork, yet. The OpenOffice planet aggregator overall sounded rather positive, and a LibreOffice planet is already in place.

So far so good. Now it’s time to pull this office suite out of the quicksand it has been slowly sinking into over the last few years. It’s not that there were not continuous improvements over that time (document comments alone were a great leap in 3.1) but there was just a lot more sand that was being added (yes, it’s a mediocre metaphor, I’m sorry).

Compatibility

By far the biggest problem is the current state of interoperability. Since 2007/2008 we have needed import/export filters for OOXML. They should have been the absolute highest priority. Now, more than two years along, we still cannot display a generic docx file (with a TOC and a few images), without the format completely breaking.

I understand that this is difficult, prone to break, and OpenDocument should solve all our problems (alas, Office 2007 SP2, has been a sobering comment on that approach) but the matter of fact is that we have lost ground, tremendously. With Word 2003 and OpenOffice 3.x, formatting bugs were rare and collaborating was not an issue. Since 2007 and especially OOXML that’s no longer possible and I, for example, have been unable to use OpenOffice for basically all collaborative tasks that have to look right; and this is not even counting viewing/editing “SmartArt” or Microsoft’s current bibliography features. I don’t want to use Office 2007, right now I have to.

Innovating

Apart from absolutely crucial elements like above, LibreOffice now also has an opportunity for innovation. Innovation that is not slavishly copying Microsoft’s ribbon interface or similar recent changes in the text-editing-world but finding ways to work more intuitively with this office suite and improving the tasks that matter, i.e. working with text, numbers and doing basic layouting, while maintaining a stable, reliable office suite with sufficient features.

The difficult question in that context sometimes is: “Am I introducing this particular feature so Microsoft documents can be edited well (e.g. SmartArt), or am I slavishly copying functionality because I think users need it or want it?” Finding the correct answer to each new feature provided by the big pink elephant in the room is going to be diffcult and take time, I just hope it won’t take too much.

A cautionary tale of why to take this time though is exemplified by weird things one still comes across in OpenOffice and LibreOffice:

Did you know you can still create WordArt? It’s called Fontwork. You can find it with View ->Toolbars -> Fontwork. It needs to go.

Here is a thumbnail browser. Cool homepage icons, eh? I’m trying to remember if there ever was a document that I recall which was visually improved by a stock clipart – this especially includes presentations. Don’t think so. Check out the “Rulers” and “Sounds”, too. It’s under Tools -> Gallery.

Once implemented, things are not easy to remove, there will always be someone who depends on them. In the end, though, I hope the community comes down on the side of offering good, modern defaults and sacrificing the odd feature to make it harder to shot yourself in the foot visually. I hope in the end it will make working with LibreOffice pleasing, rather than a chore.

Cleaning Up

One of the elements that gives me the most confidence in this project is their list of Easy Hacks. It shows that the foundation and its volunteers realize that the existing code needs to be harmonized, corrected and most of all tidied up, so that legacy code will not hold future developments back as much as they have in the past.

Addendum

So, good luck LibreOffice and please, please do not change the name back to OpenOffice.org (the .org especially has been so incredibly annoying over the years!) should Oracle give the name back. Make a fresh, bold new start.

If you want to help this community, you can contribute your time or funds.

Flash, netbooks and too much bandwidth

Fri, 23 Apr 2010 00:00:00 +0000

Several online video sites estimate the bandwidth they can deliver too you and adjust the video bitrate accordingly. A good example is the Daily Show. Click on ‘Full Episodes’ and the player has a bar indicator and actually gives you feedback on:

Current Available Bandwidth
Current Video Stream
Max Video Stream

This is really nice for testing things. From my location the maximum video stream available was 1720kbs, which should not be a problem for a modern broadband connection. However, the Linux Flash plugin has notoriously bad performance in terms of video decoding on machines with limited resources, such as netbooks. This becomes particularly evident when you download a file and it plays fine but embedded in Flash it is close to 100% CPU load and only a sequence of intermittent still images.

In these cases, where you cannot choose a video bandwidth (such as 360p on Youtube), you can still reduce it by throttling the connection on an individual network interface. Debian-based distributions can just grab the wondershaper package for that.

After installing, it’s trivial to set the bandwidth. For example:

$ sudo wondershaper wlan0 1000 500 # parameters are: interface, kbs down, kbs up

Initially I chose 500/500 but the feedback from the Daily Show player showed that 500 was insufficient to deliver their minimum quality at 450kbs due to protocol overhead and everything else. The parameters above turned out to work nicely. Afterwards you can clear the throttling with the following:

sudo wondershaper clear wlan0

Wikireader (now with stickers!)

Sat, 06 Feb 2010 00:00:00 +0000

In many of the more relaxed civilizations on the Outer Eastern Rim of the Galaxy, the Hitch Hiker’s Guide has already supplanted the great Encyclopedia Galactica as the standard repository of all knowledge and wisdom, […] it scores over the older, more pedestrian work in two important respects. First, it is slightly cheaper;…

That’s how Douglas Adams introduces the Hitchhiker’s Guide in the first novel of his series by the same name. Wikipedia has several similarities to the Guide that are highlighted in that first characteristic. Like the Guide it has supplanted the more scholarly works with its reach and breadth, as well as popularity.

I vaguely remember how years ago we were sitting in the Florida sun off-campus and Gavin made the remark that you would just have to put Wikipedia on a memory card, put it in a phone and you’d basically have the Guide, including about the same amount of accuracy.
Enter Wikireader

Granted, it’s a device that differs in many aspects from what purists would call the Guide, notably absent are the hundreds of buttons Adams described and it doesn’t have a cover, and so on, but in essence it provides the world’s knowledge summarized in over three million articles in a memory chip the size of my fingernail.

Nowadays, with smart phones that have decent browsers being pervasive, it’s not quite as impressive as it might have been even a few years ago. Nonetheless, (ignoring a few temporal paradoxes concerning Wikipedia’s existence) I’m sure that I would have killed to own such a device up until a few years ago. The potential for being a smartass (and having the sources to back it up) were just so darn limited back in my youth.

Over the holidays I spent several days without an Internet connection (if you don’t count the phone) and the Wikireader proved that Wikipedia’s addictiveness is also present in this device. The ‘random’ button is especially genius, you never know what weird topic comes up that you have never even heard of. You can then hit the button again and again, with around a second latency, until it looks interesting. The device takes getting used to, since it’s counterintuitive that the device is not backlit and thus is hardly readable in low-light conditions but I’m looking forward to trying it out in the summer.

The physical manifestation of Wikipedia in my palm is an interesting thing. As long as I can find some AAA batteries, I’ll always have that encyclopedia, no matter how many sea cables are ruptured or how much filtering a government would like to employ. Surprisingly, as long as the Openmoko company runs we even get ‘new editions’ like the Guide. Only those aren’t downloaded directly, we have to want to load the new image onto our microSD cards (or have new cards shipped to you). More importantly, though, technically everyone can make their own ‘editions’, so we don’t have to accept a “Mostly harmless”, if we don’t like it. Admittedly, I tried to ammend non-Mediawiki-content to my image but quickly abandoned that for FBReader on my phone.

…and secondly it has the words DON’T PANIC inscribed in large friendly letters on its cover.

Font-Face Kits

Mon, 02 Nov 2009 00:00:00 +0000

When you look at common practices for styling fonts on the web, the majority of sites either tend to be minimalistic to preserve consistency or a bit redundant to have it look as nice as possible with stock fonts on every system.

Helvetica

A good example is Helvetica. It’s a beautiful modern typeface, hell, they made a whole documentary about it! Anyway, it’s included in OS X and thus of course a web designer certainly likes to tell the browser to use it if it’s there. Fallback solutions are then often just Verdana, Arial and a system default sans-serif.

Actually, I should still have a copy of Helvetica from when I had a Macbook. Does anyone know to what license I agreed when I got that? Even if I’d null out the drive afterwards I’d probably break the license agreement if I’d copy it over…

Embedding

The alternative to specifying fonts one after the other and hoping that at least one is present is embedding the fonts through CSS. This is similar to specifying images within CSS and has been supported for a long time. However, some inconsistencies hampered making full use of these techniques.

For one, some browsers support TrueType/OpenType, some only embedded OpenType (EOT), and Firefox 3.6 now comes with something that’s called WOFF. Now, even if you had a license for Helvetica, you could of course not embed it since that would basically give every schmuck the font for free (which they would like about 25€ per variant for). Granted, EOT does not make it trivial to get a regular font file from it but someone who really wants to extract the glyphs will probably be able to.

Font-Face Kits

A very cool solution to the technical as well as legal problems are so called font-face kits for freely licensed fonts. Sites like Font Squirrel provide packages that can simply be copied and pasted into existing CSS files. Since they take care of the obscure things, such as converting to EOT, this can be used by basically any CSS designer without further knowledge about fonts and font formats.

All that’s needed then is to upload the provided font files and change the font-family reference for each instance, in your CSS files, where you would like that font to be used. You can change the whole look of your site in about half an hour, without resorting to ugly hacks such as embedding images.

I liked this solution so much that I converted my blog to Sansation Regular. It’s a bit playful but if you like more conservative typefaces (they have their uses, my desktop isn’t using Sansation…) they have a great selection of featured fonts for that, too.

Blocking distractions, when necessary Part 2

Tue, 21 Jul 2009 00:00:00 +0000

In the first part we created a list of things we want to block on occasion. The point of this part is to make it easy to switch from full access to limited access. Enter, OpenWrt.

OpenWrt

A large chunk of wireless routers on the market today can not only run their stock firmware but also OpenWrt. It’s useful for creating wireless mesh networks and all sorts of things but the most common task is just a free-as-in-speech router. There are countless tutorials online for installing it and resources for which devices are supported, so I’m not covering that here. Apart from being able to SSH into your router, OpenWrt now exposes many things such as buttons and LEDs, which you can use in your scripts. These examples have been tested with Kamikaze 8.09.1. In older releases, several things were elsewhere, such as /proc/diag.

Tinyproxy

Due to the size limitations on many models and feature limitations on certain solutions I ended up using Tinyproxy. Definitely also install the luci package for Tinyproxy, it makes setup a lot quicker. Due to several smaller bugs in the current releases it’s probably still necessary to move the Busybox web interface from 80 to 8080.

At this point, you should let Tinyproxy listen on your router’s IP on port 80. Also allow access from your subnet and 127.0.0.1. Most often this will be 192.168.1.0/24. Finally upload your filter list and turn your proxy on. Due to the way the current OpenWrt releases works this will not yet router your traffic through the filter. Also, if Tinyproxy doesn’t seem to start properly, make sure the logging directories are also present.

Putting it together

All that’s missing is a script, to run once a button is pushed, let an LED blink as long until it is pushed again and toggle routing the traffic through the proxy. My script is largely based on a howto from the OpenWrt wiki (as is much in this second part). It uses the SES LED and works with a Buffalo router running a 2.4. kernel due to the wireless drivers (otherwise iptables redirect could be used). Placing it into /etc/hotplug.d/button/, for example as 01-nettoggle, should make it work. Though you’ll probably have to adjust LEDs, buttons, and maybe also the router IP. Now have fun with tricking that dumb part of your brain into getting things done, to paraphrase Merlin Mann.

Blocking distractions, when necessary

Tue, 14 Jul 2009 00:00:00 +0000

The dilemma

This semester I had a handful of exams rather late in the year, weeks after classes ended. Now, the Internets are a pretty interesting place and I find it extremely difficult to not glance at the feed reader and all the other gizmos rather often, when in front of the screen.

I saw three possible solutions:

a) Work analog. (Pretty unrealistic, and think of the poor trees…)
b) Force yourself to not get distracted. (Sounds as likely as most New Year’s resolutions.)
c) Turn off only the distractions, or at least minimize them.

Naturally, I went for C. It gave me an excuse to go write the scripts for it. The first attempt consisted of a list of domains added to /etc/hosts on my main machine. It worked, but just blocking domain names often gives inconsistent results in respect to subdomains and other problems. I have split this tutorial into two parts to make it more readable. The first part deals with collecting domain names, the second with filtering them by hitting a button on an OpenWrt router.

Collecting URLs

Many programs dealing with RSS today will often export to OPML (just an XML file) and manipulating these with XmlStarlet works great. Let’s try it on Miro.

We have a list of outline elements, some are groups, some are feeds. The following will extract the URL from all outline elements.

xmlstarlet sel  -t -m //outline -v @xmlUrl -n ~/miro_subscriptions.opml

Appeding | grep . should also get rid of the empty lines. Just repeat the procedure for the other XML variants your programs produce.

Without proper XML

Sometimes, even when an XMLish notation like HTML is used you get nonvalidating files. For example with Firefox bookmarks (other browsers do it, too). Tidy is an easy way to automatically fix those problems. The following will produce a validating bookmarks file. tidy -asxml -o cleanbookmarks.html bookmarks.html With this you can run it through XmlStarlet without problems. I adapted the following somewhat cryptic line from the XmlStarlet User Guide. Additionally, | grep -v "place:" removes the lines from Firefox which aren’t URLs.

xmlstarlet  sel --html -T -t -m "//*[local-name()='a']"  -v @href -n -n a.xml

Getting domains

From the previous steps you now have a list of URLs. With these, you could already jump ahead to step 2 since the proxy in this project can also filter on URLs or patterns.

If, however, you want to broaden it to domain names, there is one more step. The following Python script takes two arguments, a file with URLs and a filename to write the domains to. Download it directly. Example usage: ./getdomains.py urllist domainlist

If you quickly look over the script, you might notice the four lines below. Since many sites have inconsistent uses of the www prefix, the script looks for all domains with a www prefix. If successful, the domain is added to the list with www and without. This does not cover any other subdomain.

domainshort=domain[4:]
o.write("%s \n" % domain)
if domain[:3] == "www":
    o.write("%s \n" % domainshort)

If you don’t have OpenWrt, you can also download a version with two tiny changes to make the ouput readable by /etc/hosts. Just append your entries at the end of that file.

Upcoming

In the next post, we’ll see how one can easily switch using this list on and off with a button, rather than meddling with configuration files each time.

Use your remote with Amarok and Last.fm

Thu, 11 Jun 2009 00:00:00 +0000

If you have an amplifier (or something similar) with a remote and your computer attached to it, there are probably several buttons that don’t do anything in this configuration. In my case, it’s the buttons available to control a CD or tape deck. Wouldn’t it be neat to repurpose those? In short, all you need is an IR receiver, LIRC and a handful of commands. There are already quite a few tutorials on the net for doing this but many are a bit dated and so I think this post still brings something new, if simply by addressing installing LIRC on Ubuntu Jaunty to use Amarok 2.

IR Receiver

First off, get an infrared receiver. I got a serial one off of ebay very cheaply, just search for “lirc.” You can often reuse receivers from other gadgets, LIRC is quite flexible that way. Let’s now begin with a plugged in receiver but nothing else. How do you know it’s even working? First, load the appropriate module. In my case, lirc_serial. You could let the LIRC script load it but /etc/modules works just as well. Put the model name for your remote in /etc/lirc/hardware.conf, start /etc/init.d/lirc and you should be mostly done. The command irw should now return correct values for the remote on individual button signals. If the device cannot be found, consider running the command with root privileges and make sure the LIRC device exists. In Ubuntu, you often have to reference /dev/lirc0 because /dev/lirc is the default many scripts assume.

If your remote doesn’t work, try adding it to /etc/serial.conf with /dev/ttyS0 uart none, of course, adjust ttyS0 as necessary and consider rebooting.

Unknown Remotes

However, what do you do if your remote isn’t in the list of remotes LIRC knows? No problem, LIRC can learn new remote signals, too. Just call irrecord and follow the on-screen instructions.

After that you can move the resulting file to /etc/lirc/lircd.conf and the remote should work. To complete the setup you still have to tell LIRC what to do when you press a button. There are several alternatives to settings things up but calling irexec -d from the autostart folder is probably one of the quickest. Irexec by default calls ~/.lircrc. Here is a short excerpt from my configuration, of course the button names of your remote might be completely different.

begin
prog = irexec
remote = abc123
button = play
config = dbus-send --type=method_call --dest=org.kde.amarok /Player org.freedesktop.MediaPlayer.Pause
end


begin
prog = irexec
remote = abc123
button = stop
config = dbus-send --type=method_call --dest=org.kde.amarok /Player org.freedesktop.MediaPlayer.Stop
end

Last.fm

Now, what was this talk about Last.fm earlier? Well, Amarok has decent support for Last.fm built-in and even has a little ❤ button on the main toolbar so you can favorite a song. You can easily call this feature over DBUS, too:

dbus-send --type=method_call --dest=org.kde.amarok /amarok/MainWindow org.kde.amarok.MainWindow.loveTrack

So, now you can flip through your music collection from your couch, skip a track and even favorite it in Last.fm without having to to use your computer. What could be improved? Lots. For one, moving to the MPRIS namespace in DBUS would make the whole thing more flexible, but I’m skeptical that it will ever support things like the ❤ button. Also, I tried org.freedesktop.MediaPlayer.Seek but it didn’t work as expected.

OpenDocument in SP 2 mostly useless

Thu, 21 May 2009 00:00:00 +0000

I actually wanted to do a whole series on Office 2007/OpenOffice 3 interoperability but it just didn’t work out. New developments, however, give me an opportunity to tackle on the more import parts: file formats.

Background information

Microsoft used a binary file format (doc/xsl/ppt) until Office 2003, then they switched to OOXML and now it is (docx/xslx/pptx). OpenOffice (OOo) has very good support for reading and writing the binary format. They have a rudimentary import filter for OOXML but it is nowhere near complete, having big gaps in such areas as SmartArt and others. Also, no export filter exists, OOo users have to use the binary format to send files back to Microsoft Office users (yes, they could now use ODF but not really, more on that later).

OOo uses ODF (odt/ods/odp) and this became an ISO standard. Microsoft promised OpenDocument support based on the current spec, which does lack some definition on formulas and OOo 3 is basing its file format on the not finalized newer version of OpenDocument.

Now

So, Service Pack 2 for Office 2007 with OpenDocument support was released and many people are not happy. Apart from the obvious Excel blunder, several things are simply not supported, rather than ‘differently implemented.’

While these changes do send negative messages, OpenDocument is also option #4 under ‘Save as’, right after the native format, template and 97-03 DOC, which at least looks like good intentions. Amongst many other items, the Office help files (referenced when you export to ODF) point out that the following things are just dropped during the export: - Track Changes, collaborative editing is now quite difficult (the main point for not using PDF!). It’s a showstopper for my frequent group assignments.

Table of Contents is converted to text, which is more or less ok.
Page break sometimes breaks other fields.

Apart from these shortcomings, some acceptable, others not, and probably not fixed until SP 3, there are the things OpenOffice does not display right. Who’s fault these are is probably impossible to tell without consulting Oasis, Microsoft, Sun and then choosing an opinion at random. Here are a few I ran into when opening 3 documents in OpenOffice:

From Word

Table of Contents: every item on a singe page…your 30 page paper is now 75.
Formatting rules: inconsistenly applied. No consistency between “style” and overriding “font” definition.

From Excel

Formula bug: forget it.
No graphs:, only an OLE placeholder is shown, at least with a simple 3D bar chart.

From PowerPoint

Empty line items: show bullet points, on Office they are just greyed out when the line is actively selected and empty otherwise.
Default placeholders: sometimes present, especially when Master slides applied and not all fields used.
Indentations: have different settings, text thus is easily flowing over regions occupied by other things. Especially if paragraph indention has been modified by hand.

All these things are not terribly bad, most could probably be fixed in 1 or 2 revision of the respective software packages. The problem, however, is time. The next Office service pack could be years away and still not solve outstanding issues.

Even OpenOffice doesn’t have a major release until Fall (and let’s face it, major changes would be necessary) and i think there is at least somewhat of a case to be made to not water down the OpenOffice implementation just because Microsoft can’t do it right. Finally, these issues are only from Office to OpenOffice, at least a few problems, and likely many more, are also present when going from OpenOffice to Office.

It’s the browser wars all over again, and I still can’t exchange a presentation without relying on the doc format to even get close to be able to run it and not be embarrassed about what pops up on the screen.

Building a Lego ATM machine

Fri, 08 May 2009 00:00:00 +0000

The end of the semester is finally inching closer and one course in particular ended with a quite successful final presentation for our team of 3. The objective was to use several crates of Lego Mindstorms material to build a working model of two atm machines.

Rebooting ATM image is CC BY-NC-NA from n2b. Our ATM not quite identical to original.

And that’s what we did. OK, not quite, our currency consisted of square wooden blocks (about 50mm*50mm*4mm) and our bank cards only contain a black and white strip that encodes a binary number. We reused some old credit card sized plastic cards and used the pattern on the right for the cards.

Then we built a PIN pad with a cut up connector cable, a project box, 6 buttons and a handful of resistors. Additionally, we had team members that were tweaking the construction until every single “bill” dropped perfectly. Then, all that was missing was an actual backend. For this we simulated a database with an array (loaded from, and saved on exit to, a text file) and set aside one of the NXT control units to serve as the bank server. Then, our ATMs send requests as a string over the weird Bluetooth stack to the server and got a reply back, depending on the account’s state. Here is the end result. It comes in at just about 1200 lines of NXC code with support for German, English and French, as well as administrative functions on the bank to manage accounts.

This and three more videos.

A student's guide to OpenOffice 3.0: Part 1

Sat, 13 Dec 2008 00:00:00 +0000

Foreword

Before the summer, hardly anyone I knew had Office 2007. Now, however, many have bought new laptops for the Fall semester and it’s no surprise that they come preloaded with Office 2007. Combine that with the fact that my new college really likes group presentations and group papers and you get a field study of OpenOffice 3.0 vs Microsoft Office 2007.

Results

Short version: OpenOffice 3.0 is in many ways inferior to Office 2007 and exchanging files has gone from nearly perfect to rather limited. After five weeks I had to bite the bullet and purchase Office 2007.

Long version: It’s not the end of the world and OpenOffice will probably catch up in most areas within a release or two. However, there are several pitfalls I encountered along the way and so I want to present a list here on what is not possible, when it will likely be fixed and how to work around it.

The posts will roughly cover the following topics:

Styles, Colors and Fonts
OOXML
Citations and Sources
Powerpoint and SmartArt
Excel vs Calc

Styles, Colors And Fonts

After you have seen half a dozen presentations and papers done in Office 2007 you will notice that they actually did a really nice job of making things look decent. I’m sure within the next five years we will all not be able to stand the default Powerpoint templates but for now they are a welcome change when compared to the often crude looks of Office 2003 and OpenOffice (especially before 3.0).

New fonts embedded in Vista and Office 2007 have been a major factor in improving visual design by mostly retiring the ancient Arial and Times New Roman combination in favor of Cambria, Calibri etc. These fonts are possibly also the most problematic thing because they cannot easily be patched away.

Here’s why: you cannot get them for free and there are no equivalent substitutes. If you google ‘office 2007 fonts linux’ you will find out a few ways to get them installed but that’s dubious at best. For one, several bloggers point out that you can get the fonts from the free PowerPoint viewer but the license expressly states that use of the fonts is only valid on the Windows operating system. I’m pretty sure that the license I paid for grants me the right to use them outside the virtual machine but without delving into the EULAs I’m not going to speculate. The fonts directly from the manufacturer cost $300 by the way.

Which then leads to the second point: substitutes. As pointed out in this article, fonts that are metrically equivalent matter. For now you’ll either have to pay for the fonts one way or another or resort to less pretty fonts. These will still become the expected standard in the near-term so expect to be the odd one out.

Therefore let’s hope (or start a fundraiser so) that the same things happens that did for the Core Fontspackage. Red Hat paid for the Liberation fonts and they are close enough so that you didn’t have to use the Core fonts if you didn’t want to. I’m not holding my breath, though, since the same firm which did the Liberation fonts worked on the C* fonts from Microsoft.

Better default color schemes are one of the additions that really add to the value of Office 2007 and where OpenOffice definitely needs to catch up. It was certainly possible to create tables which looked just as harmonious and professional in Office 2003 given enough time and talent but these defaults make that possible for the average person.

It just is really nice if you can get from a plain table such as the one on the left to a table such as on the right with just one click:

Default colors are a point where it could be comparatively easy to improve the look and feel of open source software but it is often overlooked. Take KOrganizer for example, you could take a look at the KDE/Oxygen color palette and set the colors accordingly and have a really nice looking calendar, similar to how iCal looks really nice on default on OS X, instead of the greenish nightmare that it is today. Why is this important? Because most people do not change their defaults. They often don’t know where so good defaults make a difference, especially for an office suite.

These table styles and such are also not imported very well in OpenOffice but that will be covered in Part 2. To summarize, defaults have improved, making documents better looking without having a designated designer on hand but it isn’t impossible for OpenOffice to get there. If you have such abilities to produce great defaults, consider giving your input to the OOo devs.

Check back in a few days or subscribe to the RSS feed on your right to find out even more about the mysteries of MS Office, OpenOffice and stuff…yay!

How to break your git repository

Sun, 31 Aug 2008 00:00:00 +0000

I was cleaning up a few things on my drive and moved several folders around. Unbeknownst to me at the time, one flash drive was still vfat when I thought it was ext2/3. For some reason no obvious warning such as “cannot change permissons on file xyz” appeared when moving files. So I assumed everything worked fine and I committed a small change to a file in my main git repository, which was moved, too. That wasn’t so smart. As I ran git status too see how things were looking I got a listing of every file in the repo having been modified. Next I noticed that nearly all files had the executable bit set, all in .git as well. Then I noticed it was a vfat partition and moved the directory back to a Linux partition. Then, back on the old drive, the only error message that came back was:

fatal: Not a git repository

The first thing I tried to remedy this was removing the executable bit from all files and a helpful forum post supplied the useful one-liner:

find . -type f -print0 | xargs -0 chmod a-x

Still, the fatal error message persisted. It had to be somehow connected to the .git directory which looked fine but when compared to a newly created test repository it became obvious that the one commit on the vfat partition had changed the filename HEAD to head…must be a default policy for case-insensitive filesystems… Anyway, all worked well after renaming head back to HEAD and a simple git checkout -f got rid of any remaining problems with incongruent files. Maybe this post will be a helpful shortcut to somebody else who wasn’t paying attention.