wswld

I Created an Issue in a Flask-RESTful Repository, You Won't Believe What Happened Next

2018-06-14T07:00:00-07:00

Two and a half years ago (Dec 4, 2015) I’ve written about a minor problem in Flask-RESTful, that puzzled and amused me. An issue was created on the same day. I then went on with my life, hoping for someone to step in and fix this trivial design inconsistency. Several people including the repo maintainer enthusiastically expressed support for proposed changes but no code was ever commited.

Fast forward almost a year (Oct 30, 2016), I noticed this issue while skimming through my Github activities and decided to open a PR. I wasn’t using Flask at the moment, so I was doing it for the greater good. Having commited some code and hoping for the maintainer to review it ASAP, I switched to other commitments. Needles to say there was no review in several days, weeks and even months. Feeling more and more irritated with the whole affair, I emailed the maintaner and several other people, whose PRs have been merged recently - to no avail. I gave up and decided to finally let it go.

Fast forward almost two years (Jun 6, 2018) the maintainer of the repository suddenly started reviewing the code. It is now merged for good and I’m glad to have contributed to the project, but somehow this whole ordeal of trying to commit my trivial change for almost 2.5 years makes me worried about the ever-increasing entropy of Open Source. Perhaps several thousand new libs and frameworks were created last couple of minutes. Do we have the guts to support and maintain it all? Could it be that we got carried away with a sexy idea of starting our own new thing and developed intolerance for the actual grind and commitment?

Limitations of the Beeminder Model

2018-01-09T06:00:00-08:00

I’ve been having somewhat on and off relationship with Beeminder. I’m not going to waste time explaining what it is as if it wasn’t done again and again all over the internet, but for the sake of keeping you on this page: it is a service that keeps you motivated, using commitment contracts and a bit of money pledging. On one side I’m convinced this is absurd model and we are rational enough to achieve our goals without pledging money on the progress. On the other side though: I can’t stop procrastinating on my rationally chosen goals and somehow this relatively simple behavioral trick does wonders. I can tell the difference in productivity between the time I’m actively using Beeminder and the time I’m off, and this difference is as problematic as this is great: there is no long term effect to it all. It goes back to square one the moment you decide to archive your goal or stop using the service altogether. Before I go into detail, a little disclosure is due: I’m no psychologist or economist and my understanding is based on pretty anecdotal evidence. I’m not going to pretend this all is somewhat objective and smartass, at least I don’t want you to have that impression.

So let’s start and create a goal. Actually, let’s create two goals: a Do More (steps done) and a Do Less (cups of coffee). We’re tracking them for some time and we walk a lot and are extremely decaffeinated. After a couple of months we decide we don’t need Beeminder anymore. We’ve grown over it. However, the next couple of days we suddenly notice that we just don’t do as much walking as before and drink a lot of coffee, probably even more than we used too. What happened? In my understanding Beeminder goal is a little economy in itself, an equivalent of Monopoly money. You have a commodity (cups of coffee, day of sedentary lifestyle, etc) that you attempt to limit artificially through placing a price on that. Thus the economy is born. Especially if you never retroratchet your goals it may get a little silly, as you start accumulating your buffer and making spending decisions. For example: one cup of coffee costs 1 step of Beeminder graph, you can accumulate steps available to you by saving, you can also inflate and deflate your Beeminder step, or you can get some buffer in exchange for money. You always have the price for your negligence, so that the negligence becomes a commodity in itself.

When you occasionally deflate the step too much by removing the goal or making it too easy — you create a situation where the consumer (you) is subconsciously driven towards increase in consumption. You’re creating an economical situation that has psychological consequences. In my opinion the problem is that eventually you remove your authentic self from the loop: “I don’t drink this much coffee because it’s unhealthy” becomes “I don’t drink coffee because I don’t have enough magic or real coins for that”. This substitution may be effective at first, but in the long run it eventually comes to catching up on coffee when the artificial limitation drops. When you suddenly have all the money in the world can you find enough willpower not to buy as much as you can carry? I noticed this effect with various goals: not only it all returned to the original level, it got worse for some time. You may have noticed similar problem with diets, as soon as you get off, all the weight is back and sometimes it gets even worse. Beeminder treats the symptoms without fixing the underlying problem and that is the greatest limitation of this model.

What’s most troubling about all that, I don’t really have a solution in mind, only a hint of it. In his book Existential Psychotherapy Irvin Yalom suggests a near death experience as a method of becoming a more motivated and authentic self. Of course it’s a bit extreme, although there are exercises to emulate this effect to some degree. Somehow this idea also correlates with Tim Urban’s brilliant Ted talk on procrastination which makes me think there is something about this idea. In my opinion if there is really somewhat universal key to finding your real authentic goals and sticking to them in the long run, it is somewhere in the area of realizing our finitness and creating a short feedback circuit to review every day in the perspective of the time we still have. Making every day the first day of the rest of your life. But there is still no clear methodology, that would help you avoid being lulled into the mundane, the comfort zone of seemingly endless life, full of mindless routine and the ensuing procrastination. Until we are there, let’s Beemind our way through.

Aspiring Photographers are Quitting Instagram en Masse and Here's Why

2017-03-06T00:00:00-08:00

Full disclaimer: the title is a total clickbait. Still I’m very concerned about the current state of Instagram and the direction the site has taken. Here are some of my grievances:

I haven't seen anyone but bots in a while. Sure, that's a gross exaggeration, but it certainly feels like actual people are in the minority on the site. Most accounts that like or follow me are spambots or some low-tier SMM guys from developing world.
My major grievance is lack of posting in official API. Sure, there are unofficial private API wrappers, but why should users jump through the hoops? Limiting access to CRUD operations in public API is walled garden approach at its finest.
No links. Instagram prohibits adding clickable links and copying in both descriptions and comments. I understand they're doing it to minimize spam, but it's like getting rid of intestines to avoid diarrhea. Also, see the first point. How much should it take before ineffectiveness of this approach is evident? In the meanwhile there is no way for the regular user to share a link to a bigger story.
Instagram still reduces the quality of pictures and ruins your videos. Sure, it's much better now than it used to be, but it's still far from ideal. Plus, what's up with quality of Instagram pics on Facebook? Is this by design? Because I have no other explanation how it can stay unfixed for so long.
It doesn't cater towards its original target audience - the amateur photographers. Instagram is no longer a network for good phone photography (or attempts at that), most pics one sees today are selfies, which says a lot about the current target audience of the service. Introduction of Stories (which is 100% Snapchat rip-off) cemented this trend.

I’m still using the service for sharing what I deem interesting shots and occasional videos, but I’m also using Flickr as some kind of mirror feed and to be honest Instagram starts to feel like dead weight.

TIL: Mongo Sucks

2016-10-07T06:19:06-07:00

MongoDB $nin and $ne and $exists: false queries are super expensive. I mean there is a point at which they are basically unusable. And I learned it the hard way, having developed a service, that relies heavily on this kind of queries to emulate queue-like workflow. 5+ million documents later and I’m hurriedly adding indexes for all flags that take more than 60 seconds to query for.

Information on this issue is limited to a short FAQ entry at Mongo docs and a couple StackOverflow answers. That’s it. Come on, it should be the first thing people see, when they open Mongo docs - a huge dialog you should scroll through three times before you can click accept.

Overriding Default Werkzeug Exceptions in Flask

2015-12-07T06:47:17-08:00

Update 15.06.2018: The issue described in this post is now a history, read the epic story of my struggle to get it fixed.

Let’s play a game here. What HTTP code is this exception:

{ 
  "message": "The browser (or proxy) sent a request that this server could not understand." 
}

No no, you don’t look at the code in response! That’s cheating! This is actually a default Werkzeug description for 400 code. No shit. I thought something is bad with my headers or encryption, but I would never guess simple Bad request from this message. You could use a custom exception of course, the problem is, that the very useful abort(400) object (it’s an Aborter in disguise) would stick with the default exception anyway. Let’s fix it, shall we?

There may be several possible ways of fixing that, but what I’m gonna do is just update abort.mapping. Create a separate module for your custom HTTP exceptions custom_http_exceptions.py and put a couple of overridden exceptions there (don’t forget to import abort we’ll be needing that in a moment):

from flask import abort
from werkzeug.exceptions import HTTPException


class BadRequest(HTTPException): 
    code = 400
    description = 'Bad request.'


class NotFound(HTTPException): 
    code = 404
    description = 'Resource not found.' 

These are perfectly functional, but we still need to add it to the default abort mapping:

abort.mapping.update({ 
    400: BadRequest, 
    404: NotFound 
})

Note that I import abort object from flask, not flask-restful, only the former is an Aborter object with mapping and other bells and whistles, the latter is just a function.

Now just import this module with * to your app Flask module (where you declare and run your Flask app) or someplace it would have a similar effect on runtime.

Note that you also should have the following line in your config because of this issue:

ERROR_404_HELP = False 

I’m not sure why this awkward and undocumented constant isn’t False by default. I opened an issue on GitHub, but no one seems to care.

Remote Debugging with PyCharm

2015-09-17T10:15:19-07:00

I’m working in a project now, that requires a certain (server) environment to run, hence it is developed on my local machine and then gets deployed on remote server. I thought I’m gonna say bye bye to my favorite PyCharm feature, namely the debugger, but to my surprise remote debugging has been supported for years now. It took some time to figure out (tutorials online are a bit ambiguous), so here is a short report on my findings.

For the sake of this tutorial let’s assume the following:

Remote host: foo_host.io,
Remote user: foo_usr (/home/foo_usr/),
Local user: bar_usr (/home/bar_usr/),
Path to the project on the local machine: /home/bar_usr/proj.

Here goes the step-by-step how-to:

First we need to set up remote deploy, if you haven’t done so already. Go to Tools → Deployment → Configuration. And set up access to your remote server via SSH. I’d use:
- Type: SFTP,
- SFTP host: foo_host.io (don’t forget to test the connection before applying),
- Port: 22 (obvously),
- Root path: /home/foo_usr,
- User name: foo_usr,
- Auth type: Key pair (OpenSSH or PyTTY),
- Private key file: /home/bar_usr/.ssh/id_rsa (you’d need to generate the key and ssh-copy-id it to the remote machine, which is outside of the scope of this tutorial).
Go to Mappings tab and add Deployment path on server (pehaps, the name of your project),
Now under Tools → Deployment you have an option to deploy your code to remote server. These first three steps could be replaced with simple Git-repository on the side of the server, however I sometimes prefer this way.
Now when you have the deployment set up you can go Tools → Deployment → Upload to …, note however, that it deploys only the file you have opened or the directory you selected in the project view, so if you need to sync the whole project just select your project root.
I use virtualenv, so at this step I need to ssh into the remote machine and set up virtualenv in your project directory (/home/foo_usr/test/.env), which is outside of the scope of this tutorial. If you’re planning on using the global Pyhton interpreter, just skip this step.
Now let’s go File → Settings → Project … → Project Interpreter. Using gear button select Add Remote. The following dialog window would let you set up a remote interpreter over SSH (including remote .env), Vagrant or using deployment configuration you have set up previously. For the sake of this tutorial I’m going to put something like that here (using SSH of course):
- Host: foo_host.io,
- Port: 22 (which is there by default),
- User name: foo_usr,
- Auth type: Key pair (OpenSSH or PyTTY),
- Private key file: /home/bar_usr/.ssh/id_rsa,
- Python interpreter path: /home/foo_usr/proj/.env/bin/python
If you set up everything correctly, it should list all the packages installed in your remote environment (if any) and select this interpreter for your project.
Now let’s do the last, but the most important step: configure debugging. Go to Edit Configurations… menu and set things up accordingly. For our hypothetical project I will use the following:
- Script: proj/run.py (or something along these lines),
- Python interpreter: just select the remote interpreter you have set up earlier,
- Working directory: /home/bar_usr/proj/ (note that this is working directory on local machine),
- Path mapping: create a mapping along the lines of /home/bar_usr/proj = /home/foo_usr/proj (although this seems pretty easy, it may get tricky sometimes, when you forget about mappings and move the projects around, be careful).

That’s about it. Now we should have a more or less working configuration that you could use both for debugging and running your project. Don’t forget to update/redeploy your project before running as the versions may get async and PyCharm would get all whiny about missing files.

My Take on Yandex Pre-interview Python Assignment

2015-06-08T04:19:12-07:00

I’ve applied for a junior Python position at Russian internet giant Yandex (very similar to Google). And although my application has been rejected, due to lack of experience, I think their little pre-interview test and my take on that may be of interest to any inquisitive pythonista. Note, that this has never been properly translated into English before, so this is probably exclusive in that regard.

Assignment I

There are two lists of different length. The first one contains keys, the second - values. Write a function, that would create a dict out of these lists. If the key doesn’t have a value - it should equal None, if the value doesn’t have a key, it should be omitted.

def get_dict(list1, list2):
    ret = dict(map(None, list1, list2))
    if ret.get(None, False):
        ret.__delitem__(None)
    return ret

Assignment II

Login should start with latin symbol, contain latin symbols, digits, dots and hyphens, but end only with a latin symbol or a digit. Minimum length is 1 symbol, maximum - 20 symbols. Write a function that checks strings for correspondence with these rules. Think of several methods of solving this problem and compare them.

import re
import time
def check1(login):
    ret = False
    if re.match('^[a-zA-Z][a-zA-Z0-9\-\.]{0,19}(?<;![\-\.])$', login):
        ret = True
    return ret
def check2(login):
    ret = False
    if (len(login) >= 1 or len(login) <;= 20) and login[0].isalpha() and (login[-1].isalpha() or login[-1].isdigit()):
        for a in login[1:-1]:
            if a.isalpha() or a.isdigit() or a == '-' or a == '.':
                ret = True
    return ret
def compare(login):
    tm = time.time()
    check1(login)
    print(time.time() - tm)
    tm = time.time()
    check2(login)
    print(time.time() - tm)

Assignment III

There are two tables users and messages (I changed names and messages to non-Cyrillic):

`users`
UID	name
1	Greg Laplante
2	Luisa Portillo
3	Austin Hunt

`messages`
UID	msg
1	Hello, Greg!
3	Send me the card, quickly.
3	I'm waiting on the corner of 5th and Lafayette
1	This is me again. Please message me more often.

Create a SQL query, that would return two fields: User name and Total amount of messages.

SELECT users.name AS "User name",count(*) AS "Total amount of messages"
FROM users
JOIN messages ON users.uid = messages.uid
GROUP BY users.uid

Assignment IV

Suppose you have a generic access.log. How to get 10 most frequent IP-addresses using standard terminal tools? How to do that with Python?

# BASH:
grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' access.log | sort -n | uniq -c | sort -n -r | head -10
# PYTHON:
import sys
import re
all = re.findall("[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}", open(sys.argv[1], 'r').read())
srt = sorted(all, key=all.count, reverse=True)
unq = []
for m in srt:
    if not m in unq:
        unq.append(m)
print unq[0:10]

If you can think of a better way to solve any of these, let me know.

Humble Collection of Python Sphinx Gotchas: Part II

2015-04-03T04:26:19-07:00

Gotcha 1: Release and Version

Sphinx makes a distinction between the release code and the version of the application. The idea is that it should look this way:

version = "4.0.4"
release = "4.0.4.rc.1"

Most project use a much simpler versioning convention, so they would probably do something like this:

version = "4.0.4"
release = "4.0.4"

I’d been doing this myself for some time, until I realized the conf.py file is a simple Python file (no shit!) and it is perfectly fine to do something like this:

version = "4.0.4"
release = version

Yeah, kinda obvious I know. I however missed this (though I’ve been putting much more complex code to conf.py) and some people did either. So, I’ll just leave it here.

Gotcha 2: Download as PDF

Sometimes a PDF download should be provided along with the hosted HTML version. It looks good and people can get a well-formatted file to use locally as they are trying to work with your API or product. In short: it could be done easily, using Python. I’m really pissed off at people, who know Python or Bash and they still keep asking, whether there is an automatic way of doing that. Well, it doesn’t get more automatic than that:

# generating latex and pdf
make latexpdf
# generating html
find 'index.rst' -print -exec sed -i.bak "s/.. &//g" {} \;
find 'index.rst' -print -exec sed -i.bak "s/TARGET/$UPTARGET/g" {} \;
VERSION="$(grep -F -m 1 'version = ' conf.py)";
VERSION="${VERSION#*\'}";
VERSION="${VERSION%\'*}"
sphinx-build -b html . ../$TARGET/$VERSION/
cp latex/$UPTARGET.pdf ../$TARGET/$VERSION/

Note, that there should be the following line, inserted into the index.rst in the example above:

.. &  `Download as PDF <TARGET.pdf>`_

Where .. is the comment syntax, & is here to distinguish the line from usual comments and TARGET is replaced with $UPTARGET which is the upper case version of the project name and the default name of the .tex and .pdf files. It creates a relative link to the .pdf file, which is then copied to the exact same folder, where HTML output is located. I’m not going explain much about the variables, as their sources may differ. In my work I use a python script, with exact same principle (I figured bash example would be more universal) and it gets values of $TARGET, $UPTARGET and $VERSION from a JSON file with a list of targets (more on that in the next example). In the example above, I’m stripping values off the conf.py file. In fact you can use whatever input you wish, even pass the values as arguments. What I was trying to illustrate is the concept itself.

Gotcha 3: Using Scripting to Organize the Sphinx Project as a Multiple Project Knowledge Base

Some of the companies, I’ve been working at had this huge array of active projects, that they wanted to present as a single site, or the whole variety of sites with the same theme, or the single site with PDF version for every first level subsection. Basically they wanted me to create a Sphinx-based knowledge base. Using a simple Python or Bash script there are ways to organize your project any way you want (we’ll use Python this time as it’s closer to what I’ve been using). We’re going to create a site that automatically builds PDF version for each first level subsection (project) and puts it alongside the subsection’s index.html. Basically this is a bit more complex variant of the previous example.

Let’s imagine we have a single Sphinx project with a couple first level sections corresponding to company’s projects, for example: Foo and Bar (give me that medal for originality, yeah). Basically, your folder structure will look like this:

Acme
|  index.rst
|_ Foo
|  |_1.0.0
|    |_index.rst
|  |_1.1.0
|    |_index.rst
|
|_ Bar
   |_1.0.1
     |_index.rst

Yeah, we also have versions. I use the following script for the projects with such layout. Don’t worry, it only looks kinda big. The script is rather simple. Also, I’ve commented the hell out of it so that you could figure it all out.

Note, that you also need to create a targets.json file in the root of your project, containing the following lines (assuming we’re using the structure we agreed on in the beginning):

{
  "foo" : "Foo Foo",
  "bar" : "Barrington"
}

The file will tell the script of full project names and how they correspond to target names (folder names) in the structure. Also you will need to have a temp.py file containing only the info we need for PDF building with most of the target names and version numbers represented as variables for injection (yeah, I know this is hacky, but did’t want to bother with imports, dependencies etc). First of all it should have $VRSN tags:

# The short X.Y version.
version = '&VRSN'
# The full version, including alpha/beta/rc tags.
release = '&VRSN'

It should also have tags in the LaTeX part of the settings:

latex_documents = [
  ('index', '&TRGT.tex', u'&UPTRGT',
   u'ACME', 'manual'),
]

Other than that temp.py may resemble your usual conf.py. The reason is that we use conf.py for HTML and it has preset version and project name values for the project as the whole. So we better distinguish between the file for injections and the main configuration file, so that they don’t mess with each other. Note that if you’ll need to add some additional parameters or a preamble to the LaTeX output, you should do that in temp.py as conf.py is not used for building PDFs at all.

If we prepare the project this way, the script should build PDF’s for every subproject and put them to the subproject’s HTML root. Ideally the HTML version could also be built separately for every subproject (for the right project name/version to appear for every subproject). This script is more of a proof of concept rather than out-of-the-box solution. However if you now understand Sphinx’s capability to extension and automation, you may create projects of any complexity yourself.

Humble Collection of i3wm Lifehacks: Part I

2015-03-21T13:33:20-07:00

Lifehack 1: Locking the Screen

This is kinda a nobrainer, but me myself sometimes look for a place, where to copy some of the syntax (I’m lazy and don’t always keep that in my head), so let’s start with this one. i3wm ships with beautiful and robust screen locker i3lock, which can be launched like this:

i3lock -c 000000

It will lock the screen with black overlay. The problem is, that you wouldn’t be typing this command every time you want to lock the screen. We need to add a shortcut to i3 config file:

bindsym $mod+Shift+Tab exec "i3lock -c 000000"

Now when we press the combination of mod-button (Win in my case) and Shift+Tab - our screen gets locked.

Lifehack 2: Activating/Disactivating the Second Screen

If you use i3wm on daily basis, you probably know, that the second screen is not enabled automatically. You should manage displays manually with xrandr command. If we run this command without attributes, we’re gonna get something like this:

Screen 0: minimum 320 x 200, current 3200 x 1080, maximum 8192 x 8192
LVDS1 connected 1280x800+1920+0 (normal left inverted right x axis y axis) 261mm x 163mm
   1280x800      60.02*+  50.05
   1024x768      60.00
   800x600       60.32    56.25
   640x480       59.94
VGA1 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 521mm x 293mm
   1920x1080     60.00*+
   1680x1050     59.95
   1280x1024     75.02    60.02
   1440x900      59.89
   1280x960      60.00
   1280x720      59.97
   1024x768      75.08    70.07    60.00
   832x624       74.55
   800x600       72.19    75.00    60.32    56.25
   640x480       75.00    72.81    66.67    60.00
   720x400       70.08
HDMI1 disconnected (normal left inverted right x axis y axis)
DP1 disconnected (normal left inverted right x axis y axis)

On some rare occasion writing something like this would not be a problem:

xrandr --output VGA1 --right-of LVDS1 --auto

Well, I know you’re probably well aware of how xrandr works. Just in case.

It may get pretty redundant if you use second screen on daily basis and regularly unplug it from your laptop. The best way to go would be to add script shortcut to your /usr/bin/ or /bin/ directory. Run the following lines:

printf '#!/bin/bash\n\nxrandr --output VGA1 --right-of LVDS1 --auto' > /usr/bin/screenswitch
chmod 755 /usr/bin/screenswitch

We can use screenswitch command which doesn’t make it much easier. What would certainly help us is a key shortcut, so let’s add a line similar to one in the previous hack to our i3 configuration:

bindsym XF86Display exec "screenswitch"

Now when you press on your special key combo (Fn+F7 on my ThinkPad), you enable/disable the second screen. Try some other key combination if you have no special display button. Of course the script itself is pretty basic and it would work only if your screen works well in auto and you use the same second screen daily. However, there are more complex scripts available all over the web (example).

Lifehack 3: Locking the Screen on Wake

If you use pm-utils with your i3wm setup, you’ve probably noticed that the screen is not locked, when the laptop is awakened after suspend or hibernate. It’s very insecure. Let’s try to fix it. Create file cat /etc/pm/sleep.d/91blocker and add the following lines to it:

#!/bin/sh
case "$1" in
        thaw|resume)
                su youruser -c '/usr/bin/i3lock -c 000000'
                ;;
        *) exit $NA
                ;;
esac

Don’t forget to change youruser to your username. Now let’s make sure we have all the right permissions:

chmod 755 /etc/pm/sleep.d/91blocker

Now, if we run pm-suspend or pm-hibernate our screen is going to be locked on wake. This script has one shortcoming though: it doesn’t lock the screen instantly, so you may see stuff for a couple of seconds before it gets locked. If it is not a critical issue to you, feel free to use it, othrewise you may need to work on it or find a different solution altogether. If you have any ideas how to improve it, let me know.

Short History of my Relationship with Lenovo ThinkPad

2015-03-14T12:47:10-07:00

For those, who have limited time, this post comes down to the following statement: Lenovo Thinkpad X201 is the best X-series Thinkpad created yet. What follows is my attempt at proving this point with my merely anecdotal evidence. I’m funny like that. Here comes a short story of my relationship with Lenovo Thinkpad X series.

In 2013 I’ve been working as a technical writer (more technical than a writer actually) in a medium-sized web-slash-mobile startup and the Macbook, they’d given me, failed and I decided to try something new. At that time I got increasingly interested in Lenovo Thinkpad (yeah, it hadn’t been IBM for quite a long time already). A couple of my colleagues had these X220 machines and they seemed pretty solid and professional, especially with all kinds of Linux installed on them (I worked with a bunch of Python devs and everybody used their favorite flavor of Linux). My transition from Macbook to Thinkpad was also dictated by how Macbook wouldn’t let me use i3 WM (which I was completely sold on at the time) as the main WM. So I went to my manager an he approved my order. The problem was I wasn’t really familiar with modern ThinkPads then and ordered 14” model (thought I could use all the extra screen space). I figured any Thinkpad will do, but it was a mistake.

I got T431S which was admittedly quite expensive at the time, but didn’t look like Thinkpad at all. If anything it resembled plastic version of Macbook. It had a rather disguising chiclet (island-type) keyboard, no LED indicators, thinner body and as a result much less ports (although for the record I do understand S stands for slim). The only thing it had in common with the previous generations of Thinkpads was the clit, which was kinda useless without the additional row of buttons, the device actually had no touchpad buttons at all as it mimicked the Macbook-style platform touchpad (awful, awful trend, actually). The hardware was of questionable quality and it gave me lots of headache on Linux ( especially WiFi) despite the ThinkPads traditionally being considered one of the best laptops, when it comes to compatibility with Linux. I worked on this machine until the company went under, and got used to it somehow, but it never lived up to the image of Thinkpad I had in my head.

Even after that I didn’t give up on the Thinkpad series completely, though it clearly went downhill with every subsequent model. My wife got herself X230 at work and as I got to play with the device a bit, I had an impression that this is not as bad as 431S, so as the line moved forward I decided to go in the opposite direction. At that time I started working in a medium enterprise infosec company and they had Thinkpads all over the place, and most of these were the Thinkpads as I expected them to be from the day one. These were X201 models. They aren’t as outdated as the earlier ones but they have all the right features. Here is a short comparison between some of the latest X series models:

	X201	X220	X230	X240
LED Indicators	9 on the front and 3 are mirrored on the back.	3 on the front and 2 on the back.	2 on the front and 2 on the back.	None (!)
Keyboard	Classic	Classic	Chiclet	Chiclet
Ports	VGA, Ethernet, 3 USB, separate ports for mic and headphones, phone line port, ECSC slot.	VGA, Mini DisplayPort, 3 USB (1 USB 3.0), combo audio jack, media card reader slot, ECSC slot.	VGA, Mini DisplayPort, 3 USB (2 USB 3.0), combo audio jack, media card reader slot, ECSC slot.	VGA, Mini DisplayPort, 2 USB 3.0, combo audio jack, media card reader slot, ECSC slot.
ThinkLight (Keyboard Flashlight)	Yes	Yes	Yes	The keyboard is backlit instead. Get your tongue out of Apple's ass, Lenovo!
Clit Buttons	Yes	Yes	Yes	Touchpad is a platform with areas for clit buttons, which is kinda sad.*

* - to be fair Thinkpad X250 actually went back to having hardware buttons, so X240 is not the whole new tendency, but rather disappointing stumble.

So, to sum it up for X201:

There is the right number of LED indicators (X220 and X230 have less and X240 seem to have none whatsoever) and they are mirrored on the back of the machine, which is convenient, when the lid is closed.
The classic Thinkpad keyboard is just right for coding. No trendy chiclet bullshit.
Ports and slots is the area, where the age of the machine shows the most. It doesn’t have any USB 3.0 ports and Mini DisplayPort would be actually nice. Still, it’s much better than X240.
There are two rows of buttons, one for clit mode and the other for the touchpad. Although I work with clit most of the time, I find having an additional bottom row rather convenient, yet I’d probably go with no touchpad at all if that was an option.
Flashlight!

The only problem with X201 model for me is that it’s not available for sale officially anymore (at least where I live), I even tried to buy out my office X201, when leaving the company but they wouldn’t let me. So I found a place that sold used ThinkPads for a reasonable price and bought one from there. This machine is pure magic, and it doesn’t matter that it’s a bit outdated. It has i3 CPU (which is still fair these days), up to 8GB RAM (which is usually enough), extended 6 cell battery makes up for its age (easily gives me 5 or 4 hours of relaxed coding) and overall design hints at the times when the word Thinkpad meant something more than “an ugly plastic Macbook knockoff”. Without much exaggeration I can say, that in 12.5” X line of ThinkPads (at least to me) the X201 model seems greatly superior to anything made before (due to being relatively modern) or after. It’s still relevant today and has the potential of being developer’s muse (fetishist talking) and workhorse.

An update is due: although I still think X201 is one of the best ThinkPad X-series machines, after a heated Reddit discussion X220 seems to be an even better model with all the advantages of X201 (except the amount of LEDs 😏), plus newer hardware and better screen. You probably should consider that machine if you are shopping for classic ThinkPads.

ASUS X102B and second screen on Debian

2015-02-02T10:17:36-08:00

I’ve stumbled upon a very capricious piece of hardware lately, which is ASUS X102B. Basic (very basic) video seems to work out of the box, but there are numerous problems here and there. Especially the second screen, which doesn’t seem to work out of the box. If you run xrandr it will only recognize the “default” output and even that would look pretty much broken. First thing to do was:

sudo apt-get install firmware-linux-nonfree

Actually, it’s a go-to solution (as in first thing to try) to many Debian hardware-related problems, as the distro doesn’t include non-free firmware by default. After that it recognizes most outputs the right way, but it may miss the right modes. If so, you could add your mode manually.

First, retrieve the full information about the mode:

cvt 1920 1080

You will get something like that:

# 1920x1080 59.96 Hz (CVT 2.07M9) hsync: 67.16 kHz; pclk: 173.00 MHz
Modeline "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync

Now use the info to add the mode to xrandr and then assign it to the output:

xrandr --newmode "1920x1080" 173.00  1920 2048 2248 2576  1080 1083 1088 1120 -HSync +VSync
xrandr --addmode VGA-0 "1920x1080"

Use the xrandr command to check the list of availible outputs and modes. You probably know that already, but here is how to use xrandr with the newly created mode:

xrandr --output VGA-0 --mode "1920x1080"

Well, as this one issue is officially resolved I’m off to fight the rest of a couple hundred problems, that arise, when trying to use this laptop with Debian. Wish me luck and leave a comment if you had some trouble with the machine (when used with Debian that is) — we could try to work it out together.

Update on BioAid and my Hearing in General

2014-09-24T02:20:29-07:00

This post was written about a year ago, but it has been lost among the drafts, as I haven’t been a frequent guest here lately. Now it’s time to finally publish this piece for good.

Some time ago I did a big, comprehensive review of the BioAid hearing app for iOS. A couple of months down the road, the situation has changed entirely. I’ve got some good news and some bad news for you. Let’s start with the latter.

Bad News

I’ve joined the dark side and after several months of struggling, I’ve finally bought a commercial hearing aid. It is not a spontaneous decision though, I’ve been thoroughly thinking it over for a couple of weeks. Here is what made my mind up:

I’ve started to notice some discomfort while using BioAid, for the most part it was minor headache and mental fatigue. I’ve been taking some medications, that could have caused this effect too, but I do believe BioAid had its share of responsibility in driving me into this condition. As I’ve been using Gradual HF regime, it could be a little too high frequency in my case. Not high enough to notice that at once, but it had some profound effect on me in the long run. I was feeling ruined by the weekday evenings after the full day of continuous use of BioAid at work, and I was feeling OK on weekends, when I hardly used the app at all. I think it was natural fatugue, combined with the sound irritant of BioAid. I’m not saying that you’re sure to be feeling exactly the same way, but I strongly recommend you to stop using the app as soon as you feel any side effects. You should also consider starting using the app twice if you have any sort of rare medical condition. The creators of the app warn you about that themselves.
Annoying state of being unable to use my iPhone to its full potential throughout the day and implications of using it as a hearing aid in day-to-day situations made me feel quite miserable. If you’re interested in what I’m speaking about, I’ve been writing about all the limitations in the original post. However, it could be alright, if you really can’t afford a hearing aid, or want to use the app as a temporary solution.

As a result, I went to the same center, where I’ve refused to buy an aid in the first place and bought the same exact aid I had been offered back then. It is OK (Widex, by the way), but I’ve not changed my mind completely. I do think that devices like BioAid are the future of hearing aid market, which is really underdeveloped and monopolistic in this day and age.

Good News

Some time ago I got this letter in my inbox:

Dear Vsevolod,

You contacted me a couple of months back about the original BioAid app. I’d like to let you know that I’ve been looking at the hearing app idea again recently and have just released (yesterday) a rather more powerful and flexible piece of software. Check out aud1.com for more details and don’t hesitate to get back to me if you have any questions.

Best, Dr. Nick Clark

Dr. Nick Clark is one of the scientists behind the original BioAid project (the one who wrote most of ObjC code, actually) and Aud1 is his solo project. ~~Yes, basically it’s BioAid 2.0 and it’s paid now.~~ Actually, it’s not BioAid 2.0, but rather implementation of BioAid algorithm, as Nick Clark himself explained it:

I’d just like to clear up any confusion that I may have caused by my haphazardly typed original email! Aud1 is not the new name for BioAid. BioAid is the name of a biologically-inspired open-source gain model. The original BioAid app was a particular implementation of this algorithm (confusingly also named BioAid, but referred to in-house as “the BioAid app”). Aud1 is a much more flexible framework that has been developed independently by one of the original BioAid team (me), and currently runs an optimized version of the BioAid algorithm. However, there are plans to allow the user to switch between various algorithm designs in the future, potentially making Aud1 a useful research tool for field comparisons. Switching algorithms is not like changing the processing strategy on a hearing aid, but rather more like switching out an entire part of the hearing instrument.

Aud1 is a platform for the BioAid algorithm, and potentially other algorithms in the future, allowing it to behave more like the lab scale version that we used (providing features like linked stereo processing if the user has appropriate input hardware). Aud1 is no more a hearing aid than the original BioAid app can be considered a hearing aid, because they are just a software component restrained by the limitations of the devices on which they run. I prefer the deliberately vague term “assistive hearing technology”. Limitations aside, the the BioAid app really seemed to help a select group of people, and this motivated me to push the technology further, adding many features requested by BioAid-app users. Check it out if you like.

I installed the app and field-tested it right away. I was glad to see that some of the annoying issues of the original version were gone. The app features much cleaner interface and more flexible configuration with sliders instead of fixed presets. There are no more welcome popup screens appearing every time on startup and the app seems to preserve the configuration on relaunch.

It also has introduced some new features like an ability to choose bit rate of the output, support for stereo, latency test and input/output calibration. It also provides some basic session info and a logger for the tinkerers. The application now looks more mature and ready for commercial distribution. Although no essential improvement over the original app was introduced, it looks, feels and hears much better, which is enough for me to reach for my wallet. Still, there are issues, that were ignored, like returning to hearing aid mode after a call (as the stock music app does) and some other minor problems. Regimes are the same for the most part (albeit a tad more configurable), hence it haven’t solved my headache problem. Eventually, I’ve abandoned the concept of iPhone as the everyday hearing aid. Again, it doesn’t mean it would not work for you. Give it a try.

By the end of the day I do think that this version is worth every pence, even if you’re not particularly amazed with the new features and improvements. You may consider it a little contribution to an amazing project, especially if you have been using the original BioAid for some time already. After many months of extensive BioAid usage, I was glad to pay it back. Hopefully, you would be too. If you’re completely new to this kind of apps, my advice would be to try BioAid first and see, whether you’re not experiencing any of the side effects and it does help your hearing, then you could easily migrate to Aud1.

A little year-down-the-road update is due. As of now the projects seems a abandoned: last updates on BioAid and Aud1 Facebook pages date back to September 13, 2013. It is quite unfortunate as the project showed big promise. Hopefully, Nick Clark haven’t abandoned this idea completely and works on something new in the same vein. Time will tell.

Humble Collection of Python Sphinx Gotchas: Part I

2013-09-02T08:11:05-07:00

Gotcha 1: Getting Started with Sphinx Theming

Update 02.10.2014: It seems absolutely obvious to me, but apparently it is not that obvious to some: you can’t do anything about your Sphinx theme without some basic CSS and HTML skills. You can only change some of the theme parameters, if the developer was kind enough to add some, but it is well covered in the official documentation.

Customizing Sphinx visuals was somewhat upsetting to me at first, since the process is not straightforward and documentation is scarce. It could be a little bit user-friendlier. However, as soon as you get a grip on the basics, it gets pretty smooth and simple. You may have already tried copying over the default theme only to discover, that there is nothing particularly useful in there for an inquisitive scholar. Only one of the standard Sphinx themes is in fact complete, the rest of them simply inherit its properties to add some minor alterations. This theme is called Basic and it’s a minimal sandbox template, the only theme that could be helpful for getting to the very bottom of Sphinx customization. Later you’d be able to inherit it and create a template, consisting only of alterations, but for a start it’s OK to copy the Basic in its entirety.

Hopefully, you’ve already created a folder for your Sphinx project and initiated it by issuing:

$ sphinx-quickstart

Or you may have an already existing Sphinx project, you want to theme — it’s up to you. In your project folder create the _themes directory and then rename and copy the theme folder there. Basic theme, perhaps, should be located in site_packages folder of your active Python install.

Next step would be to change conf.py of your project, accordingly. First, we should make sure, that the following line is uncommented and correct:

html_theme_path = ['_themes']

Don’t forget to check the value of the html_theme parameter above:

html_theme = 'renamed_basic'

As of now you may start making alterations to the theme. You will find that HTML files in there aren’t really HTML files, but templates (Sphinx uses Jinja template engine) with some staff automatically inserted on build. You can combine these automatic tags with basic HTML tags and as soon as you figure out how it works, you could move some of the interactive tags around, or get rid of some of them altogether. Don’t forget to check whether you’re not breaking anything though. Most visual aspects of Sphinx theme are modified through the main CSS file, which is located in the static folder. For Basic theme it would be basic.css_t. Notice, that t in the extension brings to our attention the fact, that this is a template. Other than that it could be viewed and edited as a simple CSS file. If you’re interested in the values, provided by Sphinx templates and how you cam make use of them, consult the official documentation. If you want the main CSS file to have some other name, you can change that in theme.conf, there are also some other settings, that could be of interest.

Gotcha 2: Spoiler

If you play with Sphinx templates for some time, you may want to start adding something more interactive. We could easily implement these kinds of things trough plain JavaScript or jQuery.

Let’s go trough a little example project, to recognize, what could be achieved with combined power of Sphinx and JavaScript. You may notice, that as for common elements in HTML themes there is usually a way to address them in JavaScript, it’s not that easy to introduce new classes and wrap some parts of the text into them. There is no custom div tag in Sphinx, unless you do that as .. raw directive which is not really a native way, and breaks lots of things. What I needed to do was to wrap some section of text into a div and make it collapsible on button pressed. You could achieve that by creating your own admonition and assigning it to certain HTML class.

For example:

.. admonition:: Request
		:class: splr
 		Request example and parameters.

You can then easily reference this admonition by its class in your JavaScript. For example you could put this jQuery script to your page.html template:

<script type="text/javascript">
$('.splr').css("background-color", '#F0F0F0');
$('.splr').css('box-shadow', '0px 0px 1px 1px #000000');
$('.splr > .last').hide();
$('.splr > .expanded + .last').show('normal');
$('.splr > p').click(function() {
    $(this).find('.last').hide();
    $(this).toggleClass('expanded').toggleClass('collapsed').next('.last').toggle('normal');
});
</script>

If implemented correctly, this script should turn the splr admonition into a collapsible drawer. I’ve been actively using this, when documenting HTTP APIs, since it’s very helpful to hide JSON responses by default. Note, that you could also use this method for special CSS effects. Imagine, if aside from usual note, warning and tip you could have yellow, blue and purple boxes for whatever reason you can think of? Well, it couldn’t be any simpler. Admonitions are good default containers for some parts of your text, that should differ in design or function from the rest of the page.

You may have noticed, that Sphinx often adds TOC to sidebar automatically, if it’s not explicitly placed in the page itself. While this is certainly a very useful feature, sometimes things get out of control. I didn’t use the worst case in the picture on the right, but it could get up to innumerable 1st level sections, each of them could have a number of subsections and so on. Sometimes it is a clear sign, that the page should be reorganized into multiple standalone pages, united by a category, but it’s not always possible or needed. It’s perfectly alright to have a long and deep TOC in some cases and Default Sphinx theme is terrible in that regard (as of 2014 it is fixed and children categories are shown only for the currently selected one).

You could use an updated version of the algorithm from the previous example to collapse some parts of the TOC by default. Note, how this script works with ul and li tags of the TOC tree list. Some stuff is applied recursively on the highest level of the list, some — on the subsequent levels. Especially noticeable with different styles applied to different levels of the list, so that you could tell whether it is 1st, 2nd or even 3rd level title. Here is the full script:

<script type="text/javascript">
$('.sphinxsidebar').attr('position', 'fixed');
$('.sphinxsidebar').css('position', 'fixed');
$("h3:contains('Table Of Contents')").css('border-bottom', '1px');
$("h3:contains('Table Of Contents')").text('TOC:');
$('.sphinxsidebarwrapper li').css('background-color', '#B8B8B8');
$('.sphinxsidebarwrapper li').css('box-shadow', '0px 0px 1px 1px #000000');
$('.sphinxsidebarwrapper li').css('color', 'white');
$('.sphinxsidebarwrapper li').css('border-color', '#000000');
$('.sphinxsidebarwrapper li > ul > li').css('background', '#D0D0D0');
$('.sphinxsidebarwrapper li > ul > li').css('box-shadow', '0px 0px 1px 1px #000000');
$('.sphinxsidebarwrapper li > ul > li > ul > li').css('background', '#F0F0F0');
$('.sphinxsidebarwrapper li > ul').hide();
$('.sphinxsidebarwrapper li > .expanded + ul').show('normal');
$('.sphinxsidebarwrapper li > a').click(function() {
  //hide everything
  $(this).find('li').hide();
  //toggle next ul
  $(this).toggleClass('expanded').toggleClass('collapsed').next('ul').toggle('normal');
});
</script>

Perhaps, it won’t look very well, but it is a very simplified version to illustrate the concept. If you get the idea, you may alter or add .css methods to achieve more plausible visuals. You could also work on lower title levels, but you will have to figure it all out for yourself. In this version the snippet works best only with three headings levels from <h1> to <h3>, but it could definitely handle more.

That’s it. I’ve compiled these examples as a full-featured theme project on GitHub. I’m going to polish it to some extent and perhaps, implement more interactive stuff over time. Feel free to contribute to this little project. (Nope, never happened. Well, it did happened, but it was no good, sorry.) If you come upon any issue with this little gotchas of mine, let me know. Sure, I’ve tested everything myself, but you never know. Also, you’re free to use any of these examples as a building block in your work with no attribution, since they’re rather generic and simple.

On dType Suspension

2013-08-22T00:56:04-07:00

I can safely confess, that a couple years ago I didn’t know a single thing about programming. I was aware of some fairly abstract concepts and had a basic understanding of how it all works, but it definitely wasn’t enough. My English teacher had a saying about the active vocabulary: “You may learn all the words from the dictionary by heart, but unless you use them regularly and naturally, you don’t really know them”. My situation with programming was somewhat similar to knowing lots of trivia, but having no grasp on the practical side of things. I was determined on fixing it as soon as possible. I’ve tried reading a book or two, but it never really got me going. Well, it explained a couple things here and there, but it was like learning things by heart — tedious and irrelevant (on an absolutely unrelated note: Learn Python the Hard Way is great). At that point one of my techie friends suggested me throw the book away and learn by immersion: make an objective, stumble upon problems, see docs and StackOverflow for possible solutions. That was the moment I started looking for the first project, fairly simple, yet more challenging than a mindless Hello World routine.

Once, I was typing down a big portion of plain text on my old slow Android phone, using another memory hog Office Suite, with all those controls, sets of buttons on all sides of the screen and I wished there was something like Focus Writer for Ubuntu: basic, but fairly powerful in terms of achieving that special zen state. There weren’t many such projects in Android market back then (yeah, kids, it was called that in days of old) and this is how the idea of dType has stricken me. The concept was fairly simple: a minimalistic tool, that would let you jot down some text and then pass it to some other application (Evernote, Dropbox, Email, etc.) for saving or processing. It was simple enough to get grip at basics, yet quite challenging for a person, who haven’t seen Java code (or any code) before.

It was the moment, when I started coding. Well, let’s say it was more about googling intensively for just about anything. It was hard. Most of the time I didn’t know, what I was doing and asked fairly inept questions on StackOverflow. I still do, but now at least I can tell, what most parts of my code are for. First, the immersion is like trying to play piano blindfolded — my code probably stunk a big time, but at the end of the day it worked and it was encouraging. Interest in Android development helped me to get a job as a technical writer in a bunch of Android-related projects, especially OpenCV for Android. Since, I’ve been working mainly on C++ API references, I’ve started to delve into OOP concepts. I’ve been thoroughly explained, what is a class, a method and how they relate to each other, interfaces, abstract classes and the rest of this stuff. I’m extremely grateful for my mentors at there. Later, working on some other project, I had a chance to look closer at working Java code and see these concepts applied to Java. I immediately started to refactor dType code once again in attempt to implement thorough OOP design and shake off all the redundancy. My code became a little bit more laconic and neat. Not that it couldn’t get any better, but it was still a huge leap forward for me.

As long as I remember, dType was constantly improving. It was first a bunch of undocumented spaghetti code, which was somewhat straightened out at version 0.16 — it became the earliest version I bother to keep in the repository, since everything before that was a complete disaster. Perhaps, it’s still rather bad, but I’ve managed to shorten it almost twice, provide descriptive JavaDoc (for the sake of it, I know no one will probably bother to read) and fix a lot of issues while at it. I do feel a little attached to this code emotionally, since it is my first coding experience, that has grown into a little indie project of mine. Over the course of two years it has provided me with innumerable challenges and priceless practical experience, but it’s finally time for me to move on. I’ve taken great interest in Python lately, and started a couple of projects in it. Coming back to Java code became more and more daunting to me. I was also advised by several programmers, that I’d better concentrate on getting really proficient with one language for now. My growing frustration with Java verbosity ensured, that I would end up with Python as my language of choice.

Still, it was a hard decision for me to drop dType completely. People do use it and clone it on GitHub (yeah, it had a couple of clones before the project has been moved back and forward, nevermind the actual numbers on GitHub). This project, though certainly quite niche and facile, does work for some. I decided, that this suspension is going to be more of a role shift for me: from active developer of this application to its maintainer. It will stay as an open repository at GitHub for you to clone and alter, it will stay published on Google Play. You can continue to use it in version 0.71. If people provide some relevant pull requests, I would be happy to merge them and even publish the resulting build as the new version of the app. It’s just that me myself don’t have the time or inclination for introducing new features anymore. It is now exactly the way I envisioned it, when I was starting. My big learning project has reached its objective. It’s finished. My priorities have changed, but if you do care, I would be glad to see your contributions. I’m not naive to think that it could become a huge open source project, mind you, but I do hope, that the app could continue living on its own, while I’m pursuing something new.

Update 06.10.2014: No one decided to contribute to this project yet, and probably never will as more than a year has passed since this post. Perhaps, this is for the best, as I’ve seen people complaining about a couple of nasty visual bugs on some devices. Me myself wasn’t able to work with it on Galaxy SIII as the screen gets black from time to time. So if you really want to revive the project, I can only wish you good luck with that. Seriously, if you want to try, give me your contacts, so that I could talk you out of it.

Why Static Site Generators aren't Good for My Blog

2013-08-06T06:24:27-07:00

I would be speaking about my own experiences and you may have noticed my in the title, which is there to remind you about the subjective nature of this article. I don’t dare to speak about your blog or any other blog out there, but my own. I have no intention to convert you to my side, yet I would be very glad to see some of the like-minded people out there. I know they exist. My own research on the topic has unveiled, that although static site generators have this zealous following, there are sober voices in the crowd, appealing to common sense. This is exactly why Kevin Dangoor went “From Wordpress to Octopress and Back” or why Michael Rooney is “Migrating from Octopress to WordPress” — in the exact opposite direction from the majority of switchers.

It doesn’t boil down solely to Wordpress vs Octopress debate, as the issue in hands is much broader and may be represented as dynamic site engines vs static site generators. If you’re not aware of the difference between the two, here are the basics: with dynamic site your content is generated dynamically by an application running on the server side, static sites, on contrary, are pre-generated or written outright in HTML. Basically, dynamic sites are a web-applications that could change their behaviour instantly, depending on input and other factors, while static sites: well, they’re just HTML files passively sitting there, waiting for you to open and read them. Sure, with introduction of jQuerry, Java Script and HTML 5 to the mix, the difference gets a little blurry, but let’s stop at this level.

So, static vs dynamic. It could be virtually anything: Tumblr vs Hyde, Blogger vs Pelican, Movable Type vs Jekyll, etc. Major differences between the two paradigms are more or less the same. It means that we should be really comparing the paradigms themselves, not their instances.

So, what are the lucrative advantages of the static generation model? What makes people switch so quickly and without looking back? I came up with 3 most important reasons:

Almost endless customizability.
It’s mostly plain markup text files, that you can store in a Git repository.
Increased loading speed and security (well, no features - no loopholes, obviously).

All three are valid points and at some moment I went down the static path myself with all three in mind. First I went with Jekyll, then switched to Octopress. At some point I even tried to make Sphinx-generated site (sic!) work as a personal web-page, but lack of blog awareness and increasing complexity made me abandon this idea. When they say that running this kind of site is the easiest things to do, this is complete nonsense. In terms of comfortable workflow, I only can say a couple good things about pure Jekyll paired with GitHub Pages, but the result was so raw and required so much customization, that straightforward workflow was hardly an advantage. It is positioned as a toy for true geeks, tinkerers, but I don’t see how anyone could really benefit from this kind of tinkering (yeah, even geeks and tinkerers). I work in IT and we are here mainly to solve problems, not create tons of complimentary issues. Instead of reinventing the wheel, you could as well invest your time into something, that really needs to be done, perhaps — writing good stuff for your website.

I’m not the only one who noticed it, but for several months I’ve been experimenting with static generators, I’ve hardly written half a post. It was a common problem, as I googled it. People dug so deep into the endless customization and switching routine that eventually they’ve stopped writing. I may sound conservative to some, but I still think that blogging is mainly about writing. Sure, to some extent, this problem is applicable to dynamic platforms too (ping pong between Wordpress and Blogger, anyone?), but with static generators it grows to catastrophic proportions. They are the ultimate time drain for nerds and wannabes. Sure, if your time worth basically nothing and constant tweaking your blog is the best part about having it, be my guest. To me it looks pretty much like that little joke from Fry&Laurie:

Oh, yes, my boyfriend’s a real DIY enthusiast, DIY mad. He’s decorated the whole room and he’s put up all these bookshelves, and now he’s writing all these books to put on them.

Another seeming advantage of static site generators is their reliance on plain text, that could be utilized in distributed version control systems like Git. Most switchers assume, that they already use plain text and Git for code, why not use it for a blog? The problem is, that it also complicates things instead of easing them. Each generator has its own super-easy-workflow™ with different special folders, commands and scripts — quickly it becomes a mess. In this regard Git is basically one more noose on your neck. I’d focus on one especially nasty implementation of Git in such workflow — Octopress. You should fork the original Octopress repo, the _deploy folder will be used for deploy to the pages repository and sources should be committed to the special source branch. Now imagine what would happen if a somewhat major update gets pushed to the upstream of Octopress and your copy has been significantly modified over time. As someone has put it: “Octopress is great, until it breaks”. If you have some experience with Git and seen the Octopress workflow, you may imagine the hell it could possibly be. Actually, Octopress is itself a heavily modified version of Jenkins. No offence, but it all seems like Rupe Goldberg machine to me.

If you google it, you will find lots of people, performing full-featured benchmark tests of Wordpress vs Octopress with a complete disregard for the principal differences between the two. People start speaking of security and speed benefits of static sites, forgetting about all the advantages of dynamic sites, that come at the price of increased complexity and bloat (yes, I do think Wordpress is a tad bloated). Imagine the situation: you need to jot down a post draft on a public or someone else’s PC? Will you be cloning the repo, installing ruby, Octopress — setting up all the environment to write a short post? What about mobile support? Should you attempt to pull Octopress to your mobile phone? What about preserving drafts in the cloud without publishing them, but having access to them virtually from anywhere and anytime? Can you really put a price on that? People start using Evernote or similar service for drafts, but does it really worth introducing another tool to your workflow? Does mobility and availability worth another couple seconds of load time? My own choice is comfort and efficiency. I want my blog to be complimentary to my technical endeavors, not the other way around.

I’ve started thinking that less is actually more a long time ago and it may apply to blogging as to almost any other area of our life. You may notice, that I don’t even use the standalone Wordpress install, but the pretty limited hosted Wordpress site. I prefer to pay engineers at Wordpress $13 for domain mapping and settle for less choice in themes, plugins and other options to focus on writing. We’re all too lost in the world of different platform and workflow options these days. Google it and you will see hundreds of rants about why platform A is deliberately better, than platform B, why static sites are better, than dynamic. You almost never hear that they help you in writing, no. It’s all about SEO, storage space, customization, load speed and other insignificant stuff, not directly related to blogging. We’re too obsessed with form and seem to forget about goddamn content. But, as I said before, it is all entirely subjective. You may still go down the static route, customize the ass out of your blog. You may even spend several years on writing your own static or dynamic blog engine from scratch, that will sure be absolutely unique and different from anything ever done before. Yet, I’m writing this post in a beautiful distraction-free WYSIWYG editor and my draft will be preserved online, when I press the Save button and no rake deploy is needed ever again.

BioAid Meets Life

2013-07-20T02:10:56-07:00

Couple of weeks ago me and my wife went to a hearing aid center for a free consultation before, perhaps, buying a new hearing aid. The previous one has been working just fine for five years or more and it has became unusable lately. In the center they have run all the ususal tests, created an audiogram and offered a couple variants to choose from. I was satisfied with the devices, however I haven’t recognized any major difference from the one, that I already had. I was hoping to manage with $600-$500, and I was shocked when they named their price: even the cheapest of the devices was around $1000. We could afford it, but I declined.

Being a person with inborn defect of the auditory nerve, I’m wearing a BTE hearing aid all my life since the early childhood and I still remember the day, when I put the thing on for the first time. For a boy, who haven’t been hearing the sounds of footsteps on asphalt or the birds twittering, etc, it was a marvelous to discover all those sounds for the first time. Later in life, when I was wearing my fifth of sixth pair of hearing aids this wonderful piece of technology was already taken for granted and I was actively using it in school, university and later — in the workplace. Gradually, I started to recognize quite a few shortcomings of modern hearing aids:

Most doctors would suggest you wear aids on both ears, since it is really good at helping you to locate the sound and experience the stereo or 3D hearing. Wearing two devices may be considered tempting if you don’t do anything else with your ears like using a phone, headphones and participating in all the different kinds of intense activities (sport?), when people may unwillingly flick it off. It’s a physical inconvenience of having something plugged into your ear that is not that simple to take off, but paradoxically simple to drop. People using them day-to-day would understand, what I’m talking about. Headphones and phone would also require you to take the aid off first and don’t get me started on the horrible phone regime, which is available in most modern aids. This was actually how I lost one of my aids: I needed to use a phone, took it off and missed my pocket. Never saw that device again.
Whistling. Yes, they are constantly whistling and it is a curse upon the people with limited hearing. They whistle even more as the plastic ear mold of the aid is wearing off, which means that ideally it should be replaced every year. Whistling is produced by the mic and the speaker in the ear mold since they are too close to each other and the feedback sound is being produced. Ideally the ear mold should hermetically fit in the ear, to avoid the feedback, but at times it sticks out anyway, and when it does — it’s close to unbearable.
Close sourced software and hardware parts. This industry is controlled by a bunch of electronics industry giants (Siemens, Phonak, etc) and they became to some extent monopolistic in this market, since only they had the initial resources to support research and production of hearing aids. Of course they are laying out all the rules now, which leads us to the fourth and the worst limitation of all.
The price. These devices are pricey as hell. It’s a mic, a little processor and a speaker. Yeah, the size is super-small, but it doesn’t add up to $1500-$2000 in my head, sorry. It’s just immensely overpriced. I’m not a cheap guy, but I do have a problem when people feed me up with “magic” and as I work in technology, I know that such claims are almost 100% marketing and outright bullshit. They know that most of us — the hearing impaired don’t have a choice and we’re forced to pay twice as much money for anything they come up with. If you look for the last ten years — they are going round in circles. Hearing aids haven’t seen any revolutionary improvement for decades, compared with the booming technology market in consumer electronics and it’s just sad.

Considering all the above, there is great demand for some open solution in the market. Something you could thinker with yourself and use as a temporary or permanent substitution for a commercial hearing aid. To achieve that, it should be capable not only of recording, amplifying and reproducing the sound, but it also should be smart enough to amplify only some frequency ranges, depending on type and severity of the hearing damage. It would need some computational power to process the sound. Ideally, it should also analyze the sound and get rid of background noise, while normalizing the rest of it (making it quieter or louder depending on the context). Modern smartphones are perfect candidates, since they have everything we need in a hearing aid. I started looking for solutions available as an iPhone app and stumbled upon BioAid.

BioAid is an app, implementing a full-featured hearing compensation algorithm, developed by a team of scientists in the university of Essex. They themselves stress on the fact, that this is not about an iOS app, but the algorithm at the heart of it, which took years of research and continues to evolve today.

Initially, the research was not concerned with hearing aids at all but with the construction of computer models of how hearing works at a physiological level in the auditory periphery.

However, the team has moved to working on hardware models and opted for mobile phones, since commercial hearing aids are almost impossible or too expensive to modify and require an agreement with the manufacturers, which is not that easy to obtain. Smartphones have everything, that a hearing aid needs (a mic, a processor and a speaker), they’re compact enough and modern smartphones have sufficiently long battery life to perform on-the-fly sound processing almost all day. In my case it was a godsend and I rushed to test the app in everyday situations.

First thing I needed to do was to find the most suitable mode. For me it was simple since I’ve done million audiograms and knew that my hearing lacks some of the higher frequencies. After a quick scan I have found Gradual HF mode — the one I recognized at once as it reminded me of how all my aids sounded. My advice would be to start your scan with the first variants of every mode since some of the modes may be too loud or high in frequency and it’s just unpleasant to learn it the worst way. Surprisingly, finding the right mode is not a problem at all. I was afraid the app would require audiograms and it would complicate things. It’s definitely easier this way. Depending on the headphones (they have different levels and may alter the sound a little bit) I was best off with the 2nd and 3rd variants of Gradual HF mode.

I started testing the app in a park with lots of people walking, rolling and skating around. Although it was quite a test to start with I was pretty impressed with the results. It reminded me of times, when I put my first aid on. I heard everything happening around quite distinctly. Frequencies were altered in the right way. Sure iPhone headphones mic has its problems and I’m still hoping to find a better one, but other than that, I had no problems at all. It does reduce a little bit of background noise, depending on Gate value, however I wouldn’t recommend setting it much higher than default settings as it may cripple the other, more critical sounds. The problem with the standard headphones mic boils down to missing out on sounds from behind or on the left (if you have the mic on your right side) occasionally, but it’s not critical. However, if you’re speaking with someone and the person is on the left — it may work a little less precisely than usually. The mic is also quite sensitive to wind and clothing rustle. Due to some lag you can’t use Bluetooth headphones, though. This is an iPhone issue since people watching videos with Bluetooth headphones sometimes notice that too.

Usually, I wear my hearing aid in office, since it is the only place, where most of conversations are critical and may happen almost spontaneously. I was also quite satisfied. I heard everything said on meetings, even better, than with my previous aid, and decided to use BioAid at least temporarily at work. The only problem I can imagine is people’s perception of wearing headphones all the time. Some people assume that you’re listening to music. My office is quite liberal and modern since it’s an IT company and a little Skype chat announcement worked, I can imagine, however, it couldn’t always work this well. Me personally, I find headphones more aesthetically tolerable than a BTE aid, since people are almost constantly wearing headphones nowadays. Another problem is that you may need to buy a battery extender case or enhance the battery life of your iPhone in some other way. My battery is just enough to live through an usual working day, avoiding, if possible other uses of the phone. I only listen to some occasional music on my commute in the morning and after work. If I took the phone off charger at 8 in the morning it is usually almost dead by 22 after the full 8-hour day. Battery life is my biggest concern with the app so far. I had thought of getting a separate iPod Touch player and run the app there as professor Ray Meddis does in this video.

Other minor flaw is that algorithm is implemented in mono, though in theory, stereo implementation is also possible. It is a problem since it may affect your perception of the direction, the sound is coming from. Even if it was processed as stereo, the iPhone standard headphones mic is mono, so the sound is mono by default. Perhaps, stereo would be even worse as a battery drain, so maybe it’s OK the way it is. It is specific to iPhone implementation only and not the algorithm itself. Speaking about iPhone implementation, there are also some minor issues that complicate the workflow: like the app stopping on phone call and not resuming afterwards, or welcome screen appearing every time the app is launched, but all of these are solvable.

Still, for now I’m not even thinking of getting back to commercial aids. I have a very strong impression, that BioAid approach is the future of hearing aids. Especially for people, who don’t have the hearing damage so severe, that they require deep in-canal aids or even implants, which is majority of people with hearing problems. Unloading the sound processing from the aid to a smartphone or similar device (iPod Touch?) may be the right way especially considering the fact, that going from nine to twelve channels adds up at least a thousand dollars to the price of the commercial aid and iPhone has enough computational power to process much more. Sure, there are still some problems, but most of them are of the implementation and are going to be fixed sooner or later.

Algorithm itself is entirely open-source, which means you can fork it on GitHub and create your own version, addressing all of the issues described above, or providing support for some other platform. If you’re a hearing impaired person and you’ve decided to try BioAid for yourself, don’t forget to provide your feedback to the research group, since it may turn out very useful to them.

Update 14.11.2014: I’ve written another piece on BioAid in a year’s time, where I’ve reexamined most of my earlier observations and conclusions.

Google Apps and Household Finances

2013-06-20T18:41:06-07:00

I’m in charge of household finances. By this I imply recording every single money transaction, occurring within our household, and providing on demand information about balance by account, overall household value, etc. I also issue monthly financial reports, so that me and my wife could analyze them and come up with a better spending strategy. When the whole thing works, it provides outstanding insights into the economic condition of our household. When it works.

Let’s assume you want to try it for yourself. Casual Google search will expose you to the vast amount of options, some of them are paid, some — free as in beer. But I dare you wouldn’t be completely satisfied with any of those apps and services:

Some of the superb options out there are probably unavailable in your country, if you live outside of US. For example, the famous Mint app is unavailable in my country without a good VPN.
Some of the projects don’t provide any mobile app or provide a 3rd party solution, which will have all sorts of quirks and unsupported features.
Only a small subset of projects gets it right. Experience is always to some extent limiting even with the paid applications. I’ve never seen a personal finance app that would provide all the features I need, and I’m not among the most demanding users out there, believe me.
You usually have no access to data except for export (if you’re lucky). If your data is somehow lost or ruined, it all happens under-the-hood. You won’t be able to fix the thing yourself, unless you’re a developer and the app is open source.
Platforms may be also the thing. Some developers provide only iOS and Mac binding and other try to please everyone except Apple users. I’m using Mac and iPhone, and my wife has been a die hard Android zealot (just kidding). Since she couldn’t access the service herself, she notified me about every transaction and I placed it into the system on her behalf. Can you imagine how exhausted I was after several months of this workflow? It was all because the app developer didn’t provide any sane way of syncing between Android and iOS. Moreover, when my wife has switched to the iPhone, we discovered that they don’t even provide syncing between iOS devices.

I’m speaking now about Money by Jumsoft. Not only it never really implemented syncing between different mobile devices, but it also failed to provide simple Mac to iPhone sync, the feature that is actually listed on their site. It used to sync through iCloud but when all the drama with iCloud not supporting databases occurred, they went back to Wi-Fi syncing. It never worked right and ultimately it crippled our data. Five months of carefully collected entries for every single transaction gone in a second. It was the moment I started to look for some other approach.

New approach came as an idea of using something simple and omnipresent. Something that would be available for all of the popular mobile and desktop platforms out there. I was thinking of using Google Docs. First, this idea seemed a little bit crazy (it still does), but then as it developed into a working prototype, it was actually a very smart move (still crazy though).

Let’s break the concept down in theory:

We need some kind of interface for creating entries.
We need a database for storing entries.
We need some kind of a script to process the results.

I looked closely at all the products provided by Google, and found all three components for implementing this concept:

Interface is going to be implemented as Google Forms document.
Database is best implemented as Google Spreadsheet. Jumping ahead of myself, I can also reveal that the forms may write responses to the spreadsheets. That’s exactly what we need!
Script was a little bit harder to figure out. First, I was hoping to process everything with built-in spreadsheet functions, but it never really worked for this kind of calculations. So I went with Google Apps Script. More of that in a bit.

It all look quite simple in theory, but in reality there are quite a few pitfalls here and there. I’m going to guide you trough all the major steps of implementing this concept in practice.

Creating the form and collecting responses is not actually that hard. In Google Drive create the form document and populate it with elements. However, there are a couple of minor considerations which may affect your output data to some extent:

Watch the question titles as they directly affect the number of columns and their caps. Section caption is of no interest in regard to the output data though.
Questions do not override. So, if you have question called Amount in one section and a question with the same name in another, it will result in two columns Amount in your table with different values.
Also you may want to avoid nested sections as they complicate things a little.

Here is an example of how you should not organize your form:

Section: Start
Element: Type [Bills, Food]

* Depending on type value go to one of the following:

- Section: Bills
Element: Subtype [Electricity, Cellular, Rent]
Amount: [Number]

* Commit results.

- Section: Food
Element: Subtype [Grocery, Fast Food]
Amount: [Number]

* Commit results.

Above you could probably notice an attempt to override the Subtype and Amount elements. It will result in a table with duplicate Subtype and Amount columns. It’s not that smart but it is the way it is. What I did is getting rid of subtypes and creating only the number of types I would certainly need. For example I have no Bills in types, but Electricity, Cellular, etc. So, I ended up with only one section like this:

Section: Start
Element: Type [Electricity, Cellular, Rent, Grocery, Fast Food]
Amount: [Number]
* Commit results.

I was geared towards creating the system as simple as possible, so I tried to exclude all the nice but potentially useless stuff leaving only the core functionality, that would be harder to break. In menu go Responses → Choose response destination. It will show a dialog window allowing you to choose a spreadsheet for your output data. Quite easy. If you test the form now you can see how the responses are being added to the table in your Google Drive. It even creates the human-readable timestamp column, which saves you the trouble of inputing the date and time yourself. Note, that live form may be accessed as a bookmark, or opened in mobile Google Drive apps for Android and iOS.

Creating the form and gathering the data aren’t exactly the trickiest stages of our little project. Providing somehow valuable analytics on top of that data – that’s the real challenge. Let’s imagine, that we’ve collected all the data and now we want to analyze it. We’ll need some automatically updated metrics for our project:

Household Balance
Balance by Account

As soon as we figure out the algorithm for these two, we can easily implement any other metric (Balance by Category?) using the same method. Note that built-in spreadsheet functions probably won’t work, we need something much more extensive and smart. Here comes the Google Apps Script. It’s basically a full-featured scripting API for simple JavaScript web apps. Google provides the server and the ability to bind the script with the variety of Google Products. If you haven’t heard of that before, there are lots of examples and learning materials on their site — believe me, there is a lot of magic going on over there.

Let’s see how we can apply the scripting capabilities of Google Apps to our case. Open your destination spreadsheet and in menu go: Tools → Script Editor, in the Google Apps Script dialog window select Spreadsheet. We will need a little script, that should run, when the spreadsheet is opened. Example script will leave you with onOpen and readRows functions. You can pretty much start with that.

Let’s see my take on the latter:

In the first part we start with SpreadsheetApp.getActiveSpreadsheet() and end with row values as a 2-dimensional array values. Please keep in mind, that the array is 2-dimensional.

 var ss = SpreadsheetApp.getActiveSpreadsheet();
 var sheet = ss.getSheetByName("Form Responses");
 var rows = sheet.getDataRange();
 var numRows = rows.getNumRows();
 var values = rows.getValues();

Now we should declare all the variables we will need. In my case it was the output array and a separate variable for every account within family, including the cash accounts:
```
 var arr = [];
 var hhld = 0.0;
 var vcsh = 0.0;
 var c4545 = 0.0;
```
Note that hhld here stands for the total household value, vcsh for Victor’s cash, and c4545 — a fictional credit card by its last four digits.

Let’s iterate trough every row and match the entry with the account accordingly:

 for (var i = 0; i <= numRows - 1; i++) {
     var row = values[i];
     if (row[1]=='Victor Cash')
     {
         vcsh = vcsh+parseFloat(row[3]);
         hhld = hhld+parseFloat(row[3]);
     }
     if (row[1]=='4545')
     {
         c4545 = c4545+parseFloat(row[3]);
         hhld = hhld+parseFloat(row[3]);
     }
 };

Now we may append the values to the array and return it:

 arr[0]=hhld;
 arr[2]=vcsh;
 arr[3]=c4545;
 return arr;

Let’s get to the main onOpen() function. It is shorter, but a little more tricky:

We open an active spreadsheet again, but this time use the other sheet, since no one would ever consider mixing data and results a good idea. We also should create two arrays.
```
 var ss = SpreadsheetApp.getActiveSpreadsheet();
 var targetSheet = ss.getSheetByName("Analytics");
 var array = [];
 var data = [];
```
Now let’s call the readRows function and get an array of account balances. Note how we use data[0] to create a two-dimensional array.
```
 array = readRows();
 data[0]=array;
```
Finally, we should get the sheet range and assign data to it. Note that setValues can work only with 2-dimensional arrays and it was the reason we created one in the first place:
```
 var range = targetSheet.getRange('A2:M2');
 range.setValues(data);
```

You can test your app by both opening the spreadsheet or calling the onOpen function directly by clicking Tools → Script Manager → Run. For now the script doesn’t really work with mobile devices, so you will need to open the sheet on your machine to update the metrics. There are innumerable ways you may improve this script. Please let me know if you come up with something cool.

Update 18.06.2013: I’ve found a way to automate the account counters and therefore enable full support for mobile devices. If you follow the workflow described in the post, you would end up with the script running only when opened, however for a spreadsheet paired with a form there is another kind of trigger available: on form sent. It runs the script every time, when the form is sent and unlike onOpen it seem to be performed on the side of Google, which makes our script platform-agnostic. Trigger can be enabled by going to Resources → Current project's triggers. In the dialog window add a new trigger and then select your main function (onOpen), next — From spreadsheet (yes, they have time-driven triggers too) and On form submit. You may test it now by filling the form from your phone and then checking the account counters in a couple of seconds or so.

Update 14.11.2014: We’ve spent some time with this system (a year or so), but eventually I’ve decided to make something more reliable: ~~check out my Le Ménage project on Github~~ (got rid of the project becasue YNAB).

Introduction

2013-06-07T10:51:29-07:00

My name is Vsevolod Glumov.

I’m working now as a technical writer with strong focus on code references and other developer-oriented materials. I’m also an inspired beginner developer, delving into the worlds of Java and Python on my own. You can find links to quite a few of my open-source projects on this page.

I’m not among the people who have recently joined the industry because they heard it may help them to pay their bills. I’m not among the people, constantly laying siege to conference halls in search of another million-dollar startup idea. I’m here because I’m in the state of being constantly amazed by the vast universe of knowledge the 20-year-old industry is now. I want to contribute to this and I want to be a part of it.

Along the way I’m getting stuck and puzzled. Sometimes I’m capable of fixing it myself, sometimes — some wiser guy comes and helps me. Some of these issues were real pearls and it’s a shame that they may be ultimately forgotten. Sometimes I suddenly get an idea of how I can improve my performance and implement them in my workflow. I start to perform better and it’s a shame I have no public space online to tell about this trick. Well, not anymore.

At the end of the day there is no final goal to this blog. Ideally, it should have no end, no real destination, as the learning process that it conveys. In fact, I wouldn’t even call it a blog. It should be more like the journal of the odyssey into the unknown.

Let it be the ultimate mission statement.