Update for Thursday, 10 March 2011

Virtuoso, deadlocks, prefixing SPARQL and SPARUL queries

Lately Virtuoso has been greeting me with 40001 Error SR172: Transaction deadlocked. These happen during heavy concurrent writes to database. hwilliams in forum suggested to try log_enable(2) to force autocommit of transactions. So I’d want to prefix my SPARUL queries with the following:

DEFINE sql:log-enable 2 

My code is in Python, and it’s using SuRF as ORM in front of Virtuoso’s triple store. So I’d need to somehow tell SuRF to add that DEFINE thing in front of each SPARUL query it generates and executes. Luckily, there’s virtuoso_protocol SuRF plugin by Christoph Burgmer that supports such thing for SPARQL queries. After some minor patching it supports prefixing SPARUL queries as well. So in my python code, the store and session initialization code looks something like:

# config_items have been read from .ini file previously
store = surf.Store(**config_items)
store.writer.define = "sql:log-enable 2"
session = surf.Session(store, {})

Autocommit only makes sense for queries that modify data, so I’m not putting anything in store.reader.define. But I could put some stuff there as well, for example, a directive to turn on some inferencing:

store.reader.define = 'input:inference "uri_of_my_inference_rules"'

And yes, the log_enable(2) thing seems to help, I haven’t encountered deadlocks since turning that on.

Caching → Complexity → Doom

Resisted urge to solve things by the usual “let’s just add more caching”. Caching, sharing cache between processes and machines, invalidating cache–adds tons of complexity. Complexity destroys the fun of hacking and makes brain hurt. There are alternatives to caching like, avoiding work, or grouping work so that cache can have a scope of one function or one HTTP request, or just going shopping. Yay.

hg and git and time machines

I’ve been getting along with bitbucket fine, just pushing and pulling changes from and to my lonely repositories, but as soon as I needed to push a change upstream or merge some stuff back, I’d get confused with merges, conflicts, and many heads–seriously scary stuff. Like I need to operate a time machine and I don’t know what any of the knobs do. Actually, when you think about it, for a code artifact living in VCS, VCS is a lot like a time machine, forks being parallel universes… Anyway, I read some chapters from Mercurial book, and the many heads thing is now much clearer to me. Time well spent.

Other stuff

Also some wrestling with tests, it’s complexity again, as soon as your test suite gets sufficiently big and diverse, it’s a PITA to keep it up and running, and it’s extreme PITA to get it running in new environments.

Been also looking in server monitoring services. You know, all good things come to an end, production server included. Started with registering at Was it up?, this service promises to send me an email in case the server doesn’t respond to HTTP request.

Advertisements

2 thoughts on “Update for Thursday, 10 March 2011

  1. Is it fine to insert part of this on my personal website if perhaps I publish a reference point to this web-site?

Comments are closed.