Update: In my eagerness to announce these workshops I made a scheduling error, incorrectly thinking the dates would be March 15+16 rather than 16+17. As a result I need to move one of the workshops to the weekend before, and since the Intro workshop should happen before the Advanced the new dates will be:

• Saturday March 9: Introduction to Processing and Generative Art
• Saturday March 16: Generative Art, Advanced Topics

Sorry for the confusion! On the plus side the Intro workshop might now be a smaller group which should make it nice and intimate.

I haven’t done any workshops in New York since November, so I have decided to offer my Intro and Advanced Generative Art workshops back-to-back the weekend of March 16+17 on consecutive weekends, Saturday March 9 and Saturday March 17.

The venue will be my apartment in comfortable Park Slope, Brooklyn. As usual I have 8 spots available for each workshop, they do tend to reach capacity so get in touch sooner rather than later. Reservation is by email and your spot is confirmed once I receive payment via PayPal.

The workshops will be taught using the most recent Processing 2.0 beta version (2.0b8 as of this moment), and as usual I will be using my own Modelbuilder library as a toolkit for solving the tasks we look. Familiarizing yourself with Processing 2.0 and Modelbuilder would be good preparation.

Make sure to download Modelbuilder-0019 and Control-P5 2.0.4, then run through the provided examples. Check OpenProcessing.org for more Modelbuilder examples.

Note about dataviz: I know there is a lot of interest in data vizualization and I do get asked about that frequently in workshops. I can’t promise to cover data in detail since it’s a pretty big topic.

If you’re specifically looking for data techniques I would recommend looking at the excellent workshops series taught by my friend Jer Thorp. He currently offers two such workshops, titled “Processing and Data Visualization” and “Archive, Text, & Character(s)”.

Screenshot: Manic Time, time tracking app

Possible resources of interest in connection with the upcoming Self-Ethnography and Data Visualization module at AHO, focusing on tools and inspiration. Please also refer to the lecture note and the articles I put on Dropbox.

Articles / videos

• The Data-Driven Life, Gary Wolf, New York Times, April 2010.
• Wired: Know Thyself: Tracking Every Facet of Life, from Sleep to Mood to Pain, 24/7/365
• Gary Wolf, Quantified Self TED talk
• Jan Willem Tulp: The Process of Creating Data Visualizations
• The Eyeo Festival has a Vimeo channel full of great talks that deal with infoviz topics, including speakers like Jer Thorp, Ben Fry, Moritz Stefaner, Stefanie Posavec and too many others to mention here.

Tools and software

• Two options for auto-tracking everything you do on your computer: Rescue Time and Manic Time.
• QuantifiedSelf.com has a decent overview of available self-tracking tools: Guide / iPhone products
• More obscure but Open Source and quite powerful if the documentation can be trusted: Selfspy, Python-based and runs on MacOS, outputting to a local database.
• Popular social services: FourSquare, Last.fm
• Self-tracking goes well with an Open Source, information-wants-to-be-free attitude: See Open You or search GitHub for quantified self for all kinds of related projects.
• Fitness enthusiasts, runners and bikers are avid data trackers. They drive the personal data tracking market while often remaining blissfully unaware of the Quantified Self utopia: MapMyRide, Nike Fuelband, Fitbit, Snore Lab
• For periodic on-the-run logging of any type of information: your.flowingdata and Daytum offer convenient logging with charts and data export, clearly aimed at data heads. Compare Daily Tracker, which is more aligned with the narrative of self-improvement.
• Evernote is not strictly about data tracking, but it is a very convenient tool for it. Focused on cloud-based note-taking that bridges across mobile devices, your web-based services and even normal office software, it’s a blank slate for any number of uses.
• Services built around calendars and to-do lists: Clocked In, Wunderlist, Basecamp. Not obviously related to data logging, but often relying the same mentality of self-discipline.
• Life Hacker is a respected blog discussing strategies to help you stay productive and in charge of your own life / career / inbox. Data tracking is one such strategy, popular since it promises to provide quantifiable analysis.
• Somewhat related: Getting Things Done (or GTD) is a bestseller book about time-managing, but it also has a large online following that is pretty much a subculture in its own right.
• On a more personal and emotional note, mood tracking was an early hit. It is now increasingly being presented as a possible self-help tool against mood disorders: Emoo.me, Moodpanda.
• Also: Mydrinkaware, Recovery etc. focus on battling addiction issues.

Projects / inspiration: So much to pick from, so little time. Google is your friend.

• Nick Felton’s Feltron Report is a masterpiece of well-designed subjective, quirky self-tracking.
• We feel fine by Jonathan Harris and Sep Kamvar is a classic.
• Visual Complexity is the motherlode of visualizations.
• Moritz Stefaner: Map Your Moves
• Jer Thorp about how to access the NY Times API for fun and profit. Also, the classic Just Landed Twitter viz.
• Brian Staats, data-driven home page
• Onformative: Visualization of Skype networking
• Linkedin Network Map tool
• Career advice? Enrico Bertini interviews Moritz Stefaner about How to Become a Data Visualization Freelancer

USound04Cylinder example from sound-responsive visuals workshop last weekend, see code below

I’ve just announced two more NYC workshops for the weekend of September 8+9:

• Sep 8: Sound-responsive visuals in Processing
• Sep 9: 3D printing and Parametric modeling in Processing

It’s currently looking like my busy Fall schedule will mean that I’ll be doing less of these workshops over the next few months, so if you have any interest in taking one of them this might be a good time.

The first sound-responsive visuals workshop happened last weekend and was a lot of fun. Here are some of the key elements we looked at:

• As it turns out, Minim’s FFT.logAverages() method (which divides the FFT into logarithmic averages) give a far more useful result than the raw spectrum data on their own. Using that as our starting point we built a FFT helper class to act as our source material.
• Since our focus in looking at the sound data is to turn it into a useful parameter for driving visuals, I demonstrated a series of data modulating strategies that give us greater control over the sound input.
• Adding temporal damping of the FFT data (interpolating between old and new values) allows us to control the rate of signal change, which is crucial to make the sound-driven animation match the perceived tempo of the sound space.
• Finally, we used a simple envelope shaper (a 1D Bezier interpolation) to de-emphasize the lower part of the spectrum. Bass tends to be over-represented in the FFT data, so to get a better distribution we can simply tone down the low end of the FFT by multiplying each data point with a modifier dictated by the shaper function. Initially we also scaled up the top end, but that produced noise and artificially high values, so in the end we kept the shaper function within the [0..1] range.

See the attached Processing sketch for an example that we went through in detail. The libraries in the “libraries” folder must be copied to your Processing libraries folder before running.

Download: USound04Cylinder.zip (includes Modelbuilder and ControlP5 0.5.4, requires Minim)

Update, April 2013: Getting Fitbit intraday data is possible through a back door of sorts. Clay Gregory’s jFitbit library makes this easy: https://github.com/claygregory/jfitbit.

I wrote UFitbit for a data workshop, it’s a simple Processing library that sits on top of jFitbut and facilitates batch downloading of Fitbit data for local storage. UFitbit comes with examples showing how to download from Fitbit and later to access the locally stored copies for visualization purposes.

Source and binaries can be found on GitHub:

UFitbit: github.com/mariuswatz/teaching/
Requires Processing 2.0b and Modelbuilder.

I just bought a FitBit activity tracker, thinking it could be a fun toy that I could use as a test case in workshops. At $99 it seemed like a decent deal, and from what I’ve heard it’s a pretty decent piece of technology with smart web-based statistics.

So far so good, I’ve worn the device for one day and as predicted it’s mildly satisfying to see logs confirming my pattern of sedentary desk life interspersed with occassional dog walking. Happiness all round.

But then I log on to the FitBit web site (which looks good, btw) and notice a button saying “Premium”. Curious to see what services FitBit considers worth paying an extra $49/year for, I click on it only to discover to my horror that data export apparently is one of those services.

Seriously? What the hell did I buy this thing for, if I can’t get the data? It’s a data tracking device, people.

I guess it was naive of me to believe that the data I generate would belong to me. It must be my Open Source gullibility getting the better of me. I should have known, data capitalism trumps free information any day.

On the plus time I still have time to return this cute little gizmo. It does seem like a good product, but if I can’t get the data without paying extra my enthusiasm is already gone. I’m not training for a marathon and I’m not enough of a narcissist to want to compare my stats to my FitBit “peers”. I really just wanted the data, and I’m too cheap to pay $49 for it, if only on a matter of principle.

Thanks for being yet another entry in the rogue’s gallery of data hoarders, Fitbit. You could have gone the way of the light, but you were tempted by the path of darkness. I realize you have to make money, but people’s data is the wrong place to start.

Ok, so it turns out there are some proper FitBit hacks out there, including this little Google Apps hack by John McLaughlin (LogHound on GitHub) that seems capable of auto-retrieving your data hourly. I got it running with Google Spreeadsheets but obviously haven’t tested it over any longer interval. This still only gives daily updates, which is a far cry from the minute-by-minute granularity I would like and which the device clearly supports. Turns out FitBit has an API called API-Get-Intraday-Time-Series which would give you that dataset, but it’s part of what they call a Partner API which is restricted to a case-by-case basis. So given good karma (or a lucrative app proposal) one could potentially get the data.

Hacker friend Qdot has taken a more low level approach with libfitbit, going straight to the tracking device and communicating with it over USB. This approach seems promising, but I don’t have any Linux boxes around and there’s no Windows support for now. In any case that’s not a likely avenue for students unless they’re on the seriously techy side, and my goal with the FitBit was to find a student-friendly tracking device.

The search continues, but at least I’m considering keeping the darn thing. That would save me from kicking myself for having to pay return shipping and a $15 “restocking” fee just to recoup a few dollars if I did decide to return it. The fact that FitBit at least has an API up and running helps a little, even though I’m unlikely to recover my enthusiasm for their business practices.

I revisited the code I posted for reading WordPress WXR files in Processing, adding some nice enhancements:

• Parsing is now handled by a WPPost class
• Posts are displayed sequentially on the screen
• Previously exported posts are deleted before a new export is started, using a function to delete all files in a directory (a handy hack all to itself)

Instead of reposting the code here I’ve uploaded parseWordpressWXR to OpenProcessing, a practice I intend to follow in the future since OpenProcessing makes the code more accessible and available to a larger community. I have embedded the applet version below as an experiment, hopefully it works for most readers.

In other news: OpenProcessing just went semi-pro this week, adding a Plus membership structure that gives a few perks and allows users to support the development of the OpenProcessing service. I’m sure some who read the announcement greeted it with skepticism, but I tend to believe community services work better when those developing them have a financial incentive to do so. Sinan Ascioglu has been putting some long hours supporting OpenProcessing and I for one am happy to support the project through a Plus membership.

Read the rest of this entry »

Update: I have posted a more refined version of this code: parseWordpressWXR.pde.

Impressed by Jer Thorp’s quick dissection of BLDGBLOG text data at On The Verge at the SVA last week, I sat down to see how I could get text out of my WordPress installs. The result below is the result, using files exported from WordPress in their custom WXR XML format.

Processing’s XMLElement easily parsed the WXR file, which is basically a custom RSS format and hence valid XML. From there it was easy enough to extract simple parameters like title, post id and post content. As a simple test I export each post as plaintext files, which could later be used for text analysis. One tool for such text analysis is Overview from Associated Press.

Adding code to extract categories, tags and other metadata should be easy enough. The WXR also contains post attachments like images and other media, which could easily be identified and downloaded using permalinks. The code below only deals with items which have a “wp:post_type” content of “post”.

The code is fairly basic, but should nonetheless be useful for anyone wanting to do any kind of text analysis on a WordPress blog. One would need access to a WXR file, obviously.

Read the rest of this entry »

Update: After this was posted it got blogged on Creative Applications and I’ve received quite a few responses via Twitter. See the separate post “Screensaver Culture – Twitter responds” for a summary, as well as some further thoughts on the demise of the screensaver.

• Your task is to come up with a concept for a screensaver that is both suitable to the screensaver format and updated to a 2012 understanding of interaction design. We are looking for ideas that go beyond the traditional screensaver format, or which reinvent that format by applying design thinking to a field full of visual cliche.
• Two general directions are suggested (but not required):
  1. Ambient data gadgets – screensavers as data aggregators and visualizers.
  2. Computational graphics – parametric visuals.
• If your idea is too ambitious to realize in a 3-day time frame we want to see convincing screen mockups of how the screensaver would work. But we would rather see a real demo that’s rough around the edges than a slick Photoshop sketch. You must submit at least one Processing sketch illustrating part of your screensaver’s functionality.

• Yuji Adachi: Briblo – Lego style screensaver
• Twistori, visualization of tweets with certain keywords
• Smashing Magazine: Screensavers – Best Of (2007)
• Kuler Screensaver showing latest Adobe Kuler color palettes
• SETI @ Home distributed computing screensaver
• Scott Draves: Electric Sheep, massively distributed fractal image generator
• Twingly, blogosphere visualization
• Terence M. Welsh (Really Slick) – OpenGL 3D screensavers
• Stefan Trifan: Weather, City
• GeocodEarth, Flickr, GeoRSS mashup

I wrote a useful piece of code during the recent Shakerag workshop that makes it easy to save and load parameter data from text files. To celebrate the recent launch of the new Processing Wiki I have added it there, but for completeness I will also post it here.

Use Data.beginSave() to initialize data string collection, then add data with Data.add(). To write to file, use endSave(filename). I’ve included code for auto-incrementing filenames, it’s used in the example code.

To load data, call Data.load(filename), then use readInt(), readFloat(), readString() etc. to get values from the strings read from the text file. Used properly this should give you most of the flexibility you need for simple data saving.

Read the rest of this entry »

KBGAnimD01EB 0008, still from new sound visualization

Ever since doing Stockspace project it seems I am getting asked to do data-related work. This despite the fact that my personal interests diverge from such masters of insightful infographics as Martin Wattenberg, David McCandless or Jonathan Harris.

Suffice to say that I am more concerned with exploring data structures as spaces than I am with providing new understandings of the information contained within them. Manuel Lima’s Information Visualization Manifesto calls for a seriousness on the topic of data treatments, while my projects remain comfortably frivolous.

Recently I’ve been working on a project that has required researching data sources and adapting them to illustrate a bigger idea, which has led to much Googling in the absence of good data from the client. Sometimes you find the right thing immediately, but sometimes data is hard to find in a format that is freely available and easily parsable. Since I have found some good sources I thought I’d share them here…

• PVWATSS calculator from Renewable Resource Data Center (RReDC) provides theoretical calculations of yields (hour-by-hour, 365 days/yr) from photovoltaic solar panels for most international locations.
• World airport codes with geo name data, helpfully spidered by Wiebe Cazemier so you don’t have to.
• Unattributed FTP depository of geo-related CSV files, including postal and ZIP codes, world city info etc.
• Historical weather data from Weather Underground, providing year-by-year hourly information for most cities in the world. (See also this list of weather APIs.)
• Yahoo Finance offers downloads of historical stock price data (CSV) for just about any stock symbol out there. Sadly I have yet to find an open intraday stock data source, due to the proprietary nature of that kind of information.
• The Guardian’s Datablog is a recent venture, and I have yet to use any of the data sets they provide. But I like their approach, which is clearly aimed at democratizing data with a view to public agenda.

I would be interested in hearing tips about any great data sets out there, particularly interesting time series data.

Miska Knapek recently sent me a link to a source of weather sensor data from Helsinki, including measurements of wind direction at the top of Helsinki’s Olympic Tower in 5-minute intervals. He has already made some wind visualization videos and some fabricated wind data sculptures based on this data.

I’ve uploaded some HD videos from the Stockspace series I did a while back. You can see the whole set on my Flickr: Stockspace set. There you can also see them in proper HD resolution, which helps preserve finer details of the geometry.

These were originally created for use as TV spots for Knight Capital Group. Most of the time they get aired on financial networks very early in the morning, since traders love preparing for the opening of the market by obsessively watching news.

They were also shown on the NASDAQ screen in Times Square once, on an occasion where the Knight CEO rang the morning bell to open the NY Stock Exchange that day. As a recent immigrant to New York that certainly felt gratifying.