Lately I have been interested in the process and possibilities of new dataset creation (see this example of a feminist dataset) and what punk ML (coined by Arnab) would look like.
I was thinking about how to use the web tracking and surveillance tools we have learned about for these purposes. For this week’s assignment I set out to make a webpage the re-envisions computer vision to be generated by and labeled by the user.
When people go to this website, they will see a mirror video screen. The website will prompt them to trace their face in as much detail as they want. It will explain that they won’t be able to see their traces, but not to worry, the computer can see them. Then it will prompt them to enter an adjective to describe themselves. Their traced image and their adjective will be captured on the watcher.html side.
The idea is to build an image dataset built, controlled and prescribed the users. What would a machine learning algorithm look like trained on this dataset?
I started by getting to know the code for the remote browser tracking, trying it out and editing it so that it draws instead of moving the div with the mouse.
Then I took a look at the WebRTC code. I wanted to incorporate video, but realized for what I wanted to do I didn’t need to be able to see the video on the other side. All I needed was a p5 sketch, an input box and to track the use mouse movements over that.
So I started by modifying the mouse movement tracking code to create a new div each time to create a line drawing instead of redrawing it following the mouse.
I tested this and it worked (well the first time I spelled one of the variables wrong so it looked like this —>
But then I got it to look like this! And then this!
On the left is what the user sees. On the right is what I see :)
I tested it on myself once I tweaked the appearance:
I tested it on myself and then collected some first samples:
Tom
Alive
Hannah
Messy
Mark
Kind
Veronica
Hungry
Arnab
Hyperactive
It was really interesting to me to see what words people chose to describe themselves when given such an open prompt (you can see the label in the upper left hand corner of the screenshot and I’ve also put it in the caption).
Next will be to make the server save these images and the input into a dataset: images and their labels.
Behind the scenes below…
You can also view the code on github here.
