You will probably need an audio interface and some 1/4" audio cables to use the audio output of the theremini. I just stumbled across this because I'm looking into buying one, so I don't have one yet and can't confirm this, but that's what it looks like the audio outputs are (left and right 1/4" audio jacks).
As for audio interfaces, have fun figuring out what fits your budget and meets your needs. Since it looks like the theremin is capable of producing two channels for recording in stereo, you probably want to get one that at least has two line inputs (so something like a focusrite 2i2).
The USB is mainly for tinkering with the MIDI controller functionality of the theremini. Which seems like an intriguing feature, but from what you're asking is a significantly more in depth jump than what you are wanting to do right now.