Dec 31

Solving Sneaky Audio Issues

Over the holidays, I’ve spent quite a bit of time tackling some very tricky audio issues in the rock lab. Issues that have been plaguing me for quite a while, but I didn’t have enough dedicated time to solve.

I have intentionally created an all-digital studio, and although I also have a turntable and LP’s because I love the sound of warm dynamics of analog, for the jams and recording that I do, the digital realm is infinitely more convenient…but it has it’s own set of challenges. :)

Challenge 1: Quiet Strat
Strat sounds awful via POD HD 500. My Red Strat (a Squier Stagemaster from the 90′s) is a Humbucker / Single / Single guitar, and has always sounded lovely and hollow like a typical Strat. But connected to the POD HD, it sounded weak and sad. I tried everything I could think of in the patch settings on the POD to get it to sound halfway decent, and it just never did. So I put it aside, and pulled out the Fretlight – it has fancy Shelby Designs pickups, so I figured it should sound fine. Nope. Same issue. I spent hours searching for pickups and such to find something decent…but just didn’t have the $$ to try it on a whim. So, I pulled out the blue Stagemaster (an Ibanez RG clone, Humbucker/Humbucker), on which I’d installed a Alumitone Deathbucker pickup. Sounded OK, but there were still issues. I put the whole thing aside for a couple days, and then came back and pulled the POD out and set it up out from under the desk. And lo and behold – the “Input Pad” switch was on. There was a similar switch on my old POD X3…and I had it on most of the time, but I hadn’t seen it on the HD500. They’d moved it to the top of the unit, I didn’t notice it since it was under the desk. I flipped it off of Pad and lo and behold everything sounded wonderful. Must be a stronger DB reduction than the X3 version was. Great! Except for this strange high-pitched ringing after every note.

Challenge 2: High-Pitched Ringing from POD HD 500
I have the POD connected to my MOTU 896mk3 via a S/PDIF digital cable. It saves me running more audio around, and bottom line the POD and the MOTU are both all-digital inside, so if I ran analog from the POD to the MOTU, it would be going Guitar (analog) -> POD (digital) -> Audio Cable (analog) -> MOTU Input (analog) -> MOTU mixer (digital). Saving a few A/D->D/A steps makes sense. Plus I get stereo with only one cable, and can record at very high sample rates if need be. However, there are some tricks to making S/PDIF sound right. Trick #1 is to make sure that the POD Output and the MOTU input are set to the same settings. Which I had done when originally setting it up. And if you don’t have it right, typically you hear nothing but horrible digital noise – so pretty obvious. In this case however, the settings on the MOTU side had changed (probably when I did a full reset on it a while back), and it was just slightly off – the MOTU side was at 44.1khz sample rate, and the POD at 48k. I flipped the MOTU to 48k, and voila – high-pitched ringing was gone. Looks like it was a subtle form of digital distortion. To figure this out, I listened to the guitar direct into the MOTU – no noise…I listened to the headphone output from the POD – no noise. So pretty much had to be the digital cable – checked both ends, and voila. Also, when the POD is not on (power unplugged), but the mixer is, I get whooshy digital noise coming into the cable. So I have to mute that input if the POD is unplugged.

One thing that kept eluding me here – since I run the Mac’s digital audio out into one of the ADAT ports on the MOTU via a TOSLINK cable, you have to use the Audio/MIDI Setup application on the Mac to make sure your sample rate from the Mac matches the MOTU as well, otherwise IT will have the strange distortion on it as well. Everything has to match! In my case, I set everything to 48k.

Challenge 3: DVI Audio Whine From Yamaha THR-10
I love the tones and feel of the Yamaha THR-10 – and it does have a USB output – but that output doesn’t play well with the MOTU mixer…unless I run some routing on the Mac side to take the USB audio and route it back into the mixer. Which I’ve done…and it just doesn’t sound right. So I’ve been running the headphone output of the THR into an analog input on the MOTU. When I run it with the power supply plugged into the THR, I hear the DVI Audio Whine from the various computer equipment through the headphone output. If I wasn’t having to amplify the output of the THR a bit (with the gain knob on the MOTU input channel) to get the levels to match, you’d probably never hear it, but with things turned up, it’s super-annoying. Bottom line, I need to run a balanced cable from the output of the THR into the MOTU to shield it from the RFI. However, the stereo headphone on the THR won’t support that. So I’ve ordered a unbalanced->balanced converter to try as a last resort. It will also be useful for connecting my turntable to the MOTU at some point, so it’s not a one-trick pony. :)

So there you go, three ‘simple’ problems that took me a dozen hours of hair pulling to sort through…I hope if you had any of these issues that this helped you!

Oct 14

Podiobooks Gondor-Hosted Performance Analysis with NewRelic

So Podiobooks.com is finally stabilizing after our initial push to get the critical features up and working again.

While we are still formulating the best plan to add the features that require some sort of user authentication, the ‘anonymous’ features have stabilized. One of my major concerns from our emergency launch was performance. While we’d been cooking up the Django version of the Podiobooks codebase for three years, performance tuning was hardly our biggest concern.

So when we set up our Django hosting at Gondor, I opted for a pretty big setup – two dedicated instances with 1GB of RAM each, with Django/gUnicorn app servers running on one, and the Redis cache/Postgres database instances running on the other.  While I think that Gondor’s prices for such instances are very good. Podiobooks is a site that primarily subsists on donations, so the lower we can get costs, the more money we can give to the authors and the folks that keep the site running.

To try and get a feel for how the site is performing, I installed the NewRelic application performance monitoring suite on the Podiobooks production instance. With NewRelic set up as a filter on top of the Podiobooks wsgi.py, it has amazing powers to analyze pretty much every aspect of your application’s performance, from the time it takes the browser to load the page, process the DOM and load assets, to the time it takes database queries to run. For queries it sees as running slowly, it automatically runs an Explain Plan on them, so you can quickly determine how to optimize them.

Here’s the chart that I find the most interesting. Along the Y axis is the response time of the application – how long it took to process the request and return data to the browser. This is the purest measure of your app’s performance, since it only includes your code, not the impacts of the network, their browser, loading images, etc.  We’ll look at that impact in a minute. For now, take a look at the X axis. This shows the number of requests handled per minute.

So, why is this important? In short – it shows clearly that the more requests per minute that Podiobooks is getting, the better the response time is. So, we’re not getting swamped with requests and getting slower the more people that hit the site.  This is super good news.

You might wonder how it’s possible that the performance is better with more simultaneous hits, and the answer is caching.  The Redis cache is set to last 5 minutes for most pages right now, so if you get a lot of hits within a 5 minute period, few of them will have to wait for the page to get cooked up by the database and app server, they just get a cached version streamed out of Redis back to their browser.  As requests slow down, the chances that any given user is going to get a ‘stale’ page that has to get refreshed and not just served out of the cache increases.

You can also look at the dot color to see that right around 8PM mountain time is when we get the highest simultaneous traffic to the site.

Once thing that we’ve noticed looking at the Google Analytics traffic to the site is that in terms of pure hits to the site, the iTunes Music Store is by far our biggest ‘user’. Since most of the titles on Podiobooks.com are also listed in the Music Store (as podcasts), the Music Store crawler is regularly checking on all the feeds to see if anything has changed. So making those RSS feed views as low-impact to folks browsing the site as possible was important.

Unfortunately, when I first looked at the ‘Slow SQL’ display in NewRelic, the queries that were underneath the RSS feeds were some of the most impactive. I had spent zero time optimizing those views and queries, and yet the vast majority of hits to the site were going through them! Luckily, a quick application of Django’s ‘select related‘ smoothed out that issue.  Long-term, we should probably be caching those views longer than 5 minutes.

Here’s the database-only equivalent of the application report above:

Query time is pretty flat with load still…again likely due to caching, both at the Django level, and natively within Postgres.

And here’s one for just the CPU time being consumed:

Good news all around. While I’m of course helpful that we can get the number of users on the site to increase to the point where we’d need to add more capacity…right now I think we have too much capacity, and can likely save some money by going to down a single dedicated instance.

Finally, if you are interested in the total time it takes to load pages, this graph covers that:

The tan color is how long it takes from when the network is done loading the HTML to when the browser declares the page to be loaded (DOM Ready), then then teal is the time from that point until the end of the ‘load’ time in the browser, so after all the images are loaded and such. Pages that have fancier CSS calculations, more images, and more JavaScript take longer in that teal zone.  Since that describes most of our pages, it’s the biggest contribution to load time. It’s also the least noticeable to most users, since the page is ‘doing something’ during that time.

Take note that even though about 1/3 of Podiobooks.com traffic is from mobile devices (often on 3G or slower networks), the network time is rarely a factor compared to the page rendering time. That’s on purpose – the pages have minimal HTML (thanks to @brantsteen), so they load over the network quickly, but then the complex CSS and JavaScript for the responsive layout kicks in, and it can take a second or two for everything to look perfect.

Let me know via Twitter if you have any questions about the site or performance tuning!

Cheers,

Tim

Oct 07

Twitter Weekly Updates for 2012-10-07

  • Mindset you get in when reading academic papers is like the Shakespeare mindset; once you are in it you can't stop talking pompous. #
  • Oh safety training videos from the early 80's. SO HILARIOUS. Oh wait, they are serious. Oooh, Hantavirus training? This is getting creepy. #
  • Celebrating completely hack-free @podiobooks Google results! :) #
  • Interesting thoughts on why posted software professionals' salaries don't appear to be going up, despite high demand: http://t.co/vwjUg5re #
  • This may affect me next week. http://t.co/KCheSxuh #
  • I am in serious need of jam time. I have drums, keyboards, basses, guitars, and mics. I will play any or all of them. Who's in? #
  • To be clear folks, if you want to come Jam with me this weekend, knowing what you are doing is not a requirement. I will be happy to teach! #
  • Class instructor: "Format your paper per APA 7th Edition guidelines". Me: "APA is only up to 6th Edition." Them: "Right." #
  • Me: "APA 6th doesn't call out how to handle this, but what it does say, seems to indicate X." Instructor: "Do it Y, like on this site." #
  • Me: "That site says to do it X." Instructor: "No, do it Y, like the example on the site, not like the guidelines." #
  • Me: "That example says 'this is APA *5th* Edition format'". Instructor: "Just do it way Y. I tire of you engineers and your need for logic." #
  • This is what American higher education teaches you: Think critically, until the teacher tires of you. #
  • Well, One Direction on the iTunes Festival is cute, but I expect some short dresses and hilarious attempted dancing from my brit pop. #
  • I do think @CaptSpectacular could pull off Harry Styles from One Direction as a Halloween costume. #
  • finished Hidden Things by Doyce Testerman and gave it 5 stars http://t.co/CSL1gVe2 #Kindle #
  • Snarky @github comment exchanges are my new favorite thing. #
  • Today's jam time with @Jsquared1975 was excellent. JJ knows how to rock. #
  • School paper complete, @podiobooks calmed down…time to nap, I think. :) #
  • Hmm…latest contingency planning hotline update was "encouraged by the progress." That sounds promising. #
  • Contract extension! Tomorrow is for partying! #
Sep 30

Twitter Weekly Updates for 2012-09-30

  • Just completed a 0.27 mi elliptical workout with @RunKeeper. Check it out! http://t.co/dVm8q3fd #RunKeeper #
  • Beautiful rain, the sound of nothing, the sound of soothing, the sound of gutters, pipes, tubes, windows, wheels, shingles and growth. #
  • Great story of a great-sounding car. "How to Build an American Car" — the making of the 2013 @Cadillac ATS, http://t.co/YumztgSA #
  • From "@CaptSpectacular: I'm 90% through @FinalFormGames ' JamesTown! But not on Legendary, 'cause that game is *hard*!" #
  • Fitocracy app brings an RPG touch to working out | Ars Technica http://t.co/6AX25v8P #
  • From @CaptSpectacular: "Our waiter looks like the sniper from Team Fortress 2. Does he have an Australian accent? And a hat?" #
  • It is interesting to me that http://t.co/ol1XW27l gets > 225,000 visits a day, but that 98% of those visits are to the RSS feeds. #
  • Class homework took a while this week…realizing I should do the next 2-3 assignments in prep for upcoming work stoppage duty. #
  • I wish. RT @gigaom: Hey IT — embrace, don’t stifle, developers’ flight to cloud http://t.co/ZettY1Ke #
  • I-70…you are a cruel, cruel road. What would Ike think of this behavior? #ForShame #
  • Colorado Kolsch from @stmworks – my favorite beer of the summer! http://t.co/O2kmfFLp #
Sep 23

Twitter Weekly Updates for 2012-09-23

  • Put the ol' DSL to the test tonight – two people playing LOTRO, Google Hangout + deployment validation testing. #GotABitSlow #
  • Oddly – it wasn't the network that got slow, it was my Mac, once I killed the hangout it was fine. Odd. 40MB DSL, but still. :) #
  • A successful emergency @podiobooks deployment. 3 years of work came to fruition in 3 days. Thanks @brantsteen, @evoterra and @codeshaman ! #
  • Procrastination: @CaptSpectacular was here since Friday night, and waited till 40 min before we had to leave to announce his homework. #
  • Arrr. #piratedayboxchecked #
  • In a plentiful culture, many are liked by few. In a rare culture, few are all the many have to like, so they adapt or starve. #
  • The Internet makes plentiful culture possible at a scale we are just now glimpsing. #
  • I will admit to being pretty delighted with the @github Issue Tracker. #
  • I just uploaded "Dirty Girl Run 2012" to Vimeo: http://t.co/x6u87V37 #
  • Oh unit tests. You've saved me from myself again. #
Sep 16

Twitter Weekly Updates for 2012-09-16

  • Thanks to cooler weather, I was able to get more overhead storage put up in the garage and scrape peeling paint off the house! #
  • Still not sure what I think of Manning in Orange. #ObligatoryFootballTweet #
  • Of late between the day job and school; I've spent a great deal of time with Word. Mostly cordial; rare cursing. We understand each other. #
  • My favorite tweet from 9/12/2011: "Indifference is an underrated virtue". #
  • Word Doc corruption: conquered. #
  • Sometimes the story is better than the truth. But although the story may haunt you, the truth will kill you. #
  • There is little more heartbreaking than a long-running batch job that fails just before the end. #
  • As usual, @brantsteen brings the heat when its needed. @evoterra owes him many beers. #ButtKicker #
  • I must admit that @github 's "close a bug tracker item with a commit comment" feature makes me squee a little. #
  • 300 commits to the @podiobooks codebase * 6 hrs avg dev per commit = 1800 hrs of work! RT @pb2dev: SUCCESS:pb2-ci-master $300 – #
  • http://t.co/bxOOXNOc #
  • http://t.co/Zj8Pq4CF #
  • http://t.co/4AYwHk0q #
  • http://t.co/tMXqQYAU #
  • http://t.co/pX6hrzAQ #
  • I might have to get prints of some of these vintage motivational posters. #SnarkyButTrue #
  • My breadnbutter! RT @TheAtlanticWire: A guide to the rise of the Portmanteau http://t.co/MAQn3Veo #
  • Alas, Brother/Nephew visit was too short. But, gave Nephew D&D books, so hopefully this winter will be dungeon crawl-tastic! #
  • I know you are saying "Skrillex in the morning and Styx at noon? Isn't like like beer after liquor never sicker?" I say NO IT'S FINE. #
Sep 09

Twitter Weekly Updates for 2012-09-09

  • RT @ForbesTech: Bruce Willis To Sue Apple Over Right To Bequeath His iTunes Library http://t.co/Fg4xFwvF #
  • My 2nd Monday of online classes at @RegisUniversity , and the 2nd Monday in that the online system is down! Coincidence? #IBrokeItAgain #
  • OK BoxBot! It's time to test your mettle! http://t.co/6bzysZdr #
  • The iTunes festival has introduced me to Ed Sheeran. U.N.I. from his "+" album is particularly interesting. #
  • BoxBot: "Don't put my GPS antenna so close to my compass!" Me: "It's just an antenna!" BB: "Fine! That's North!" Me: "That's East!" #
  • BoxBot: "That antenna is freaking me out!" Me: "Hmm, there is a giant magnet on the bottom of this GPS antenna…" BoxBot: "See!!!" #
  • https://t.co/nfJomHzi #
  • An unexpected benefit of Denver Athletic Club membership: a great Table Tennis lesson with a former Iraqi National Team member. ;) #
  • A little Iraqi guy shouting "Move! Move!" while a robot shoots balls at you at high speed is suprisingly motivating, and sweat-inducing! #
  • More iTunes festival tonight. Olly Murs and The Milk. Olly Murs = Wham! + Ska, The Milk = Southern Rock + Soul + Ska. #BothExcellent #
  • Tonight's iTunes Festival highlight was the dubstep beatboxer. Haven't figured out his name yet, but amazing! #
  • Love like a muscle grows/with use and time strength is sown/and when trumpet is again blown/the fibers leap/to hold, caress, ache and weep #
  • Yeah, @EdSheeran is about as much of a bard as can be. Proof of what one man with an audience, passion, stories, and a loop pedal can do. #
  • .@Trello is exactly what I've been wanting to organize information for running an RPG session. Web/Mobile multi-media index cards. #
  • So, which is the worse earworm? The Love Boat theme, or the Price is Right theme? #
  • Hmm. Seen 3 brand-new Bentleys in various parts of town this week. #EconomyPickingUp ? #
  • The school exit signs glow red; perhaps to persuade the students that leaving school is the gateway to hell. http://t.co/4Q5PnNKt #
  • You are responsible, forever, for what you have tamed. #
  • This is what folks that make robots celebrate. Not what was intended, but they'll take it. http://t.co/D60kggeI #
  • Proving that @doycet is only 1/2 hipster, his writing netbook only *looks* like a typewriter. http://t.co/q3EHw4t1 #
  • Uh oh. Action Time. #DragonAge http://t.co/ugQSyvA5 #
  • Dang – watching @DaruJones rip it up on drums with Jack White on the iTunes festival is humbling and inspiring at the same time. #
  • It would be fascinating to be able
    to track my cortisol levels on days when I have to drive in parking lots and ones where I don't. #
Sep 02

Twitter Weekly Updates for 2012-09-02

  • Telcos will suffer because of "subscription myopia". http://t.co/VlA9ESIb #
  • "It's doubtful the gamification of politics will reduce disillisionment" MTV fantasy election game | http://t.co/tH225vLH #
  • Back to school after my quarter off! Online course this time…interesting. :) #
  • What to cats do outside for fun? Kill, and leave the corpses to rot. Cat point of view camera footage, shows the truth: http://t.co/v4v3PdqN #
  • Day 1 of my first online class at Regis. Day 1 of Regis online class system outage. Coincidence? I think not. #Flex #IBrokeIt #
  • "The accumulation of knowledge is a two-edged weapon; it helps to determine what is possible but it may also define what is impossible." #
  • Poodle Moth of Dooooom! http://t.co/S9Y1RQFp #
  • Schoolwork for the week done, time to work on the Robot! #
Aug 26

Twitter Weekly Updates for 2012-08-26

  • If telephone companies really wanted to stop people from dropping their landlines, outlawing robocalls would be a good starting point. #
  • Look what arrived on my Kindle this morning from @doycet ! http://t.co/W2l4hend #
  • Does liking "cheesy" works mean that you like more or less truth in your fiction? To me, the nod to the camera is acknowledgement of truth. #
  • 80lb barbell full squats. 5 reps. x 3 sets. Yep. Definitely getting stronger! #
  • The order you experience things affects your opinion; you tend to like the first option best http://t.co/S21zYg6y @doycet #hiddenthingscover #
  • Her Diary / His Diary http://t.co/oKGR98qi #