I can imagine getting myself into a similar fix. I'd like to think I'd calmly clarify that while I enjoy it I don't get through as many as quickly as I'd like; I'm currently reading blah, and previously blah and blah, but I can't recall the last ten.

Because they're presumably just trying to call bullshit, since it can sound like such an easy probably oft-recomended 'hobby' to say you have, so it's 'oh yeah well what have you actually read recently then', not actually 'I now therefore expect you to have perfect recall over your read catalogue'.