The relationships between absolute lymphocyte counts (ALC), drug- related toxicities, and clinical responses remain unclear in cancer patients treated with PD-1 (programmed cell death 1) inhibitors. We performed a retrospective review of 167 adult solid tumor patients treated with nivolumab or pembrolizumab at a single institution between January 2015 and November 2016. Patients with an ALC >2000 at baseline had an increased risk of irAE (OR 1.996, p<0.05) on multivariate analysis. In a multivariate proportional hazards model, a shorter time to progression was noted in patients who were lymphopenic at baseline (HR 1.45 (p<0.05)) and at three months (HR 2.01 (p<0.05)). Patients with baseline lymphopenia and persistent lymphopenia at month 3 had a shorter time to progression compared to those who had baseline lymphopenia but recovered with ALC > 1000 at 3 months (HR 2.76, p<0.05). Prior radiation therapy was the characteristic most strongly associated with lymphopenia at 3 months (OR 2.24, p<0.001). These data suggest that patients with higher baseline lymphocyte counts have a greater risk for irAE, whereas patients with lymphopenia at baseline and persistent lymphopenia while on therapy have a shorter time to progression on these agents. These associations require further validation in additional patient cohorts.