Dynamic treatment regimes (DTRs), which comprise a series of decisions taken to select adequate treatments, have attracted considerable attention in the clinical domain, especially from sepsis researchers. Existing sepsis DTR learning studies are mainly based on offline reinforcement learning (RL) approaches working on electronic healthcare records data. However, a trained policy may choose a treatment different from a human clinician's prescription. Furthermore, most of them do not consider: 1) heterogeneity in sepsis; 2) short-term transitions; and 3) the relationship between a patient's health state and the prescription. We propose a novel framework, an adaptive decision transformer for DTR (ADT² R), which recommends an optimal treatment action for each time step depending on the heterogeneity of the sepsis and a patient's evolving health states. Specifically, we devise a trajectory-optimization-based module to be trained with supervision for treatments and adaptively aggregate the multihead self-attentions by deliberating on various inherent time-varying patterns among sepsis patients. Furthermore, we estimate the patient's health state by adopting an actor-critic (AC) algorithm and inform the treatment recommendation by learning about its short-term changes. We validated the effectiveness of the proposed framework on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, an extensive intensive care database, by demonstrating performance comparable to the state-of-the-art methods.