Uncertain environment on multi-lane highway, e.g., the stochastic lane-change maneuver of surrounding vehicles, is a big challenge for achieving safe automated highway driving. To improve the driving safety, a heuristic reinforcement learning decision-making framework with integrated risk assessment is proposed. First, the framework includes a long short-term memory model to predict the trajectory of surrounding vehicles and a future integrated risk assessment model to estimate the possible driving risk. Second, a heuristic decaying state entropy deep reinforcement learning algorithm is introduced to address the exploration and exploitation dilemma of reinforcement learning. Finally, the framework also includes a rule-based vehicle decision model for interaction decision problems with surrounding vehicles. The proposed framework is validated in both low-density and high-density traffic scenarios. The results show that the traffic efficiency and vehicle safety are both improved compared to the common dueling double deep Q-Network method and rule-based method.