A search for pair production of up-type vector-like quarks (T) with a significant branching ratio into a top quark and either a Standard Model Higgs boson or a Z boson is presented. The same analysis is also used to search for four-top-quark production in several new physics scenarios. The search is based on a dataset of pp collisions at root s = 13TeV recorded in 2015 and 2016 with the ATLAS detector at the CERN Large Hadron Collider and corresponds to an integrated luminosity of 36.1 fb(-1). Data are analysed in the lepton+jets final state, characterised by an isolated electron or muon with high transverse momentum, large missing transverse momentum and multiple jets, as well as the jets+E-T(miss) final state, characterised by multiple jets and large missing transverse momentum. The search exploits the high multiplicity of jets identified as originating from b-quarks, and the presence of boosted, hadronically decaying top quarks and Higgs bosons reconstructed as large-radius jets, characteristic of signal events. No significant excess above the Standard Model expectation is observed, and 95% CL upper limits are set on the production cross sections for the different signal processes considered. These cross-section limits are used to derive lower limits on the mass of a vector-like T quark under several branching ratio hypotheses assuming contributions from T -> Wb, Zt, Ht decays. The 95% CL observed lower limits on the T quark mass range between 0.99TeV and 1.43TeV for all possible values of the branching ratios into the three decay modes considered, significantly extending the reach beyond that of previous searches. Additionally, upper limits on anomalous four-top-quark production are set in the context of an effective field theory model, as well as in an universal extra dimensions model.