Implicit Interaction takes advantage of the rise of predictive algorithms, trained on our behaviour over weeks, months and years, and employs them to streamline our interactions with devices from smartphones to Internet connected appliances. Implicit Interaction provides users the advantage of systems that learn from their actions, while giving them the feedback and controls necessary to both understand and influence system behaviour without having to rely on an application for every connected device. This is an active area of research and as such presents challenges for interaction design due, in part, to the use of user-facing machine learning algorithms. This paper discusses the challenges posed by designing in accountability for system actions and predictions, the privacy concerns raised by both the sensing necessary to power these predictions and in how the predictions and systems actions themselves can expose behavioural patterns, and the challenges inherent in designing for the reality of machine learning techniques rather than the hype.