In this PhD thesis we will present some new insights in different problems in the field of stochastic processes. A stochastic resonance system is studied using path integral techniques, originally developed in quantum field theory, to recover the optimal means through which noise self-organises before a rare transition from one potential well to the other. These results allow one to determine precursors to a rare events in such system.We then study the survival probability of an autonomous Ornstein-Uhlenbeck process using the asymptotic matching techniques developed in fluid dynamics. Here, we obtain a simple analytical expression for this quantity that exhibits a good agreement with numerical determination.Next, rare events in similar systems are studied using a recurrent neural network to model the noisy part of the signal. The neural network facilitates the prediction of future noise realisations and hence rare transitions.Using a combination of analytical and numerical techniques a low-dimensional model is constructed and it is able to predict and to reproduce the main dynamical and equilibrium features of the El Ni\~no and Southern Oscillation (ENSO), the largest inter-annual variability phenomenon in the tropical Pacific which has a global impact on climate.Using the results obtained for the survival probability of the Ornstein-Uhlenbeck process, an approximate analytical solution for the probability density function and the response is derived for a stochastic resonance system in the non-adiabatic limit.Finally, the Landauer principle is applied to investigate the thermodynamics of finite time information erasure, using a model of a Brownian particle in a symmetric double-well potential. Analytical tools are derived to calculate the distribution of the work required to erase information through an arbitrary continuous erasure protocol, and the theoretical findings are numerically validated.