This book presents fundamental relations between random walks on graphs and field theories of mathematical physics. Such relations have been explored for several decades and remain a rapidly developing research area in probability theory. The main objects of study include Markov loops, spanning forests, random holonomies, and covers, and the purpose of the book is to investigate their relations to Bose fields, Fermi fields, and gauge fields. The book starts with a review of some basic notions of Markovian potential theory in the simple context of a finite or countable graph, followed by several chapters dedicated to the study of loop ensembles and related statistical physical models. Then, spanning trees and Fermi fields are introduced and related to loop ensembles. Next, the focus turns to topological properties of loops and graphs, with the introduction of connections on a graph, loop holonomies, and Yang-Mills measure. Among the main results presented is an intertwining relation between merge-and-split generators on loop ensembles and Casimir operators on connections, and the key reflection positivity property for the fields under consideration. Aimed at researchers and graduate students in probability and mathematical physics, this concise monograph is essentially self-contained. Familiarity with basic notions of probability, Poisson point processes, and discrete Markov chains are assumed of the reader.