In this article, we study the problem of digital pre/postcoding design in multiple-input multiple-output (MIMO) systems with 1-b resolution per complex dimension. The optimal solution that maximizes the received signal-to-noise ratio relies on an NP-hard combinatorial problem that requires exhaustive searching with exponential complexity. By using the principles of alternating optimization and quantum annealing (QA), an iterative QA-based algorithm is proposed that achieves near-optimal performance with polynomial complexity. The algorithm is associated with a rigorous mathematical framework that casts the pre/postcoding vector design to appropriate real-valued quadratic unconstrained binary optimization (QUBO) problems. Experimental results in a state-of-the-art D-WAVE QA device validate the efficiency of the proposed algorithm. To further improve the efficiency of the D-WAVE quantum device, a new preprocessing technique, which preserves the quadratic QUBO matrix from the detrimental effects of the Hamiltonian noise through nonlinear companding, is proposed. The proposed preprocessing technique significantly improves the quality of the D-WAVE solutions as well as the occurrence probability of the optimal solution.