Posts Tagged ‘VictorUriaValle’
[VoxxedDaysTicino2026] Training My Rival in Java: A Deep Q-Learning AI to Play Azul
Lecturer
Victor Uria Valle is a Full Stack Software Engineer at CERN, specializing in modernizing and developing enterprise applications to support laboratory operations. He holds a degree from Universidad de Oviedo and pursues personal projects in artificial intelligence, including game-playing agents. Relevant links include his LinkedIn profile (https://ch.linkedin.com/in/vuria).
Abstract
This article scrutinizes Victor Uria Valle’s endeavor to construct a Deep Q-Network (DQN) in Java for mastering the board game Azul. It delineates the game’s mechanics, Q-learning fundamentals, neural network integration, and training methodologies. Through analytical exploration, it assesses environmental modeling, state representation, reward engineering, and performance implications, underscoring Java’s viability for reinforcement learning without specialized frameworks.
Contextualizing Azul and Reinforcement Learning
Victor commences by elucidating Azul’s rules, a tile-placement game involving a shared board with factories displaying colored tiles and a central area. Players select tiles of one color from a factory or center, placing them on their personal board’s pattern lines—a right-angled triangular grid. Completed lines transfer tiles to a wall, scoring points based on adjacency, while excess tiles incur penalties on the floor line. Code structures reflect this: enums define tile colors, factories hold tile lists, player boards track pattern lines, walls, floor lines, scores, and penalties. Legal moves constrain placements—tiles must match line colors, avoid filled wall positions, and handle overflows. This setup frames reinforcement learning (RL), where agents learn via trial-and-error interactions. Victor employs Q-learning, estimating action values (Q-values) in states to maximize cumulative rewards. Traditionally tabular, Q-learning suits discrete spaces but falters in Azul’s vast state space (exponential due to board configurations). Contextually, AI in games traces to chess and Go, but Victor opts for Java sans PyTorch or TensorFlow, demonstrating accessibility. Implications: Democratizes RL for Java developers, though lacks optimized libraries.
Deep Q-Network Architecture and Innovations
To address state complexity, Victor integrates deep neural networks, approximating Q-values via function approximation. The DQN processes states as inputs, outputting Q-values per action. State encoding vectorizes boards: binary representations for tile presence in factories, centers, pattern lines, walls, and floors. Actions encode selections (factory/center, color, line/floor). The neural network, a multi-layer perceptron, uses rectified linear units (ReLU) for non-linearity, trained via backpropagation with mean squared error loss. Innovations include experience replay—storing state-action-reward-next-state tuples in a buffer, sampling mini-batches to break correlations—and a target network, periodically updated to stabilize learning. Epsilon-greedy policy balances exploration (random actions) and exploitation (highest Q-value actions), decaying epsilon over episodes. Code snippets illustrate:
public double[] forward(double[] input) {
// Input to hidden layer
double[] hidden = new double[hiddenSize];
for (int i = 0; i < hiddenSize; i++) {
double sum = 0;
for (int j = 0; j < inputSize; j++) {
sum += input[j] * weightsInputHidden[j][i];
}
hidden[i] = Math.max(0, sum + biasHidden[i]); // ReLU
}
// Hidden to output
double[] output = new double[outputSize];
for (int i = 0; i < outputSize; i++) {
double sum = 0;
for (int j = 0; j < hiddenSize; j++) {
sum += hidden[j] * weightsHiddenOutput[j][i];
}
output[i] = sum + biasOutput[i];
}
return output;
}
This forward pass computes Q-values, enabling action selection. Implications: Custom implementation fosters understanding, though less efficient than frameworks; suitable for educational prototypes.
Training Methodology and Reward Design
Training employs a DQN agent and trainer. The agent selects actions via epsilon-greedy, executes them, computes rewards, and stores experiences. Replay samples batches, updating the network; target updates occur periodically. Rewards equal score deltas minus amplified floor penalties, incentivizing efficient placements. Trainer parameters include episodes, steps per save, epsilon decay (initial 1.0 to minimum 0.05). Live demo showcases training: loading pre-trained models, executing moves in a terminal UI, observing random explorations (5% minimum) for continual learning. Methodologically, Victor iterates games, refining via replays. Contexts: Balances short-term gains (line completions) with long-term strategies (wall synergies). Implications: Reward shaping critical—penalizing floors curbs suboptimal habits; serialization enables resuming, vital for lengthy trainings.
Analytical Implications for Java-Based AI
Victor’s approach validates Java for RL sans external dependencies, leveraging enums, lists for game logic, arrays for neural computations. Innovations: Pure Java DQN demystifies black-box frameworks, aiding pedagogy. Challenges: Manual matrix operations less performant; state space demands efficient encoding. Future: Enhance UI, explore advanced variants (Double DQN). Broader ramifications: Empowers non-ML specialists in game AI; implications for scalable, interpretable agents in constrained environments.