Research apprentice · 2022 – 2023 · Institut de Mathématiques de Marseille (I2M) · 📄 Report (PDF)
1- Overview
During my first year at École Centrale Marseille, I followed a research work-study program at the Institut de Mathématiques de Marseille (I2M). I first learned the mathematical foundations of neural networks, then used TensorFlow to study a precise question: how well can a neural network approximate a function with very steep gradients, and how can we help it?
- Role: Research apprentice (work-study alongside the engineering curriculum)
- Period: 2022 – 2023
- Stack: Python, TensorFlow / Keras, NumPy, Matplotlib
- Reference: B. Després, Neural Networks and Numerical Analysis (De Gruyter)
2- Theoretical Background
A feed-forward neural network is a composition of affine maps and activation functions:
$$f = f_p \circ g_{p-1} \circ f_{p-1} \circ \dots \circ g_0, \qquad f_i(x) = W_i x + b_i,$$where the $W_i$ are weight matrices, the $b_i$ biases and $g_i$ a sigmoid activation. Training minimizes the quadratic loss over a dataset $D = \{(x_i, y_i)\}_{i=1}^N$:
$$J(W, b) = \sum_{i=1}^{N} \big| y_i - f(x_i; W, b) \big|^2.$$3- Experiments
Approximating a one-variable function
I trained a network with one hidden layer of 50 sigmoid neurons (Adam optimizer, mean squared error) to learn
$$f_{\text{obj}}(x) = 50\,x\, e^{-7x^2} \quad \text{on } [-10, 10].$$The fit is good where the function varies slowly, but poor around the sharp peaks.
Gradient-aware sampling
Rather than simply adding more training points, which is costly, I redistributed the training points so that they are denser where the function varies sharply. The local step between two points is shrunk where the second derivative is large:
$$h_k \propto \frac{1}{1 + \log\big(1 + |f''(x_k)|\big)},$$then normalized so that the points still cover the whole interval. With the same number of points, the network approximates the high-gradient region much better (left: uniform sampling, right: gradient-aware sampling):
A five-variable function
I extended the experiments to learning the product of five real numbers in $[-1, 1]$ with a network of two hidden layers of 10 neurons, trained on a 5-dimensional grid.
4- Next Steps
The natural continuation is to generalize the gradient-aware sampling algorithm to functions of several variables.
5- Takeaways
This first research experience gave me a rigorous understanding of what a neural network actually computes, before using it as a tool, and my first hands-on practice of TensorFlow.