So many years later on a new laptop I had exaclty the same issue. Solution is this simple script:
cat matlab.sh
xdpyinfo | grep resolution
xrandr --dpi 96
xdpyinfo | grep resolution
and it produces
./matlab.sh
resolution: 143x144 dots per inch
resolution: 96x96 dots per inch
So the resolution of 143x144 dots per inch was way too big for matlab en with 96x96 dots per inch it looks pretty OK.