How to use Matlab to fill in some fields in a website, execute a function of the website and then return a result field?

I have a concrete question. I want to use matlab to automatically calculate the driving distance between two addresses. Here is a website that provides such calculation:

http://christianvarga.com/driving_distance.html

How can I use matlab to input the fields "start" and "end" and then execute a click on the "Calculate Route" button, then return the "Distance" field result?

Answer 1

[Marc Jakobi]

I wouldn't recommend internal Matlab functions such as urlread() and urlwrite(). Instead, take a look at cURL:You can use the function system() to control cURL in Matlab. If you don't have experience with the cURL syntax, you can use the Firefox plugin firebug to generate cURL commands.

Here's a rough example of how you would do it:

1. Download cURL from here (You will either need a Win32 version or a Win64 version, depending on your operating system's architecture).
2. Extract the command line tool and put it in Matlab's current directory.
3. Install Firebug for Firefox
4. In Firefox, navigate to the website you would like to automate the commands for.
5. You need to activate Firebug with the plugin button on the toolbar. A window will pop up below.
6. Fill out the form/s and hit send or search or whatever to send off the form.
7. In Firebug, under "console", look for POST requests. Right click on them and "copy as cURL".
8. Past the command into a text editor (I recommend Notepad++) that has a "find and replace" feature. The Matlab editor is sufficient for this task, too.
9. Replace all ' in the command with " (the Linux version of cURL uses ' while the Windows version uses " in it's syntax)
10. This is where it gets tricky: You need to analyze the cURL code. In Matlab, copy and paste the cURL commands variables in Matlab. Use the concatenation function to make the text more readable.

cmd = ['-curl "https://www.example.com', ...
  /more_text_here', ...
  /even_more_text_here']

Search the command for the text you put into the forms (WARNING: Be careful not to accidentally reveal any login information or other sensitive information!)

You can use concatenation to turn those strings into variables. For example, I would like to send a request repeatedly for data of various dates.

    cmd1 = ['-curl "https://example.com/more_curl_command/',...];
    cmd2 = datestr(timestamp(i), 'dd.mm.yyyy'); % replace part of command containing the input with a variable
    cmd3 = ['+-hstart+0+-hstop+24+-hinc+0.25', rest_of_command];
  [status, cmdout] = system([cmd1, cmd2, cmd3]);

If you find something like a sessionID in the first command output "cmdout", you may need to extract that and replace all sessionIDs in any further commands with the one returned in cmdout.

That's pretty much it. I would suggest you give it a try. It will take some time to figure out, but once you do, it works quite reliably. With a little practice, you can automate almost everything you can do using a browser.

[Dashao Luo]

Hi Marc, I got stuck with "Extract the command line tool and put it in Matlab's current directory.". I found no command line tool from the unzipped folder and also dont know what to do with the unzipped folder (name: curl-7.50.3). Could you please provide more guidance? Thanks a lot in advance!

[Jayesh Khatri]

Hey Marc

Since the 'firebug' has been discontinued by Firefox since 2017, do you know any other solution for the exact same problem?

Thanks in advance.

[Andrew Short]

Thanks Marc for this super useful method!

@Dashao Luo: newer versions of MATLAB already have cURL capabilities. You can do [~,status] = system('cURL ...');, then examine status; it should have the commands you entered in the form (if it's not xhl, or other more modern web interfaces).

@Jayesh Khatri: the firefox web developer version has all the old firebug tools. You can go to the "three horizontal lines" tab in the top right, select web developer, and select Network, then find the Post command there. You can right click the Post command (or any command) and copy as cURL, then go from there.

Answer 2

[Dashao Luo]

Thanks! Is it possible to provide an example for a quick start?

[Marc Jakobi]

I updated my answer with a step by step guide. I can't send you an exact example, because it does involve a lot of analyzing of commands, requests and returns, which takes a long time.

P. S. please use the comment function to reply to answers in this forum.

[Dashao Luo]

Hi Marc, thanks a lot for updating the answer with the example! I will try it out and contact you if I have further questions.