2020.22080 - Win 10 - WebClient DAT "losing" track of responses with asynchronous calls?

Hello,

I unfortunately can’t share a working debug file since I encountered the problem on some private API.

I’ll try to give as much details as possible so that you can reproduce in your test setup:

  • Trigger on the same frame 2 or more asynchronous .request() calls.
  • You might observe that the callback “onResponse” would trigger only once, and the other calls are getting “lost”.

I tested with Postman against the same API to trigger plenty of calls in a row, and always got the expected responses.

I tested in Touch to add a delay of multiple frames using run(), and was able to get the response.

The API in question is throttled at 5 calls per seconds, triggering 2 requests on the same frame should not be a problem.

I noticed, in the responses header, that the first response has a specified size in kb, while the other one doesn’t. Could this cause an issue ?

When triggering both requests from the Textport, one after the other (not on the same frame) I get both responses as expected. Triggering from, let’s say, a DAT execute or CHOP execute, both on the same frame, cause the issue.

I was wondering, with the little knowledge I have on the backend, if it could be a threading issue ? Could the asynchronous call send back responses at almost the same time, and while the WebClient DAT is busy parsing a response, it is locked and the second response is silently failing / getting lost.

I know it’s very little to help debug, but maybe you are already aware of something, or it can give you an idea where to look at…

Let me know if I can help more,

Thanks,
Michel

I can successfully reproduce the issue against a second API. I have confirmation that the API is not rate-limiting. The Spotify API is supposed to return a statusCode of 429 in the response headers when receiving too many calls.

I was working this afternoon on adding Analysis and Features of the Spotify API to my Spotify component.

Here is the order of things to understand the code:

  • When receiving a new track Id, call both the Analysis and Features endpoints of the Spotify API.
  • “Sent call for” is printed, twice on the same frame.
  • The callback is never triggered and the print messages “Setting analysis” or “Setting features” are never printing.

If I replace those 2 API calls by a run(… , delayFrames=600) and run(… , delayFrames=1200), I receive both answers correctly.

You can check it out by:

Thank you for your time,
Michel

Hey @JetXS

thanks for the great examples! We’ll have a look at it.

Hope you are doing very well
Best
Markus

Hi @snaut, @JetXS,

I’m having similar issues with the Web Client DAT.

It seems like it doesn’t allow multiple requests at the same time. If a request is still pending, another call to request(url, method, ...) will cancel (not give a response to) the first request. I’m not sure if this is an error or the designed behaviour of the Web Client DAT.

If this expected behaviour, then there seem an error related to this. When I do a PUT request:

webClient.request('http://test.com/upload', 'PUT', data=data)

and while this request is still pending, I follow up with a get:

webClient.request('http://test.com/data', 'GET')

it seems to execute the GET with a PUT method. The response from the server will be:

Cannot PUT http://test.com/data

So it looks like it cancels the PUT and does a new request with the new URL, but the old method.

If the DAT would allow multiple asynchronous requests, there is also the question of how to relate a response to a request. Normally, in passing a callback or using an async function, you would still be executing in the same scope of code or you can bind arguments to the callback. With the callbacks DAT it’s a bit different…

Could it be an idea to pass a originalRequest parameter in the onResponse callback? That could maybe give more information about the original request, like:

onResponse(webClientDAT, statusCode, headerDict, data, originalRequest):
    # originalRequest.url
    # originalRequest.method
    # originalRequest.header
    # originalRequest.data

Best,
Thijs

1 Like

+1000 to that. It’s a big setback of the webclient DAT at the moment. I was going to take time to write an RFE for that but Thijs went ahead in this bug report haha.

1 Like

The way im doing it currently is building a request query. So instead of sending the request directly I put the request in a query and then check the current query item. If that is empty I actually send the request and set the current query item to the just send request.
Now I just wait for the recieve and handle the response with the data of the current request data. After everything is handled I just set the current query item back to null and run the query check again.

Still working out how to handle actualy timeouts but I have the feeling thats basicly the same way it works in other components. Or you could just spawn a new WebClientDAT (which is basicly the same as an AJAX Object in JS) and delete it after the response.

Hi @alphamoonbase,

Building a queue of sorts also passed my mind, but it’s not really the same as being able to fire multiple async requests at a time and handling the responses accordingly. Which is normally the kind of the behaviour you’d implement in web development.

Spawning a Web Client DAT per request could be an alternative, but somehow it feels a lot more heavy to do this in TouchDesigner than to create an XMLHttpRequest in JS.

Best,
Thijs

1 Like

There always is the request library which, in fact, is already inside of touch. :slight_smile:
https://requests.readthedocs.io/en/master/

But seems like it is on the list at Derivative to handle the connections internaly.

Oh thanks that is good to know!
Haha I was thinking of maybe installing it on my local Python to try it out, but always better if it’s already included. However I’m afraid I can’t find anything about async in requests… Hope to use async / await sometime soon in Touch :slight_smile:

With request python lib you have to manage your own threads and use queues to communicate .

Native support for an asyncio event loop would be great indeed . Unreal seems to allow that https://github.com/20tab/UnrealEnginePython/blob/master/tutorials/AsyncIOAndUnrealEngine.md#adding-an-asyncio-loop-engine-in-the-ue4-core

1 Like

Maybe take a look at markus’ WebComp (which is the basis for the WebClientDAT). Hes using requests there so maybe something inside to find.

@alphamoonbase any new developments on that ? didn’t dig too much in it lately. Is it still a hassle to track those asynchronous calls ?

I built a que system which appends the the requests with custom meta data and callbacks. Still one after another but thats ok for my usecase. Will see if I might upload a general version of that.

Mmm yeah not too much of a fan of that TBH… It’s patching / putting a bandaid on something that is broken.

@snaut did you guys look into that ? Did you encounter that challenge recently with Twitcher ?

Hey @JetXS,

we are looking at ways to make this possible. Currently the webClient DAT is just not set up to deal with multiple open connections the way that for example the webServer or webSocket DAT can handle them.
An idea would be to return an identifier per request() Method call and pass that identifier into the callback. Your logic doing the requests and dealing with the responses can then match them up.
Display wise, the webClient DAT is only dealing with a single response. There is pros and cons to keeping old data around. The possibility to show previous responses goes against readability while only showing the last response obscures data.

Cheers
Markus

Hey Markus,

Thanks for getting back to me.

To me what would make the most sense is to have a combo Table / Callback just like the MQTT client.

I would imagine in the table to have eventually an ID like you say, but also maybe a column where the initial request is displayed, and then the response and maybe bonus like absolute frame number on request and absolute frame number on response…

To be honest, I don’t imagine anyone doing REST / JSON stuff with just DATs and that all the responses and logic would be handled in the callback anyways.

Exposing all those infos in the callback function would be great.

Thank you for your time,
Michel