How TD read a dynamic content website?

Beginners
1

Hi, everyone, I am very young in TD but have to handle a complicated project. I’m facing a very critical issue hoping for help.

I’m using the webclient Dat to read a dynamic content website hopefully to read out the source code content in TD. However, it turns out that TD only read the blank index page but not the dynamic content with the same website link. This is because different source codes are showing between the Inspect view and the page source view!! TD only read the blank page source!

I need TD can read out the dynamic content once it has changed the content in real-time. Are there other solutions other than using the webclient Dat? Or another approach on website is also welcome.

the website link : https://main.d2h8jmeaabvib5.amplifyapp.com/
I am very frustrated now because it is the last step to make all things happen.

Untitled-1
Untitled-11854×1325 193 KB

Untitled-2
Untitled-22100×1482 403 KB

Untitled-3
Untitled-31770×743 136 KB

2

Did you build the website yourself?
The problem with SPA-Webapps from frameworks like this is that they generate the content in the browser using javaScript. The webclientDAT does not execute javascript.
If you have built the website yourself you can try to transition to NuxtJS to enable serverSideRendering. Instead of the WebBrowser generating the website the server will do that (but stil being dynamic).

You could also try hacking around with the webrenderTOP which is a full browser with Javascript. You can inject some javascript to relay certain information. Or maybe it is possible to inspect the raw HTML from TD via infoDAT? (@selina ?)

1 Like
3

The Info DAT does not provide the raw HTML. You can inject javascript but no return values.

1 Like
4

@tkwing, were you able to figure this out? I’m needing the same thing.

5

Hi @paulfehr,

you will have to analyze the javascript that populates the website. In the above example there is a reference to a javascript file that contains a variety of functions that seem to pull relevant content. For example:

...
generatePoem() {
                (this.animationStarted = !0),
                  fetch(
                    "https://9qmr9y8ngd.execute-api.ap-east-1.amazonaws.com/default/questionnaire/get_answer",
                    {
                      method: "POST",
                      headers: { "Content-Type": "application/json" },
                      body: JSON.stringify({
                        func: "generatePOEM",
                        questionnaireid: this.questionnaire_id,
                      }),
                    }
                  )
                    .then((e) => e.json())
                    .then((e) => {
                      console.log(e.body),
                        e.body.result.length > 0 &&
                          ((this.poemtitle = e.body.result["0"].title),
                          (this.poem_array[0] = e.body.result["0"].line1));
                      for (var t = 1; t <= e.body.result.length; t++)
                        this.poem_array[t] = e.body.result[t - 1].poem_line;
                      this.animationStarted = !1;
                    })
                    .catch((e) => console.error("Error:", e));
              },
...

Included in there is a post request which you could potentially duplicate with the Web Client DAT and retrieve the information from the returned data.

cheers
Markus